v2.0.0

Summary

The main change for this version is the remodeling of DNS data (including new node types, e.g. HostName), inclusion of a lot of new datasets, new reference attributes for relationships, and a lot of code cleaning and bug fixes.

List of changes

• new datasets:
  • DNS resolution chain (OpenINTEL)
  • DNS resolution for umbrella, NS, and MX (openINTEL)
  • URL classification (Citizenlab)
  • sibling ASes (InetIntel)
  • Atlas probe (RIPE)
  • Atlas measurement (RIPE)
  • IXP (PCH)
  • url2hostname (post-process)
  • umbrella (CISCO)
  • IPv6 AS Hegemony (IHR)
  • AS Relationship IPv4 & IPv6 (BGPkit)
  • Alice looking glass
  • RoVista (Virginia Tech)
• support for node with multiple labels
• new reference attributes (reference_time_modification reference_time_fetch and reference_url_data, reference_url_info) replacing reference_time and reference_url
• most (all?) crawlers push nodes and links in batches
• docker service for public instance
• pre-commit checks
• automatically add neo4j constrains and indexes
• updated to neo4j 5.16
• code cleaning and numerous bug fixes

v1.1.0

Summary

Change the labels for nodes to be conform with Neo4j naming convention.

Features

• Renaming of node labels (e.g. DOMAIN_NAME is now DomainName)
• Simplified docker usage with docker_compose file

v1.0.0

Summary

First stable working version with Neo4j.

Features

Including 13 data sources:

• APNIC: AS population estimates​
• BGPKIT: AS relationship, route collector peers stats, prefix to ASN​
• Bgp.tools: AS names, AS tags​
• CAIDA: AS rank​
• Cloudflare: top100 domain names​
• IHR: Country ranking, AS Hegemony, ROV​ (including prefix dependencies)
• MANRS: members​
• NRO: delegated stats​
• OpenIntel: Tranco top 1M​
• PeeringDB: organization, IX, IX members, peering LAN, facilities​
• RIPE NCC: AS names, ROA​
• Stanford: ASdb​
• Tranco: top 1M

And two post-processing scripts that adds:

• IP addresses and prefixes address family
• mapping IP addresses to their most specific prefixes