GSOC idea: app to select RIS/Routeviews collectors

[romain-fontugne]

The main goal here is to provide a tool for researchers to select a good set of collectors. Each collector is connected to networks and receive BGP data from these networks (also called peers).
Because of the large amount of data, usually researchers pick only a few collectors to conduct studies with these datasets. This is often arbitrary so we'd like to help them to make better decisions. For example selecting collectors that don't have too much overlap, selecting collectors that include most Tier1 networks, or the minimum number of collectors for a given set of ASes (peers).
To help them make these decision we could also display stats related to the peers of a collector, how many are they? how much data they give? what type of network are they (IYP could help for that)? etc..
Data can be retrieved from the Internet Yellow Pages (IYP)

[pankaj87072]

hey @romain-fontugne as I am able to understand the idea we need to build a web app which have interactive ui and tools which help in the selecting appropriate data which is gathered from internet router for the analysis purpose .
Am I right ??.
if I am right then i think i can do this project can you guide me in this.

[romain-fontugne]

Yes, but the hardest part is to select data in a way that is meaningful for the user. This means a good understanding of RIS, Route views, and basic understanding of BGP.

[pankaj87072]

yes I understand your concern can you tell me is this same idea as discussed in this Network topology overview #36 because after reading it looks similar so if someone is into this project then i think i shouldn't interfere

[romain-fontugne]

Actually the two projects are quite different. This one is a bit lower priority

[pankaj87072]

Ohk i got the idea now so this project is bit complex as for me now because in this analyzing complex peering relationships between ASes and collectors is challenge for me right as i have worked in networking project earlier but developing algorithm is different so i think i should take straightforward one rather than out of my domain.

[Manasvini]

Hi Romain,
If this project is still being worked on, I'd like to contribute. I have a few of questions:

1. What is the larger goal here? Are we studying changes to peering behavior/ route changes?
2. What statistics at the collector are we interested in?
3. Are collector stats gathered by consuming the RIS feed via a Kafka pipe? The RIS feed seems like a firehose, so I imagine we need some aggregation window for stats?

Thanks,
Manasvini

[romain-fontugne]

Hi,

1. The main goal here is to provide a tool for researchers to select a good set of collectors. As you said RIS and Routeviews have tons of data, so usually researchers pick only a few collectors to conduct studies with these datasets. This is often arbitrary so we'd like to help them to make better decisions. For example selecting collectors that don't have too much overlap, or selecting collectors that include most Tier1 networks.
2. Each collector is connected to networks and receive BGP data from these networks (also called peers). The stats are mainly related to these peers, how many are they? how much data they give? what type of network are they (IYP could help for that)? etc..
3. Data can be retrieved from the Internet Yellow Pages (IYP)

[Manasvini]

If I understand correctly we need to:

1. Find a good set of collectors for a set of ASes.
   The metric for deciding a collector depends on the 1) vertex properties that depend on only looking at the nodes and edges corresponding to the collector and 2) network properties that involve picking a set of collectors (something along the lines of representing the no. of ASes shared between collectors as edges and the collectors as vertices, and then find the minimum spanning tree to find the set of collectors with the least overlap)?
2. Surface statistics related to collectors (and their peers) once the collectors have been identified. It seems like this could help with (1).

IYP provides a mechanism to query and gather the data (BGPKit, BGPTools and CAIDA ASRank to start with?), but I imagine this data needs to be exposed as an API endpoint or will it also need to be visualized?