Broker doesn´t close connections

[nrodday]

Dear BGPStream developers,

I am observing a strange behaviour for quite a while now but I couldn´t figure out why this is. I use pybgpstream but I also confirmed the same behaviour with the bgpreader command line interface.

For the following command (which I use every 24h for the respective last day) the connection transfers data but idles at some point, keeping the connection open but not transferring anything anymore:
bgpreader -p routeviews -p ris -t ribs -k "147.28.240.0/20" -w 1575676800,1575763199

Since transmissions seemed not to have ended, my scripts do not finish their analysis. I observe the following (and the processes stay alive for days until I kill them manually):

I should also mention that I am using the same code on a different machine and there I also observed this behaviour for a couple of days, killed all remaining processes and restartet them and since then it works fine, but it seems undeterministic to me when the problem occurs and how I can stop/avoid it. I think it is unrelated to the BGPReader version and my assumption is that the broker doesn´t close the transmissions properly sometimes.

I have connections open at the moment where the issue persists. The IP of the machine I am using is: 137.193.62.14.

Thanks!

Kind regards,
Nils Rodday

[waehlisch]

@alistairking, @digizeph any ETA when you have time to look into this?

[cmosig]

I have the same issue right now using the command:

bgpreader -w 1583020800,1585699199 -t updates -k 45.132.188.0/24 -k 45.132.189.0/24 -k 45.132.190.0/24 -k 45.132.191.0/24 -k 147.28.32.0/24 -k 147.28.33.0/24 -k 147.28.34.0/24 -k 147.28.35.0/24 -k 147.28.36.0/24 -k 147.28.37.0/24 -k 147.28.38.0/24 -k 147.28.39.0/24 -k 147.28.40.0/24 -k 147.28.41.0/24 -k 147.28.42.0/24 -k 147.28.43.0/24 -k 147.28.48.0/24 -k 147.28.49.0/24 -k 147.28.50.0/24 -k 147.28.51.0/24 -k 147.28.52.0/24 -k 147.28.53.0/24 -k 147.28.54.0/24 -k 147.28.55.0/24 -k 147.28.44.0/24 -k 147.28.45.0/24 -k 147.28.46.0/24 -k 147.28.47.0/24

sorry, this command is a bit long, but I don't have time to create a smaller example to reproduce the issue. bgpreader has been running for >2 days and does not output anything anymore. The output so far was only for the first 12 hours of the specified 1 month time period.

I have had the same issue for a different command, also for a month, where bgpreader stopped outputting data after 12 days and then just quit.

Is there a limit to how much data one can fetch using the bgpreader?

[alistairking]

I'm not sure that this is due to the broker not closing a connection -- the connections to the broker are typically very short lived and are only to retrieve metadata about which dump files should be processed. (Indeed the hostname the in the screenshot above is not that of the server that runs the broker, but instead is a server that was being used at that time to serve data directly, working around an issue with the RIS archives.)

I suspect that this is instead libbgpstream getting stuck trying to read from an HTTP connection to either RV or RIS that has already been terminated by the server. We've seen this happen occasionally with our own continuous processing and have a rough idea of what is going wrong.

Unfortunately it seems that we haven't actually created an issue to track this bug. We'll do that and then we can try and come up with a plan to fix the problem.

[waehlisch]

Unfortunately it seems that we haven't actually created an issue to track this bug. We'll do that and then we can try and come up with a plan to fix the problem.

what about this issue, #175 ;)

[cmosig]

I'm not sure that this is due to the broker not closing a connection -- the connections to the broker are typically very short lived and are only to retrieve metadata about which dump files should be processed. (Indeed the hostname the in the screenshot above is not that of the server that runs the broker, but instead is a server that was being used at that time to serve data directly, working around an issue with the RIS archives.)

I suspect that this is instead libbgpstream getting stuck trying to read from an HTTP connection to either RV or RIS that has already been terminated by the server. We've seen this happen occasionally with our own continuous processing and have a rough idea of what is going wrong.

Unfortunately it seems that we haven't actually created an issue to track this bug. We'll do that and then we can try and come up with a plan to fix the problem.

Thanks! I am not sure if this explains that only parts of requested data are being output.

[alistairking]

@waehlisch touché :)

@cmosig, if the partial output you are talking about is partial in terms of the time range (i.e., the first N hours of the range), then yeah, this explains it. libbgpstream would get stuck reading data from time N and then never be able to move past it.

[cmosig]

@cmosig, if the partial output you are talking about is partial in terms of the time range (i.e., the first N hours of the range), then yeah, this explains it. libbgpstream would get stuck reading data from time N and then never be able to move past it.

ahh, thanks!

[alistairking]

If you guys want to live on the edge, you could try patching wandio like this: https://github.com/wanduow/wandio/pull/46/files and see if it helps.

[cmosig]

Thank you, I will try that!

[digizeph]

If you guys want to live on the edge, you could try patching wandio like this: https://github.com/wanduow/wandio/pull/46/files and see if it helps.

FYI, the patch above has made its way to wandio 4.2.3 release: https://github.com/wanduow/wandio/releases/tag/4.2.3-1. (a bit farther from the edge now 😃)

[cmosig]

sry for the delay. I tested this again with the patch. One of the two issues has been solved. bgpreader now does not run indefinitely anymore which is good, but for query above I am still only getting data for the first 12 out of 30 days. And I am getting a whole lot of these errors:

HTTP ERROR: Couldn't resolve host name (6)
2020-05-23 19:33:04 1872226: bgpstream_reader.c:169: WARNING: Could not open (http://data.ris.ripe.net/rrc01/2020.04/updates.20200411.0115.gz). Attempt 1 of 5

I am not sure why, but there were not second attempts (only Attempt 1 of 5) and a few minutes later bgpready quit. The problem above seems unrelated, probably #171 and #166.

I believe the original issue is fixed and should be closed, but I let you be the judge :) . Thanks for the quick help!