Http client should have some retry

[josepowera]

I was running on stream from #3 and after some time I can see that some variant subProcesses fail.
running [email protected]

I did some log parsing and found some kind of variant hls fetch error... "Connection closed by remote peer" or similar (I have rewritten it from memory, not copy pasted). Upstream server was CDN from Akamai, therefore probably solid connection..

Is there any easy way to add some http retrys with minimal backoff to http request on some errors (most probably http code 5xx, timeout, connection failure)? something like in https://stackoverflow.com/questions/9446387/how-to-retry-urllib2-request-when-fails
Why - Fetching Two variant streams makes double trouble ... But six - ... :)

uporabnik@ubuntu:~/umzzoutput$ cat 0/index.m3u8 | grep MEDIA
#EXT-X-MEDIA-SEQUENCE:1126
uporabnik@ubuntu:~/umzzoutput$ cat 1/index.m3u8 | grep MEDIA
#EXT-X-MEDIA-SEQUENCE:1123
uporabnik@ubuntu:~/umzzoutput$ cat 2/index.m3u8 | grep MEDIA
#EXT-X-MEDIA-SEQUENCE:465
uporabnik@ubuntu:~/umzzoutput$ cat 3/index.m3u8 | grep MEDIA
#EXT-X-MEDIA-SEQUENCE:1188
uporabnik@ubuntu:~/umzzoutput$ cat 4/index.m3u8 | grep MEDIA
#EXT-X-MEDIA-SEQUENCE:46
uporabnik@ubuntu:~/umzzoutput$ cat 5/index.m3u8 | grep MEDIA
#EXT-X-MEDIA-SEQUENCE:145

[josepowera]

Got example from logs: [Errno 104] Connection reset by peer

/tmp/shareddata/umzz/3/seg37.ts:   start: 73545.727200   end: 73551.727200   duration: 6.000000
{'tags': {'#Iframe': '73491.7272', '#EXT-X-PROGRAM-DATE-TIME': '2023-04-11T21:51:03.050204Z', '#EXTINF': '6.000000,'}, 'name': '/tmp/shareddata/umzz/3/seg28.ts', 'file': 'seg28.ts', 'num': 28}
deleting /tmp/shareddata/umzz/3/seg28.ts
throttling 5.65
[Errno 104] Connection reset by peer
https://rbmn-live.akamaized.net/hls/live/590964/BoRB-AT/master_578.m3u8 ...........Failed
/tmp/shareddata/umzz/0/seg38.ts:   start: 73548.727200   end: 73554.727200   duration: 6.000000
{'tags': {'#Iframe': '73494.7272', '#EXT-X-PROGRAM-DATE-TIME': '2023-04-11T21:51:05.364482Z', '#EXTINF': '6.000000,'}, 'name': '/tmp/shareddata/umzz/0/seg29.ts', 'file': 'seg29.ts', 'num': 29}
deleting /tmp/shareddata/umzz/0/seg29.ts

[futzu]

This is the situation that isn't going to work, downloading all the variants at once over a network.
It will try and reload the manifest several times, but I hadn't run into failed segments, Connection reset by peer.... I am not sure.

Let's stick to local files for now, and make sure everything works before using a network,
it will be easier to diagnose problems.