Better Handling for Failed/Timed Out Requests from Tidal

[cleonyc]

When a Getting Track Details task times out or fails, it just gets stuck, it doesn't produce any feedback for the user. This results in a confusing error that only appears once you look at the files being downloaded. Might be a good idea to implement a timeout and retry for this step, and if that's not feasible, then at least having some feedback and (maybe, lmk if this is a good idea) stopping the download entirely might work best.

[MinisculeGirraffe]

Internally at least there's already a retry with exponential backoff for API requests. When it's stuck on Getting Track Details it's because of this retry process. But to your point this doesn't display any user feedback and it probably should.

I think the ideal solution is that we need to track the number of API requests we can make before rate limit and only initiate a call if it wouldn't cause a rate limit.

If making a new request would cause a rate limit we sleep that specific task until it would succeed and display a message saying that's what's going on and that if it keeps happening to lower the concurrency.

This weekend I should have some free time and I'll try and implement this and a few other issues i've been putting off.

[MinisculeGirraffe]

The retry values for the API requests are found here.

They aren't user configurable though, and they probably should be. Another issue with using the middleware system is that it's entirely transparent to the caller, so when getting the details of a track, the caller has no idea if 1 request was made or 5. It just resolves to an error once the retry policy has been exhausted.

For a quick and dirty fix you can modify the max_n_retries value to be significantly higher and set backoff_exponent to be 1.

You'll send overall more http requests, but given a high enough value they'll all go through eventually without failure.

[cleonyc]

Went ahead and looked at the debug logs, seems like almost all of the errors coming from tidal when downloading from https://listen.tidal.com/artist/3565245 are 429s. These settings fixed the failed downloads, but as you said it's not ideal since there is little feedback to the user.

   let retry_policy = ExponentialBackoff {
        max_n_retries: 100,
        max_retry_interval: std::time::Duration::from_millis(10000), // doesn't really matter
        min_retry_interval: std::time::Duration::from_millis(1000),
        backoff_exponent: 1,
    };

[MinisculeGirraffe]

Yeah. It needs a rework. Tonight and tomorrow I'm going to work on implementing

• User configurable retry settings with better default values.
• Visible Feedback when a retry occurs.
• Displaying a report once all the downloading tasks have completed to show if there were any failures and other useful information like total size, number of downloads, etc.

[popeye2468]

how do i implement these settings? please

[cleonyc]

Hey do you want any help implementing this? I've been really busy recently but next weekend and the weekend after i might be able to write something. Anything you figured out or that I should keep in mind?

[cleonyc]

@MinisculeGirraffe Just in case you missed it, see above.