The golang GCS client supports automatic retrying here via keeping track of how many bytes the client has already seen and creating a new ranged read request to resume the read where the client left off:

https://github.com/googleapis/google-cloud-go/blob/9afb797d75499807b29c372ec375668be4d2995e/storage/http_client.go#L1268-L1275

So apparently object_store::buffered::BufReader might be the answer to this, at least it prevents long-running connections to the backend.

I feel like the docstring of into_stream() should be extended to point it out that you might want to use object_store::buffered::BufReader instead.

I'm not that happy yet object_store::buffered::BufReader doesn't do any "readahead", meaning we'd have to wait the summed up round-trip times whenever the buffer gets empty, but that's probably also tricky to do in the general case.

I'm also not sure if the calls to get_range it does are actually retried, in case of temporary connectivity issues / rate-limiting.

take

take

I've made a PR #6519 similar to the closed PR #5383, but only permits reqwest::Error::Decode to be retried, since this is the error type that we see associated with this issue (see #5882)

EDIT: I am not sure this is exactly the same issue that @flokli is seeing, but it is the one being seen in the #5882 issue so maybe I should associate this PR with that issue instead?

Given the subtle nature of errors and retries in the context of streaming requests, I think the only practical way forward will be to create an example/ test case that shows the problem.

The example would likely look something like:

1. A mock http server that pretends to be an S3 endpoint
2. Some settings in the mock server that can inject errors (like abort after some bytes have already been returned)
3. Testing the object store client then with errors / how it would automatically retry.

This test would also let us explore the various issues / corner cases that could be encountered

It would be likely that writing the test harness would be a significant undertaking, but that would be the best way to inform a proposal for API changes

We already have all of this setup as part of the retry tests - https://github.com/apache/arrow-rs/blob/master/object_store/src/client/retry.rs#L505. It would be a relatively straightforward enhancement to inject a failure mid-stream.

Edit: As for what the API would look like, given the only streaming request is get, and the corresponding logic is already extracted into GetClientExt which is shared across all implementations. This would be the obvious place to put any retry logic for streaming requests.

Perfect! So maybe @erratic-pattern you can make a PR with the failure scenario ? It would be valuable to document in code the current behavior, even if we decide we don't have the time to fix it.