Streaming reads?

[JohnEmhoff]

I spent a little time looking through the code, so I apologize if I just missed something -- is it possible to stream the contents of a blob without reading the whole thing to memory or disk? The only methods I see accept a file-like object and write all content directly into it, or similarly a filename.

Both necessitate reading the whole blob before handing control back to the caller, so working with large files is extremely awkward / impossible. I suppose a workaround could be to use this library in a thread and write to a file-like object that acts as in intermediary, but that seems like a huge onus to put on the caller.

[scottwernervt]

is it possible to stream the contents of a blob without reading the whole thing to memory or disk?

No, it looks like libraries like boto3 load it into memory first. The AWS S3 Driver can definitely support it according to the boto3 documentation. I need to investigate the other driver libraries. Which cloud service are you using?

[JohnEmhoff]

I'm using google's platform, but I think this will be an issue for any of the backends due to the API that cloudstorage exposes, if I'm not mistaken.