How would you use this for multiple remote files?

[JTalarico]

It is not clear based on the documentation how I would use stream_zip() for multiple remote files.
I understand that any iterator that yield tuples should work, but I am having difficulty with the following

remote_file_list = [...]

def fetch_remote_file(filename):
   resp = get_remote_file(filename)
   yield (filename, modified_at, perms, ZIP_64, (resp.data,))

files_iterator = iter(fetch_remote_file(file) for file in remote_file_list)

stream_zip(files)

It seems to work for 1 file, but more than one gives ValueError: not enough values to unpack (expected 5, got 1)

I'm not sure what I am doing wrong

[michalc]

Lots of different ways to do it depending on what you need to do per file, but this is one that I think should work:

remote_file_list = [...]
files = (
    (filename, modified_at, perms, ZIP_64, (get_remote_file(filename).data,))
    for filename in remote_file_list
)
stream_zip(files)

This does look like it's loading the contents of each uncompressed member file into memory - so potentially it can be improved upon to make it more stream-y. But how to do that depends on what get_remote_file returns.

[michalc]

Also have now tweaked some of the documentation to have at least some multi-file examples

[JTalarico]

Seems to have resolved my issue. Thank you!

[JTalarico]

Still having issues with the files loading into memory before the stream starts

def get_bulk_download_stream(minio_uris):
    minio_client = get_client()

    # Define generators for remote files
    # These will download when the stream starts, instead of before
    def minio_content_generator(uri):
            bucket_name = get_bucket(uri)
            object_name = get_object(uri)
            minio_resp = minio_client.get_object(bucket_name, object_name)
            yield minio_resp.data

    def generate_all_files(minio_uris):
        modified_at = datetime.now()
        perms = 0o600

        for uri in minio_uris:
            filename = uri.split("/")[-1]
            yield (filename, modified_at, perms, ZIP_64, minio_content_generator(uri))

    return stream_zip(generate_all_files(minio_uris))

[michalc]

Ah suspect it'll be the minio_resp.data... looking at https://min.io/docs/minio/linux/developers/python/API.html#get-object-bucket-name-object-name-offset-0-length-0-request-headers-none-ssec-none-version-id-none-extra-query-params-none, get_object returns a urllib3.response.HTTPResponse, and looking at https://urllib3.readthedocs.io/en/stable/reference/urllib3.response.html#urllib3.response.HTTPResponse.data, its .data property is a single bytes instance - so it would be all the bytes of the response, and in memory at once.

However, there is a stream function https://urllib3.readthedocs.io/en/stable/reference/urllib3.response.html#urllib3.response.HTTPResponse.stream that instead of a single bytes instance, returns an iterable of bytes

And having a look around https://min.io/docs/minio/linux/developers/python/API.html#object-operations, looks like it's recommended to close/release connection.

So putting that all together:

def get_bulk_download_stream(minio_uris):
    minio_client = get_client()

    # Define generators for remote files
    # These will download when the stream starts, instead of before
    def minio_content_generator(uri):
        bucket_name = get_bucket(uri)
        object_name = get_object(uri)
        minio_resp = minio_client.get_object(bucket_name, object_name)
        try:
            yield from minio_resp.stream()
        finally:
            minio_resp.close()
            minio_resp.release_conn()

    def generate_all_files(minio_uris):
        modified_at = datetime.now()
        perms = 0o600

        for uri in minio_uris:
            filename = uri.split("/")[-1]
            yield (filename, modified_at, perms, ZIP_64, minio_content_generator(uri))

    return stream_zip(generate_all_files(minio_uris))

[JTalarico]

This seems to have solved it. Thank you!