Continuation of a zip file

[Xalag]

Hi everyone,
I am wondering if there is already a way to continue a zip file.
The use case would be:

• I want to pack a bunch of files into a huge zip file, let's say 10GB in final zipped size.
• After 8GB, my process get's a shutdown signal and has to stop.
• When I restart the process it should be able to read the last state form the already existing zip file and continue to pack the rest of the files into the zip, so that the zipping of the first 8GB does not have to be redone.
  Is there a way to achieve this?

[michalc]

Interesting problem!

So there is definitely nothing in stream-zip that does this

To do it perfectly would probably be impossible: you would always have to recompress the file that was interrupted at the very least, just from the nature of the deflate compression algorithm. But to even do that, you would have to keep track (in a separate file?) of where the zip process got to. A bit complex, but doable.

I am so far pretty much leaning to saying this is out of scope due to the complexity and the fact it seems like a fairly niche situation (e.g. I personally wouldn’t use this I suspect, and instead always restart from the beginning). But do you have more details on your case and whether restarting from the beginning of a member file would be acceptable?

[Xalag]

I agree, it would be quite complex to implement. For now, I only wanted to check if there is a solution already. I now opted for restarting from the beginning and will see how long this will be ok.

From my use case, I try to export machine learning data (images + mask-annotations) and zip the files afterwards for download. This can produce quite big zip files. In the ML world it seems quite normal to distribute datasets in huge zip-files, e.g. the COCO dataset: https://cocodataset.org/#download

While searching for a solution I also found a similar use case for zip continuation, which is different in the way that the continuation is not necessarily due to the size of the zip file but instead due to the way how new files come in: https://learn.microsoft.com/en-us/answers/questions/1427619/append-files-to-a-zip-inside-azure-storage-contain

I think the code of stream-zip would be a good starting point to implement e.g. zip-file continuation based on Azure Storage block blobs, but I agree that this topic would be too specialized as to implement it within stream-zip itself.