Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compare zipping performance of zipfile and zip #2221

Open
tcompa opened this issue Jan 28, 2025 · 0 comments
Open

Compare zipping performance of zipfile and zip #2221

tcompa opened this issue Jan 28, 2025 · 0 comments

Comments

@tcompa
Copy link
Collaborator

tcompa commented Jan 28, 2025

Branching from #2215 to have a clean benchmark.

We created a heavy folder to be zipped:

$ find ./xxx/ -type f | wc -l
177818

$ du -sh ./xxx/
4.0G	./xxx/

And we zip this folder with two approaches.
In Python

from fractal_server.zip_tools import _create_zip
import time

t0 = time.perf_counter()
_create_zip(folder="/somewhere/xxx", output="/somewhere/zipbyfractal.zip")
t1 = time.perf_counter()
print(f"{t1-t0=}")

and with a native command

$ zip -qr zipbyzip.zip ./xxx/

Results

$ time zip -qr zipbyzip.zip ./xxx/

real	0m31.580s
user	0m24.308s
sys	0m2.715s

$ python zip_with_fractal.py 
t1-t0=45.33480614399923

$ ls -lh *zip
-rw-rw-r-- 1 tommaso tommaso 236M Jan 28 11:21 zipbyfractal.zip
-rw-rw-r-- 1 tommaso tommaso 246M Jan 28 11:20 zipbyzip.zip

TLDR:

  • zip is indeed faster, on this specific case
  • This difference is not enough to justify switching to zip right away, due to other costs (the actual implementation, and the additional external dependency which may not be available)
  • Further explorations are possible, and especially by re-running the benchmark on a network filesystem (as opposed to the ext4 I used)
  • There is a small difference in size of the two outputs, like due to the different compression level
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

1 participant