Add compress kwarg to CQM.to_file() #1296
Conversation
We use zipfile.ZIP_DEFLATED to be consistent with NumPy.
| kwargs = dict(compression=zipfile.ZIP_DEFLATED) if compress else dict() | ||
| with zipfile.ZipFile(file, mode='a', **kwargs) as zf: |
There was a problem hiding this comment.
My only doubt in this PR was about the compression method and compression level to use.
We have bzip2 and lzma at our disposal, in addition to the default gzip.
Looks like bzip2 provides 15% better compression over gzip, but it's ~8x slower at (de)compression.
lzma halves the archive size over gzip, but it's a lot slower than bzip2 during compression, although roughly as fast as gzip during decompression.
gzip/deflate dominates in memory overhead.
In addition, bzip2/lzma modules might not be available on all systems, especially if user self-compiled Python, because system packages need to installed beforehand.
Taking all that into consideration, looks like ZIP_DEFLATED with the default compression level (6 - balance between speed and size) is the optimal choice here. It's widely supported, it's fast, with minimal memory overhead, and provides decent compression (at least to handle obviously redundant sequences).
Address the CQM part of #1235.
Because we use
zipfilefor our CQMs, we get backwards compatibility for free.