Zstd compression for better performance

[Glandos]

Hi!
I just open a simple discussion because I was unable to find why ZIP format was selected. When Yarn2 was made public, zstandard was quite mature, and much more performant than ZIP. Here is a small benchmark on one of my project dependency

> hyperfine "zip -r -9 typescript-patch-30b732d1e2-6bf45caf84.manual.zip node_modules/" "zstd -9 -f typescript-patch-30b732d1e2-6bf45caf84.tar"
Benchmark #1: zip -r -9 typescript-patch-30b732d1e2-6bf45caf84.manual.zip node_modules/
  Time (mean ± σ):      4.538 s ±  0.090 s    [User: 4.510 s, System: 0.024 s]
  Range (min … max):    4.463 s …  4.746 s    10 runs
 
Benchmark #2: zstd -9 -f typescript-patch-30b732d1e2-6bf45caf84.tar
  Time (mean ± σ):      1.528 s ±  0.028 s    [User: 1.531 s, System: 0.026 s]
  Range (min … max):    1.490 s …  1.569 s    10 runs
 
Summary
  'zstd -9 -f typescript-patch-30b732d1e2-6bf45caf84.tar' ran
    2.97 ± 0.08 times faster than 'zip -r -9 typescript-patch-30b732d1e2-6bf45caf84.manual.zip node_modules/'

-rw-r--r-- 1 user user 11426008 15 avril 14:11 typescript-patch-30b732d1e2-6bf45caf84.manual.zip
-rw-r--r-- 1 user user 64993280 15 avril 14:06 typescript-patch-30b732d1e2-6bf45caf84.tar
-rw-r--r-- 1 user user  9453575 15 avril 14:06 typescript-patch-30b732d1e2-6bf45caf84.tar.zst
-rw-r--r-- 1 user user 11416285 15 avril 10:54 typescript-patch-30b732d1e2-6bf45caf84.zip

So the output is smaller than the original ZIP file, and it compressed 3 times faster.

For decompression, I included the file extraction:

hyperfine "unzip -o typescript-patch-30b732d1e2-6bf45caf84.manual.zip" "tar xf typescript-patch-30b732d1e2-6bf45caf84.tar.zst"
Benchmark #1: unzip -o typescript-patch-30b732d1e2-6bf45caf84.manual.zip
  Time (mean ± σ):     350.3 ms ±   4.1 ms    [User: 304.7 ms, System: 45.6 ms]
  Range (min … max):   346.3 ms … 359.1 ms    10 runs
 
Benchmark #2: tar xf typescript-patch-30b732d1e2-6bf45caf84.tar.zst
  Time (mean ± σ):     135.6 ms ±   4.9 ms    [User: 78.3 ms, System: 108.1 ms]
  Range (min … max):   126.7 ms … 145.3 ms    21 runs
 
Summary
  'tar xf typescript-patch-30b732d1e2-6bf45caf84.tar.zst' ran
    2.58 ± 0.10 times faster than 'unzip -o typescript-patch-30b732d1e2-6bf45caf84.manual.zip'

The same result are achieved.

Yarn could greatly benefit of such performance, but why ZIP was selected? Because it can compress and index multiple files?

[arcanis]

why ZIP was selected? Because it can compress and index multiple files?

• Very portable
• The format doesn't belong to any single owner
• Integrated by default into many OS, IDEs, etc
• Reasonable compression
• Mature implementation (libzip)
• Random file access

Perhaps we could have chosen a different format, but development cycles were better spent elsewhere at the time. Perhaps at some point the cache format will become the bottleneck and it'll be revisited, but even now it's still not a priority for the core team compared to other stuff (which isn't to say we wouldn't be interested to see other cache formats be implemented by other people, just that it's a significant amount of work and we can't guarantee it will be merged if the improvements end up too small for the complexity cost).

[merceyz]

Note that libzip supports zstd if we were to look into that.

[arcanis]

Interesting 🤔 That could be a good experiment indeed, then

[MrChrisRodriguez]

@Glandos did you run any benchmarks after training a zstd dictionary? I wonder how much more significant the improvement could be with a generally-applicable domain-specific dictionary like brotli.

[Glandos]

I'm on holidays right now, but I'll keep this on my todo list.
However, I did some benchmarks on zstd with dictionary, but it is only significant on small dataset (<1MB). Of course, given the size of some packages, it can worth it.

[Glandos]

After a thought, and because I didn't find the time to run a benchmark, it seems not really useful to have a dictionary for this, if you want to maintain compatibility with other tools. A regular zstd -d call won't work unless you provide the dictionary.
Do you still think it's worthy?

[MrChrisRodriguez]

@Glandos -- my assumption was that all access to the compressed dependencies would go through yarn's modified fslib and that as a result, interoperability would flow through accordingly. But if that's not the case, then interoperability is certainly much more important!

In any case, using a dictionary is just an optimization to explore on top of your very good suggestion and can be deferred until it's determined if zstandard is something the Yarn team would consider.