-
Notifications
You must be signed in to change notification settings - Fork 30.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
zlib: add zstd support #52100
base: main
Are you sure you want to change the base?
zlib: add zstd support #52100
Conversation
Review requested:
|
ec19b6e
to
28914c7
Compare
Alright, got up to making the basics (seemingly) work: $ out/Debug/node -p 'zlib.zstdDecompressSync(zlib.zstdCompressSync("Hello World")).toString()'
Hello World |
383ff46
to
47c7ab8
Compare
Fixes: nodejs#48412 PR-URL: nodejs#52100
Fixes: nodejs#48412 PR-URL: nodejs#52100
Fixes: nodejs#48412 PR-URL: nodejs#52100
Fixes: nodejs#48412 PR-URL: nodejs#52100
Fixes: nodejs#48412 PR-URL: nodejs#52100
Fixes: nodejs#48412 PR-URL: nodejs#52100
Fixes: nodejs#48412 PR-URL: nodejs#52100
cc @nodejs/zlib @nodejs/security-wg |
Fixes: nodejs#48412 PR-URL: nodejs#52100
Fixes: nodejs#48412 PR-URL: nodejs#52100
Fixes: nodejs#48412 PR-URL: nodejs#52100
Rebased & updated zstd to the latest release (1.5.5 --> 1.5.6). |
Fixes: nodejs#48412 PR-URL: nodejs#52100
|
||
CompressionError ZstdCompressContext::Init(uint64_t pledged_src_size) { | ||
pledged_src_size_ = pledged_src_size; | ||
cctx_.reset(ZSTD_createCCtx()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why cctx?
cctx_.reset(ZSTD_createCCtx()); | |
ctx.reset(ZSTD_createCCtx()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The c
is for compression (cctx vs dctx). Given that we only ever deal with one or the other, we could drop that marker from the fields. But it seems worthwhile to match the naming used by zstd itself.
Can you also update license_builder? |
The changes to the top-level LICENSE file were generated by running |
Fixes: nodejs#48412 PR-URL: nodejs#52100
Moved the init errors into C++ where we can add more context to them. |
720bf6c
to
7d18a7c
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #52100 +/- ##
========================================
Coverage 87.89% 87.89%
========================================
Files 651 651
Lines 183364 183680 +316
Branches 35714 35756 +42
========================================
+ Hits 161171 161453 +282
- Misses 15465 15494 +29
- Partials 6728 6733 +5
|
Hello. Any ETA for this? |
@jkrems ... can you give this a rebase? |
@jasnell I can but at this point I have spend quite bit of time on this PR without a clear signal on whether this would be likely to land (in this form or another). So it would be easier to justify if there was a tentative "yep, the high level approach looks good". By high level I mean something like "roughly following brotli integration in terms of API, using the facebook zstandard native implementation as the backing". |
Adds
ZstdCompress
andZsdDecompress
to thezlib
module which can be used to compress/decompress with the Zstandard ("zstd") algorithm.Notable omissions:
zlib.md
don't call out any params beyond the basic compression level.The code follows similar patterns to the PR that added Brotli support. Just that instead of brotli, it adds the equivalent zstd APIs. Just like Brotli, this required separate compression/decompression context objects.
Zstd itself has been around and stable for multiple years but this PR is early in terms of web support: It only just starts shipping by default in Chrome 123. On the other hand, by shipping in Chrome it will soon be supported quite widely on the web. Firefox also signaled support (https://bugzilla.mozilla.org/show_bug.cgi?id=1301878#c65).Zstd is now enabled on the web and supported by both Chrome and Firefox: https://caniuse.com/zstdOfficial support in node.js would allow passing additional WPTs around
fetch
(nodejs/undici#2847).