-
-
Notifications
You must be signed in to change notification settings - Fork 2.9k
fetch, tls, and http fixes #24740
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fetch, tls, and http fixes #24740
Conversation
It's a bit counter-intuitive, but there are two streams here: the implementation here, and the connected output stream. When we say "unflushed" we mean don't flush the connected output stream because that's managed externally. But an "end" operation should always flush the implementation stream.
simplifies the logic & makes it respect limit
.zstd => std.compress.zstd.default_window_len, | ||
.gzip, .deflate => std.compress.flate.max_window_len, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
General comment mostly borne out of my own confusion: this concept of "buffer capacity" for std.compress
seems to be defined in different ways many different places, and some of them appear to disagree with eachother.
For zstd
, these doc comments would lead me to believe that default_window_len
is not enough:
zig/lib/std/compress/zstd/Decompress.zig
Lines 35 to 36 in 3fb8684
/// The output buffer is asserted to have capacity for `window_len` plus | |
/// `zstd.block_size_max`. |
zig/lib/std/compress/zstd/Decompress.zig
Lines 77 to 80 in 3fb8684
/// When connecting `reader` to a `Writer`, `buffer` should be empty, and | |
/// `Writer.buffer` capacity has requirements based on `Options.window_len`. | |
/// | |
/// Otherwise, `buffer` has those requirements. |
For deflate
, these constants are just generally confusing to me:
zig/lib/std/compress/flate.zig
Lines 3 to 8 in 3fb8684
/// When decompressing, the output buffer is used as the history window, so | |
/// less than this may result in failure to decompress streams that were | |
/// compressed with a larger window. | |
pub const max_window_len = history_len * 2; | |
pub const history_len = 32768; |
Seems like some more definitive recommended_buffer_size
or min_buffer_size
constant should be defined per-compression-algorithm and used consistently.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah I've been figuring this stuff out as I go along and only just starting to get a handle of it. Once I'm confident in the flow chart I'll do another pass over the doc comments, constants, and make sure everything is clear and agrees with each other.
even in this PR you can see I changed the rules for Reader.stream
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried to do all this stuff in a branch and only inflict it on master once I had a fully done, coherent change, but it's just too massive. everybody has to come along for the ride and suffer with me, sorry
Follow-up from #24698. Progress towards #24732.
cc @ianprime0509
it's still not quite all the way there yet:
error return trace: