compressable:Zstd comp support #4657

Athishpranav2003 · 2024-10-03T18:13:50Z

Which issue(s) this PR fixes:
Fixes #4162

What this PR does / why we need it:
Adds new compression method support to handle messages.

Add zstd to compress option.
- Buffer
- out_file
- out_forward
in_forward: support zstd format. (Experimental)

Docs Changes:

add zstd compression support fluentd-docs-gitbook#591

Release Note:

Buffer: add zstd to compress option.
out_file: add zstd to compress option.
out_forward: add zstd to compress option.
in_forward: support zstd format. (Experimental)

Athishpranav2003 · 2024-10-03T18:16:41Z

@daipom @ashie need some comments on this since its a very new compression method. I tried to make sure the existing support is not broken and added this feature additionally. Need to perform some more work but wanted to get some comments from you guys

daipom · 2024-10-04T01:30:20Z

Thanks! I will see this this weekend!

Athishpranav2003 · 2024-10-16T15:58:29Z

@daipom did u get chance to look at this?

daipom · 2024-10-17T07:50:20Z

Sorry, I haven't made time for this. 😢
I'll make time for it this week.

Athishpranav2003 · 2024-10-17T11:24:16Z

Its fine i guess,
Even i was very busy in last 2 weeks 👍

daipom · 2024-10-21T10:41:05Z

@Athishpranav2003
I'm sorry for my late response. 😢

I have confirmed the entire direction!
It's great! Thanks for starting this enhancement!

Sorry I haven't made time to see the detailed implementation, such as Compressable module,
but the overall design of Fluentd would be essential for us now.
So, now, I comment on the overall direction of further modification.

It looks basically good as in_forward support!
All that is left is to support Output/Buffer/Chunk logic for output plugins. (Currently, these logic assumes gzip only)
To do so, it would be a good idea to support out_forward first.

The core logic would be the following.

EventStream#to_compressed_msgpack_stream

fluentd/lib/fluent/event.rb

Lines 65 to 68 in 78a7972

    
           def to_compressed_msgpack_stream(time_int: false, packer: nil) 
        
             packed = to_msgpack_stream(time_int: time_int, packer: packer) 
        
             compress(packed) 
        
           end

Output#generate_format_proc

fluentd/lib/fluent/plugin/output.rb

Lines 1016 to 1027 in 78a7972

    
           FORMAT_MSGPACK_STREAM = ->(e){ e.to_msgpack_stream(packer: Fluent::MessagePackFactory.thread_local_msgpack_packer) } 
        
           FORMAT_COMPRESSED_MSGPACK_STREAM = ->(e){ e.to_compressed_msgpack_stream(packer: Fluent::MessagePackFactory.thread_local_msgpack_packer) } 
        
           FORMAT_MSGPACK_STREAM_TIME_INT = ->(e){ e.to_msgpack_stream(time_int: true, packer: Fluent::MessagePackFactory.thread_local_msgpack_packer) } 
        
           FORMAT_COMPRESSED_MSGPACK_STREAM_TIME_INT = ->(e){ e.to_compressed_msgpack_stream(time_int: true, packer: Fluent::MessagePackFactory.thread_local_msgpack_packer) } 
        
           def generate_format_proc 
        
             if @buffer && @buffer.compress == :gzip 
        
               @time_as_integer ? FORMAT_COMPRESSED_MSGPACK_STREAM_TIME_INT : FORMAT_COMPRESSED_MSGPACK_STREAM 
        
             else 
        
               @time_as_integer ? FORMAT_MSGPACK_STREAM_TIME_INT : FORMAT_MSGPACK_STREAM 
        
             end 
        
           end

Chunk::Decompressable

fluentd/lib/fluent/plugin/buffer/chunk.rb

Lines 184 to 235 in 78a7972

    
           module Decompressable 
        
             include Fluent::Plugin::Compressable 
        
             def append(data, **kwargs) 
        
               if kwargs[:compress] == :gzip 
        
                 io = StringIO.new 
        
                 Zlib::GzipWriter.wrap(io) do |gz| 
        
                   data.each do |d| 
        
                     gz.write d 
        
                   end 
        
                 end 
        
                 concat(io.string, data.size) 
        
               else 
        
                 super 
        
               end 
        
             end 
        
             def open(**kwargs, &block) 
        
               if kwargs[:compressed] == :gzip 
        
                 super 
        
               else 
        
                 super(**kwargs) do |chunk_io| 
        
                   output_io = if chunk_io.is_a?(StringIO) 
        
                                 StringIO.new 
        
                               else 
        
                                 Tempfile.new('decompressed-data') 
        
                               end 
        
                   output_io.binmode if output_io.is_a?(Tempfile) 
        
                   decompress(input_io: chunk_io, output_io: output_io) 
        
                   output_io.seek(0, IO::SEEK_SET) 
        
                   yield output_io 
        
                 end 
        
               end 
        
             end 
        
             def read(**kwargs) 
        
               if kwargs[:compressed] == :gzip 
        
                 super 
        
               else 
        
                 decompress(super) 
        
               end 
        
             end 
        
             def write_to(io, **kwargs) 
        
               open(compressed: :gzip) do |chunk_io| 
        
                 if kwargs[:compressed] == :gzip 
        
                   IO.copy_stream(chunk_io, io) 
        
                 else 
        
                   decompress(input_io: chunk_io, output_io: io) 
        
                 end 
        
               end 
        
             end

It is complicated, but the compression of Buffer and Chunk and the flush process of output plugins are closely related.
The following existing implementations may be helpful for how output plugins behaves:

fluentd/lib/fluent/plugin/out_forward.rb

Lines 253 to 259 in 78a7972

    
           unless @as_secondary 
        
             if @compress == :gzip && @buffer.compress == :text 
        
               @buffer.compress = :gzip 
        
             elsif @compress == :text && @buffer.compress == :gzip 
        
               log.info "buffer is compressed.  If you also want to save the bandwidth of a network, Add `compress` configuration in <match>" 
        
             end 
        
           end

fluentd/lib/fluent/plugin/out_forward.rb

Lines 664 to 668 in 78a7972

    
           chunk.open(compressed: @compress) do |chunk_io| 
        
             entries = [0xdb, chunk_io.size].pack('CN') 
        
             sock.write entries.force_encoding(Encoding::UTF_8) # 2. entries: String (str32) 
        
             IO.copy_stream(chunk_io, sock)                     #    writeRawBody(packed_es) 
        
           end

fluentd/lib/fluent/plugin/out_file.rb

Lines 214 to 225 in 78a7972

    
           writer = case 
        
                    when @compress_method.nil? 
        
                      method(:write_without_compression) 
        
                    when @compress_method == :gzip 
        
                      if @buffer.compress != :gzip || @recompress 
        
                        method(:write_gzip_with_compression) 
        
                      else 
        
                        method(:write_gzip_from_gzipped_chunk) 
        
                      end 
        
                    else 
        
                      raise "BUG: unknown compression method #{@compress_method}" 
        
                    end

fluentd/lib/fluent/plugin/out_file.rb

Lines 245 to 268 in 78a7972

    
           def write_without_compression(path, chunk) 
        
             File.open(path, "ab", @file_perm) do |f| 
        
               if @need_ruby_on_macos_workaround 
        
                 content = chunk.read() 
        
                 f.puts content 
        
               else 
        
                 chunk.write_to(f) 
        
               end 
        
             end 
        
           end 
        
           def write_gzip_with_compression(path, chunk) 
        
             File.open(path, "ab", @file_perm) do |f| 
        
               gz = Zlib::GzipWriter.new(f) 
        
               chunk.write_to(gz, compressed: :text) 
        
               gz.close 
        
             end 
        
           end 
        
           def write_gzip_from_gzipped_chunk(path, chunk) 
        
             File.open(path, "ab", @file_perm) do |f| 
        
               chunk.write_to(f, compressed: :gzip) 
        
             end 
        
           end

Athishpranav2003 · 2024-10-21T11:31:26Z

@daipom thanks for the review.

Yah, this will kick a lot more changes in the overall events pipeline.
For now i have just added the compression support as a module in Compressible. And used this in in-forward. I will try to look at Buffer as well in meantime(might need more time since i got busy with a project). I guess maybe splitting it into multiple PRs will be easy since i can finish small chunks one by one(seperate functionality) and easy to review

Athishpranav2003 · 2024-10-23T11:55:49Z

@daipom can you please check the change in the Compression module as well. Once if we merge this we can proceed with the other development around this

daipom · 2024-10-25T07:55:47Z

@Athishpranav2003

can you please check the change in the Compression module as well

Sure! I will review the Compressable module!

Once if we merge this we can proceed with the other development around this

To judge if we can merge this only with the support of the module and in_forward, we need to reach an agreement on updating Forward Protocol Specification.

It would be necessary to update CompressedPackedForward Mode.
By allowing zstd as the value for the key compressed, it would be possible to add Zstd support while keeping compatibility, but I'm not sure yet.

Athishpranav2003 · 2024-10-25T08:33:39Z

Sure @daipom

If it's needed then I can split the PR into 2 where this will only focus on zstd and other one for in-forward

Athishpranav2003 · 2024-10-29T15:38:19Z

@daipom how to proceed in this PR?

daipom · 2024-10-30T09:39:39Z

Sorry, please give me a few more days 😢

daipom

@Athishpranav2003
Sorry for my late response 😢
I have commented on some points, but basically it looks good! Thanks!

About our future policies, I think it is better not to separate the PRs, as these changes need to be considered in combination with updating Forward-Protocol.

How about supporting the out_forward by implementing Output/Buffer/Chunk logic in this PR?

My concern is whether we can support Buffer/Chunk zstd compression.
To support it, the compressed data must be able to be concatenated.
gzip allows it.
It appears that we can concat zstd files without any problems, but I wonder if this is an officially supported specification of zstd and zstd-ruby gem.

$ echo "Hello world" > 1
$ echo "Hello Fluentd" > 2
$ zstd -f 1
$ zstd -f 2
$ cat 1.zst 2.zst > 3.zst
$ zstd -d 3.zst
$ cat 3
Hello world
Hello Fluentd

lib/fluent/event.rb

lib/fluent/plugin/compressable.rb

lib/fluent/plugin/in_forward.rb

Athishpranav2003 · 2024-11-03T05:03:02Z

bundle exec rake test TEST=test/plugin/test_compressable.rb                      2856ms  Sun 03 Nov 2024 11:29:54 AM IST
/usr/bin/ruby -w -I"lib:test" -Eascii-8bit:ascii-8bit /usr/share/gems/gems/rake-13.2.1/lib/rake/rake_test_loader.rb "test/plugin/test_compressable.rb" 
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/rubygems_ext.rb:250: warning: method redefined; discarding old encode_with
/usr/local/share/ruby/site_ruby/rubygems/dependency.rb:341: warning: previous definition of encode_with was here
/usr/share/ruby/bundled_gems.rb:75:in `require': libruby.so.3.2: cannot open shared object file: No such file or directory - /usr/lib64/gems/ruby/yajl-ruby-1.4.3/yajl/yajl.so (LoadError)
	from /usr/share/ruby/bundled_gems.rb:75:in `block (2 levels) in replace_require'
	from /usr/share/gems/gems/yajl-ruby-1.4.3/lib/yajl.rb:1:in `<top (required)>'
	from /usr/share/ruby/bundled_gems.rb:75:in `require'
	from /usr/share/ruby/bundled_gems.rb:75:in `block (2 levels) in replace_require'
	from /home/aggressive_racer1/projects/fluentd/lib/fluent/config/literal_parser.rb:20:in `<top (required)>'
	from /usr/share/ruby/bundled_gems.rb:75:in `require'
	from /usr/share/ruby/bundled_gems.rb:75:in `block (2 levels) in replace_require'
	from /home/aggressive_racer1/projects/fluentd/lib/fluent/config/element.rb:18:in `<top (required)>'
	from /usr/share/ruby/bundled_gems.rb:75:in `require'
	from /usr/share/ruby/bundled_gems.rb:75:in `block (2 levels) in replace_require'
	from /home/aggressive_racer1/projects/fluentd/test/helper.rb:42:in `<top (required)>'
	from /home/aggressive_racer1/projects/fluentd/test/plugin/test_compressable.rb:1:in `require_relative'
	from /home/aggressive_racer1/projects/fluentd/test/plugin/test_compressable.rb:1:in `<top (required)>'
	from /usr/share/ruby/bundled_gems.rb:75:in `require'
	from /usr/share/ruby/bundled_gems.rb:75:in `block (2 levels) in replace_require'
	from /usr/share/gems/gems/rake-13.2.1/lib/rake/rake_test_loader.rb:21:in `block in <main>'
	from /usr/share/gems/gems/rake-13.2.1/lib/rake/rake_test_loader.rb:6:in `select'
	from /usr/share/gems/gems/rake-13.2.1/lib/rake/rake_test_loader.rb:6:in `<main>'
rake aborted!
Command failed with status (1): [ruby -w -I"lib:test" -Eascii-8bit:ascii-8bit /usr/share/gems/gems/rake-13.2.1/lib/rake/rake_test_loader.rb "test/plugin/test_compressable.rb" ]
/usr/share/gems/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/cli/exec.rb:58:in `load'
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/cli/exec.rb:58:in `kernel_load'
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/cli/exec.rb:23:in `run'
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/cli.rb:455:in `exec'
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/cli.rb:35:in `dispatch'
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/cli.rb:29:in `start'
/usr/share/gems/gems/bundler-2.5.16/exe/bundle:28:in `block in <top (required)>'
/usr/share/gems/gems/bundler-2.5.16/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
/usr/share/gems/gems/bundler-2.5.16/exe/bundle:20:in `<top (required)>'
/usr/bin/bundle:25:in `load'
/usr/bin/bundle:25:in `<main>'
Tasks: TOP => test => base_test
(See full trace by running task with --trace)

@daipom i addressed your comments but couldnt test it locally due to some issue.
yajl-ruby is already installed in my system but some other issue

daipom

Did not this code work?

#4657 (comment)

lib/fluent/plugin/in_forward.rb

Athishpranav2003 · 2024-11-05T02:44:29Z

I guess there is a small careless
Problem is local testing is not working at all. Could help me with that later

I will check after 18th
Got packed with exams now :(

daipom · 2024-11-05T02:44:30Z

Hmm, these changes would be necessary, but looks like the error in your environment is caused by another reason.

#4657 (comment)

/usr/share/ruby/bundled_gems.rb:75:in `require': libruby.so.3.2: cannot open shared object file: No such file or directory - /usr/lib64/gems/ruby/yajl-ruby-1.4.3/yajl/yajl.so (LoadError)

daipom · 2024-11-05T02:46:38Z

I will check after 18th
Got packed with exams now :(

OK! I'm sorry my response was so slow, and it took so long.
Good luck with the exams!

Athishpranav2003 · 2024-11-18T08:49:43Z

@daipom the gem issue got resolved after update,prolly some mismatched versions,

All tests are passing now

Signed-off-by: Athish Pranav D <[email protected]>

Athishpranav2003 · 2024-11-18T12:13:55Z

@daipom some doubts here

in chuck open method for decompress we dont have any type identification.
how to identify the chunk type while open/read method is called

fluentd/lib/fluent/plugin/buffer/chunk.rb

Lines 201 to 226 in 78a7972

    
           def open(**kwargs, &block) 
        
             if kwargs[:compressed] == :gzip 
        
               super 
        
             else 
        
               super(**kwargs) do |chunk_io| 
        
                 output_io = if chunk_io.is_a?(StringIO) 
        
                               StringIO.new 
        
                             else 
        
                               Tempfile.new('decompressed-data') 
        
                             end 
        
                 output_io.binmode if output_io.is_a?(Tempfile) 
        
                 decompress(input_io: chunk_io, output_io: output_io) 
        
                 output_io.seek(0, IO::SEEK_SET) 
        
                 yield output_io 
        
               end 
        
             end 
        
           end 
        
           def read(**kwargs) 
        
             if kwargs[:compressed] == :gzip 
        
               super 
        
             else 
        
               decompress(super) 
        
             end 
        
           end

Athishpranav2003 · 2024-12-20T08:04:55Z

Yah,
I was seeing ur contributions for a while.
Its fine for me

daipom · 2024-12-26T07:11:14Z

@Athishpranav2003
Sorry for my late response.
Overall, it looks great! Thanks so much!

About out_file and out_forward, it would be better that we do not support setting a different compression format from the buffer.
There would be no particular benefit in allowing it.
Those plugins should raise ConfigError if a different compression format is set (except :text).

For example, for out_forward, we should check the format as follows.

        if @buffer.compress == :text
          @buffer.compress = @compress unless @compress == :text
        else
          if @compress == :text
            log.info "buffer is compressed.  If you also want to save the bandwidth of a network, Add `compress` configuration in <match>"
          elsif @compress != @buffer.compress
            raise Fluent::ConfigError, "You cannot specify different compression formats for Buffer (Buffer: #{@buffer.compress}, Self: #{@compress})"
          end
        end

As for implementation, there appears to be no other problems.
(There are some points where it would be nice to make some things simpler, but not essential.)

If you have any other concerns, please let me know.

We need to update the protocol before we merge this.
I will consider how to proceed it.

To judge if we can merge this only with the support of the module and in_forward, we need to reach an agreement on updating Forward Protocol Specification.

It would be necessary to update CompressedPackedForward Mode. By allowing zstd as the value for the key compressed, it would be possible to add Zstd support while keeping compatibility, but I'm not sure yet.

Athishpranav2003 · 2024-12-26T11:16:31Z

@daipom i have addressed ur comments. Feel free to point out the nits as well.

lib/fluent/plugin/out_file.rb

Athishpranav2003 · 2025-01-03T03:20:24Z

@daipom
we need to precisely update this section ryt? https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5#compressedpackedforward-mode

lib/fluent/plugin/out_file.rb

daipom · 2025-01-06T05:27:09Z

@daipom we need to precisely update this section ryt? https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5#compressedpackedforward-mode

Yes!!

daipom · 2025-01-06T05:39:32Z

@daipom we need to precisely update this section ryt? https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5#compressedpackedforward-mode

Yes!!

I have made an issue for this.

Update Forward Protocol Specification for zstd compression support #4758

Let's summarize the new protocol there, for now.

daipom · 2025-01-06T05:51:18Z

@daipom we need to precisely update this section ryt? https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5#compressedpackedforward-mode

Yes!!

I have made an issue for this.
* [Update Forward Protocol Specification for zstd compression support #4758](https://github.com/fluent/fluentd/issues/4758)
Let's summarize the new protocol there, for now.

@Athishpranav2003
I have made a draft. Could you please check it?

#4758 (comment)

Signed-off-by: Athish Pranav D <[email protected]>

daipom

LGTM! Thanks for this significant improvement!!
This should be merged into the next minor version v1.19.
Also, we need to wait #4758.

daipom

LGTM! Thanks so much!

In #4758, we now have a clear direction for protocol changes, so I merge this PR.

**Which issue(s) this PR fixes**: * Related to #4758 **What this PR does / why we need it**: When using `compress zstd` for `out_forward`, output the following warning log. ``` zstd compression feature is an experimental new feature supported since v1.19.0. Please make sure that the destination server also supports this feature before using it. in_forward plugin for Fluentd supports it since v1.19.0. ``` In #4758, it was agreed that we would refrain from updating [the forward protocol v1.5](https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1.5) for the time being and instead output a log message indicating that this feature is experimental. > for now, we will make zstd available as an exprimenatal feature on the Fluentd side, and note in the documentation that it is an experimental feature that cannot be used unless the server supports it. > Just displaying as a warning log to tell: "This is experimental feature and not standardized yet" or similar message should be enough. * #4758 (comment) * #4758 (comment) * #4758 (comment) **Docs Changes**: Not needed. (It should be done as #4657) **Release Note**: Not needed. (A note for #4657 would be enough.) Signed-off-by: Daijiro Fukuda <[email protected]>

**Which issue(s) this PR fixes**: Fixes #5005 **What this PR does / why we need it**: This PR will fix RuntimeError with large zstd compressed data **Docs Changes**: Not needed because this feature has not been released yet. **Release Note**: Should be described with #4657 Signed-off-by: Shizuo Fujita <[email protected]>

Athishpranav2003 force-pushed the zstd_comp_support branch from 2b8c9d1 to 4143998 Compare October 3, 2024 18:15

Athishpranav2003 marked this pull request as draft October 3, 2024 18:16

Athishpranav2003 force-pushed the zstd_comp_support branch 2 times, most recently from 6bf74ae to 0b79317 Compare October 3, 2024 19:04

daipom self-requested a review October 4, 2024 01:29

daipom mentioned this pull request Oct 21, 2024

Add Zstd compression support to S3 plugin fluent/fluent-plugin-s3#439

Merged

daipom reviewed Nov 2, 2024

View reviewed changes

Athishpranav2003 force-pushed the zstd_comp_support branch from 0b79317 to dbbd2f6 Compare November 3, 2024 05:01

daipom requested changes Nov 5, 2024

View reviewed changes

lib/fluent/plugin/in_forward.rb Outdated Show resolved Hide resolved

lib/fluent/plugin/in_forward.rb Outdated Show resolved Hide resolved

Athishpranav2003 added 2 commits November 18, 2024 14:20

Added zstd support for msgs

139ecad

Signed-off-by: Athish Pranav D <[email protected]>

Added UTs

7d837b6

Signed-off-by: Athish Pranav D <[email protected]>

Athishpranav2003 force-pushed the zstd_comp_support branch from dbbd2f6 to 7d837b6 Compare November 18, 2024 08:50

Athishpranav2003 force-pushed the zstd_comp_support branch from 9696e0d to c44175c Compare December 26, 2024 11:16

Athishpranav2003 marked this pull request as ready for review December 26, 2024 11:16

daipom reviewed Dec 27, 2024

View reviewed changes

lib/fluent/plugin/out_file.rb Outdated Show resolved Hide resolved

Athishpranav2003 force-pushed the zstd_comp_support branch from c44175c to dc13f85 Compare December 28, 2024 05:12

Athishpranav2003 requested a review from daipom December 28, 2024 05:14

daipom reviewed Jan 6, 2025

View reviewed changes

lib/fluent/plugin/out_file.rb Outdated Show resolved Hide resolved

daipom mentioned this pull request Jan 6, 2025

Update Forward Protocol Specification for zstd compression support #4758

Open

Athishpranav2003 added 2 commits January 6, 2025 14:11

out_file+out_forward - zstd addition

6bbb5a9

Signed-off-by: Athish Pranav D <[email protected]>

UTs for Out-forward and Out-file

cdb963f

Signed-off-by: Athish Pranav D <[email protected]>

Athishpranav2003 force-pushed the zstd_comp_support branch from dc13f85 to cdb963f Compare January 6, 2025 08:41

daipom added this to the v1.19.0 milestone Jan 6, 2025

daipom previously approved these changes Jan 6, 2025

View reviewed changes

Merge branch 'master' into zstd_comp_support

bb48e4e

daipom dismissed their stale review via bb48e4e January 21, 2025 02:04

daipom approved these changes Jan 21, 2025

View reviewed changes

daipom merged commit 30c3ce0 into fluent:master Jan 21, 2025
10 checks passed

daipom mentioned this pull request Feb 19, 2025

Use zstandard for buffer compression #3545

Closed

daipom mentioned this pull request Apr 9, 2025

out_forward: add warning for zstd as experimental #4896

Merged

This was referenced Jun 19, 2025

in_forward: RuntimeError with large zstd compressed data #5005

Closed

compressable: fix RuntimeError with large zstd compressed data #5006

Merged

daipom mentioned this pull request Jun 27, 2025

add zstd compression support fluent/fluentd-docs-gitbook#591

Open

compressable:Zstd comp support #4657

compressable:Zstd comp support #4657

Uh oh!

Conversation

Athishpranav2003 commented Oct 3, 2024 • edited by daipom Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Athishpranav2003 commented Oct 3, 2024

Uh oh!

daipom commented Oct 4, 2024

Uh oh!

Athishpranav2003 commented Oct 16, 2024

Uh oh!

daipom commented Oct 17, 2024

Uh oh!

Athishpranav2003 commented Oct 17, 2024

Uh oh!

daipom commented Oct 21, 2024

Uh oh!

Athishpranav2003 commented Oct 21, 2024

Uh oh!

Athishpranav2003 commented Oct 23, 2024

Uh oh!

daipom commented Oct 25, 2024

Uh oh!

Athishpranav2003 commented Oct 25, 2024

Uh oh!

Athishpranav2003 commented Oct 29, 2024

Uh oh!

daipom commented Oct 30, 2024

Uh oh!

daipom left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Athishpranav2003 commented Nov 3, 2024

Uh oh!

daipom left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Athishpranav2003 commented Nov 5, 2024

Uh oh!

daipom commented Nov 5, 2024

Uh oh!

daipom commented Nov 5, 2024

Uh oh!

Athishpranav2003 commented Nov 18, 2024

Uh oh!

Athishpranav2003 commented Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Athishpranav2003 commented Dec 20, 2024

Uh oh!

daipom commented Dec 26, 2024

Uh oh!

Athishpranav2003 commented Dec 26, 2024

Uh oh!

Uh oh!

Athishpranav2003 commented Jan 3, 2025

Uh oh!

Uh oh!

daipom commented Jan 6, 2025

Uh oh!

daipom commented Jan 6, 2025

Uh oh!

daipom commented Jan 6, 2025

Uh oh!

daipom left a comment

Choose a reason for hiding this comment

Uh oh!

daipom left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Athishpranav2003 commented Oct 3, 2024 •

edited by daipom

Loading

daipom left a comment •

edited

Loading

Athishpranav2003 commented Nov 18, 2024 •

edited

Loading

daipom left a comment •

edited

Loading