Truncate long lines more than buffer_max_size for Tail Input Plugin

[prafgup]

Is your feature request related to a problem? Please describe.

Currently the Tail Input plugin doesn't process log messages that are more than the buffer_max_size and just skips them in case skip_long_lines is enabled.

• https://docs.fluentbit.io/manual/dev-3.0/pipeline/inputs/tail

Describe the solution you'd like

There should be an option to truncate the message (maybe a new parameter max_log_size) to either default to buffer_max_size or be configurable to a given size.

Currently the CloudWatch Logs Output plugin has this truncating feature

• https://github.com/fluent/fluent-bit/blob/master/plugins/out_cloudwatch_logs/cloudwatch_api.c#L340-L366

The AWS CloudWatch Agent does the same logic of truncating big log lines to 256KB

• https://github.com/aws/amazon-cloudwatch-agent/blob/9516943b3ff95c72b949d2a155c014758db0c891/plugins/inputs/logfile/logfile.go#L85-L88

After truncation it reopens the file for tailing -

• https://github.com/aws/amazon-cloudwatch-agent/blob/9516943b3ff95c72b949d2a155c014758db0c891/plugins/inputs/logfile/tail/tail.go#L466-L471

Describe alternatives you've considered

Additional context

I am trying to use a Tail Input Plugin with a HTTP Output plugin where the input log line can be arbitrary long but the HTTP Output can only process 256KB of a single log line.
I would want FluentBit Tail plugin to be able to truncate the big log line >256KB. Also for my use case the mem_buf_limit can not be more than 1MB thus we can not fit very large log lines into the buffer and use Filters to truncate the log lines.