Add support for writing each batch of records as a CLP IR file; Replace Fluent Bit Go MessagePack decoder with our own implementation.

[davemarco]

Description

Plugin can now retrieve chunks from the fluent-bit engine and encode them as compressed IR. After encoding, the plugin will write the chunk to a local file. I added the following files:

• decoder.go to fix the msgpack string decoding issue mentioned in the last PR
• ir.go to help with conversion to IR

The next step is to send this same chunk to S3.

single key

A fluent-bit record is a JSON-like object, and while CLP can parse JSON into IR it is not recommended. To resolve this issue, the user can specify a key, and the plugin will look for the key in the fluent-bit record, and output the key's value instead of the whole record. Typically, fluent-bit configurations with no parser setup will have the key "log" in the record containing the unstructured log, so the user should specify the key "log" to convert their unstructured logs into IR. As a fallback for logs missing a key, and for users who do want to compress parsed JSON, the plugin does support converting the entire JSON record.

warning

Do not run this development version for long durations. It will generate a small file every second to your local system.

Validation performed

Confirmed that CLP package can compress and search the compressed IR outputted by the plugin

[davemarco]

reverted context back to config since wasn't being used

[davidlion]

Some small tweaks to doc strings / comments.

[davidlion]

Add support for writing each batch of records as a CLP IR file; Replace Fluent Bit Go MessagePack decoder with our own implementation.