out_opentelemetry: add option to preserve fields in log body

[niedbalski]

Summary

• Add new configuration option logs_body_preserve that allows record fields to be retained in the log body when they're used/processed by the OpenTelemetry output plugin

Details

1. The OpenTelemetry Protocol (OTLP) format separates data into two main parts:

   • A log record body (typically containing the main message)
   • Attributes (key-value pairs for metadata and other fields)

2. The OpenTelemetry output plugin converts Fluent Bit records to this format by:

   • Setting the log body based on logs_body_key settings
   • Moving remaining fields to attributes when logs_body_key_attributes: true is enabled

3. The key issue is in log_record_set_attributes function (line ~231):

   if (ctx->logs_body_key_attributes == FLB_TRUE && ctx->mp_accessor && ra_match) {
       /* Code that removes fields from the record */
       ret = flb_mp_accessor_keys_remove(ctx->mp_accessor, event->body, &out_buf, &out_size);
       /* ... */
   }

4. When the processor_calyptia adds the field "hello: world", this happens:

   • The field is added to the record
   • When processed by the OpenTelemetry output, this field is:
     • Not matched by any logs_body_key pattern
     • So it gets moved to the attributes section completely
     • The field is removed from the record body

5. In contrast, the stdout output just prints whatever it receives, so you see all fields preserved.

6. The logs_body_preserve option now strictly does what its name suggests - it preserves record fields in the log body without forcing them to be added as attributes.

7. Fields are only added to attributes if:

   • logs_body_key_attributes=true is set
   • AND either:
     • Fields were removed from the record (unpacked is true), OR
     • Fields were preserved (logs_body_preserve=true) and we're still adding them to attributes

With this change:

• If you set only logs_body_preserve=true, fields will remain in the record but won't be added to attributes
• If you set both logs_body_key_attributes=true and logs_body_preserve=true, fields will remain in the record AND be added to attributes

Configuration Example

# Sender configuration with logs_body_preserve enabled
pipeline:
  inputs:
    - name: dummy
      tag: dummy.log
      dummy:
        rate: 1
        samples: |
          {"message":"dummy"}
      processors:
        logs:
          - name: calyptia
            actions:
              - type: put
                opts:
                  key: hello
                  value: world
  outputs:
    - name: stdout
      match: "*"

    - name: opentelemetry
      match: "*"
      host: receiver
      port: 4318
      logs_uri: /v1/logs
      logs_body_preserve: true

Expected Output

When the OpenTelemetry output plugin sends logs to a receiver with logs_body_preserve: true, the log body should contain all fields from the original record:

[0] v1_logs: [[1744281011.1334685613, {"otlp"=>{"observed_timestamp"=>0, "timestamp"=>1744281011912789344, "trace_flags"=>0}}], {"message"=>"dummy", "hello"=>"world"}]

Without this option, the field used to populate the log body would be removed, resulting in only one of the fields being present in the output.

[edsiper]

@niedbalski thanks for providing this PR.

After reviewing the usability aspect, it seems to me that the main purpose of this is to have the ability to retain certain keys as part of the record body and today we only support one key.

Based on the expected output provided, it seems we want the body to support multiple keys and optionally set the others as attributes.

Would this high-level-poc-config do what is intended ?

pipeline:
  inputs:
    - name: dummy
      dummy: '{"message":"this is the message"}'
      processors:
        logs:
          - name: content_modifier
            action: insert
            key: hello
            value: world
          - name: content_modifier
            action: insert
            key: attr
            value: some_attr_value

  outputs:
    - name: stdout
      match: "*"

    - name: opentelemetry
      match: "*"
      host: 127.0.0.1
      port: 4328
      logs_uri: /v1/logs
      logs_body_key: 
        - message
        - hello
      logs_body_key_attributes: true

so attributes will have the attr key while the record body a map with message and hello ?