Custom ITextFormatter BodyFormatter for log message bodies

[DevProJSSV]

Hello team,

I am trying to find a way to format otlp logs when using the opentelemetry sink :

Log.Logger = new LoggerConfiguration()
    .WriteTo.OpenTelemetry(options =>
    {
        options.Endpoint = "http://127.0.0.1:4317";
        options.ResourceAttributes = new Dictionary<string, object>
        {
            ["service.name"] = "test-logging-service",
            ["index"] = 10,
            ["flag"] = true,
            ["value"] = 3.14
        };
        options.Format=json;   -------- > # Something like this
    })
    .CreateLogger();

I know there is a way to do it with.WriteTo.Console() bust as I am leveraging otlp communication I would like to send all structured in Json using WriteTo.OpenTelemetry().

May you recommend me any approach?. Thank you in advance @nblumhardt

[DevProJSSV]

Based on the official documentation (https://github.com/open-telemetry/opentelemetry-proto/tree/v1.1.0/examples/) we can add Content-Type: application/json header to set the encoding to json I am trying to add it in the options.Headers like:

options.Headers = new Dictionary<string, string>
                {
                    ["Authorization"] = "Basic dXNlcjphYmMxMjM=", // user:abc123
                    ["Content-Type"] = "application/json"
                };

Unhandled exception. System.InvalidOperationException: Misused header name, 'Content-Type'. Make sure request headers are used with HttpRequestMessage, response headers with HttpResponseMessage, and content headers with HttpContent objects. at System.Net.Http.Headers.HttpHeaders.GetHeaderDescriptor(String name) at Serilog.Sinks.OpenTelemetry.HttpExporter..ctor(String endpoint, IReadOnlyDictionary2 headers, HttpMessageHandler httpMessageHandler) in /app/src/Serilog.Sinks.OpenTelemetry/Sinks/OpenTelemetry/HttpExporter.cs:line 48
at Serilog.Sinks.OpenTelemetry.OpenTelemetrySink..ctor(String endpoint, OtlpProtocol protocol, IFormatProvider formatProvider, IReadOnlyDictionary2 resourceAttributes, IReadOnlyDictionary2 headers, IncludedData includedData, HttpMessageHandler httpMessageHandler) in /app/src/Serilog.Sinks.OpenTelemetry/Sinks/OpenTelemetry/OpenTelemetrySink.cs:line 41
at Serilog.OpenTelemetryLoggerConfigurationExtensions.OpenTelemetry(LoggerSinkConfiguration loggerSinkConfiguration, Action1 configure) in /app/src/Serilog.Sinks.OpenTelemetry/OpenTelemetryLoggerConfigurationExtensions.cs:line 42 at Example.Program.NewGetLoggerOne(OtlpProtocol protocol) in /app/example/Example/Program.cs:line 110 at Example.Program.Main() in /app/example/Example/Program.cs:line 48
but I am getting an error like this :

[nblumhardt]

HI! This sink only supports the HttpProtobuf and Grpc protocol flavors (options.Protocol covers this). What collector are you using that requires HTTP/JSON?

[DevProJSSV]

Hello @nblumhardt I am using opentelemetry collector and I would like to be able to send logs from my services pods that are instrumented with serilog using the otlp opentelemetry sink as you saw above.

I mean structuring our otlp output logs to JSON as we can with :

.WriteTo.Console(formatter: new CompactJsonFormatter())

I was hoping there was an options.Encoding or something similar to send my logs in JSON.

I found in this link https://github.com/open-telemetry/opentelemetry-proto/tree/v1.1.0/examples/ from otel official docs we have to pass the Content-type header to set the encoding. I can assume.WriteTo.Console(formatter: new CompactJsonFormatter()) does it under the methods but how can I do it with :

.WriteTo.OpenTelemetry(options =>
    {
        options.Endpoint = "http://127.0.0.1:4317";
        options.ResourceAttributes = new Dictionary<string, object>
        {
            ["service.name"] = "test-logging-service",
            ["index"] = 10,
            ["flag"] = true,
            ["value"] = 3.14
        };
        options.Format=json;   -------- > # Something like this
    })

[DevProJSSV]

@nblumhardt please let me know if my details make sense for you, thank you in advance

[nblumhardt]

Hi @DevProJSSV,

I think this might be something like a "body formatter" - which isn't supported right now, but might also not be needed.

The body/message field from a source application is intended to be human-readable text, with all structured data included in the attributes sent along with the payload.

The payload sent via the sink to the collector is fully-structured - it might be that over in the collector, you need to transform the whole thing to JSON before sending it on to its ultimate destination. What's the final destination/exporter intended for the collector's output?

[DevProJSSV]

Thank you for the details @nblumhardt I am using a splunk exporter so I can send the logs to splunk platform backend. We are not doing anything on the collector side to encode the input data received at the otlp receiver. We were expecting to handle the logs format structure from the code side as we did wit the .WriteTo.Console(formatter: new CompactJsonFormatter()). But then which will be the approach I should consider for the collector configuration? Could you provide us with some references similar to my context? The only change related to the collector I could find was https://github.com/open-telemetry/opentelemetry-proto/tree/v1.1.0/examples/ which mentions we need to from the code side establish the request/response communication letting the collector know the encoding.

[RogerVFbr]

I have a similar scenario. We use a custom JSON formatter that implements a company wide logging contract. We need this specific structure to be passed over to any exporters. Without this feature we can't use the sink. It looks like this would be an important upgrade. Is there any alternative?

[nblumhardt]

I'm open to the idea of specifying an ITextFormatter BodyFormatter {..} option that would support using any of the current Serilog formatters (such as the compact JSON one) to produce the OTLP payload's body property.

Just one thing to be aware of, this would effectively put a stringified JSON object into the output location that normally contains the Message property. Is that what's needed, here?

[RogerVFbr]

@nblumhardt correct! In the context of the "LogRecord" object, this seems like the most reasonable option. I've locally forked the project and implemented the necessary changes and it worked well from the OTEL Collector's perspective. The changes I made:

• Create a new property in the OpentelemetrySinkOptions class of type ITextFormatter? called Formatter
• Create a new private property of the same type in the class OpenTelemetrySink populated via constructor.
• Adjust OpenTelemetryLoggerConfigurationExtensions accordingly.
• Receive this text formatter In the ProcessMessage method of the LogRecordBuilder class.
• If it's null, proceed as already implemented.
• If it's not, format the log event using it and populate a StringWriter object.
• Then assign the StringWriter's string output to the StringValue of an AnyValue object,
• Pass the AnyValue object to the Body property of the LogRecord object.

[nblumhardt]

Thanks for digging into it @RogerVFbr! 👍

I think the property might need to be named BodyFormatter to avoid confusion if we at some future point enable the JSON variant of OTLP, but otherwise, sounds good to me.

Are you interested in sending a PR with those changes and some minimal tests?

[DevProJSSV]

Hello is there any plan to take this option for a new release?

[nblumhardt]

Hi @DevProJSSV; it doesn't look like this will make it into 4.1.0 (just about to publish) but a minimal, tested PR should be fairly quick to get through the works if anyone watching along is able to send one.

Scanning this again briefly, I think the only major implementation point not covered so far is whether BodyFormatter would have some parallel for span names (subtly distinct from log record bodies), or whether these would somehow be combined and the supplied formatter (perhaps with a different name) used for both. MessageFormatter might be a better name/formulation if we go this direction 🤔