Integrate Tracing (derived from OpenTelemetry)

[CMCDragonkai]

Specification

OpenTelemetry is an overly complicated beast. It's far too complex to adopt into a logging system. However the basic principles of tracing makes sense. Here I'm showing how you can set one up for comparison testing, for us to derive a tracing schema and later visualise it ourselves or by passing it into an OTLP compatible visualiser.

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.36

The above command runs jaeger. Take note of 4318 port which is the OTLP protocol over HTTP.

Visit localhost:16686 to be able to view the jaeger system.

Then any example code, like for example https://github.com/open-telemetry/opentelemetry-js/blob/main/examples/basic-tracer-node/index.js can run and push traces directly to the docker container.

What is frustrating is:

1. OpenTelemetry code only exports to stderr as an afterthought, it's not considered first class usage
2. The stderr exporters output via console.log and produce pretty printed results that are not actual JSON. Thus you cannot just pipe it to a relevant location.
3. The schema of the span data isn't clear, it seems different parts of the documentation still have old data, or maybe the JS implementation itself is hasn't been updated to the new schema.

The plan:

1. Create your own "span" derived from opentelemetry and output as just regular structured JSON
2. Massage it to be compatible to open telemetry viewers like jaeger
3. Use jaeger's 4318 to stream the JSON and view data in the interim
4. Find an easier way to visualise traces, maybe something that can be used CLI or in the GUI
5. For production usage, feed to any structured log capturer, and then feed into a viewer that understands trace information

Additional context

• Structured JSON Logging #14
• Integrate Structured Logging #3 (comment)
• Integrate Structured Logging #3 (comment)
• Integrate Structured Logging #3 (comment)
• ENG-100

Tasks

1. ...
2. ...
3. ...

[CMCDragonkai]

It seems alot of the complexity is due to the vendors fragmentation and they are trying to make everything compatible.

[CMCDragonkai]

Most tracing tools like https://nodejs.org/api/tracing.html and chrome:://tracing expect a finite dataset, that is expected that a trace has a beginning and end. That's why it's always been "request" driven. Open telemetry is just deriving stuff that came before like in https://github.com/gaogaotiantian/viztracer https://github.com/janestreet/magic-trace https://github.com/kunalb/panopticon and more.

I'm interested in more than just request-driven tracing but live infinite traces (call it continuous tracing that shows finished and live spans at the same time), and correlates them too. I'm guessing we need zoomable levels of detail the ability to filter out irrelevant information dynamically.

Open telemetry in particular does not appear to emit a span until it is done. I'd imagine knowing when a span started even if it did not end yet would be useful for live continuous tracing.