THREESCALE-9682: Add support for OpenTelemetry

[jlledom]

This adds OpenTelemetry instrumentation for Sinatra in Apisonator.

How to verify:

1. Follow the instructions on the docs to launch a Jaeger container.
2. Set OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" and the new var CONFIG_OPENTELEMETRY_ENABLED=true.
3. Work normally, e.g. send some transaction requests to the listener.
4. Visit Jaeger UI at http://localhost:16686 and verify the requests are reported.
5. Verify that not setting CONFIG_OPENTELEMETRY_ENABLED or setting it to false disables the metrics and you can't see the requests in Jaeger.

Jira Issue:
https://issues.redhat.com/browse/THREESCALE-9682

[akostadinov]

Is it possible by any chance to test that the endpoint is operational with the said env variable? Maybe not, just asking.

[jlledom]

Is it possible by any chance to test that the endpoint is operational with the said env variable? Maybe not, just asking.

I don't think there's a way other than pinging the endpoint host and port directly.

[akostadinov]

Fair enough. But how do we know whether this fixes the actual JIRA issue THREESCALE-9669?

[jlledom]

Fair enough. But how do we know whether this fixes the actual JIRA issue THREESCALE-9669?

The issue is not only about apisonator. As I understood it, the only apisonator work required is:

3scale Backend should be instrumented using opentelemetry SDK. Sinatra has instrumentation in place implemented in https://github.com/open-telemetry/opentelemetry-ruby-contrib/tree/main/instrumentation/sinatra

So there's no need to check any endpoint AFAIK. @eguzki ?

[eguzki]

That's the one jira issue describing the work being done here: https://issues.redhat.com/browse/THREESCALE-9682

[eguzki]

LGTM

I left one suggestion, but it's a nit.

[eguzki]

It would be nice to drop an snapshot from jaeger UI with the trace being recorded from backend listener

[eguzki]

This is only executed from the backend listener?

[jlledom]

I'd say it only makes sense for the listener, which is who listens to sinatra routes

[eguzki]

the worker is also importing this file.

What about lib/3scale/backend/server.rb ?? I did not try myself, though. Looks like lib/3scale/backend/listener.rb is being imported by both worker and listener 🤷

Not important, though.. up to you.. not a request for change. The worker will not have the opt-in switch enabled, so it is harmless.

[jlledom]

Fixed: a45900b

[jlledom]

It would be nice to drop an snapshot from jaeger UI with the trace being recorded from backend listener

Some screenshots: