Add OpenTelemetry tracing to track and understand what's happening during web stack activity

[anjackson]

Our services are quite complex, and in particular the main website access stack involves some long service chains when doing things like screenshot generation.

Following https://opentelemetry.io/blog/2022/instrument-nginx/ we have to use our own Docker image to install OTEL, which is a bit sub-optimal as it makes it harder to keep things up to date (OTEL module is now > v1.0.0). This is acceptable for the stack-level NGINX instance, because it runs behind our 'front door' NGINX which is kept up to date in other ways.

Note that we can simplify that recipe because https://medium.com/jaegertracing/introducing-native-support-for-opentelemetry-in-jaeger-eb661be8183c and we can just set OTEL_EXPORTER_OTLP_ENDPOINT etc to point directly at Jaeger. As we don't have that much traffic, it should be fine to run a single-container Jaeger service for each web stack.

The OTEL project has modules and documentation for things like FastAPI and requests in various languages, so it should be possible to integrate the components without too much work.