Nginx: Improve tracing

[tiithansen]

1. Add possibility to use HTTP OTLP exporter.
2. Improve tracing so that module creates internal or client span depending if the request is handled by Nginx directly or passed upstream.
3. Set span status depending on result. If status code is between 400 - 599 set status as error.
4. Catch internal processing failures in log phase.
5. Rewrite tests in JS

Because of how Nginx works we can't determine what module will serve the actual request before its served. For what purpose I have implemented ProxyRecordable which makes it possible to change the span kind after processing in log phase.

[tiithansen]

@esigo any plans with this one?

[vainikkaj]

I've got mixed feelings regarding span status. For example, it isn't always clear whether 4xx status codes are "errors". Does OTEL specification have clear stance on these?

Perhaps configuration flag for defining which status codes are viewed as span errors would be enough?

[huggsboson]

Just wanted to add an upvote on this one, I reached out to the nginx team and they were resistent to adding client spans to the instrumentation itself, so this seems like the best path forward:
nginxinc/nginx-otel#17

[huggsboson]

I've got mixed feelings regarding span status. For example, it isn't always clear whether 4xx status codes are "errors". Does OTEL specification have clear stance on these?

Perhaps configuration flag for defining which status codes are viewed as span errors would be enough?

I wouldn't consider 4xx's errors either, every time we did it in our other instrumentation it lead to a lot of false positives / noise.

[johanneswuerbach]

@esigo / @seemk what would be required to move this forward?