Outbound http permits metrics

[rylev]

Partially fixes #3307

I didn't implement a gauge of available_permits for two reasons:

• We don't yet support gauges in spin_telemetry
• The semaphore has no API for getting the number of total permits which means we have to thread that book keeping through which I didn't feel like doing.

Curious what folks think about this.

[lann]

I'll take it over nothing! It would be good to have available_permits as a leading indicator of (potential) problems but the waited count should be useful on its own as a trailing indicator.

[rylev]

@lann @calebschoepp the only thing I'm unsure about is whether monotonic_counter! is the right way to emit metrics like these. It seems like the implementation is simply using tracing::trace! which I don't think would do what we want.

[calebschoepp]

@lann @calebschoepp the only thing I'm unsure about is whether monotonic_counter! is the right way to emit metrics like these. It seems like the implementation is simply using tracing::trace! which I don't think would do what we want.

It should work. It is using tracing_opentelemetry::MetricsLayer. Maybe a comment could be thrown in spin_telemetry to make that more obvious? Also looking at the docs I linked it seems to support gauges so you could unblock yourself on that front