[Bug]: Expected value after metric, got "#" ("INVALID") while parsing: "Resilience_polly_str#"

[sumjas]

Describe the bug

I'm facing a recurring issue in my deployed environments while scraping metrics using Prometheus. The error message reads:
"expected value after metric, got "#" ("INVALID") while parsing: "Resilience_polly_str#"

I'm using Polly library version 8 for resiliency, which internally uses OpenTelemetry for instrumentation. The metrics are being exported using a Prometheus exporter.

here is the error details

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

Exception(s) (if any)

No response

Polly version

8.2.1

.NET Version

net6.0

Anything else?

No response

[martincostello]

Based on our documentation, the metrics that correlate with what you've shared in your screenshot should be resilience.polly.strategy.*.

Is something in your code changing the metric names and corrupting them as part of the Prometheus export process?

If not, we'll need a minimal reproducible example as a GitHub repository that demonstrates the issue you're experiencing so we can look into this further.

[martincostello]

Also, try upgrading to the latest version (8.4.1) to see if that makes any difference.

[sumjas]

@martincostello, we are using only the Polly inbuilt metrics not any custom metrics and no code in the system changes the name of any of these metrics. We are using the OpenTelemetry.Exporters.Promotheus.AspnetCore library. Not sure whether it corrupts the name of the metrics internally, I have also created a bug with OpenTelemetry Dev community. Let's see if we can get some information from them.

I will try to have a minimal reproducible example if that can help.