Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After update to 1.8 version and when using the ASP .Net Core Prometheus exporter and /metrics end point I get error on Prometheus some times. #5506

Closed
jnormen opened this issue Apr 4, 2024 · 3 comments
Labels
bug Something isn't working pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package

Comments

@jnormen
Copy link

jnormen commented Apr 4, 2024

Bug Report

List of all OpenTelemetry NuGet
packages
and version that you are
using (e.g. OpenTelemetry 1.0.2):

Metrics I use:
.AddMeter("Microsoft.AspNetCore.Hosting",
"Microsoft.AspNetCore.Routing",
"Microsoft.AspNetCore.Diagnostics",
"Microsoft.AspNetCore.Server.Kestrel",
"Polly",
"MassTransit"

Symptom

PrometheusTargetMissing ***** (http monitor/prometheus critical)[FIRING:3]

With info:
A Prometheus target has disappeared. An exporter might be crashed.

The text mention:
"expected equal, got INVALID"

When I revert to version 1.7 the problem stops with the same meters on.

Same error in beta and rc1 version of 1.8 exporter. Not a valid output.

What did you see instead?

I get alarm from Prometheus with:
description = A Prometheus target has disappeared. An exporter might be crashed. VALUE = 0 LABELS =

Reproduce

Use 1.8 version of Open Telemetry and the 1.8 rc1 of the Prometheus asp.net core exporter

@jnormen jnormen added the bug Something isn't working label Apr 4, 2024
@vishweshbankwar vishweshbankwar added the pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package label Apr 4, 2024
@Abrynos
Copy link

Abrynos commented Apr 7, 2024

This issue is caused by Prometheus adding the Accept: application/openmetrics-text header. I've got this reproducible with both Prometheus version 2.45.3+ds-3 (currently in Debian testing) and with

curl -H "Accept: application/openmetrics-text" {protocol}://{host}:{port}/{configured_endpoint} # incorrect response body

In comparison, the response comes out flawlessly using either command-line below or OpenTelemetry.Exporter.Prometheus.AspNetCore version 1.7.0-rc1 or earlier

curl {protocol}://{host}:{port}/{configured_endpoint} # works

Here's a sample response camptured via curl: test.txt


Related PR: JustArchiNET/ArchiSteamFarm#3183

Additional thanks to @ezhevita for investing your time as well.

@martinjt
Copy link
Member

I believe this the same as #5515, which is being fixed. If that seems right, we can close this in favour of that.

@reyang
Copy link
Member

reyang commented May 21, 2024

Duplicated with #5515. The bug was introduced by #5107 and fixed by #5623.

@reyang reyang closed this as not planned Won't fix, can't repro, duplicate, stale May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package
Projects
None yet
Development

No branches or pull requests

5 participants