server returned HTTP status 400 Bad Request: err: out of order sample.

[pmanno]

Hello all,

Just installed Grafana Agent 0.25.1 and seeing a lot of these in syslog:

grafana-agent[39968]: ts=2022-06-17T01:52:47.527374814Z caller=dedupe.go:112 agent=prometheus instance=5583daf8ce37d70e10717cc35cbfdeb7 component=remote level=error remote_name=5583da-a5a3bb url=https://mimir.company.com/api/v1/push msg="non-recoverable error" count=52 exemplarCount=0 err="server returned HTTP status 400 Bad Request: user=anonymous: err: out of order sample. timestamp=2022-06-17T01:52:42.466Z, series={__name__=\"prometheus_tsdb_wal_segment_current\", instance=\"127.0.0.1:9090\", instance_group_name=\"b0e0a30b02ac1e5e22f23e9309ff7a61\", job=\"grafana_agent\"}"

Any thoughts on what I should be looking at to fix this?

Thanks,
Paul

[pmanno]

Think I figured it out. The instance label was colliding with several other instances, did a relabel on that and it seems to have fixed the problem.