Clarification on DataDog Observability Metrics

[rimbesat]

Our Dashboard currently does show the majority of our experiments. However, on a few experiments the chaos.controller.disruptions.count displays a value below 1.
For example:

In most of these cases the experiment has a count metric of 0.1.

We would like to know if you have come across this issue before, or any ideas of what has gone wrong?

[ptnapoleon]

These screenshots are from a DataDog dashboard? The metrics themselves are hopefully being emitted correctly, but they might be rolled up incorrectly

[ptnapoleon]

Could you please confirm a way to validate if the metrics are being emitted correctly?

I don't know a way other than using DataDog to confirm we're seeing metrics when expected.

Also, Could you provide more context on metrics getting rolled up incorrectly?

If the metrics are being evaluated as a rate rather than a count on the query side, it would result in these small floating points

[rimbesat]

Alright thanks,
Is there an option to have a metrics endpoint for the controller so we can see metrics, or its only done via the push method, dogstatsd?

[ptnapoleon]

@Devatoria Do you know? ^ I do not

[Devatoria]

Excuse me for the long delay in response. As of today we can't do that, we're using the Datadog statsd library directly. However, the metrics package uses and interface so we can implement other ways to expose metrics. We can imagine implementing another way (prometheus for instance) of exposing metrics. It is not that complex as it is just implementing the interface.

[nikos912000]

Hi @ptnapoleon @Devatoria. Thanks for responding.
We have filed a DataDog ticket for this. After spending some time on it, this might have to do with a problematic cluster that causes the decimals. I hope the agent flare will give us more context.