Remove Prometheus Metrics When Kubernetes Pods Are Deleted

[borg-z]

Motivation

Currently, Falcosidekick integrates with Prometheus to expose metrics related to Falco events. However, there is an issue where metrics associated with Kubernetes pods are not being cleaned up when those pods are deleted from the cluster. For example, when a pod named dns-cz89z is removed, the corresponding metric:

falco_events{k8s_ns_name="default", k8s_pod_name="dns-cz89z", rule="Exec from shell in container", tier="cluster"}

remains in Prometheus. Over time, this leads to an accumulation of metrics for pods that no longer exist, resulting in a cluttered metrics database and potential confusion when monitoring the cluster's state. This issue affects the reliability and accuracy of monitoring, making it difficult to distinguish between active and inactive resources.

Feature

Implement one of the following solutions:

• Cleanup Mechanism:
  Automatically remove Prometheus metrics associated with Kubernetes pods when those pods are deleted from the cluster.

• Change Metric Type:
  Modify the falco_events metric from a CounterVec to a GaugeVec

Alternatives

Additional context

Example of the issue in action:

After deleting the pod dns-cz89z, the following metric remains:

falco_events{k8s_ns_name="default", k8s_pod_name="dns-cz89z", rule="Exec from shell in container", tier="cluster"}

Over time, these stale metrics accumulate, leading to performance degradation and cluttered monitoring dashboards.

[Issif]

Hi,

I understand your point, but I don't see any way to tell falcosidekick that the source pod doesn't exist anymore. And I'll not change the metric type, to avoid any breaking change for the users. I don't know how other applications handle this situation.

[borg-z]

Thanks for the quick reply! I will think of ways to solve the problem

[Issif]

Maybe a periodic restart might help.