Ignore metrics read over prometheus with NaN value (#1231)

* Skip reading counters and gauges with NaN value. Signed-off-by: Caleb Metz <[email protected]> * Spelling Signed-off-by: Caleb Metz <[email protected]> * Adds changelog entry --------- Signed-off-by: Caleb Metz <[email protected]> Co-authored-by: Scott Opell <[email protected]>
DataDog · Feb 6, 2025 · b7b446f · b7b446f
1 parent b06706c
commit b7b446f
Show file tree

Hide file tree

Showing 2 changed files with 11 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Introduced the ability for users to configure lading's sample rate,
   configuration option `sample_period_milliseconds` in `lading.yaml`.
 - Users can now configure expvar scraping on https endpoints, skipping certificate validation.
+- Fixes issue when parsing `NaN` values from a prometheus endpoint
 
 ## [0.25.4]
 ## Changed

diff --git a/lading/src/target_metrics/prometheus.rs b/lading/src/target_metrics/prometheus.rs
@@ -276,6 +276,11 @@ pub(crate) async fn scrape_metrics(
                     }
                 };
 
+                if value.is_nan() {
+                    warn!("Skipping NaN gauge value");
+                    continue;
+                }
+
                 gauge!(format!("target/{name}"), &all_labels.unwrap_or_default()).set(value);
             }
             Some(MetricType::Counter) => {
@@ -287,6 +292,11 @@ pub(crate) async fn scrape_metrics(
                     }
                 };
 
+                if value.is_nan() {
+                    warn!("Skipping NaN counter value");
+                    continue;
+                }
+
                 let value = if value < 0.0 {
                     warn!("Negative counter value unhandled");
                     continue;