feat(slo): Avoid false positive burn rate alerting with partial rolled-up data #203279

kdelemme · 2024-12-06T16:33:20Z

Resolves #190143

🏇🏻 Summary

This PR makes the burn rate rule less prone to false positive alerting when the rolled-up data is not fully computed for the burn rate window or when the rolled-up data consist of intermittent data, e.g. low-traffic service's SLO.

This is achieved by using the observed bad events, e.g. observed total events - observed good events, and the total slices expected for the given window duration and SLO timeslice duration, e.g. there are 60 1min-slices in a 1h window or 30 2min-slices in a 1h window.

For example, if we have only 1 total event observed and 0 good event observed during a 1h window (using 1min slices), the new burn rate becomes 1/60/(1-objective) instead of 1/(1-objective). The new burn rate is 60x smaller than the previous, which would avoid triggering the alert.

Note

Did some housecleaning in the burn rate rule as well: Adding slo.revision term filter, reusing function to generate aggs keys

🧬 Testing

buildQuery tests snapshots have been updated with the new expected aggs

…ange

elasticmachine · 2024-12-06T19:46:05Z

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

kdelemme · 2024-12-06T20:02:10Z

x-pack/plugins/observability_solution/slo/server/lib/rules/slo_burn_rate/lib/build_query.ts

+  const source = burnRateWindows
+    .map((_windDef, index) => {
+      const windowId = `${WINDOW}_${index}`;
+      return `(params.${generateAboveThresholdKey(
+        windowId,
+        SHORT_WINDOW
+      )} == 1 && params.${generateAboveThresholdKey(windowId, LONG_WINDOW)} == 1)`;
+    })
+    .join(' || ');


💬 I find map().join() easier to follow than a reduce with a ternary operator

github-actions · 2024-12-06T20:02:22Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

kdelemme · 2024-12-06T20:03:08Z

.../plugins/observability_solution/slo/server/lib/rules/slo_burn_rate/lib/summary_repository.ts

-                term: { 'slo.instanceId': instanceId },
-              },
+              { term: { 'slo.id': slo.id } },
+              { term: { 'slo.revision': slo.revision } },


💬 Filtering on the slo.revision won't change much in most cases, but cannot hurt when the SLO is updated and the previous data is still available when the rule runs

kdelemme · 2024-12-06T20:04:01Z

x-pack/plugins/observability_solution/slo/server/lib/rules/slo_burn_rate/lib/build_query.ts

+  // For timeslice budgeting method, we always compute the burn rate based on the observed bad slices, e.g. total observed  - good observed = bad slices observed,
+  // And we compare this to the expected slices in the whole window duration
+  const burnRateAgg = isTimesliceBudgetingMethod


💬 The main change is here.

elasticmachine · 2025-01-07T18:14:01Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: a460bc1
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-203279-a460bc1b7374

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #6 / discover/security/context_awareness security root profile cell renderers host.name DataView mode should open host.name flyout
[job] [logs] Jest Tests #1 / StepDefinePackagePolicy default API response should display vars coming from package policy

Metrics [docs]

✅ unchanged

History

💚 Build #264417 succeeded 0d6a281
💚 Build #263791 succeeded 54b3e71
💚 Build #263418 succeeded 08deb4d
💚 Build #257943 succeeded fba2696

cc @kdelemme

dominiqueclarke

LGTM! Alert did not fire immediately.

dominiqueclarke · 2025-01-07T18:13:08Z

...solutions/observability/plugins/slo/server/lib/rules/slo_burn_rate/lib/summary_repository.ts

-                term: { 'slo.instanceId': instanceId },
-              },
+              { term: { 'slo.id': slo.id } },
+              { term: { 'slo.revision': slo.revision } },


kibanamachine · 2025-01-07T20:21:42Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12658708629

…d-up data (elastic#203279) (cherry picked from commit 0e13d86)

kibanamachine · 2025-01-07T20:27:09Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…d-up data (elastic#203279)

… rolled-up data (#203279) (#205806) # Backport This will backport the following commits from `main` to `8.x`: - [feat(slo): Avoid false positive burn rate alerting with partial rolled-up data (#203279)](#203279)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Kevin Delemme <[email protected]>

…d-up data (elastic#203279)

kdelemme added 7 commits December 5, 2024 14:11

Remove unused dependency

4dc21ef

Filter with the slo revision as well

e7839b2

Use function instead of hardcoding keys

317fce9

Simplify source script builder

64f27df

Use total slices as computed from the timeslice window and the date r…

b6622bc

…ange

rename slices util function

560d70b

Update tests

5316080

kdelemme added release_note:skip Skip the PR/issue when compiling release notes backport:prev-minor Backport to (8.x) the previous minor version (i.e. one version back from main) Team:obs-ux-management Observability Management User Experience Team v8.18.0 labels Dec 6, 2024

kdelemme self-assigned this Dec 6, 2024

Merge branch 'main' into feat/burn-rate-rule-improvement

fba2696

kdelemme marked this pull request as ready for review December 6, 2024 19:46

kdelemme requested a review from a team as a code owner December 6, 2024 19:46

kdelemme commented Dec 6, 2024

View reviewed changes

botelastic bot added the ci:project-deploy-observability Create an Observability project label Dec 6, 2024

kdelemme commented Dec 6, 2024

View reviewed changes

kdelemme added 3 commits December 30, 2024 12:37

Merge branch 'main' into feat/burn-rate-rule-improvement

08deb4d

Merge branch 'main' into feat/burn-rate-rule-improvement

54b3e71

Merge branch 'main' into feat/burn-rate-rule-improvement

0d6a281

dominiqueclarke self-requested a review January 7, 2025 16:28

Merge branch 'main' into feat/burn-rate-rule-improvement

a460bc1

dominiqueclarke approved these changes Jan 7, 2025

View reviewed changes

kdelemme merged commit 0e13d86 into elastic:main Jan 7, 2025
8 checks passed

kdelemme deleted the feat/burn-rate-rule-improvement branch January 7, 2025 20:21

kibanamachine added the v9.0.0 label Jan 7, 2025

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jan 7, 2025

feat(slo): Avoid false positive burn rate alerting with partial rolle…

32e3626

…d-up data (elastic#203279) (cherry picked from commit 0e13d86)

kibanamachine mentioned this pull request Jan 7, 2025

[8.x] feat(slo): Avoid false positive burn rate alerting with partial rolled-up data (#203279) #205806

Merged

kowalczyk-krzysztof pushed a commit to kowalczyk-krzysztof/kibana that referenced this pull request Jan 7, 2025

feat(slo): Avoid false positive burn rate alerting with partial rolle…

901a404

…d-up data (elastic#203279)

crespocarlos pushed a commit to crespocarlos/kibana that referenced this pull request Jan 8, 2025

feat(slo): Avoid false positive burn rate alerting with partial rolle…

e79d328

…d-up data (elastic#203279)

CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Jan 13, 2025

feat(slo): Avoid false positive burn rate alerting with partial rolle…

14d3ea8

…d-up data (elastic#203279)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(slo): Avoid false positive burn rate alerting with partial rolled-up data #203279

feat(slo): Avoid false positive burn rate alerting with partial rolled-up data #203279

kdelemme commented Dec 6, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Dec 6, 2024

kdelemme Dec 6, 2024 •

edited

Loading

github-actions bot commented Dec 6, 2024

kdelemme Dec 6, 2024

kdelemme Dec 6, 2024

elasticmachine commented Jan 7, 2025 •

edited

Loading

dominiqueclarke left a comment

dominiqueclarke Jan 7, 2025

kibanamachine commented Jan 7, 2025

kibanamachine commented Jan 7, 2025

feat(slo): Avoid false positive burn rate alerting with partial rolled-up data #203279

feat(slo): Avoid false positive burn rate alerting with partial rolled-up data #203279

Conversation

kdelemme commented Dec 6, 2024 • edited by kibanamachine Loading

🏇🏻 Summary

🧬 Testing

elasticmachine commented Dec 6, 2024

kdelemme Dec 6, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Dec 6, 2024

🤖 GitHub comments

kdelemme Dec 6, 2024

Choose a reason for hiding this comment

kdelemme Dec 6, 2024

Choose a reason for hiding this comment

elasticmachine commented Jan 7, 2025 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

dominiqueclarke left a comment

Choose a reason for hiding this comment

dominiqueclarke Jan 7, 2025

Choose a reason for hiding this comment

kibanamachine commented Jan 7, 2025

kibanamachine commented Jan 7, 2025

💚 All backports created successfully

Questions ?

kdelemme commented Dec 6, 2024 •

edited by kibanamachine

Loading

kdelemme Dec 6, 2024 •

edited

Loading

elasticmachine commented Jan 7, 2025 •

edited

Loading