feat(validator): add support to validate essential metrics produced by Kepler

[vprashar2929]

This commit introduces functionality to validate essential metrics produced by Kepler
The following comparisons are included:

• Node Exporter Comparison

  • Validates node_rapl_<package|core|dram> metrics against kepler_node_<package|core|dram>{dev}

• Kepler Process Comparison

  • Compares kepler_process_<package|core|dram|platform|other|uncore>{latest} metrics to
    kepler_process_<package|core|dram|platform|other|uncore>{dev}

• Kepler Node Comparison

  • Validates kepler_node_<package|core|dram|platform|other|uncore>{latest} against
    kepler_node_<package|core|dram|platform|other|uncore>{dev}

Additionally, the following changes are made to existing functionality:

• Adds a new metric_validations.yaml file which includes promql queries for comparisons along with threshold values
• Update the existing stressor.sh script to now support few more parameters to make it more flexible
  • warmup time: time to wait before starting the stressor
  • cooldown time: time to wait after the stressor is finished
  • repeats: number of times to repeat the stressor. Since for
    regression test we don't want to repeat the stressor multiple times
• Adds a new validator-regression.yaml file which includes the configuration for the regression test

[github-actions]

🤖 SeineSailor

Here's a concise summary of the pull request changes:

Summary: This pull request enhances the validator module by introducing a new validate_metrics command to the CLI, which validates Kepler metrics and generates reports. Key modifications include:

• Added validate_metrics command with --duration and --report-dir options
• Modified PrometheusJob named tuple to include dev and latest fields
• Updated load function to initialize new fields when loading configuration from a file
• Introduced max_mae parameter to validation_from_yaml function for Maximum Absolute Error computation
• Added new functions: validate_metrics, ScriptResult, and write_md_report

Impact: These changes expand the validator's capabilities for handling and validating Kepler metrics, but do not affect exported function signatures or global data structures. The external interface and behavior are modified, requiring configuration file updates.

Observations/Suggestions:

• The changes seem well-structured and organized, with clear intentions to enhance the validator's features.
• It would be beneficial to include tests for the new validate_metrics command and its associated functions to ensure their correctness and robustness.
• Consider adding documentation or comments to explain the purpose and usage of the new max_mae parameter and the ScriptResult and write_md_report functions.
• Review the configuration file updates required for these changes to ensure a smooth transition for users.

[sthaha]

Can we also validate these invariants in the same version of dev

• kepler_node_<pkg|core|uncore|dram|other>{dev} = sum of ( process_<pkg|core|uncore|dram|other>{dev} )
• kepler_node_<pkg|core|..> = node_exporter_rapl_<pkg|core...>
  *sum( kepler_process_bpf_cpu ) = node_exporter_cpu_time

[vprashar2929]

for kepler_node_<pkg|core|dram...>{dev} = sum of (process_<pkg|core|dram....>){dev} do you mean
MAE of sum(rate(kepler_node<pkg|core|dram>){dev}[20s]) and sum(rate(process_<pkg|core|dram>{dev}[20s]))?

[vprashar2929]

Explicitly using rate interval as 60s because:

Prometheus scrape Interval = 3s
Data points for 12s Interval(i.e 4* scrape interval) = 12/3 = 4 data points
Data points for 60s interval = 60/3 = 20 data points

With 20 data points, we get a smoother and more reliable estimate. When comparing two sum(rate(...)) a stable rate reduces the variability in MAE calculations leading to more accurate assessments.

[vimalk78]

check why changed scrape interval, and update comment accordingly

[vprashar2929]

setting scrape every 3 seconds rather than every 5 seconds, over a typical time window will collect significantly more data

[vprashar2929]

Here is sample CI run that would look like for reference once we have this merged: https://github.com/sustainable-computing-io/kepler-metal-ci/actions/runs/12366281744/job/34512777104

My idea is to use the equinix runners on demand on PR's. Reviewers or authors can add a comment in the PR something like /test-regression which will trigger a workflow like this which can test if metrics produced by PR code base Kepler are off to what is already present in latest