Build the env var layer for the nodetreemodel config

[dustmop]

What does this PR do?

Build the env layer from environmental variables. Merge the layers that currently exist (default, file, env). Some functionality is missing so the associated unit test is skipped for now.

Motivation

Configuration can come from environment variables.

Describe how to test/QA your changes

Test exists in pkg/config/nodetreemodel/config_test.go but is currently disabled because the layers are not being properly merged. That will either be fixed before this PR is merged, or in a follow-up PR.

To manually test, use DD_CONF_NODETREEMODEL=true. Not recommended for any purpose other than development of the nodetreemodel config. The agent currently won't run past initialization due to the log level not being configured.

Possible Drawbacks / Trade-offs

The env vars are all read at once, at startup. This adds a bit of additional start-up time, because viper instead would read the associated env var (potentially) every time a setting was accessed. Because we're reading all env vars at startup, this also means we will no longer support env vars changing during runtime, which seems to be more in line with the behavior that we want.

Additional Notes

[cit-pr-commenter]

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: cc092e14-3a19-4ff0-8fb8-1f9aed8881e0

Baseline: 053bded
Comparison: 1f959a7
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	idle_all_features	memory utilization	+1.00	[+0.90, +1.11]	1	Logs bounds checks dashboard
➖	file_tree	memory utilization	+0.36	[+0.23, +0.50]	1	Logs
➖	idle	memory utilization	+0.17	[+0.12, +0.22]	1	Logs bounds checks dashboard
➖	file_to_blackhole_300ms_latency	egress throughput	+0.09	[-0.09, +0.27]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	+0.01	[-0.21, +0.24]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	+0.01	[-0.32, +0.34]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.01, +0.01]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.01	[-0.11, +0.09]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	-0.06	[-0.18, +0.05]	1	Logs bounds checks dashboard
➖	file_to_blackhole_500ms_latency	egress throughput	-0.16	[-0.41, +0.09]	1	Logs
➖	quality_gate_idle	memory utilization	-0.18	[-0.23, -0.13]	1	Logs bounds checks dashboard
➖	tcp_syslog_to_blackhole	ingress throughput	-0.38	[-0.43, -0.33]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	-0.46	[-0.94, +0.02]	1	Logs
➖	basic_py_check	% cpu utilization	-0.80	[-4.70, +3.09]	1	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-1.28	[-2.00, -0.56]	1	Logs
➖	pycheck_lots_of_tags	% cpu utilization	-4.72	[-8.17, -1.27]	1	Logs

Bounds Checks: ❌ Failed

perf	experiment	bounds_check_name	replicates_passed	links
❌	file_to_blackhole_300ms_latency	memory_usage	9/10
❌	quality_gate_idle_all_features	memory_usage	9/10	bounds checks dashboard
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	idle	memory_usage	10/10	bounds checks dashboard
✅	idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[agent-platform-auto-pr]

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=48510451 --os-family=ubuntu

Note: This applies to commit 1f959a7

[hush-hush]

Looks great

[hush-hush]

I think we can remove this once #30868 is merged and all tree are always initialized.

[hush-hush]

Should we refuse a new replacer once the config is "ready" ? Maybe returning an error for now ?

[hush-hush]

could we use transformer or similar instead of xform ?

[hush-hush]

We should use something else than the official Datadog prefix, DD_, to avoid potential conflict with the test runner in the CI.

[dustmop]

Good idea, changed to TEST_

[dustmop]

/merge

[dd-devflow]

Devflow running: /merge

View all feedbacks in Devflow UI.

---

2024-11-07 22:32:17 UTC ℹ️ MergeQueue: pull request added to the queue

The median merge time in main is 24m.