Post process e2e tests logs to make them more readable #31615

KevinFairise2 · 2024-11-29T14:20:13Z

What does this PR do?

This PR add a step in e2e tests that will read the json output of e2e tests result. And parse it to produce the test logs in a format where they are not mixed all together.
If the logs are not too long they will be displayed directly in the PR logs with a dropdown: like here
If the logs are too long we do not display them directly in the logs to avoid deteriorating the logs output (collapsed lines still count in the limit of output size). We just indicate that all the logs can be found in job artifacts: like here + artifacts
The logs are always stored in the artifacts under e2e-output/logs

The E2E_LOGS_PROCESSING_TEST_DEPHT allows you to configure the logs post processing.
If the test use a single TestSuite and instantiate subtests to run different tests. Like new-e2e-installer does for example. You should increase this value, so it consider not only the parent test when splitting logs

Motivation

Describe how to test/QA your changes

Possible Drawbacks / Trade-offs

Additional Notes

agent-platform-auto-pr · 2024-11-29T14:21:51Z

Gitlab CI Configuration Changes

⚠️ Diff too large to display on Github.

Changes Summary

Removed	Modified	Added	Renamed
0	98	0	0

ℹ️ Diff available in the job log.

cit-pr-commenter · 2024-11-29T15:41:54Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 6bc8d6e1-747f-487f-8c0b-c11d2b71c20d

Baseline: 8d89378
Comparison: 596c242
Diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	quality_gate_logs	% cpu utilization	+1.97	[-0.97, +4.92]	1	Logs
➖	file_tree	memory utilization	+0.48	[+0.36, +0.60]	1	Logs
➖	file_to_blackhole_1000ms_latency_linear_load	egress throughput	+0.47	[+0.01, +0.94]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	+0.46	[-0.33, +1.25]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	+0.27	[+0.16, +0.39]	1	Logs bounds checks dashboard
➖	uds_dogstatsd_to_api	ingress throughput	+0.01	[-0.08, +0.11]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.01, +0.01]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	-0.02	[-0.75, +0.72]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	-0.03	[-0.92, +0.85]	1	Logs
➖	otel_to_otel_logs	ingress throughput	-0.04	[-0.73, +0.65]	1	Logs
➖	file_to_blackhole_300ms_latency	egress throughput	-0.04	[-0.67, +0.58]	1	Logs
➖	file_to_blackhole_0ms_latency_http1	egress throughput	-0.05	[-0.91, +0.81]	1	Logs
➖	file_to_blackhole_0ms_latency_http2	egress throughput	-0.07	[-0.88, +0.74]	1	Logs
➖	file_to_blackhole_500ms_latency	egress throughput	-0.36	[-1.14, +0.41]	1	Logs
➖	quality_gate_idle	memory utilization	-0.40	[-0.44, -0.35]	1	Logs bounds checks dashboard
➖	tcp_syslog_to_blackhole	ingress throughput	-0.78	[-0.84, -0.72]	1	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-1.80	[-2.51, -1.08]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http1	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http1	memory_usage	10/10
✅	file_to_blackhole_0ms_latency_http2	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency_http2	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency_linear_load	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_300ms_latency	lost_bytes	10/10
✅	file_to_blackhole_300ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_logs	lost_bytes	10/10
✅	quality_gate_logs	memory_usage	10/10

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.

…/post-process-e2e-logs

CelianR · 2024-12-03T12:08:19Z

tasks/new_e2e_tests.py

+                color_message(
+                    "Too many logs to print, skipping pretty print. You can still find them properly organized in the job artifacts",
+                    "yellow",
+                )


🥜 nitpick: ‏I prefer to print WARNING: <message> with WARNING in yellow and the message as usual

Or maybe we can color all the message in bold?

agent-platform-auto-pr · 2024-12-03T12:46:41Z

[Fast Unit Tests Report]

On pipeline 50699042 (CI Visibility). The following jobs did not run any unit tests:

Jobs:

tests_deb-arm64-py3
tests_deb-x64-py3
tests_flavor_dogstatsd_deb-x64
tests_flavor_heroku_deb-x64
tests_flavor_iot_deb-x64
tests_rpm-arm64-py3
tests_rpm-x64-py3
tests_windows-x64

If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help

pducolin · 2024-12-03T13:05:29Z

.gitlab/e2e/e2e.yml

  script:
-    - inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password) --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
+    - inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com -c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password) --junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer --logs-folder=$E2E_OUTPUT_DIR/logs --logs-post-processing --logs-post-processing-test-depth=$E2E_LOGS_PROCESSING_TEST_DEPTH


💬 suggestion
What about parsing E2E_LOGS_PROCESSING_TEST_DEPTH at python time?

We could, but I prefered to keep it as a parameter so it is more discoverable. Otherwise you can only use it if you know it exists

tasks/new_e2e_tests.py

…/post-process-e2e-logs

agent-platform-auto-pr · 2024-12-10T13:30:32Z

Package size comparison

Comparison with ancestor 8d89378971e1074a64be7313c3bba7034a91fcc4

Diff per package

package	diff	status	size	ancestor	threshold
datadog-agent-amd64-deb	0.00MB	✅	1270.67MB	1270.67MB	140.00MB
datadog-iot-agent-amd64-deb	0.00MB	✅	113.20MB	113.20MB	10.00MB
datadog-dogstatsd-amd64-deb	0.00MB	✅	78.32MB	78.32MB	10.00MB
datadog-heroku-agent-amd64-deb	0.00MB	✅	526.45MB	526.45MB	70.00MB
datadog-agent-x86_64-rpm	-0.00MB	✅	1279.91MB	1279.91MB	140.00MB
datadog-agent-x86_64-suse	-0.00MB	✅	1279.91MB	1279.91MB	140.00MB
datadog-iot-agent-x86_64-rpm	0.00MB	✅	113.26MB	113.26MB	10.00MB
datadog-iot-agent-x86_64-suse	0.00MB	✅	113.26MB	113.26MB	10.00MB
datadog-dogstatsd-x86_64-rpm	0.00MB	✅	78.40MB	78.40MB	10.00MB
datadog-dogstatsd-x86_64-suse	0.00MB	✅	78.40MB	78.40MB	10.00MB
datadog-agent-arm64-deb	0.00MB	✅	1004.85MB	1004.85MB	140.00MB
datadog-iot-agent-arm64-deb	0.00MB	✅	108.67MB	108.67MB	10.00MB
datadog-dogstatsd-arm64-deb	0.00MB	✅	55.59MB	55.59MB	10.00MB
datadog-agent-aarch64-rpm	0.00MB	✅	1014.06MB	1014.06MB	140.00MB
datadog-iot-agent-aarch64-rpm	0.00MB	✅	108.74MB	108.74MB	10.00MB

Decision

✅ Passed

spencergilbert

ADEL owned file lgtm

KevinFairise2 · 2024-12-10T14:47:35Z

/merge

dd-devflow · 2024-12-10T14:48:23Z

Devflow running: `/merge`

View all feedbacks in Devflow UI.

2024-12-10 14:48:22 UTC ℹ️ MergeQueue: pull request added to the queue

The median merge time in main is 24m.

Post process e2e tests logs to make them more readable

5f4b757

github-actions bot added team/agent-devx-loops short review PR is simple enough to be reviewed quickly labels Nov 29, 2024

Update logs folder param name

88dc105

KevinFairise2 force-pushed the kfairise/post-process-e2e-logs branch from cd259a1 to 88dc105 Compare November 29, 2024 14:54

KevinFairise2 added 3 commits November 29, 2024 17:14

Handle better test depth

219f23b

Merge branch 'main' of github.com:DataDog/datadog-agent into kfairise…

b37c8e6

…/post-process-e2e-logs

Fix test depth and add it for installer tests

807705c

github-actions bot added medium review PR review might take time and removed short review PR is simple enough to be reviewed quickly labels Dec 2, 2024

KevinFairise2 added 2 commits December 2, 2024 15:48

Improve processing by adding parent test logs for nested tests

f00bc0f

Fix args of post process function

0736aaf

KevinFairise2 force-pushed the kfairise/post-process-e2e-logs branch from 39103f4 to 0736aaf Compare December 2, 2024 16:42

KevinFairise2 added 2 commits December 2, 2024 18:26

Remove print

f156d50

Pretty print max logs

1865c3d

KevinFairise2 marked this pull request as ready for review December 3, 2024 10:26

KevinFairise2 requested review from a team as code owners December 3, 2024 10:26

Merge branch 'main' into kfairise/post-process-e2e-logs

0c46490

KevinFairise2 added changelog/no-changelog qa/no-code-change No code change in Agent code requiring validation labels Dec 3, 2024

Remove print

27b99a2

CelianR approved these changes Dec 3, 2024

View reviewed changes

pducolin approved these changes Dec 3, 2024

View reviewed changes

KevinFairise2 added 2 commits December 10, 2024 13:07

Adress comments

596cffb

Merge branch 'main' of github.com:DataDog/datadog-agent into kfairise…

596c242

…/post-process-e2e-logs

spencergilbert approved these changes Dec 10, 2024

View reviewed changes

dd-mergequeue bot merged commit 04e117d into main Dec 10, 2024
310 checks passed

dd-mergequeue bot deleted the kfairise/post-process-e2e-logs branch December 10, 2024 15:12

github-actions bot added this to the 7.62.0 milestone Dec 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Post process e2e tests logs to make them more readable #31615

Post process e2e tests logs to make them more readable #31615

KevinFairise2 commented Nov 29, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Nov 29, 2024 •

edited

Loading

cit-pr-commenter bot commented Nov 29, 2024 •

edited

Loading

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CelianR Dec 3, 2024

CelianR Dec 3, 2024

agent-platform-auto-pr bot commented Dec 3, 2024 •

edited

Loading

pducolin Dec 3, 2024

KevinFairise2 Dec 10, 2024

agent-platform-auto-pr bot commented Dec 10, 2024

spencergilbert left a comment

KevinFairise2 commented Dec 10, 2024

dd-devflow bot commented Dec 10, 2024 •

edited

Loading

Post process e2e tests logs to make them more readable #31615

Post process e2e tests logs to make them more readable #31615

Conversation

KevinFairise2 commented Nov 29, 2024 • edited Loading

What does this PR do?

Motivation

Describe how to test/QA your changes

Possible Drawbacks / Trade-offs

Additional Notes

agent-platform-auto-pr bot commented Nov 29, 2024 • edited Loading

Gitlab CI Configuration Changes

Changes Summary

cit-pr-commenter bot commented Nov 29, 2024 • edited Loading

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

CelianR Dec 3, 2024

Choose a reason for hiding this comment

CelianR Dec 3, 2024

Choose a reason for hiding this comment

agent-platform-auto-pr bot commented Dec 3, 2024 • edited Loading

pducolin Dec 3, 2024

Choose a reason for hiding this comment

KevinFairise2 Dec 10, 2024

Choose a reason for hiding this comment

agent-platform-auto-pr bot commented Dec 10, 2024

Package size comparison

Decision

spencergilbert left a comment

Choose a reason for hiding this comment

KevinFairise2 commented Dec 10, 2024

dd-devflow bot commented Dec 10, 2024 • edited Loading

Devflow running: /merge

KevinFairise2 commented Nov 29, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Nov 29, 2024 •

edited

Loading

cit-pr-commenter bot commented Nov 29, 2024 •

edited

Loading

agent-platform-auto-pr bot commented Dec 3, 2024 •

edited

Loading

dd-devflow bot commented Dec 10, 2024 •

edited

Loading

Devflow running: `/merge`