add agent 6 schedule to create rc pr workflow #32073
Conversation
sabrina-datadog
added
changelog/no-changelog
qa/no-code-change
sabrina-datadog
force-pushed
the
sabrina/a6-workflow
branch
from
e7547a0 to
34aba93
Compare
|
[Fast Unit Tests Report] On pipeline 51021755 (CI Visibility). The following jobs did not run any unit tests: Jobs:
If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help |
sabrina-datadog
changed the title
Package size comparisonComparison with ancestor Diff per package
Decision✅ Passed |
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: f483cc4 Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | tcp_syslog_to_blackhole | ingress throughput | +0.45 | [+0.39, +0.51] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | +0.45 | [+0.32, +0.57] | 1 | Logs bounds checks dashboard |
| ➖ | otel_to_otel_logs | ingress throughput | +0.43 | [-0.24, +1.11] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | +0.37 | [-0.34, +1.09] | 1 | Logs |
| ➖ | file_tree | memory utilization | +0.33 | [+0.20, +0.45] | 1 | Logs |
| ➖ | quality_gate_logs | % cpu utilization | +0.30 | [-2.63, +3.22] | 1 | Logs |
| ➖ | file_to_blackhole_300ms_latency | egress throughput | +0.20 | [-0.44, +0.83] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | +0.19 | [-0.57, +0.96] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency_http1 | egress throughput | +0.15 | [-0.77, +1.07] | 1 | Logs |
| ➖ | quality_gate_idle | memory utilization | +0.15 | [+0.11, +0.19] | 1 | Logs bounds checks dashboard |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | +0.03 | [-0.84, +0.90] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.00 | [-0.01, +0.01] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency_http2 | egress throughput | -0.01 | [-0.86, +0.84] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | -0.01 | [-0.09, +0.07] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | -0.03 | [-0.72, +0.66] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency_linear_load | egress throughput | -0.05 | [-0.51, +0.42] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.06 | [-0.84, +0.71] | 1 | Logs |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | links |
|---|---|---|---|---|
| ✅ | file_to_blackhole_0ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http1 | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http1 | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http2 | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_0ms_latency_http2 | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_1000ms_latency_linear_load | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_300ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_300ms_latency | memory_usage | 10/10 | |
| ✅ | file_to_blackhole_500ms_latency | lost_bytes | 10/10 | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | |
| ✅ | quality_gate_idle | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | bounds checks dashboard |
| ✅ | quality_gate_logs | lost_bytes | 10/10 | |
| ✅ | quality_gate_logs | memory_usage | 10/10 |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
| if: ${{ env.IS_AGENT6_RELEASE == 'false' }} | ||
| uses: actions/checkout@0ad4b8fadaa221de15dcec353f45205ec38ea70b # v4.1.4 | ||
| with: | ||
| sparse-checkout: 'tasks' | ||
| persist-credentials: false | ||
|
|
||
| - name: Install python | ||
| if: ${{ env.IS_AGENT6_RELEASE == 'false' }} | ||
| uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0 | ||
| with: | ||
| python-version: 3.11 | ||
| cache: "pip" | ||
|
|
||
| - name: Install Python dependencies | ||
| if: ${{ env.IS_AGENT6_RELEASE == 'false' }} |
There was a problem hiding this comment.
Why is it disabled when releasing agent 6?
There was a problem hiding this comment.
Since the agent 6 will only have one unrelease branch (6.53.x) there is no need to find them. I tried adding an if condition at the job level, but it doesn't have access to the env variables at that point (docs).
| fi | ||
|
|
||
| - name: Create RC PR | ||
| if: ${{ steps.check_for_changes.outputs.CHANGES == 'true'}} | ||
| if: ${{ steps.check_for_changes.outputs.CHANGES == 'true' || env.IS_AGENT6_RELEASE == 'true' }} |
There was a problem hiding this comment.
Should we remove the previous step if the output is ignored when releasing agent 6 ?
There was a problem hiding this comment.
No, I think we should keep it because the check_for_changes function not only checks for changes but can also modify the tags. There is additional logic that needs to be executed as part of the process here.
| def get_git_config(key): | ||
| result = subprocess.run(['git', 'config', '--get', key], capture_output=True, text=True) | ||
| return result.stdout.strip() if result.returncode == 0 else None | ||
|
|
||
|
|
||
| def set_git_config(key, value): | ||
| subprocess.run(['git', 'config', key, value]) | ||
|
|
||
|
|
||
| def revert_git_config(original_config): | ||
| for key, value in original_config.items(): | ||
| if value is None: | ||
| subprocess.run(['git', 'config', '--unset', key]) | ||
| else: | ||
| subprocess.run(['git', 'config', key, value]) |
There was a problem hiding this comment.
💬 suggestion: Could you use ctx.run instead? (e.g. ctx.run('...', hide=True).stdout). Note that this will raise an error if the process exits with non zero value
| if github_action: | ||
| set_git_config('user.name', 'github-actions[bot]') | ||
| set_git_config('user.email', 'github-actions[bot]@users.noreply.github.com') | ||
| upstream = f"https://x-access-token:{os.environ.get('GITHUB_TOKEN')}@github.com/{GITHUB_REPO_NAME}.git" |
There was a problem hiding this comment.
Are we sure this will not leak the token when executing the command?
There was a problem hiding this comment.
GitHub masks the secret by default, so the token isn't exposed (example). let me know if there are any concerns
| do_retry = ( | ||
| False | ||
| if github_action | ||
| else yes_no_question("Do you want to retry this operation?", color="orange", default=True) | ||
| ) |
There was a problem hiding this comment.
It feels like this could be replaced by checking if a TTY is attached rather that "if running under a CI"
| def get_git_config(key): | ||
| result = subprocess.run(['git', 'config', '--get', key], capture_output=True, text=True) | ||
| return result.stdout.strip() if result.returncode == 0 else None | ||
|
|
||
|
|
||
| def set_git_config(key, value): | ||
| subprocess.run(['git', 'config', key, value]) |
There was a problem hiding this comment.
These helpers are a good idea! They could even be split in their own PR and be used throughout our tasks if this PR takes long to be merged (but definitely not required)
There was a problem hiding this comment.
I will make a ticket to update the rest of our tasks
github-actions
bot
added
long review
sabrina-datadog
force-pushed
the
sabrina/a6-workflow
branch
2 times, most recently
from
56c6e63 to
a719857
Compare
sabrina-datadog
force-pushed
the
sabrina/a6-workflow
branch
from
89a5b46 to
fd0ef74
Compare
What does this PR do?
Add a weekly schedule to the create rc pr workflow for agent 6 release candidates. ACIX-454
find_release_branchesjob to return[6.53.x]when it is for a6--patch-versionforinv release.create-rcto bump to the right version after a final version (6.53.x-rc-1)yes_no_questionuser interaction because it is a github workflowset_git_configfunction from this OTel PRMotivation
We want to ensure that agent 6 can be released at any point. So a weekly rc will be run to test that the agent 6 pipeline works.
Describe how you validated your changes
created a test branch to simulate when:
Possible Drawbacks / Trade-offs
Additional Notes