Store test diffs in job summaries and improve analysis formatting #138531

Kobzol · 2025-03-15T10:17:15Z

This PR stores the test diffs that we already have in the post-merge workflow also into individual job summaries. This makes it easier to compare test (and later also other) diffs per job, which will be especially useful for try jobs, so that we can actually see the test diffs before we merge a given PR.

As a drive-by, I also made a bunch of cleanups in citool and in the formatting of the summary and post-merge analyses. These changes are split into self-contained commits.

The analysis can be tested locally with the following command:

$ curl https://ci-artifacts.rust-lang.org/rustc-builds/<current-sha>/metrics-<job-name>.json > metrics.json
$ cargo run --manifest-path src/ci/citool/Cargo.toml postprocess-metrics metrics.json --job-name <job-name> --parent <parent-sha> > out.md

For example, for this PR:

$ curl https://ci-artifacts.rust-lang.org/rustc-builds/282865097d138c7f0f7a7566db5b761312dd145c/metrics-aarch64-gnu.json > metrics.json
$ cargo run --manifest-path src/ci/citool/Cargo.toml postprocess-metrics metrics.json --job-name aarch64-gnu --parent d9e5539a39192028a7b15ae596a8685017faecee > out.md

Best reviewed commit by commit.

r? @marcoieni

try-job: aarch64-gnu
try-job: dist-x86_64-linux

This allows the code to be simplified a little bit.

So that we can also observe them for try builds, before merging a PR.

…ation of the post merge report

Kobzol · 2025-03-15T10:18:06Z

@bors try

Store test diffs in job summaries and improve analysis formatting This PR stores the test diffs that we already have in the post-merge workflow also into individual job summaries. This makes it easier to compare test (and later also other) diffs per job, which will be especially useful for try jobs, so that we can actually see the test diffs *before* we merge a given PR. As a drive-by, I also made a bunch of cleanups in `citool` and in the formatting of the summary and post-merge analyses. These changes are split into self-contained commits. The analysis can be tested locally with the following command (you can replace : ```bash $ curl https://ci-artifacts.rust-lang.org/rustc-builds/<current-sha>/metrics-<job-name>.json > metrics.json $ cargo run --manifest-path src/ci/citool/Cargo.toml postprocess-metrics metrics.json --job-name <job-name> --parent <parent-sha> > out.md ``` For example, for [this PR](rust-lang#138523): ```bash $ curl https://ci-artifacts.rust-lang.org/rustc-builds/282865097d138c7f0f7a7566db5b761312dd145c/metrics-aarch64-gnu.json > metrics.json $ cargo run --manifest-path src/ci/citool/Cargo.toml postprocess-metrics metrics.json --job-name aarch64-gnu --parent d9e5539 > out.md ``` Best reviewed commit by commit. r? `@marcoieni` try-job: aarch64-gnu try-job: dist-x86_64-linux

bors · 2025-03-15T10:19:17Z

⌛ Trying commit e845318 with merge 34fd0c6...

bors · 2025-03-15T10:29:18Z

💔 Test failed - checks-actions

Kobzol · 2025-03-15T10:35:06Z

@bors try

Store test diffs in job summaries and improve analysis formatting This PR stores the test diffs that we already have in the post-merge workflow also into individual job summaries. This makes it easier to compare test (and later also other) diffs per job, which will be especially useful for try jobs, so that we can actually see the test diffs *before* we merge a given PR. As a drive-by, I also made a bunch of cleanups in `citool` and in the formatting of the summary and post-merge analyses. These changes are split into self-contained commits. The analysis can be tested locally with the following command (you can replace : ```bash $ curl https://ci-artifacts.rust-lang.org/rustc-builds/<current-sha>/metrics-<job-name>.json > metrics.json $ cargo run --manifest-path src/ci/citool/Cargo.toml postprocess-metrics metrics.json --job-name <job-name> --parent <parent-sha> > out.md ``` For example, for [this PR](rust-lang#138523): ```bash $ curl https://ci-artifacts.rust-lang.org/rustc-builds/282865097d138c7f0f7a7566db5b761312dd145c/metrics-aarch64-gnu.json > metrics.json $ cargo run --manifest-path src/ci/citool/Cargo.toml postprocess-metrics metrics.json --job-name aarch64-gnu --parent d9e5539 > out.md ``` Best reviewed commit by commit. r? `@marcoieni` try-job: aarch64-gnu try-job: dist-x86_64-linux

bors · 2025-03-15T10:36:17Z

⌛ Trying commit 4801dba with merge 91320fb...

bors · 2025-03-15T13:48:31Z

☀️ Try build successful - checks-actions
Build commit: 91320fb (91320fb03530fce8d4960e2f4b7b279e3b03b03e)

marcoieni

I have just this small comment. you can r=me

.github/workflows/ci.yml

src/ci/citool/src/metrics.rs

Kobzol · 2025-03-17T21:20:40Z

Thank you!

@bors r=marcoieni

bors · 2025-03-17T21:20:43Z

📌 Commit c9d3147 has been approved by marcoieni

It is now in the queue for this repository.

…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#138384 (Move `hir::Item::ident` into `hir::ItemKind`.) - rust-lang#138508 (Clarify "owned data" in E0515.md) - rust-lang#138531 (Store test diffs in job summaries and improve analysis formatting) - rust-lang#138533 (Only use `DIST_TRY_BUILD` for try jobs that were not selected explicitly) - rust-lang#138556 (Fix ICE: attempted to remap an already remapped filename) - rust-lang#138608 (rustc_target: Add target feature constraints for LoongArch) - rust-lang#138619 (Flatten `if`s in `rustc_codegen_ssa`) r? `@ghost` `@rustbot` modify labels: rollup

Rollup merge of rust-lang#138531 - Kobzol:test-diff-try-build, r=marcoieni Store test diffs in job summaries and improve analysis formatting This PR stores the test diffs that we already have in the post-merge workflow also into individual job summaries. This makes it easier to compare test (and later also other) diffs per job, which will be especially useful for try jobs, so that we can actually see the test diffs *before* we merge a given PR. As a drive-by, I also made a bunch of cleanups in `citool` and in the formatting of the summary and post-merge analyses. These changes are split into self-contained commits. The analysis can be tested locally with the following command: ```bash $ curl https://ci-artifacts.rust-lang.org/rustc-builds/<current-sha>/metrics-<job-name>.json > metrics.json $ cargo run --manifest-path src/ci/citool/Cargo.toml postprocess-metrics metrics.json --job-name <job-name> --parent <parent-sha> > out.md ``` For example, for [this PR](rust-lang#138523): ```bash $ curl https://ci-artifacts.rust-lang.org/rustc-builds/282865097d138c7f0f7a7566db5b761312dd145c/metrics-aarch64-gnu.json > metrics.json $ cargo run --manifest-path src/ci/citool/Cargo.toml postprocess-metrics metrics.json --job-name aarch64-gnu --parent d9e5539 > out.md ``` Best reviewed commit by commit. r? `@marcoieni` try-job: aarch64-gnu try-job: dist-x86_64-linux

…iaskrgr Rollup of 7 pull requests Successful merges: - rust-lang#138384 (Move `hir::Item::ident` into `hir::ItemKind`.) - rust-lang#138508 (Clarify "owned data" in E0515.md) - rust-lang#138531 (Store test diffs in job summaries and improve analysis formatting) - rust-lang#138533 (Only use `DIST_TRY_BUILD` for try jobs that were not selected explicitly) - rust-lang#138556 (Fix ICE: attempted to remap an already remapped filename) - rust-lang#138608 (rustc_target: Add target feature constraints for LoongArch) - rust-lang#138619 (Flatten `if`s in `rustc_codegen_ssa`) r? `@ghost` `@rustbot` modify labels: rollup

Kobzol added 11 commits March 15, 2025 09:24

Refactor metrics generation step

899eed1

Do not fail the build if metrics postprocessing or DataDog upload fails

301c384

Print metrics postprocessing to stdout

09d44a4

This allows the code to be simplified a little bit.

Refactor metrics and analysis in citool to distinguish them better

e757dea

Print number of found test diffs

413fd52

Use first-level heading for test differences header

6c24c9c

Print test diffs into GitHub summary

30d5757

So that we can also observe them for try builds, before merging a PR.

Add bootstrap stage to test names

634a11e

Add a helper function for outputting details

232be86

Put test differences into a <details> section and add better explan…

b4cccf0

…ation of the post merge report

Do not error out on missing parent metrics

e845318

rustbot assigned marcoieni Mar 15, 2025

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. labels Mar 15, 2025

This comment has been minimized.

Sign in to view

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Mar 15, 2025

Reformat code

4801dba

marcoieni approved these changes Mar 17, 2025

View reviewed changes

.github/workflows/ci.yml Outdated Show resolved Hide resolved

marcoieni reviewed Mar 17, 2025

View reviewed changes

src/ci/citool/src/metrics.rs Outdated Show resolved Hide resolved

Small review improvements

c9d3147

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Mar 17, 2025

matthiaskrgr mentioned this pull request Mar 17, 2025

Rollup of 7 pull requests #138630

Merged

bors merged commit 5786233 into rust-lang:master Mar 18, 2025
6 checks passed

rustbot added this to the 1.87.0 milestone Mar 18, 2025

Kobzol deleted the test-diff-try-build branch March 18, 2025 09:13

Uh oh!

Store test diffs in job summaries and improve analysis formatting #138531

Store test diffs in job summaries and improve analysis formatting #138531

Uh oh!

Conversation

Kobzol commented Mar 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kobzol commented Mar 15, 2025

Uh oh!

bors commented Mar 15, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

bors commented Mar 15, 2025

Uh oh!

Kobzol commented Mar 15, 2025

Uh oh!

bors commented Mar 15, 2025

Uh oh!

bors commented Mar 15, 2025

Uh oh!

marcoieni left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Kobzol commented Mar 17, 2025

Uh oh!

bors commented Mar 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Kobzol commented Mar 15, 2025 •

edited

Loading