tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) #143002

Enselic · 2025-06-25T06:10:21Z

Normally a run-fail ui test shall not be terminated by a signal like SIGABRT. So begin having that as a hard requirement.

Some of our current tests do terminate by a signal however. Introduce and use ~~run-fail-without-exit-code~~ run-crash for those tests.

This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as SIGABRT in tests/ui/panics/panic-main.rs (shown as Aborted (core dumped) in the logs attached to that issue, and I have also been able to reproduce this locally).

TODO

Q: what about on Windows?
A: we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix.
test all permutations of actual vs expected
Done: See tests: Require run-fail ui tests to have an exit code (SIGABRT not ok) #143002 (comment).
Handle targets without unwind support

Zulip discussion

See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235

try-job: aarch64-apple
// try-job: x86_64-msvc-1
// try-job: x86_64-gnu
// try-job: dist-i586-gnu-i586-i686-musl
// try-job: test-various

rustbot · 2025-06-25T06:10:25Z

r? @petrochenkov

rustbot has assigned @petrochenkov.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

rustbot · 2025-06-25T06:10:29Z

Some changes occurred in src/tools/compiletest

cc @jieyouxu

workingjubilee · 2025-06-25T08:18:58Z

why not simply run-abort

also would accept run-crash

jieyouxu · 2025-06-25T10:10:37Z

@bors2 delegate=try

rust-bors · 2025-06-25T10:10:40Z

@Enselic can now perform try builds on this pull request

Enselic · 2025-06-25T11:37:25Z

@bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2

rust-bors · 2025-06-25T11:37:29Z

⌛ Trying commit 17be091 with merge 873ecba…

To cancel the try build, run the command @bors2 try cancel.

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-fail-without-exit-code` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [ ] what about on Windows? - [ ] also update docs at https://rustc-dev-guide.rust-lang.org/tests/directives.html#controlling-outcome-expectations - [ ] clean up the code ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: x86_64-msvc-1 try-job: x86_64-msvc-2

Enselic · 2025-06-25T13:34:03Z

An only-windows test that was run-fail now must be run-crash. Fixed now. Trying again:

@bors2 try jobs=x86_64-msvc-1,x86_64-msvc-2

rust-bors · 2025-06-25T13:34:07Z

⌛ Trying commit ad4e082 with merge e685982…

To cancel the try build, run the command @bors2 try cancel.

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-fail-without-exit-code` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [ ] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [ ] also update docs at https://rustc-dev-guide.rust-lang.org/tests/directives.html#controlling-outcome-expectations - [ ] clean up the code - [ ] test all permutations of actual vs expected ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: x86_64-msvc-1 try-job: x86_64-msvc-2

…signal, r=petrochenkov tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use ~`run-fail-without-exit-code`~ `run-crash` for those tests. This adds further (cc rust-lang#142304, rust-lang#142886) protection against the regression in rust-lang#123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [x] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See rust-lang#143002 (comment). ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235

…rors Rollup of 8 pull requests Successful merges: - #139858 (New const traits syntax) - #140809 (Reduce special casing for the panic runtime) - #142963 (Skip unnecessary components in x64 try builds) - #142974 (Update stage0 to 1.89.0-beta.1) - #142987 (rustdoc: show attributes on enum variants) - #143002 (tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok)) - #143092 (const checks for lifetime-extended temporaries: avoid 'top-level scope' terminology) - #143096 (tag_for_variant: properly pass TypingEnv) r? `@ghost` `@rustbot` modify labels: rollup

matthiaskrgr · 2025-06-27T20:09:54Z

@bors r-
failed in #143105 (comment)

matthiaskrgr · 2025-06-27T20:11:40Z

@bors try

bors · 2025-06-27T20:12:51Z

⌛ Trying commit 4573c84 with merge 0cd719d...

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use ~`run-fail-without-exit-code`~ `run-crash` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [x] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See #143002 (comment). ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 // try-job: aarch64-apple try-job: x86_64-msvc-1 try-job: x86_64-gnu try-job: dist-i586-gnu-i586-i686-musl try-job: test-various

compiler-errors · 2025-06-27T20:14:46Z

Yeah, the test that failed in that rollup should?? probably be marked run-crash too

bors · 2025-06-27T21:12:08Z

💔 Test failed - checks-actions

Enselic · 2025-06-28T13:59:20Z

For the test-various job the problem seems to be that platforms without unwind support will abort instead, but aborting means "crashing", so run-fail will fail.

Let's take for example tests/ui/async-await/issues/issue-65419/issue-65419-coroutine-resume-after-completion.rs which is marked as run-fail.

The test will pass on x86_64-unknown-linux-gnu:

$ ./x test tests/ui/async-await/issues/issue-65419/issue-65419-coroutine-resume-after-completion.rs 
...
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 19329 filtered out; finished in 70.26ms

but fail with exit code 134 (128 + 6 for SIGABRT) with wasm32-wasip1:

$ PATH="/home/martin/opt/wasmtime-v19/wasmtime-v19.0.0-x86_64-linux:$PATH" WASI_SDK_PATH=/home/martin/opt/wasi-sdk-25/wasi-sdk-25.0-x86_64-linux ./x test --set rust.lld=true --target wasm32-wasip1 tests/ui/async-await/issues/issue-65419/issue-65419-coroutine-resume-after-completion.rs
...
error: test did not exit with failure! code=Some(134)
...
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 19329 filtered out; finished in 90.22ms

Solution proposal

I think the best way to handle this is to adjust compiletest to allow crashes with run-fail if the target does not support unwinding, but I need to experiment a bit with it.

…t ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use `run-crash` for those tests.

Enselic · 2025-06-30T05:29:43Z

@bors2 try jobs=test-various

rust-bors · 2025-06-30T05:29:47Z

⌛ Trying commit fa105d1 with merge bfdbb3f…

To cancel the try build, run the command @bors2 try cancel.

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use ~`run-fail-without-exit-code`~ `run-crash` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [x] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See #143002 (comment). - [ ] Handle targets without unwind support ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 // try-job: aarch64-apple try-job: x86_64-msvc-1 try-job: x86_64-gnu try-job: dist-i586-gnu-i586-i686-musl try-job: test-various try-job: test-various

rust-bors · 2025-06-30T07:57:45Z

💔 Test failed

CI ❌

Enselic · 2025-06-30T09:37:56Z

The strange windows failure again. Retrying without windows.

@bors2 try jobs=test-various

rust-bors · 2025-06-30T09:38:01Z

⌛ Trying commit fa105d1 with merge 4e44930…

To cancel the try build, run the command @bors2 try cancel.

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use ~`run-fail-without-exit-code`~ `run-crash` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [x] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See #143002 (comment). - [ ] Handle targets without unwind support ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 // try-job: aarch64-apple // try-job: x86_64-msvc-1 // try-job: x86_64-gnu // try-job: dist-i586-gnu-i586-i686-musl try-job: test-various try-job: test-various

Enselic · 2025-06-30T09:41:27Z

I see now that job already is green. Will try the apple job next.

@bors2 try cancel

rust-bors · 2025-06-30T09:41:29Z

Try build cancelled. Cancelled workflows:

https://github.com/rust-lang/rust/actions/runs/15969383449

Enselic · 2025-06-30T09:43:06Z

@bors2 try jobs=aarch64-apple

rust-bors · 2025-06-30T09:43:10Z

⌛ Trying commit fa105d1 with merge 04a6c3c…

To cancel the try build, run the command @bors2 try cancel.

…try> tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) Normally a `run-fail` ui test shall not be terminated by a signal like `SIGABRT`. So begin having that as a hard requirement. Some of our current tests do terminate by a signal however. Introduce and use ~`run-fail-without-exit-code`~ `run-crash` for those tests. This adds further (cc #142304, #142886) protection against the regression in #123733 since that bug also manifested as `SIGABRT` in `tests/ui/panics/panic-main.rs` (shown as `Aborted (core dumped)` in the logs attached to that issue, and I have also been able to reproduce this locally). ### TODO - [x] **Q:** what about on Windows? **A:** we'll treat any exit code outside of 1 - 127 as "crashed", and we'll do the same on unix. - [x] test all permutations of actual vs expected **Done:** See #143002 (comment). - [ ] Handle targets without unwind support ### Zulip discussion See https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/compiletest.3A.20terminate.20by.20signal.20vs.20exit.20with.20error/with/525611235 try-job: aarch64-apple // try-job: x86_64-msvc-1 // try-job: x86_64-gnu // try-job: dist-i586-gnu-i586-i686-musl // try-job: test-various try-job: aarch64-apple

rust-bors · 2025-06-30T10:20:40Z

💔 Test failed

CI ❌

rustbot assigned petrochenkov Jun 25, 2025

Enselic added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 25, 2025

Enselic mentioned this pull request Jun 25, 2025

RUST_BACKTRACE=full loop with -Cpanic=abort on aarch64-unknown-linux-gnu #123733

Open

rustbot added the T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue. label Jun 25, 2025

This comment has been minimized.

Sign in to view

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from 6ad1973 to 17be091 Compare June 25, 2025 10:12

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from 17be091 to ad4e082 Compare June 25, 2025 13:30

This comment was marked as resolved.

Sign in to view

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from ad4e082 to 2499dac Compare June 25, 2025 16:29

rustbot added the A-rustc-dev-guide Area: rustc-dev-guide label Jun 25, 2025

compiler-errors mentioned this pull request Jun 27, 2025

Rollup of 8 pull requests #143105

Closed

bors added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Jun 27, 2025

This comment has been minimized.

Sign in to view

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from 4573c84 to 6c6845f Compare June 30, 2025 02:51

This comment has been minimized.

Sign in to view

Enselic force-pushed the tests-ui-run-fail-exit-vs-signal branch from 6c6845f to fa105d1 Compare June 30, 2025 04:02

tests: Require run-fail ui tests to have an exit code (SIGABRT not ok) #143002

Are you sure you want to change the base?

tests: Require run-fail ui tests to have an exit code (SIGABRT not ok) #143002

Conversation

Enselic commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Zulip discussion

Uh oh!

rustbot commented Jun 25, 2025

Uh oh!

rustbot commented Jun 25, 2025

Uh oh!

This comment has been minimized.

workingjubilee commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jieyouxu commented Jun 25, 2025

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

Enselic commented Jun 25, 2025

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

This comment was marked as resolved.

Enselic commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rust-bors bot commented Jun 25, 2025

Uh oh!

This comment was marked as resolved.

matthiaskrgr commented Jun 27, 2025

Uh oh!

matthiaskrgr commented Jun 27, 2025

Uh oh!

bors commented Jun 27, 2025

Uh oh!

compiler-errors commented Jun 27, 2025

Uh oh!

This comment has been minimized.

bors commented Jun 27, 2025

Uh oh!

Enselic commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Solution proposal

Uh oh!

This comment has been minimized.

Enselic commented Jun 30, 2025

Uh oh!

rust-bors bot commented Jun 30, 2025

Uh oh!

rust-bors bot commented Jun 30, 2025

Uh oh!

Enselic commented Jun 30, 2025

Uh oh!

rust-bors bot commented Jun 30, 2025

Uh oh!

Enselic commented Jun 30, 2025

Uh oh!

rust-bors bot commented Jun 30, 2025

Uh oh!

Enselic commented Jun 30, 2025

Uh oh!

rust-bors bot commented Jun 30, 2025

Uh oh!

rust-bors bot commented Jun 30, 2025

Uh oh!

Uh oh!

tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) #143002

tests: Require `run-fail` ui tests to have an exit code (`SIGABRT` not ok) #143002

Enselic commented Jun 25, 2025 •

edited

Loading

workingjubilee commented Jun 25, 2025 •

edited

Loading

Enselic commented Jun 25, 2025 •

edited

Loading

Enselic commented Jun 28, 2025 •

edited

Loading