Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider an A/B test config for e2e jobs #2061

Open
aarongreig opened this issue Sep 5, 2024 · 1 comment
Open

Consider an A/B test config for e2e jobs #2061

aarongreig opened this issue Sep 5, 2024 · 1 comment

Comments

@aarongreig
Copy link
Contributor

The proposed config would run the tests once with the unmodified sycl branch and once with the tag update and compare the results, ignoring any fails that happen in both runs. This could help to identify unrelated fails, which we do have a high rate of due to how fast intel-llvm moves contribution wise. Ideally this would extend to running all the e2e jobs like this but in practice our hardware resources likely wouldn't support that much extra load, we'd probably need to pick one or two jobs to double up instead.

@pbalcer
Copy link
Contributor

pbalcer commented Sep 5, 2024

The big issue where e2e failures are a problem is L0 on PVC. It should get better hopefully now that intel/llvm is also running tests on PVC. Nvm, that intel/llvm patch no longer includes adding a PVC job... :(

What we could maybe do is have a nightly job that runs against latest intel/llvm (maybe even runs twice, to detect flaky tests), which would then create a list of tests to xfail for the normal runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants