-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Backport of [ui, tests] Various acceptance test fixups (v main) into release/1.9.x #25177
Backport of [ui, tests] Various acceptance test fixups (v main) into release/1.9.x #25177
Conversation
Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement Learn more about why HashiCorp requires a CLA and what the CLA includes 1 out of 2 committers have signed the CLA.
temp seems not to be a GitHub user. Have you signed the CLA already but the status is still pending? Recheck it. |
* retain artifacts from test runs including test timing * Pinning commit hashes for action helpers * trigger for ui-test run * Trying to isolate down to a simple upload * Once more with mkdir * What if we just wrote our own test reporter tho * Let the partitioned runs handle placement * Filter out common token logs, add a summary at the end, and note failures in logtime * Custom reporter cannot also have an output file, he finds out two days late * Aggregate summary, duration, and removing failure case * Conditional test report generation * Timeouts are errors * Trying with un-partitioned input json file * Remove the commented-out lines for main-only runs * combine-ui-test-results as its own script
* Add factory hooks for jobs to have previously stable versions and stopped status * Since #24973 node-read isn't presupposed and so should regex match only on the common url parts * Job detail tests for title buttons are now bimodal and default to having previously-stable version in history * prettier plz * Breaking a thing on purpose to see if my other broken thing is broken * continue-on-error set to false to get things red when appropriate * OK what if continue-on-error=true but we do a separate failure reporting after the fact * fail-fast are you the magic incantation that I need? * Re-fix my test now that fast-fail is off * Fix to server-leader by adding a region first, and always()-append to uploading partition results * Express failure step lists failing tests so you don't have to click back into ember-exam step * temporary snapshot and logging for flakey test in service job detail * Bunch of region and tasklogs test fixups * using allocStatusDistribution to ensure service job always has a non-queued alloc
Backporting 3 test-related PRs here:
For 1.8 and 1.7 I will manually just pull back the test fixes themselves, rather than the audit removal and test-ui.yml changes. |
Backport
This PR is auto-generated from #25031 to be assessed for backporting due to the inclusion of the label backport/1.9.x.
🚨
The person who merged in the original PR is:
@philrenaud
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.
The below text is copied from the body of the original PR.
In #24555, I changed up our
ember-exam
step intest-ui
tests to use a custom test runner. In CI, it runs in parallel in 4 partitions.This code had a bug: my tests were being reported locally in each partition, but because
continue-on-error
was set, they would also report back with a green checkmark / passing. A PR author or reviewer would have to click through passed test logs to see the failures.This PR does two main things:
Express failure
step afterwardember-exam
that summarizes failures and acts as the red-x for failing tests/checks.continue-on-error
(so ember-exam doesn't stop after a single failure) and setsfail-fast
to disabled (to allow follow-up steps, like this one, to run)Summary of tests that needed fixing
read node
ACL policy requirement. Several of our acceptance tests looked for specific URLs that presupposed the existence of the node.httpAddr for the logs being streamed, so rather than change all the hardcoded URLs, I gave the test runner a token that has node read access.withPreviousStableVersion
property to our mock job factory) and had our job-detail tests add it where appropriate.server.isLeader
property to work more nicely in multi-region federated clusters. This means that for each region, we now make a request to the/status/leader?region=${region}
endpoint. This endpoint requires sufficient permissions, so those are added to the test runner conditionally.TODO:
Overview of commits