Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backport of [ui, tests] Various acceptance test fixups (v main) into release/1.9.x #25177

Conversation

hc-github-team-nomad-core
Copy link
Contributor

Backport

This PR is auto-generated from #25031 to be assessed for backporting due to the inclusion of the label backport/1.9.x.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@philrenaud
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/nomad/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.


In #24555, I changed up our ember-exam step in test-ui tests to use a custom test runner. In CI, it runs in parallel in 4 partitions.

This code had a bug: my tests were being reported locally in each partition, but because continue-on-error was set, they would also report back with a green checkmark / passing. A PR author or reviewer would have to click through passed test logs to see the failures.

This PR does two main things:

  • Fixes the failing tests that have resulted from (and been suppressed during) new feature work since this has been in place
  • Adds an Express failure step afterward ember-exam that summarizes failures and acts as the red-x for failing tests/checks.
    • In order to do this, this PR keeps continue-on-error (so ember-exam doesn't stop after a single failure) and sets fail-fast to disabled (to allow follow-up steps, like this one, to run)

image

Summary of tests that needed fixing

  • generally: failing tests were isolated to mirage-related issues (mocked data, not user-facing). The features that caused these failures all had their own tests that were validated at development time, and the failures here had dependencies on things these features changed in various ways. Details:
  • streaming task logs: In ui: Remove unrequired node read from task log streaming page. #24973, we removed the read node ACL policy requirement. Several of our acceptance tests looked for specific URLs that presupposed the existence of the node.httpAddr for the logs being streamed, so rather than change all the hardcoded URLs, I gave the test runner a token that has node read access.
  • the start button: In [ui] Multi-condition start/revert/edit buttons when a job isn't running #24985, we changed the "Start Job" button to be conditional (If the job has stop=true, it behaves as it ever did. Otherwise, it checks for versions to see if there is a previously stable one to fall back to). Several tests (of the job page title bar, etc.) checked for the existence of the start button to prove that a job was stopped, among other things. This is kind of checking for a symptom rather than a cause, but to not shake things up too much, I added a withPreviousStableVersion property to our mock job factory) and had our job-detail tests add it where appropriate.
  • server leadership is determined across all regions: In [ui] Show ALL regions' leaders when viewing servers route #24723 we changed the server.isLeader property to work more nicely in multi-region federated clusters. This means that for each region, we now make a request to the /status/leader?region=${region} endpoint. This endpoint requires sufficient permissions, so those are added to the test runner conditionally.

TODO:


Overview of commits

Copy link

hashicorp-cla-app bot commented Feb 20, 2025

CLA assistant check

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes


1 out of 2 committers have signed the CLA.

  • philrenaud
  • temp

temp seems not to be a GitHub user.
You need a GitHub account to be able to sign the CLA.
If you have already a GitHub account, please add the email address used for this commit to your account.

Have you signed the CLA already but the status is still pending? Recheck it.

* retain artifacts from test runs including test timing

* Pinning commit hashes for action helpers

* trigger for ui-test run

* Trying to isolate down to a simple upload

* Once more with mkdir

* What if we just wrote our own test reporter tho

* Let the partitioned runs handle placement

* Filter out common token logs, add a summary at the end, and note failures in logtime

* Custom reporter cannot also have an output file, he finds out two days late

* Aggregate summary, duration, and removing failure case

* Conditional test report generation

* Timeouts are errors

* Trying with un-partitioned input json file

* Remove the commented-out lines for main-only runs

* combine-ui-test-results as its own script
* Add factory hooks for jobs to have previously stable versions and stopped status

* Since #24973 node-read isn't presupposed and so should regex match only on the common url parts

* Job detail tests for title buttons are now bimodal and default to having previously-stable version in history

* prettier plz

* Breaking a thing on purpose to see if my other broken thing is broken

* continue-on-error set to false to get things red when appropriate

* OK what if continue-on-error=true but we do a separate failure reporting after the fact

* fail-fast are you the magic incantation that I need?

* Re-fix my test now that fast-fail is off

* Fix to server-leader by adding a region first, and always()-append to uploading partition results

* Express failure step lists failing tests so you don't have to click back into ember-exam step

* temporary snapshot and logging for flakey test in service job detail

* Bunch of region and tasklogs test fixups

* using allocStatusDistribution to ensure service job always has a non-queued alloc
@philrenaud
Copy link
Contributor

Backporting 3 test-related PRs here:

For 1.8 and 1.7 I will manually just pull back the test fixes themselves, rather than the audit removal and test-ui.yml changes.

@philrenaud philrenaud marked this pull request as ready for review February 21, 2025 05:27
philrenaud
philrenaud previously approved these changes Feb 21, 2025
@philrenaud philrenaud merged commit 6cfbf44 into release/1.9.x Feb 21, 2025
39 of 40 checks passed
@philrenaud philrenaud deleted the backport/b-ui/2025-02-acceptance-test-fixups-main/nicely-evolving-turkey branch February 21, 2025 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants