manual-e2e-run #112
manual-e2e-run #112
Conversation
| e2e_containers: | ||
| description: "The number of tests that you want Cypress to run in parallel (ex. 1, 2, 3, ...)" | ||
| type: string | ||
| default: '["1"]' |
There was a problem hiding this comment.
The default value doesn't match the example provided in the description
There was a problem hiding this comment.
In terms of the very first example from web-home, that's the old flow and is also based on the e2e_container input shown here for that repo. Each repo gets to decide how many containers they need for running their tests to optimize their test runs.
In terms of the other examples referenced above, these are from tests being run in specific repos, web-navbar for the Application utility flow and web-query-results for the Utility runtime flow.
The value shown here is for the global workflow and as demonstrated by remarks above, can be overridden by the consumer.
There was a problem hiding this comment.
Sorry, should've been more clear. The description says "ex. 1,2,3", ie: a string, but the default value is a string that is an array of strings ["1"]. So does the user need to provide an array of strings, or just a string?
There was a problem hiding this comment.
Okay updated the comment to the following:
| const name = 'pr / E2E Status'; | ||
| const url = '${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}'; | ||
| const success = '${{ inputs.job_status }}' === 'success'; | ||
| const body = `${name}: ${success ? 'succeeded ✅' : 'failed ❌'}\n${url}`; |
There was a problem hiding this comment.
This is a nice touch, thank you for adding it.
| | Name | Required | Description | | ||
| | --------------------------- | --------- | ----------------------------------------- | | ||
| | `NPM_TOKEN` | True | A J1 npm.com Publish token | ||
| | `CYPRESS_RECORD_KEY` | False | The record key associated with the project in Cypress. |
There was a problem hiding this comment.
Why are these not required?
There was a problem hiding this comment.
Good call. They weren't initially required because the E2E flow was a part of the overall PR flow, and not ever repo runs E2E tests. Now that these are split out they should be required. These have since been updated.
The Issue
As discussed here, running our E2E tests every time a PR is opened and changed is a significant load on our system and can be a burden on the engineer.
This update will change our approach when it comes to how E2E tests are run. As shown in this example, E2E tests are currently baked into our PR workflow. Meaning every time an engineer opens or updates a PR, we are running our E2E tests.
Unlike our other workflow jobs, running E2E tests is costly as they are much slower than our other jobs and also have the potential to consume a lot of runners at once as we allow parallel testing (to help speed up the overall test run). To remedy this, we are taking the E2E test runs out of the PR flows and moving them into flows that can be triggered by leaving a PR comment. In this way, engineers can decide when they are ready to run the tests and we can reduce the burden on the overall system.
The Solution
PR Opened
When a PR is now opened, all the status checks will get updated minus
pr / E2E Statuswhich will stay in a state ofWaiting for status to be reportedas nothing has triggered this status check:Test Triggered
Upon leaving a comment of
/run-e2e-testson the PR, the status check will be updated to indicate its status is Pending (this means the tests are now getting run):Test Complete
Once the test have passed/failed, the status check will finally enter into a state where the PR can be merged (assuming it was a successful run):
Updates
Application Runtime Flows
As demonstrated here, the flow for application runtime PRs has been simplified and no longer contains E2E test jobs:
As demonstrated here, a
new flowwas added for running application E2E tests:These tests will get triggered when a comment containing
/run-e2e-testsis left on the PR.Utility Runtime Flows
As demonstrated here, the flow for utility runtime PRs has been simplified and no longer contains E2E test jobs:
As demonstrated here, a
new flowwas added for triggering tests from utility runtimes:These tests will get triggered when a comment containing
/run-e2e-testsis left on the PR.Rollout Strategy
v3. All current consumers are onv2, so there is no risk of breaking changes here.#dev-frontendof this change and we will make any adjustments necessary once we go live.Note: I have kept the status check regarding the
E2E Statusthe same, so no modifications should be necessary regarding the current Github rulesets.Rollback Plan
If necessary we can leverage the [update-github-workflow-version job])https://github.com/JupiterOne/ops-platform-jobs/tree/main/scripts/mass-repo-jobs/update-github-workflow-version) to rollback our workflows and leverage the current v2 release.