feat: run e2e tests on vsc release #7190

Patrick-Erichsen · 2025-08-15T23:59:27Z

Summary by cubic

Run VS Code E2E tests in the release workflow and add a reusable E2E workflow + composite action to remove duplicated CI logic. Releases are now validated by the same test matrix used in PRs.

New Features
- Added composite action to build the VS Code extension, compile E2E tests, cache deps, and output a test-file matrix. Excludes SSH tests on forks.
- Added reusable e2e-tests workflow that generates the matrix and runs tests for both e2e:ci:run and e2e:ci:run-yaml.
- Hooked e2e-tests into the release workflow so publish waits for E2E to pass.
Refactors
- Replaced bespoke PR E2E jobs with the reusable workflow, removing duplicate matrix/build steps.
- Centralized secrets handling and is_fork logic in the reusable workflow.
- Updated job dependencies so build and publish wait on vscode-e2e-tests.

github-actions

This PR refactors E2E test workflows by extracting common logic into reusable components. The changes improve maintainability and reduce duplication between main and PR workflows. However, there are several issues with the YAML syntax and potential workflow configuration problems that need to be addressed.

💡 To request a new detailed review, comment @continue-detailed-review

github-actions · 2025-08-16T00:01:32Z

.github/workflows/e2e-tests.yml

@@ -0,0 +1,58 @@
+name: 'E2E Tests'
+description: 'Reusable workflow for running VS Code E2E tests'


Remove 'description' field - it's not valid at the workflow level. Use 'name' only for workflow identification.

github-actions · 2025-08-16T00:01:32Z

.github/actions/run-vscode-e2e-tests/action.yml

+      uses: actions/upload-artifact@v4
+      with:
+        name: vscode-extension-build-Linux
+        path: extensions/vscode/build


Missing newline at end of file. Add a newline character to follow YAML best practices.

github-actions · 2025-08-16T00:01:32Z

.github/workflows/e2e-tests.yml

+          command: ${{ matrix.command }}
+          ssh_key: ${{ secrets.GH_ACTIONS_SSH_TEST_KEY_PEM }}
+          ssh_host: ${{ secrets.GH_ACTIONS_SSH_TEST_DNS_NAME }}
+          is_fork: ${{ inputs.is_fork }}


Missing newline at end of file. Add a newline character to follow YAML best practices.

github-actions · 2025-08-16T00:01:32Z

.github/actions/run-vscode-e2e-tests/action.yml

+        npm run e2e:compile
+        if [[ "${{ inputs.is_fork }}" == "true" ]]; then
+          # Exclude SSH tests for forks
+          FILES=$(ls -1 e2e/_output/tests/*.test.js | grep -v "SSH" | jq -R . | jq -s .)


Consider using 'shopt -s nullglob' before the ls command to handle the case when no test files match the pattern, preventing potential errors.

github-actions · 2025-08-16T00:01:32Z

.github/actions/run-vscode-e2e-tests/action.yml

+          # Include all tests for non-forks
+          FILES=$(ls -1 e2e/_output/tests/*.test.js | jq -R . | jq -s .)
+        fi
+        echo "test_file_matrix<<EOF" >> $GITHUB_OUTPUT


Using heredoc syntax for GitHub outputs can be problematic with JSON. Consider using a single line: echo "test_file_matrix=$FILES" >> $GITHUB_OUTPUT

github-actions · 2025-08-16T00:01:33Z

.github/workflows/e2e-tests.yml

+          is_fork: ${{ inputs.is_fork }}
+
+  vscode-e2e-tests:
+    name: ${{ matrix.test_file || 'unknown' }} (${{ matrix.command }})


The fallback 'unknown' for matrix.test_file suggests potential issues. Consider adding validation to ensure test_file_matrix is properly populated.

github-actions · 2025-08-16T00:01:33Z

.github/workflows/main.yaml

@@ -135,10 +136,23 @@ jobs:
          token: ${{ secrets.CI_GITHUB_TOKEN }}
          repository: continuedev/continue

+  vscode-e2e-tests:
+    needs: check_release_name
+    if: needs.check_release_name.outputs.should_run == 'true' || github.event_name == 'workflow_dispatch'


Consider adding 'if: always()' or 'if: success()' to clarify when E2E tests should run in the main workflow.

github-actions · 2025-08-16T00:01:33Z

.github/actions/run-vscode-e2e-tests/action.yml

+      run: |
+        node ./scripts/build-packages.js
+        cd extensions/vscode
+        npm ci


Add error handling: 'npm ci || exit 1' to ensure the workflow fails properly if npm install fails.

github-actions · 2025-08-16T00:01:33Z

.github/actions/run-vscode-e2e-tests/action.yml

+        echo "EOF" >> $GITHUB_OUTPUT
+      env:
+        # https://github.com/microsoft/vscode-ripgrep/issues/9#issuecomment-643965333
+        GITHUB_TOKEN: ${{ inputs.ci-github-token }}


Add a comment explaining why CI_GITHUB_TOKEN is used here instead of the regular github-token input.

github-actions · 2025-08-16T00:01:33Z

.github/workflows/pr-checks.yaml

-          is_fork: ${{ github.event.pull_request.head.repo.fork == true || github.actor == 'dependabot[bot]' }}
+    uses: ./.github/workflows/e2e-tests.yml
+    with:
+      is_fork: ${{ github.event.pull_request.head.repo.fork == true || github.actor == 'dependabot[bot]' }}


Good refactoring! Consider adding a comment explaining that SSH tests are excluded for forks in the reusable workflow.

github-actions · 2025-08-16T00:01:41Z

Code Review Summary

✅ Strengths

Modularization: Great refactoring to extract e2e test logic into reusable workflows and actions
Caching Strategy: Comprehensive caching of npm dependencies and node_modules across different components
Fork Safety: Properly handles fork restrictions by excluding SSH tests when is_fork is true
Matrix Testing: Smart test file matrix generation that dynamically discovers test files
Artifact Management: Good use of artifacts to share build outputs between jobs

⚠️ Issues Found

Medium

Incorrect Workflow Trigger Type: The e2e-tests.yml file uses description: field which is invalid for workflow_call triggers. This should be removed as it only applies to workflow_dispatch triggers.
Missing Error Handling: In the composite action, when building packages fails, the workflow continues without proper error propagation. The shell scripts should use set -e or check exit codes.
Hardcoded Fallback PR Number: In main-build.yaml, there's a hardcoded fallback PR number (6723) which could cause confusion:
```
PR_NUMBER=$(echo "$COMMIT_MSG" | grep -o "Merge pull request #[0-9]*" | grep -o "[0-9]*" || echo "6723")
```

Low

Redundant Output Debug: The "Debug Outputs" step in the composite action could be removed in production or gated behind a debug flag
Test File Discovery Improvements: The test file discovery uses basic ls and grep. Consider using find with more robust patterns for better reliability

💡 Suggestions

Add Workflow Documentation: Consider adding comments in the reusable workflow explaining the expected secrets and when to use it
Timeout Configuration: The hardcoded 15-minute timeout for e2e tests might need to be configurable for different test suites
Better Error Messages: When SSH tests are skipped for forks, consider logging a message to make it clear why certain tests were excluded
Consider Using GitHub Environments: For managing secrets like SSH keys, GitHub Environments could provide better security and audit trails

🚀 Overall Assessment

APPROVE

This is a well-structured refactoring that improves code reusability and maintainability. The extraction of e2e test logic into reusable workflows follows GitHub Actions best practices. The only critical issue is the incorrect workflow syntax which should be fixed before merging, but it's a simple fix. The changes demonstrate good understanding of GitHub Actions patterns and proper handling of security concerns around forks and secrets.

💡 To request a new review, comment @continue-general-review

cubic-dev-ai

cubic analysis

4 issues found across 4 files • Review in cubic

_{React with 👍 or 👎 to teach cubic. You can also tag @cubic-dev-ai to give feedback, ask questions, or re-run the review.}

cubic-dev-ai · 2025-08-16T00:11:03Z

.github/workflows/e2e-tests.yml

@@ -0,0 +1,58 @@
+name: 'E2E Tests'
+description: 'Reusable workflow for running VS Code E2E tests'


Remove the top-level 'description' key; it is not a valid workflow property and will cause workflow syntax validation errors

Prompt for AI agents

Address the following comment on .github/workflows/e2e-tests.yml at line 2: <comment>Remove the top-level 'description' key; it is not a valid workflow property and will cause workflow syntax validation errors</comment> <file context> @@ -0,0 +1,58 @@ +name: 'E2E Tests' +description: 'Reusable workflow for running VS Code E2E tests' + +on: </file context>

cubic-dev-ai · 2025-08-16T00:11:03Z

.github/workflows/pr-checks.yaml

-          is_fork: ${{ github.event.pull_request.head.repo.fork == true || github.actor == 'dependabot[bot]' }}
+    uses: ./.github/workflows/e2e-tests.yml
+    with:
+      is_fork: ${{ github.event.pull_request.head.repo.fork == true || github.actor == 'dependabot[bot]' }}


If this workflow runs on a push event (where the pull_request payload is absent), the expression throws because github.event.pull_request is undefined. Guard the access with a check on github.event_name first to avoid runtime evaluation errors.

Prompt for AI agents

Address the following comment on .github/workflows/pr-checks.yaml at line 198: <comment>If this workflow runs on a push event (where the pull_request payload is absent), the expression throws because github.event.pull_request is undefined. Guard the access with a check on github.event_name first to avoid runtime evaluation errors.</comment> <file context> @@ -192,112 +192,15 @@ jobs: AZURE_OPENAI_GPT41_API_KEY: ${{ secrets.AZURE_OPENAI_GPT41_API_KEY }} VOYAGE_API_KEY: ${{ secrets.VOYAGE_API_KEY }} - vscode-get-test-file-matrix: - runs-on: ubuntu-latest - outputs: - test_file_matrix: ${{ steps.vscode-get-test-file-matrix.outputs.test_file_matrix }} - steps: - - uses: actions/checkout@v5 </file context>

Suggested change

is_fork: ${{ github.event.pull_request.head.repo.fork == true || github.actor == 'dependabot[bot]' }}

is_fork: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == true) || github.actor == 'dependabot[bot]' }}

cubic-dev-ai · 2025-08-16T00:11:03Z

.github/actions/run-vscode-e2e-tests/action.yml

+  ssh_key:
+    description: 'SSH key for SSH tests'
+    required: false
+  ssh_host:


Input ssh_host is declared but not used anywhere in the action; unused inputs make the contract misleading and add maintenance overhead.

Prompt for AI agents

Address the following comment on .github/actions/run-vscode-e2e-tests/action.yml at line 13: <comment>Input ssh_host is declared but not used anywhere in the action; unused inputs make the contract misleading and add maintenance overhead.</comment> <file context> @@ -0,0 +1,99 @@ +name: 'Run VS Code E2E Tests' +description: 'Runs VS Code E2E tests with test matrix generation and execution' +inputs: + github-token: + description: 'GitHub token for accessing private repositories' + required: true + ci-github-token: + description: 'CI GitHub token for accessing private repositories' + required: true </file context>

cubic-dev-ai · 2025-08-16T00:11:03Z

.github/actions/run-vscode-e2e-tests/action.yml

+        npm run e2e:compile
+        if [[ "${{ inputs.is_fork }}" == "true" ]]; then
+          # Exclude SSH tests for forks
+          FILES=$(ls -1 e2e/_output/tests/*.test.js | grep -v "SSH" | jq -R . | jq -s .)


If the glob e2e/_output/tests/*.test.js matches no files, ls exits with a non-zero status and the whole composite step fails even though an empty test set would be acceptable. Redirecting the error or using a safer file discovery method avoids false negatives.

Prompt for AI agents

Address the following comment on .github/actions/run-vscode-e2e-tests/action.yml at line 68: <comment>If the glob e2e/_output/tests/*.test.js matches no files, ls exits with a non-zero status and the whole composite step fails even though an empty test set would be acceptable. Redirecting the error or using a safer file discovery method avoids false negatives.</comment> <file context> @@ -0,0 +1,99 @@ +name: 'Run VS Code E2E Tests' +description: 'Runs VS Code E2E tests with test matrix generation and execution' +inputs: + github-token: + description: 'GitHub token for accessing private repositories' + required: true + ci-github-token: + description: 'CI GitHub token for accessing private repositories' + required: true </file context>

RomneyDa

will increase confidence in main releases

tomasz-stefaniak

Looks like the tests are failing to run: https://github.com/continuedev/continue/actions/runs/17064326036

feat: run e2e tests on vsc release

021a876

Patrick-Erichsen requested a review from a team as a code owner August 15, 2025 23:59

Patrick-Erichsen requested review from tomasz-stefaniak and removed request for a team August 15, 2025 23:59

github-project-automation bot added this to Issues and PRs Aug 15, 2025

github-project-automation bot moved this to Todo in Issues and PRs Aug 15, 2025

dosubot bot added the size:L This PR changes 100-499 lines, ignoring generated files. label Aug 16, 2025

github-actions bot reviewed Aug 16, 2025

View reviewed changes

cubic-dev-ai bot reviewed Aug 16, 2025

View reviewed changes

RomneyDa previously approved these changes Aug 16, 2025

View reviewed changes

github-project-automation bot moved this from Todo to In Progress in Issues and PRs Aug 16, 2025

dosubot bot added the lgtm This PR has been approved by a maintainer label Aug 16, 2025

fix broken workflow

3ce945e

Patrick-Erichsen dismissed RomneyDa’s stale review via 3ce945e August 16, 2025 00:45

RomneyDa approved these changes Aug 19, 2025

View reviewed changes

RomneyDa closed this Aug 19, 2025

RomneyDa reopened this Aug 19, 2025

github-project-automation bot moved this from In Progress to Done in Issues and PRs Aug 19, 2025

github-project-automation bot moved this from Done to In Progress in Issues and PRs Aug 19, 2025

github-actions bot locked and limited conversation to collaborators Aug 19, 2025

tomasz-stefaniak requested changes Aug 19, 2025

View reviewed changes

dosubot bot removed the lgtm This PR has been approved by a maintainer label Aug 19, 2025

		@@ -0,0 +1,58 @@
		name: 'E2E Tests'
		description: 'Reusable workflow for running VS Code E2E tests'

	is_fork: ${{ github.event.pull_request.head.repo.fork == true \|\| github.actor == 'dependabot[bot]' }}
	is_fork: ${{ (github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == true) \|\| github.actor == 'dependabot[bot]' }}

feat: run e2e tests on vsc release #7190

Are you sure you want to change the base?

feat: run e2e tests on vsc release #7190

Conversation

Patrick-Erichsen commented Aug 15, 2025 • edited by cubic-dev-ai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by cubic

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 16, 2025

Code Review Summary

✅ Strengths

⚠️ Issues Found

Medium

Low

💡 Suggestions

🚀 Overall Assessment

Uh oh!

cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

cubic analysis

Uh oh!

cubic-dev-ai bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai bot Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

RomneyDa left a comment

Choose a reason for hiding this comment

Uh oh!

tomasz-stefaniak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Patrick-Erichsen commented Aug 15, 2025 •

edited by cubic-dev-ai bot

Loading