Skip to content

Commit

Permalink
CI(pre-merge-checks): add required checks (#9700)
Browse files Browse the repository at this point in the history
## Problem
The Merge queue doesn't work because it expects certain jobs, which we
don't have in the `pre-merge-checks` workflow.
But it turns out we can just create jobs/checks with the same names in
any workflow that we run.

## Summary of changes
- Add `conclusion` jobs
- Create `neon-cloud-e2e` status check
- Add a bunch of `if`s to handle cases with no relevant changes found
and prepare the workflow to run rust checks in the future
- List the workflow in `report-workflow-stats` to collect stats about it
  • Loading branch information
bayandin authored Nov 9, 2024
1 parent ecde8d7 commit 2fcac0e
Show file tree
Hide file tree
Showing 2 changed files with 55 additions and 7 deletions.
61 changes: 54 additions & 7 deletions .github/workflows/pre-merge-checks.yml
Original file line number Diff line number Diff line change
@@ -1,22 +1,26 @@
name:
name: Pre-merge checks

on:
merge_group:
branches:
- main

defaults:
run:
shell: bash -euxo pipefail {0}

# No permission for GITHUB_TOKEN by default; the **minimal required** set of permissions should be granted in each job.
permissions: {}

jobs:
get-changed-files:
runs-on: ubuntu-22.04
outputs:
any_changed: ${{ steps.src.outputs.any_changed }}
python-changed: ${{ steps.python-src.outputs.any_changed }}
steps:
- uses: actions/checkout@v4
- uses: tj-actions/changed-files@c3a1bb2c992d77180ae65be6ae6c166cf40f857c # v45.0.3
id: src
- uses: tj-actions/changed-files@4edd678ac3f81e2dc578756871e4d00c19191daf # v45.0.4
id: python-src
with:
files: |
.github/workflows/pre-merge-checks.yml
Expand All @@ -26,10 +30,13 @@ jobs:
- name: PRINT ALL CHANGED FILES FOR DEBUG PURPOSES
env:
ALL_CHANGED_FILES: ${{ steps.src.outputs.all_changed_files }}
run: echo "${ALL_CHANGED_FILES}"
PYTHON_CHANGED_FILES: ${{ steps.python-src.outputs.all_changed_files }}
run: |
echo "${PYTHON_CHANGED_FILES}"
check-build-tools-image:
if: needs.get-changed-files.outputs.python-changed == 'true'
needs: [ get-changed-files ]
uses: ./.github/workflows/check-build-tools-image.yml

build-build-tools-image:
Expand All @@ -40,8 +47,48 @@ jobs:
secrets: inherit

check-codestyle-python:
needs: [ build-build-tools-image ]
if: needs.get-changed-files.outputs.python-changed == 'true'
needs: [ get-changed-files, build-build-tools-image ]
uses: ./.github/workflows/_check-codestyle-python.yml
with:
build-tools-image: ${{ needs.build-build-tools-image.outputs.image }}-bookworm
secrets: inherit

# To get items from the merge queue merged into main we need to satisfy "Status checks that are required".
# Currently we require 2 jobs (checks with exact name):
# - conclusion
# - neon-cloud-e2e
conclusion:
if: always()
permissions:
statuses: write # for `github.repos.createCommitStatus(...)`
needs:
- get-changed-files
- check-codestyle-python
runs-on: ubuntu-22.04
steps:
- name: Create fake `neon-cloud-e2e` check
uses: actions/github-script@v7
with:
# Retry script for 5XX server errors: https://github.com/actions/github-script#retries
retries: 5
script: |
const { repo, owner } = context.repo;
const targetUrl = `${context.serverUrl}/${owner}/${repo}/actions/runs/${context.runId}`;
await github.rest.repos.createCommitStatus({
owner: owner,
repo: repo,
sha: context.sha,
context: `neon-cloud-e2e`,
state: `success`,
target_url: targetUrl,
description: `fake check for merge queue`,
});
- name: Fail the job if any of the dependencies do not succeed or skipped
run: exit 1
if: |
(contains(needs.check-codestyle-python.result, 'skipped') && needs.get-changed-files.outputs.python-changed == 'true')
|| contains(needs.*.result, 'failure')
|| contains(needs.*.result, 'cancelled')
1 change: 1 addition & 0 deletions .github/workflows/report-workflow-stats.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ on:
- Test Postgres client libraries
- Trigger E2E Tests
- cleanup caches by a branch
- Pre-merge checks
types: [completed]

jobs:
Expand Down

1 comment on commit 2fcac0e

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5446 tests run: 5205 passed, 9 failed, 232 skipped (full report)


Failures on Postgres 16

  • test_pgbench_intensive_init_workload[neon_off-github-actions-selfhosted-1000]: release-x86-64
  • test_sharded_ingest[github-actions-selfhosted-1]: release-x86-64
  • test_compaction[github-actions-selfhosted]: release-x86-64
  • test_download_churn[github-actions-selfhosted-1-tokio-epoll-uring-30]: release-x86-64
  • test_download_churn[github-actions-selfhosted-1-std-fs-30]: release-x86-64
  • test_download_churn[github-actions-selfhosted-10-tokio-epoll-uring-30]: release-x86-64
  • test_download_churn[github-actions-selfhosted-10-std-fs-30]: release-x86-64
  • test_download_churn[github-actions-selfhosted-100-tokio-epoll-uring-30]: release-x86-64
  • test_download_churn[github-actions-selfhosted-100-std-fs-30]: release-x86-64
# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_pgbench_intensive_init_workload[neon_off-release-pg16-github-actions-selfhosted-1000] or test_sharded_ingest[release-pg16-github-actions-selfhosted-1] or test_compaction[release-pg16-github-actions-selfhosted] or test_download_churn[release-pg16-github-actions-selfhosted-1-tokio-epoll-uring-30] or test_download_churn[release-pg16-github-actions-selfhosted-1-std-fs-30] or test_download_churn[release-pg16-github-actions-selfhosted-10-tokio-epoll-uring-30] or test_download_churn[release-pg16-github-actions-selfhosted-10-std-fs-30] or test_download_churn[release-pg16-github-actions-selfhosted-100-tokio-epoll-uring-30] or test_download_churn[release-pg16-github-actions-selfhosted-100-std-fs-30]"
Flaky tests (1)

Postgres 17

Code coverage* (full report)

  • functions: 31.7% (7868 of 24806 functions)
  • lines: 49.4% (62278 of 126036 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
2fcac0e at 2024-11-09T02:51:49.886Z :recycle:

Please sign in to comment.