-
Notifications
You must be signed in to change notification settings - Fork 105
Run all tests for new libraries versions checking at once #675
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
dnestoro
wants to merge
7
commits into
master
Choose a base branch
from
dnestoro/run-all-new-versions-tests-at-once
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+254
−231
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
9a7203f
Run all tests for new libraries versions checking at once
dnestoro 5850f41
Run parallel job for each library
dnestoro c1a2913
Add retry logic for failng tests
dnestoro 24ada51
Skip tests if there is no fix since last run
dnestoro 2e8a058
Schedule job for run once a week
dnestoro 1e6e6fe
Remove matrix limits
dnestoro ffd72c9
Use println for output of fetching new versions task
dnestoro File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
203 changes: 203 additions & 0 deletions
203
.github/workflows/check-new-library-versions-in-batch.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,203 @@ | ||
| name: "Check new libraries versions" | ||
|
|
||
| on: | ||
| schedule: | ||
| - cron: '0 0 * * 7' | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: write | ||
| actions: write | ||
|
|
||
| concurrency: | ||
| group: "workflow=${{ github.workflow }},ref=${{ github.event.ref }},pr=${{ github.event.pull_request.id }}" | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| get-all-libraries: | ||
| name: "📋 Get list of all supported libraries with newer versions" | ||
| if: github.repository == 'oracle/graalvm-reachability-metadata' | ||
| runs-on: ubuntu-22.04 | ||
| timeout-minutes: 5 | ||
| permissions: write-all | ||
| outputs: | ||
| matrix: ${{ steps.set-matrix.outputs.matrix }} | ||
| branch: ${{ steps.set-branch-name.outputs.branch }} | ||
| steps: | ||
| - name: "☁️ Checkout repository" | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: "🔧 Prepare environment" | ||
| uses: graalvm/setup-graalvm@v1 | ||
| with: | ||
| java-version: '21' | ||
| distribution: 'graalvm' | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
|
|
||
| - name: "📅 Set branch name" | ||
| id: set-branch-name | ||
| run: | | ||
| BRANCH_NAME="check-new-library-versions/$(date '+%Y-%m-%d')" | ||
| echo "branch=$BRANCH_NAME" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: "🕸️ Populate matrix" | ||
| id: set-matrix | ||
| run: | | ||
| echo "matrix=$(./gradlew fetchExistingLibrariesWithNewerVersions --quiet | sed -n '/\[/,$p')" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: "🔨 Create branch" | ||
| run: | | ||
| git config --local user.email "[email protected]" | ||
| git config --local user.name "Github Actions" | ||
| git checkout -b "${{ steps.set-branch-name.outputs.branch }}" | ||
| git push origin "${{ steps.set-branch-name.outputs.branch }}" | ||
|
|
||
| test-all-metadata: | ||
| name: "🧪 ${{ matrix.item.name }}" | ||
| runs-on: ubuntu-22.04 | ||
| needs: get-all-libraries | ||
| permissions: write-all | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| item: ${{ fromJson(needs.get-all-libraries.outputs.matrix) }} | ||
| steps: | ||
| - name: "☁️ Checkout repository" | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: "🔧 Setup java" | ||
| uses: actions/setup-java@v4 | ||
| with: | ||
| distribution: 'oracle' | ||
| java-version: '21' | ||
|
|
||
| - name: "🔧 Prepare environment" | ||
| uses: graalvm/setup-graalvm@v1 | ||
| with: | ||
| set-java-home: 'false' | ||
| java-version: 21 | ||
| distribution: 'graalvm' | ||
| github-token: ${{ secrets.GITHUB_TOKEN }} | ||
| native-image-job-reports: 'true' | ||
|
|
||
| - name: "Check for existing failure issue and skip if found" | ||
| id: check_existing_issue | ||
| run: | | ||
| GROUP_ID="$(echo "${{ matrix.item.name }}" | cut -d: -f1)" | ||
| ARTIFACT_ID="$(echo "${{ matrix.item.name }}" | cut -d: -f2)" | ||
|
|
||
| readarray -t VERSIONS < <(echo '${{ toJson(matrix.item.versions) }}' | jq -r '.[]') | ||
| FIRST_TESTING_VERSION="${VERSIONS[0]}" | ||
|
|
||
| TITLE="Failure detected for $GROUP_ID:$ARTIFACT_ID" | ||
| BODY="First failing version $GROUP_ID:$ARTIFACT_ID:$FIRST_TESTING_VERSION" | ||
|
|
||
| ISSUE_NUMBER=$(gh issue list --repo "${{ github.repository }}" --state open --search "$TITLE" --json number,title,body --jq \ | ||
| '.[] | select(.title == "'"$TITLE"'") | select(.body == "'"$BODY"'") | .number') | ||
|
|
||
| if [[ -n "$ISSUE_NUMBER" ]]; then | ||
| echo "There is no progress since last time this version was tested. Skipping further steps." | ||
| exit 0 | ||
| fi | ||
|
|
||
| - name: "Extract test path and library version" | ||
| id: extract-params | ||
| run: | | ||
| LIBRARY_PATH=$(echo "${{ matrix.item.name }}" | sed 's/:/\//g') | ||
| LATEST_VERSION=$(find tests/src/$LIBRARY_PATH/* -maxdepth 1 -type d | sort -V | tail -1 | cut -d '/' -f5) | ||
| TEST_PATH="$LIBRARY_PATH/$LATEST_VERSION" | ||
| TEST_COORDINATES=$(echo "$TEST_PATH" | tr / :) | ||
|
|
||
| echo "LATEST_VERSION=$LATEST_VERSION" >> ${GITHUB_ENV} | ||
| echo "TEST_PATH=$TEST_PATH" >> ${GITHUB_ENV} | ||
| echo "TEST_COORDINATES=$TEST_COORDINATES" >> ${GITHUB_ENV} | ||
|
|
||
| - name: "Pull allowed docker images" | ||
| run: ./gradlew pullAllowedDockerImages --coordinates="${{ env.TEST_COORDINATES }}" | ||
|
|
||
| - name: "Disable docker" | ||
| run: | | ||
| sudo apt-get install openbsd-inetd | ||
| sudo bash -c "cat ./.github/workflows/discard-port.conf >> /etc/inetd.conf" | ||
| sudo systemctl start inetd | ||
| sudo mkdir /etc/systemd/system/docker.service.d | ||
| sudo bash -c "cat ./.github/workflows/dockerd.service > /etc/systemd/system/docker.service.d/http-proxy.conf" | ||
| sudo systemctl daemon-reload | ||
| sudo systemctl restart docker | ||
|
|
||
| - name: "🧪 Run '${{ env.TEST_COORDINATES }}' tests" | ||
| id: runtests | ||
| run: | | ||
| bash ./.github/workflows/run-consecutive-tests.sh "${{ env.TEST_COORDINATES }}" '${{ toJson(matrix.item.versions) }}' 2>&1 | tee test_results.txt || true | ||
|
|
||
| # Extract successful versions | ||
| grep "^PASSED:" test_results.txt | sed 's/PASSED://g' > successful_versions.txt | ||
| echo "successful_versions<<EOF" >> $GITHUB_OUTPUT | ||
| cat successful_versions.txt >> $GITHUB_OUTPUT | ||
| echo "EOF" >> $GITHUB_OUTPUT | ||
|
|
||
| # Extract failed version | ||
| FAILED_VERSION=$(grep "^FAILED:" test_results.txt | sed 's/FAILED://g') | ||
| echo "failed_version=$FAILED_VERSION" >> $GITHUB_OUTPUT | ||
|
|
||
| - name: "✔️ New library is supported" | ||
| if: steps.runtests.outputs.successful_versions != '' | ||
| run: | | ||
| git config --local user.email "[email protected]" | ||
| git config --local user.name "Github Actions" | ||
| git fetch origin ${{ needs.get-all-libraries.outputs.branch }} | ||
| git checkout ${{ needs.get-all-libraries.outputs.branch }} | ||
|
|
||
| while read version; do | ||
| if [ -n "$version" ]; then | ||
| ./gradlew addTestedVersion --coordinates="${{ matrix.item.name }}:$version" --lastSupportedVersion="${{ env.LATEST_VERSION }}" | ||
| fi | ||
| done < successful_versions.txt | ||
|
|
||
| git add -u | ||
| git commit -m "Update tested versions for ${{ matrix.item.name }}" | ||
| git push origin ${{ needs.get-all-libraries.outputs.branch }} | ||
|
|
||
| - name: "❗ New library is not supported" | ||
| if: steps.runtests.outputs.failed_version != '' | ||
| run: | | ||
| git config --local user.email "[email protected]" | ||
| git config --local user.name "Github Actions" | ||
|
|
||
| FAILED_VERSION="${{ steps.runtests.outputs.failed_version }}" | ||
| REPO="${{ github.repository }}" | ||
| TITLE="Failure detected for ${{ matrix.item.name }}" | ||
| BODY="First failing version ${{ matrix.item.name }}:$FAILED_VERSION" | ||
|
|
||
| ISSUE_NUMBER=$(gh issue list --repo "$REPO" --state open --search "$TITLE" --json number,title --jq '.[] | select(.title == "'"$TITLE"'") | .number') | ||
| if [ -n "$ISSUE_NUMBER" ]; then | ||
| echo "Updating existing issue #$ISSUE_NUMBER" | ||
| gh issue edit "$ISSUE_NUMBER" --repo "$REPO" --body "$BODY" | ||
| else | ||
| echo "Creating new issue" | ||
| gh issue create --repo "$REPO" --title "$TITLE" --body "$BODY" | ||
| fi | ||
|
|
||
| process-results: | ||
| name: "🧪 Process results" | ||
| runs-on: ubuntu-22.04 | ||
| permissions: write-all | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} | ||
| if: ${{ always() }} | ||
| needs: | ||
| - get-all-libraries | ||
| - test-all-metadata | ||
| steps: | ||
| - name: "☁️ Checkout repository" | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: "✏️ PR for supported versions" | ||
| run: | | ||
| git config --local user.email "[email protected]" | ||
| git config --local user.name "Github Actions" | ||
| git fetch origin ${{ needs.get-all-libraries.outputs.branch }} | ||
| git checkout ${{ needs.get-all-libraries.outputs.branch }} | ||
| gh pr create --title "Update supported library versions" --body "This pull request updates supported versions of the existing libraries in the repo" | ||
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need this? What if I fork this repo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Previously this workflow was scheduled for execution every two weeks, and therefore we didn't want this to be executed every two weeks on user's forks (we would spend their repo resources that way). We can remove it now if necessary, but I don't see a reason why this job should be executed anywhere else than in this repo.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting, just leave it as a comment. Is this standard practice?
Will this prevent me from running this task manually on my fork, and should it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, that will prevent you from running this workflow on your fork (that is the purpose of this). As long as this is not a scheduled, but manually triggered job, we can remove this line. If we decide to make it scheduled at some point, I will keep this line.