Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(github): coverage workflow using evmone script #503

Merged
merged 1 commit into from
Jun 5, 2024
Merged

Conversation

winsvega
Copy link
Collaborator

@winsvega winsvega commented Apr 12, 2024

🗒️ Description

git job to measure coverage of ported tests
if a new line introduced to converted_ethereum_tests, the script will run coverage and make an artifact

demonstration: #504

What does it do:

Evmone docker image

The main script is in this repo: https://github.com/winsvega/evmtest_coverage/tree/main/dockerhub
There are 3 docker dependencies that are used in this git workflow:

  1. evmone image built in coverage mode with coverage software installed
  2. retesteth docker image used to trigger test runs via t8n (can be switched to native evmone in the future)
  3. coverage script that manages test execution and coverage verification

Coverage script

With the latest github docker workflow refactor, the script speaks for itself, but I put here some comments:

This command mounts evmtest_coverage/coverage github folder into docker /tests folder
PATCH_TESTS is a folder inside that already prepared with the .json tests to cover
run: /entrypoint.sh --mode=cover --testpath=/tests/PATCH_TESTS --outputname=PATCH
will create coverage_PATCH.lcov and PATCH folder with html report
same is done for the BASE tests

run: /entrypoint.sh --mode=diff --basefile=coverage_BASE.lcov --patchfile=coverage_PATCH.lcov
will create DIFF folder with html report on coverage. which will later be uploaded as artifact.

      - name: Run coverage of the PATCH tests
        uses: addnab/docker-run-action@v3
        with:
          image: winsvega/evmone-coverage-script:latest
          options: -v ${{ github.workspace }}/evmtest_coverage/coverage:/tests
          run: /entrypoint.sh --mode=cover --testpath=/tests/PATCH_TESTS --outputname=PATCH

      - name: Run coverage DIFF of the PATCH tests compared to BASE tests
        uses: addnab/docker-run-action@v3
        with:
          image: winsvega/evmone-coverage-script:latest
          options: -v ${{ github.workspace }}/evmtest_coverage/coverage:/tests
          run: /entrypoint.sh --mode=diff --basefile=coverage_BASE.lcov --patchfile=coverage_PATCH.lcov

🔗 Related Issues

Feauture

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@winsvega winsvega changed the base branch from eip1153 to main April 12, 2024 12:38
@winsvega winsvega requested a review from marioevz April 12, 2024 12:39
@winsvega winsvega added the type:test Type: Test label Apr 12, 2024
@winsvega winsvega added type:feat type: Feature scope:pytest Scope: Pytest plugins labels Apr 23, 2024
@winsvega winsvega force-pushed the coverage branch 4 times, most recently from c55a0ec to 708dabe Compare April 26, 2024 08:09
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments.

.github/workflows/coverage.yaml Show resolved Hide resolved
.github/workflows/coverage.yaml Outdated Show resolved Hide resolved
.github/workflows/coverage.yaml Outdated Show resolved Hide resolved
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the simplification effort, it's much cleaner now!

@marioevz marioevz changed the title git workflow evmone coverage script feat(github): coverage workflow using evmone script Jun 5, 2024
@marioevz marioevz merged commit 4852638 into main Jun 5, 2024
9 checks passed
@marioevz marioevz deleted the coverage branch June 5, 2024 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope:pytest Scope: Pytest plugins type:feat type: Feature type:test Type: Test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants