add option to generate test coverage report without codecov #207

zacharyburnett · 2024-06-09T17:03:19Z

using the guide here: https://hynek.me/articles/ditch-codecov-python/

This does not create coverage diffs or report to PRs, just creates a markdown summary table

closes #189

EDIT: here's what the summary table looks like:

Cadair · 2024-06-10T09:45:23Z

I tried this out on sunpy and it seems to explode. sunpy/sunpy#7668

Cadair · 2024-06-10T09:55:32Z

It seems that it's not picking up the coverage.xml to upload here: https://github.com/sunpy/sunpy/actions/runs/9445798095/job/26014155532?pr=7668#step:14:16 🤔

Cadair

Thanks so much for taking a swing at this!

.github/workflows/tox.yml

Cadair · 2024-06-10T09:58:06Z

.github/workflows/tox.yml

+      - run: python -Im pip install --upgrade coverage[toml]
+      - run: python -Im coverage combine
+      - run: python -Im coverage report --format=markdown >> $GITHUB_STEP_SUMMARY
+


I think it would be nice to add the ability to fail this job based on either total coverage % or diff coverage %.

pllim · 2024-06-10T09:59:05Z

tl;dr -- this does not generate patch coverage, right?

Cadair · 2024-06-10T11:43:50Z

Not yet, but I found a way to make it.

zacharyburnett · 2024-06-10T17:39:37Z

tl;dr -- this does not generate patch coverage, right?

Not yet, but I found a way to make it.

For patch coverage we'd need to store the coverage from main somewhere, to compare it to; some people I've seen online have done that by storing it in a hidden page in the repository's wiki, reading and writing it with a github token with ${{ secrets.GITHUB_TOKEN }}

tools/tox_matrix.py

pllim · 2024-06-11T10:02:31Z

I am trying this out too at spacetelescope/acstools#205 and it still crashes:

https://github.com/spacetelescope/acstools/actions/runs/9463279566/job/26068438067?pr=205

zacharyburnett · 2024-06-11T12:49:16Z

I am trying this out too at spacetelescope/acstools#205 and it still crashes:

https://github.com/spacetelescope/acstools/actions/runs/9463279566/job/26068438067?pr=205

oh whoops, I fixed the artifact issue:

but didn't pass any filenames to coverage combine 😅

pllim · 2024-06-12T09:13:49Z

Even when CI passes, it failed with the same error.

Coverage XML written to file coverage.xml

mv: cannot stat '.coverage': No such file or directory

zacharyburnett · 2024-06-12T12:58:45Z

Even when CI passes, it failed with the same error.

Coverage XML written to file coverage.xml

mv: cannot stat '.coverage': No such file or directory

it should be uploading / downloading the .coverage database, to be able to combine multiple machine types, so if that's not present then something is messed up

zacharyburnett · 2024-06-12T13:06:57Z

it looks like --cov-report=xml is what's creating the coverage.xml; however, it should still be creating a .coverage database file... I'll so some local testing to see what files it spits out
https://github.com/spacetelescope/acstools/actions/runs/9463279566/job/26118526244#step:10:105

pytest --pyargs acstools --cov-report=xml --cov=acstools /home/runner/work/acstools/acstools/doc --cov --remote-data -v

zacharyburnett · 2024-06-12T13:18:05Z

running pytest --pyargs acstools --cov-report=xml --cov=acstools --cov --remote-data -v locally does appear to create the .coverage file:

❯ ls -a
.							CITATION.md
..							CODE_OF_CONDUCT.md
.bandit.yaml						LICENSE.md
.coverage						MANIFEST.in
.coverage.urelialia2022.local.stsci.edu.35749.XCMKciAx	README.rst
.git							__pycache__
.github							acstools
.gitignore						acstools.egg-info
.mypy_cache						conftest.py
.pytest_cache						coverage.xml
.readthedocs.yaml					doc
.tmp							pyproject.toml
.tox							tox.ini

pllim · 2024-06-12T13:33:50Z

What is the difference between .coverage and coverage.xml?

zacharyburnett · 2024-06-12T14:25:07Z

What is the difference between .coverage and coverage.xml?

I'm not intimately familiar with it, but as far as I know .coverage is a SQLite database file with support for parallel-mode (storing coverage for multiple machine configurations / runs) while coverage.xml is report-focused and so is expected to only contain results from a single run.

We could use coverage.xml, but then we'd need to have a summary table of coverage for each and every run, instead of a single table collating all runs

pllim · 2024-06-12T15:19:19Z

Tried to debug at #209 but no luck. I see the coverage ran but the the upload failed: https://github.com/spacetelescope/acstools/actions/runs/9483014271/job/26134042811?pr=205

zacharyburnett · 2024-06-12T15:23:24Z

Tried to debug at #209 but no luck. I see the coverage ran but the the upload failed: https://github.com/spacetelescope/acstools/actions/runs/9483014271/job/26134042811?pr=205

could you also add -a to the ls?

pllim · 2024-06-12T15:49:49Z

Ah, yes, added. New log: https://github.com/spacetelescope/acstools/actions/runs/9483014271/job/26138000458?pr=205

for more information, see https://pre-commit.ci

Cadair · 2024-12-04T13:55:15Z

.github/workflows/tox.yml

+
+      - if: ${{ (success() || failure()) && contains(matrix.coverage, 'github') && matrix.pytest == 'true' }}
+        name: add run info to coverage filename
+        run: mv $(find ~ -name .coverage) .coverage.${{ github.sha }}-${{ runner.os }}-${{ runner.arch }}-${{ matrix.toxenv }}


I seem to be having trouble with this line.

I am trying this out on the dkist repo: DKISTDC/dkist#471 and by default tox puts .coverage in the .tmp/py312/ directory. Even if I move the .coverage file up into the root repo directory this still doesn't seem to find it.

Should this in fact be this:

Suggested change

run: mv $(find ~ -name .coverage) .coverage.${{ github.sha }}-${{ runner.os }}-${{ runner.arch }}-${{ matrix.toxenv }}

run: mv $(find . -name .coverage) .coverage.${{ github.sha }}-${{ runner.os }}-${{ runner.arch }}-${{ matrix.toxenv }}

i.e. look in the current directory?

ah, good catch, it should be using the working directory

What I can't figure out is if there's an way of doing the equivalent of this:

rf"--cov-report=xml:${{GITHUB_WORKSPACE}}{sep}coverage.xml ")

Where we can set the .coverage path?

coverage makes a .coverage database by default, if you don't specify a --cov-report type. I don't think you can set a location for .coverage though. You CAN specify --cov-report= to be html xml json lcov annotate (see https://pytest-cov.readthedocs.io/en/latest/reporting.html), but as far as I can tell the markdown table generator (python -Im coverage report -i -m --format=markdown) can only ingest the .coverage database so another format report is not needed.

Yeah, it's the inability to specify the path of the coverage db that seems to be a problem.

Even with the last commit I don't seem to be finding the db file: https://github.com/DKISTDC/dkist/actions/runs/12161554758/job/33916444661?pr=471#step:14:1

Testing this out locally seems to work.

I hope you don't mind, but I pushed a couple of experimental commits to make it search from the root workspace dir down. Let's see what happens.

no problem!

Ok well the plot thickens: https://github.com/DKISTDC/dkist/actions/runs/12161554758/job/33918322027#step:14:1

The find command finds the file, but then it fails to upload?

ah! Maybe add an ls in there to see if the mv command actually worked?

I tried to go a different way and let the upload action figure it out: https://github.com/DKISTDC/dkist/actions/runs/12161554758/job/33921111684#step:14:16

It did not go well.

Co-authored-by: Stuart Mumford <[email protected]>

.github/workflows/tox.yml

Cadair · 2024-12-05T11:38:22Z

pre-commit.ci autofix

zacharyburnett force-pushed the tox/coverage_github_actions branch from 8a31636 to fba02a5 Compare June 9, 2024 17:06

Cadair reviewed Jun 10, 2024

View reviewed changes

zacharyburnett force-pushed the tox/coverage_github_actions branch from 325b539 to 156160e Compare June 10, 2024 17:59

zacharyburnett commented Jun 10, 2024

View reviewed changes

tools/tox_matrix.py Outdated Show resolved Hide resolved

pllim mentioned this pull request Jun 11, 2024

EXP: Test coverage without codecov spacetelescope/acstools#205

Closed

zacharyburnett force-pushed the tox/coverage_github_actions branch from d8c63d1 to c5b70cf Compare June 11, 2024 13:38

zacharyburnett mentioned this pull request Jun 11, 2024

test uploading coverage as github actions summary spacetelescope/romancal#1268

Draft

6 tasks

zacharyburnett force-pushed the tox/coverage_github_actions branch from c5b70cf to f891927 Compare June 11, 2024 13:47

zacharyburnett mentioned this pull request Jun 11, 2024

test uploading coverage as github actions summary spacetelescope/stcal#262

Draft

5 tasks

zacharyburnett force-pushed the tox/coverage_github_actions branch from f891927 to 430ed7e Compare June 11, 2024 14:03

This comment was marked as resolved.

Sign in to view

This comment was marked as outdated.

Sign in to view

pllim mentioned this pull request Jun 12, 2024

EXP: PR to PR 207 #209

Closed

zacharyburnett and others added 15 commits December 2, 2024 08:53

update matrix script

2ff2795

[pre-commit.ci] auto fixes from pre-commit.com hooks

f9713a8

for more information, see https://pre-commit.ci

upload .coverage.* file to combine

c4ba2c9

list files

9786a39

use runner.os instead of matrix value

19c88a4

combine and report coverage

bcfbd6b

undo style formatting

6740255

set run shell

a6f284a

set job name as report coverage to be explicit

d626757

use conditional ifs instead of continue-of-error

64f33ca

reorder ls and mv

80bec60

find .coverage

a44231b

use find to find .coverage

35381ff

rename job to indicate that it is reporting overall coverage

5ec90ff

always run coverage report

88ebba9

zacharyburnett force-pushed the tox/coverage_github_actions branch from 242272f to 88ebba9 Compare December 2, 2024 13:56