trying to fix coverage

[pavelkomarov]

The coverage badge is broken in the readme, and it looks like the last time coverage was successfully used was 2019! I'm attempting a fix by following an old example of mine where I figured this out. I'll need one of you repo maintainers to sign in to the coverage account and generate a token and put it as a secret in this repo, or go look at the repo secrets and tell me what the token is named. (It may already exist.)

[mandli]

I think I just added it...?

[pavelkomarov]

Thanks! In the repo's Settings, there's a Security section, and it goes in there. It should be named COVERALLS_REPO_TOKEN to work with the specific commit I just made, but you can name it whatever, and I can update the script to agree.

If all is right, then rerunning the Action should result in a push to coveralls.io (and some useful stuff there).

[mandli]

I called it COVERALLS_REPO_TOKEN and added it there so if that does not work we will have to see if I used the right token.

[pavelkomarov]

I don't think I have the power to rerun the job, and I'd rather not make a dummy commit. You'll have to test it out.

[mandli]

I cannot tell if anything worked but I think we should separate out the coverage test from the primary test to make things clear.

[ketch]

I'm not sure how to test this. I ran the same commands locally and I get

Error running coveralls: Not on TravisCI. You have to provide either repo_token in .coveralls.yml or set the COVERALLS_REPO_TOKEN env var.

Sorry, I haven't used coveralls in a long time and didn't take the time to look up what to do. @mandli the point of this is to report what fraction of the code is being tested. So in a sense they can't be separated. Or do you mean to run all the tests twice?

[mandli]

You might be able to add the COVERALLS_REPO_TOKEN as an environment variable but it is added correctly as a secret as far as I can tell. I also was mistaken about separating, I thought it could be made more independent of the tests than we can.

[pavelkomarov]

So when the build runs, we should end up with info here https://coveralls.io/github/clawpack/pyclaw. If you go under the Actions tab for the repo and rerun the latest build, then hopefully it succeeds at pushing information to coveralls at the end. It failed to do so when I opened this PR (thus kicking off a CI job), because the COVERALLS_REPO_TOKEN didn't yet exist. I don't have the power to rerun that job directly from the Actions, but you probably do. I could push some other commit to this PR to start another job, but I'd rather not create an extra commit.

[mandli]

I did restart the action but it just failed out again. I am guessing that it isn't pushing the information to coveralls because of the failure but that sort of does not make sense to me given that you would probably want to have coverage info regardless of the tests failing.

[pavelkomarov]

Hmm. The tests are run with a coverage call, so the coverage info should be generated, and then the next call in the CI script is to upload that info. Does the build choose to fail before that next call is made? I think it makes some sense to not have coverage info for broken builds cluttering the coverage history. The assumption is the build shouldn't be broken most of the time.

[mandli]

I just restarted it again with debug logging turned on (not sure what that actually does TBH). coveralls is called after the pytest call so that might be why. We may want to modify the pytest call to not stop at a particular failure as well.

[pavelkomarov]

I don't think there's a way to call coveralls except for after the tests are done. First the coverage info is accumulated during tests, then it's uploaded. If you indeed would like coverage uploaded regardless of failure status, is there a way to bypass the failure? I'm having trouble figuring out whether coveralls is even meant to do coverage for failing builds. Here's what Google's AI says:

[mandli]

Looks like we just need to fix that test.

[pavelkomarov]

Okay, I spent a while playing with the yml script. If I make the coveralls call its own task and invoke always(), then I can get it to run even when tests fail, and it comes through in the coverage history 🎉 But hold your celebration, because the status is "pending completion", which I take to mean coverage didn't finish running, so there isn't really a report. Somebody else asked about forcing a report even in the presence of failing tests and didn't get any answers.

[pavelkomarov]

I suspect the fact the test is failing with a segfault may make wrapping up the coverage report less likely. If I run locally, one of the examples/advection_1d_variable/test_variable_coefficient_advection.py ....F tests fails, but the shallow water tests don't segfault. In this case, I do get a .coverage file output.

[pavelkomarov]

Ahha. There is indeed no .coverage file created, likely due to the segfault.

[ketch]

@pavelkomarov If you rebase this onto main (now that I merged #736) it should avoid the segfault.

[pavelkomarov]

Okay, something curious now: The tests finish, but .coverage isn't where I expected. I think I have to re-cd down to the right directory. So maybe it was actually there even with segfault? Unclear. Trying again.

[pavelkomarov]

Okay! The job finished, and it just showed "pending completion" again.

But then after giving it a second, it updated.

Back in business, boys!

[ketch]

Thank you and nice work, @pavelkomarov !