How to fix Pre-commit and Benchmarks run failures?

[evevkovacs]

I have setup my package using the project template and am getting errors on the benchmarks and pre-commit runs. I have tried googling the errors but haven't found anything useful (that I can understand).
For the workflow run entitled "Run benchmarks for PR / asv-pr", the fatal errors seem to be:
Running '/usr/bin/git name-rev --name-only --exclude=remotes/* --no-undefined
and
_Benchmark list file results/benchmarks.json missing!
Use asv run --bench just-discover to regenerate benchmarks.json
A traceback is available, but I won't reproduce it here.

For the workflow run entitled "Run pre-commit hooks / pre-commit-ci ", it produces thousands of lines of output and ends with:
Error: Process completed with exit code 1.
There is no indication of what the error is.

Does anyone have any advice on how to proceed?

[evevkovacs]

Addendum: I installed pre-commit locally and ran it. There were a number of linter errors, some of which I don't want to fix (eg. mixed case variable names). Anyway, I will chip away at these errors to see if that helps, and report back.

[evevkovacs]

As a follow-up, running pre-commit (and various required helper packages) locally was helpful. I either resolved all the linter errors or changed the precommit yaml file to ignore the ones I didn't want to change and was able to get the pre-commit to pass all tests. After I pushed the changes to the repository, the test succeeded.

[jeremykubica]

We just debugging a similar sounding error in one of our repositories. The root cause for us was a missing dependency in the pyproject.toml file. We were using matplotlib in one of the Python files, but had only included it in the dev dependencies. Without the debug trace, I can't say if this is happening to you as well. But you can scan for any mention of missing libraries.

[camposandro]

Hi @evevkovacs, thanks for reaching out to us!

Hopefully I am looking at the correct workflow. It seems like the issue might be that the Python version specified for the benchmarking tool (ASV) is not compatible with the one specified for the package:

ERROR: Package 'diffaux' requires a different Python: 3.10.12 not in '>=3.12'

I suspect that upgrading the Python version here to "3.12" should help. Can you please give it a try?

[camposandro]

Thanks for all your patience!

I had not noticed but this project is currently set to use the template v2.0.1. I suggest updating to its latest version (2.0.5) as we introduced several fixes to the workflows.

To upgrade your project you can simply run copier update <project directory>. I also strongly recommend you go with the customized installation to select the appropriate Python 3.12. You can find more information in our documentation.

For reference, I tested this upgrade on a fork and the benchmark workflows worked as expected.

Please let me know if that also works for you!

[evevkovacs]

I updated the template as you suggested with copier. When I ran the CI tests, there was an error in the .pre-commit.yml file:
" <<<<<<< before updating"
=======
exclude: ^docs/pre_executed
" >>>>>>> after updating"

It looks like copier put something spurious in the yaml file.
I removed the offending lines above (retaining exclude: ^docs/pre_executed) and reran. I got past that, but now there are several other problems resulting from the update. See here
Could you please take a look? They look like problems with the updated configuration files and you would be able to identify the issues more quickly than me. Thanks! (SORRY about the format above, I can't seem to fix it)

Note also, that a number of tests are being skipped now. I don't think that is a good default.

[evevkovacs]

FYI: the copier update made a mess in the asv* yaml files. They contain the following code:
"<<<<<<< before updating"
PYTHON_VERSION: "3.12"
WORKING_DIR: ${{ github.workspace }}/benchmarks
=======
PYTHON_VERSION: "3.10"
ASV_VERSION: "0.6.4"
WORKING_DIR: ${{github.workspace}}/benchmarks
">>>>>>> after updating"

I think copier did not clean up properly. I went ahead and removed the extraneous lines and got the benchmarks to run successfully. So I think I am all set, thanks for your help.

[evevkovacs]

UPDATE: asv_nightly is failing. See here
The error message is not helpful to me.

[camposandro]

That action seems to have been triggered two days ago, with the outdated workflow configuration. Its latest run succeeded! I'll keep an eye out for the next run tomorrow morning to make sure that no other issues arise. Thank you!