Using python coverage from an avocado run

[pevogam]

Hi all, I am running some unit tests for our plugin and use avocado for all of them. However, I generally also submit coverage reports which worked easily with unit tests but no longer seem to be effective with avocado. What I run is

PYTHONPATH=:. coverage run --append --source=avocado_i2n $(which avocado) run --nrunner-max-parallel-tasks=4 selftests/isolation/test_*

but the resulting coverage is 0% throughout. Is there a standard way to invoke avocado together with python3-coverage? Would you suggest a better way then the one above?

[arif-ali]

I am also keen to understand this too, as per the documentation it suggests that we should get coverage via invoking coverage and then pass on avocado but I get no-data-collected

<snip>/lib/python3.12/site-packages/coverage/control.py:887: CoverageWarning: No data was collected. (no-data-collected)
  self._warn("No data was collected.", slug="no-data-collected")

TiA

[richtja]

Hi @arif-ali, I have investigated this, and I can confirm that we have some issues here and the coverage doesn't work as is written in our documentation. Therefore, I have created #5919 to track this issue and I will be working on a solution.

I have found out that if you will set env variable COVERAGE_RUN before testing, the coverage will work as expected, see this. I hope this will help you until I fix it and put everything into a good shape.

[arif-ali]

@richtja I had a go at this, but didn't have much luck.

I've tried in the tox environment, manually and even tried avocado directly with the env variable, but didn't have much luck. Below is a paste of the output of the current run, which shows the env, and the coverage runs.

https://dpaste.com/F9R86XMZH

[richtja]

Hi @arif-ali can you please also provide your avocado config and .coveragerc file. I will investigate it. Thank you

[arif-ali]

Hi @richtja below are the requested details

• avocado config: https://dpaste.com/CXRWYXUDM
• .coveragerc: https://dpaste.com/BN6UABPES

[richtja]

Hi @pevogam and @arif-ali the #5944 should solve the issues with the coverage.py on Avocado. Can you please try it out and let me know if it works for you? Thank you.

[pevogam]

Alright then, indeed it could be made to work right away this way give or take the No data was collected. which may confuse unknowing users. I am still a bit concern about the big difference in the running time I obtained though - a run with avocado (4 workers) without coverage will take ~8 minutes:

while a run with avocado and coverage will instead take ~30 minutes:

which is a bit surprising even considering that coverage could take longer to count all calls and code. I could of course compare this with coverage from standard python unittest but it is running in serial mode so a comparison is not exactly useful.

Perhaps we need additional setting for parallel runs?

[richtja]

@pevogam, thanks for running this test.

Alright then, indeed it could be made to work right away this way give or take the No data was collected. which may confuse unknowing users.

Yes, you are right, I will update our documentation to inform users about this.

I am still a bit concern about the big difference in the running time I obtained though - a run with avocado (4 workers) without coverage will take ~8 minutes: while a run with avocado and coverage will instead take ~30 minutes: which is a bit surprising even considering that coverage could take longer to count all calls and code. I could of course compare this with coverage from standard python unittest but it is running in serial mode so a comparison is not exactly useful.

Perhaps we need additional setting for parallel runs?

Can you please compare it with the coverage on standard python unittest, because I don't see anything like this on avocado selftests. The coverage test run for 9m 17s and the regular test run for 7m 39s.

[pevogam]

Unfortunately as I wrote above I don't think a comparison with a unittest variant is possible despite having an executed workflow to compare with since it runs serially while I use avocado with --max-parallel-tasks=4. From the comparison above it seems like the coverage job is ~4 times slower, could it be that the parallel run is not really possible and the tasks end up waiting for each other?

[pevogam]

I did read somewhere in the diff about previous configuration that needed special parallel handling with coverage. In any case perhaps I am wrong and looking at the variants separately it seems the time for each has more or less doubled. So it could also be some result of variation in the servers load, I simply see that form ~8 mins per run before now multiple runs are at an average of ~25-26 minutes.

[richtja]

The parallel handling is not needed from the point of avocado. Avocado will run separated coverage.py process for each test and the results will be stored into separated file, therefore you have to run coverage combine after the testing to merge the results into one report. On the other hand, if one of your tests is concurrent, then you need to set concurrency value in .covaregerc file. Unfortunately, IIUIC this shouldn't have any impact on the runtime, only on the coverage report.