Running coverage package over briefcase dev --test

[BartStolarek]

Hi Everyone,

I'm wondering if its possible to run the coverage package over the briefcase dev --test command.

So far the closest I've managed to come is the following setup:

.coveragerc

[run]
source = <root_project_folder>/<application_name>/src/<application_name>
omit = *tests*

[report]
exclude_lines =
    pragma: no cover
    if __name__ == .__main__.:

<root_project_folder>\run_tests_with_coverage.py

import os
import sys
import coverage

print("Initialising coverage")
cov = coverage.Coverage()
cov.start()

print("Running briefcase dev --test -r with coverage")
os.chdir("<application_name>")
os.system("briefcase dev --test -r")
os.chdir("..")

print("Stopping coverage")
cov.stop()
print("Saving coverage")
cov.save()
print("Generating coverage report")
cov.report()

Console Error

Initialising coverage
Running briefcase dev --test -r with coverage

[<app_name>] Installing requirements...
REMOVED INSTALLED REQUIREMENTS ALL WERE SUCCESS
Installing dev requirements... done
[<app_name>] Running test suite in dev environment... ===========================================================================
============================= test session starts =============================
platform win32 -- Python 3.10.6, pytest-7.3.1, pluggy-1.0.0 -- C:...\beeware-venv\Scripts\python.exe
cachedir: C:...\AppData\Local\Temp.pytest_cache
rootdir: C:...<root_project_folder><app_name>
plugins: anyio-3.6.2
collecting ... collected 23 items

tests/test_app.py::test_first PASSED [ 4%]
tests/lib/test_account.py::test_first PASSED [ 8%]
tests/lib/test_baseclass.py::test_first PASSED [ 13%]
tests/lib/test_controller.py::TestController::test_add_flow_daily_amounts_to_users PASSED [ 17%]
tests/lib/test_controller.py::TestController::test_add_income_daily_amount_to_user PASSED [ 21%]
tests/lib/test_controller.py::TestController::test_calculate_user_split PASSED [ 26%]
tests/lib/test_controller.py::TestController::test_check_password PASSED [ 30%]
tests/lib/test_controller.py::TestController::test_convert_all_flows_to_daily PASSED [ 34%]
tests/lib/test_controller.py::TestController::test_convert_amount_frequency_to_daily PASSED [ 39%]
tests/lib/test_controller.py::TestController::test_find_user_by_name PASSED [ 43%]
tests/lib/test_controller.py::TestController::test_get_account_by_id PASSED [ 47%]
tests/lib/test_controller.py::TestController::test_get_all_accounts PASSED [ 52%]
tests/lib/test_controller.py::TestController::test_get_all_flows PASSED [ 56%]
tests/lib/test_controller.py::TestController::test_get_all_original_flows PASSED [ 60%]
tests/lib/test_controller.py::TestController::test_get_all_scheduled_transfers PASSED [ 65%]
tests/lib/test_controller.py::TestController::test_get_all_users PASSED [ 69%]
tests/lib/test_controller.py::TestController::test_get_flow_by_id PASSED [ 73%]
tests/lib/test_controller.py::TestController::test_get_user_names_as_list PASSED [ 78%]
tests/lib/test_controller.py::TestController::test_refresh PASSED [ 82%]
tests/lib/test_databaseoperations.py::test_first PASSED [ 86%]
tests/lib/test_dataformating.py::test_first PASSED [ 91%]
tests/lib/test_flow.py::test_first PASSED [ 95%]
tests/lib/test_user.py::test_first PASSED [100%]

============================= 23 passed in 16.00s =============================

[<app_name>] Test suite passed!
Stopping coverage
Saving coverage
C:...\beeware-venv\lib\site-packages\coverage\inorout.py:507: CoverageWarning: Module <root_project_folder>/<app_name>/src/<app_name>was never imported. (module-not-imported)
self.warn(f"Module {pkg} was never imported.", slug="module-not-imported")
C:...\beeware-venv\lib\site-packages\coverage\control.py:858: CoverageWarning: No data was collected. (no-data-collected)
self._warn("No data was collected.", slug="no-data-collected")
Generating coverage report
Traceback (most recent call last):
File "c:..<root_project_folder>\run_tests_with_coverage.py", line 19, in
cov.report()
File "C:..<root_project_folder>\beeware-venv\lib\site-packages\coverage\control.py", line 1059, in report
return reporter.report(morfs, outfile=file)
File "C:..<root_project_folder>\beeware-venv\lib\site-packages\coverage\summary.py", line 181, in report
for fr, analysis in get_analysis_to_report(self.coverage, morfs):
File "C:...<root_project_folder>\beeware-venv\lib\site-packages\coverage\report.py", line 94, in get_analysis_to_report
raise NoDataError("No data to report.")
coverage.exceptions.NoDataError: No data to report.

I will admit that importing modules, packages, etc is my absolute weakness, and the reason why I moved my project from a Kivy and buildozer to Beeware 😆

[freakboy3742]

So - yes it's possible, but it's not trivial, especially if you potentially want to deploy to iOS or Android.

There's an example of a working configuration in Toga's testbed app; That testbed has a bunch of additional complication because it needs to do some weird async and threading, but .

Essentially you can't use pyproject.toml (or .coveragerc, or any other file) as the source of your coverage configuration, because that file won't be in your packaged app (or on your path when briefcase dev executes); you need to use the Coverage API to configure a coverage session and generate a report. The mapping from configuration file to API is relatively straightforward, so it shouldn't be too hard to map your coverage needs to that API.

[BartStolarek]

Thank you! For those who come here in the future, and want to know how to accomplish what @freakboy3742 kindly provided here are the steps that worked for me:

1. Copy this code in to a file (name it whatever you like), and put it in your
   root/<application_name>/tests/ directory.

2. Add the following at the top for imports:

src_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'src'))
sys.path.insert(0, src_path)
from <application_name>.app import main

3. I also removed the toga backend testing, but going to the line source_pkgs=[toga_backend] and changing it to source_pkgs=[] #removed toga_backend.

4. To add the report and HTML do the following you can add cov.save() after the line cov.stop(), and then find the line

if total <100.0:
        print("Test coverage is incomplete")

and add after it:

html_report_dir = "tests\htmlcov"
cov.html_report(directory=html_report_dir)
print(f"HTML coverage report is generated in the '{html_report_dir}' directory")

so it looks like this:

                if total < 100.0:
                    print("Test coverage is incomplete")
                    # Uncomment the next line to enforce test coverage
                    # TODO: app.returncode = 1
                html_report_dir = "tests\htmlcov"
                cov.html_report(directory=html_report_dir)
                print(f"HTML coverage report is generated in the '{html_report_dir}' directory")

This will put your coverage report files in to folder called 'htmlcov' in your tests folder. Just open 'index.html' in a web-browser and you can see your coverage.

[freakboy3742]

A couple of clarifications:

2. Add the following at the top for imports:

src_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'src'))
sys.path.insert(0, src_path)
from <application_name>.app import main

Are you sure this is required? The app's code should be on your path as a result of running through Briefcase. If it's not, the output of briefcase dev --test -vv will give you the exact invocation of Python that is being used (and the PYTHONPATH that was used during that invocation), and might help diagnose the problem.

4. To add the report and HTML do the following you can add cov.save() after the line cov.stop(), and then find the line
   ...

This may work for dev deployment, but will be problematic when when you run as an app - the tests location won't necessarily be writable.