Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

With relative_files = true, I'm still getting absolute paths in .coverage due to $PWD-sensitivity #1674

Open
jab opened this issue Aug 25, 2023 · 10 comments
Labels
enhancement New feature or request wontfix This will not be worked on

Comments

@jab
Copy link

jab commented Aug 25, 2023

(Same symptom as #1147 but different cause.)

I need to produce relative paths in the coverage data that coverage.py produces for smoother interoperability with various consumers of this data (e.g., the VSCode Coverage Gutters extension, Code Coverage for Bitbucket Server, etc.).

I have the constraint that coverage.py is invoked inside a temporary working directory (specifically, a bazel test sandbox) that does not contain the package to be measured. Rather, the package to be measured is installed in some virtualenv's site-packages directory.

I tried using coverage.py's path remapping with relative_files = true to produce relative paths in coverage.py's output, but because coverage.py's path remapping is sensitive to the current working directory, and the package to be measured is not inside the current working directory, this is not working.

It seems this could be fixed by allowing the base directory for source resolution to be explicitly configured (defaulting to $PWD if not provided), and/or adding support for an additional, final path remapping step that applies configured substitutions without first checking for file existence.

Reproduction steps:
~cd $(mktemp -d)

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweRvim pyproject.toml

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweRcat pyproject.toml
[tool.coverage.run]
source_pkgs = ["demo"]
relative_files = true

[tool.coverage.paths]
source = [
  "demo",
  "/**/demo",
]

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweRmkdir demo

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweRvim demo/__init__.py

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweRcat demo/__init__.py
def foo():
    return 42

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweRvim test_demo.py

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweRcat test_demo.py
from unittest import TestCase, main

from demo import foo


class TestFoo(TestCase):
    def test_foo(self):
        self.assertEqual(foo(), 42)


if __name__ == "__main__":
    main()


xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweRpython3 -m venv .venv

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR.venv/bin/pip install -q coverage[toml]

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweRset -gx SRC_DIR $(pwd)


xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweRcd $(mktemp -d)

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hM$SRC_DIR/.venv/bin/coverage run --rcfile=$SRC_DIR/pyproject.toml $SRC_DIR/test_demo.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hM$SRC_DIR/.venv/bin/coverage lcov --rcfile=$SRC_DIR/pyproject.toml --debug=pathmap  # no luck:
Aliases (relative=True):
 Rule: '/**/demo' -> 'demo/' using regex '[/\\\\](.*[/\\\\])?demo[/\\\\]'
Rule '/**/demo' changed '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/__init__.py' to 'demo/__init__.py' which doesn't exist, continuing
No rules match, path '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/__init__.py' is unchanged
Wrote LCOV report to coverage.lcov

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hMvim $SRC_DIR/pyproject.toml  # try to fix by making the first path in "paths" absolute, so coverage.py can resolve it independently of $PWD:

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hMcat $SRC_DIR/pyproject.toml
[tool.coverage.run]
source_pkgs = ["demo"]
relative_files = true

[tool.coverage.paths]
source = [
  # Make this first path absolute:
  "${SRC_DIR-.}/demo",
  "/**/demo",
]

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hM$SRC_DIR/.venv/bin/coverage lcov --rcfile=$SRC_DIR/pyproject.toml --debug=pathmap  # Coverage now resolves the path and applies the transformation, but the transformation now includes the absolute path, so we haven't gotten anywhere:
Aliases (relative=True):
 Rule: '/**/demo' -> '/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/' using regex '[/\\\\](.*[/\\\\])?demo[/\\\\]'
Matched path '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/__init__.py' to rule '/**/demo' -> '/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/', producing '/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/__init__.py'
Wrote LCOV report to coverage.lcov

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.gkJlk9hMcat coverage.lcov  #  paths are absolute:
TN:
SF:/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.QW8ElweR/demo/__init__.py
DA:1,1,8W6LTH60TaV5ZrrD7DkxQQ
DA:2,1,KjVIXJCtSHJ6Jnfwt67A8Q
LF:2
LH:2
end_of_record

Perhaps some new setting like relative_to could complement relative_paths = true to enable this use case?

@nedbat
Copy link
Owner

nedbat commented Aug 27, 2023

I haven't tried to reproduce this, but: the [paths] setting is only used by the coverage combine command. Can you try using that command to process the data files? Also "/**/demo" I think can just be "*/demo".

@jab
Copy link
Author

jab commented Sep 7, 2023

Thanks very much for taking a look at this, @nedbat.

I went through the reproduction steps above, but this time changed /**/demo to */demo and added --parallel-mode to the coverage run invocation (which I think is necessary for coverage combine). I'm still getting absolute paths with coverage combine, and similar "doesn't exist...No rules match" output.

click for repro using "coverage combine"
xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnOls -a
./  ../

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnO$SRC_DIR/.venv/bin/coverage run --parallel-mode --rcfile=$SRC_DIR/pyproject.toml $SRC_DIR/test_demo.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnOls -a
./                             ../                            .coverage.jabmbp.19368.160174

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnO$SRC_DIR/.venv/bin/coverage combine --rcfile=$SRC_DIR/pyproject.toml --debug=pathmap
Aliases (relative=True):
 Rule: '*/demo' -> 'demo/' using regex '(.*[/\\\\])?demo[/\\\\]'
Rule '*/demo' changed '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py' to 'demo/__init__.py' which doesn't exist, continuing
No rules match, path '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py' is unchanged
Combined data file .coverage.jabmbp.19368.160174

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnOls -a
./         ../        .coverage

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnO$SRC_DIR/.venv/bin/coverage lcov --rcfile=$SRC_DIR/pyproject.toml --debug=pathmap
Aliases (relative=True):
 Rule: '*/demo' -> 'demo/' using regex '(.*[/\\\\])?demo[/\\\\]'
Rule '*/demo' changed '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py' to 'demo/__init__.py' which doesn't exist, continuing
No rules match, path '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py' is unchanged
Wrote LCOV report to coverage.lcov

xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.UJlDitnOcat coverage.lcov
TN:
SF:/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py
DA:1,1,8W6LTH60TaV5ZrrD7DkxQQ
DA:2,1,KjVIXJCtSHJ6Jnfwt67A8Q
LF:2
LH:2
end_of_record

Please let me know if there's anything further I can provide.

@jab
Copy link
Author

jab commented Sep 7, 2023

Note, I see the same --debug=pathmap output from coverage combine as I do from lcov, which made me think that there's some shared $PWD-sensitive logic when resolving paths.

@nedbat
Copy link
Owner

nedbat commented Mar 17, 2024

Bug #1752 was recently fixed, but it doesn't improve this situation. Here the problem is this line:

Rule '*/demo' changed '/private/var/folders/_t/xpfjyxwx727_zrtwcrhyfyjm0000gn/T/tmp.DT7IE3yo/demo/__init__.py' to 'demo/__init__.py' which doesn't exist, continuing

The important part is which doesn't exist: the relative path has to be relative to the current directory when the reporting happens.

@nedbat
Copy link
Owner

nedbat commented Mar 17, 2024

I'm going to close this: relative filenames are file names, not module names. I'm not sure how coverage could determine the relative file name to a file in a completely different directory.

@nedbat nedbat closed this as completed Mar 17, 2024
@nedbat nedbat added wontfix This will not be worked on and removed needs triage labels Mar 17, 2024
@jab
Copy link
Author

jab commented Mar 17, 2024

Perhaps some new setting like relative_to could complement relative_paths = true to enable this use case?

Would this be worth considering?

Note, I’m not currently affected by this, but expect to be next time I have to use coverage.py with Bazel.

@jab
Copy link
Author

jab commented Mar 17, 2024

(and/or support applying arbitrary transformations to paths in a final post-processing step)

@nedbat
Copy link
Owner

nedbat commented Mar 17, 2024

Perhaps some new setting like relative_to could complement relative_paths = true to enable this use case?

Would this be worth considering?

I don't use Bazel, so I don't understand the changing of directories. Instead of a relative_to, could you use an environment variable to add a path to the [path] section that would do the right thing?

(BTW: what are you quoting from?)

@nedbat nedbat reopened this Mar 17, 2024
@jab
Copy link
Author

jab commented Mar 17, 2024

(I was quoting myself in the last line of this issue's description after you expand "reproduction steps". Will move that outside the <details> block now.)

Last time I had to do this I was using an older version of Bazel with Python rules that didn't support this out-of-the-box (so we had to roll our own and hit this issue). Now I see https://bazel.build/configure/coverage#python links to https://github.com/bazelbuild/rules_python/blob/main/docs/sphinx/coverage.md which says there is now builtin coverage support via coverage.py, so I think the next step here is to see if this is still an issue with that.

If you want to close this in the meantime, please feel free, though if the idea of supporting better path remapping/truncating would be useful for addressing other issues, feel free to use this to track that if it makes sense.

@jab
Copy link
Author

jab commented Mar 26, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

2 participants