Change pytest collection logic for quirky test suite setup

[pmeier]

The project I'm working on uses a quirky setup for the test suite. Consider a module test_foo.py with the content

import unittest


class TestBar(unittest.TestCase):
    def test_baz(self, parameter):
        pass


instantiate_tests(TestBar)

The actual test case is used as template and instantiated based on some parameter. Imagine it as a crude homebrew version of pytest.mark.parametrize. After the instantiate_tests() call, test_foo.py contains multiple test cases postfixed with the parameter (TestBarSpam, TestBarHam, ...). Furthermore, each test case now has multiple tests also postfixed with the parameter (test_baz_spam, test_baz_ham, ...).

Selecting TestBar::test_baz for all parameter combinations can be done by with pytest test_foo.py -k "TestBar and test_baz". My problem arises when I want to this within my IDE. The IDE offers a nifty feature that lets the user run a test by clicking an icon next to its definition. In case I do that for test_baz this invokes pytest test_foo.py::TestBar::test_baz. For a normal setup that would work fine. Unfortunately, neither TestBar nor test_baz exist after the module is loaded.

Of course I could alter the run config of the IDE, but doing so for every test (case) is tedious and error prone. Thus, I thought I write a small pytest plugin to achieve this. And this is where I'm stuck. Basically, I need the following:

• load the module test_foo.py
• check the requested test case / test (TestBar::test_baz) for matching instantiated versions (TestBar(Spam|Ham)::test_baz_(spam|ham))
• add the matching test cases / tests to the collection

I've looked through the documentation of collection hooks. My first choice was pytest_collection_modifyitems:

Called after collection has been performed. May filter or re-order the items in-place.

Unfortunately, this will never be invoked in case the collection turns up empty. IMO, the other candidates are

• pytest_collect_file,
• pytest_pycollect_makemodule, and
• pytest_pycollect_makeitem.

The documentation on them is fairly minimal, so I'm not sure which one I could use or how to use it. Could someone help me out here or at least give me a nudge in the right direction?

[flyingangel2013]

hi, pmeier
for your requirements, you can add the hook function "pytest_collection_modifyitems" at conftest.py, the example like below:

def pytest_collection_modifyitems(items):
print('pytest collector for all test cases：\n', items)
for item in items:
print('---' * 10)
print('case name：', item.name)
print('case node：', item.nodeid)

In there, you can modify the case list or order which is executed by pytest, also if you want to adjust them dynamically, maybe you need a configuration file I think, actually, I am seeking a good solution for this issue in my project, but it's more complex, except adjusting the case list, I need to generate the case group continually until my stability test is timeout. Hope this can help you :)

[pmeier]

Are you sure about pytest_collection_modifyitems() ? In my tests this was not called when the collection did not find anything. Unfortunately, this will always be the case.

[flyingangel2013]

Sure, but you need to put this hook function in conftest.py and you can add the logic in it to handle the default cases list

[pmeier]

@flyingangel2013 I did a little more digging and now I'm sure that pytest_collection_modifyitems() is not the way to go for my problem. The main function will bail out in case something was not found

pytest/src/_pytest/main.py

Lines 623 to 628 in b9c9876

	if self._notfound:
	errors = []
	for arg, cols in self._notfound:
	line = f"(no name {arg!r} in any of {cols!r})"
	errors.append(f"not found: {arg}\n{line}")
	raise UsageError(*errors)

before the pytest_collection_modifyitems() is called:

pytest/src/_pytest/main.py

Lines 637 to 639 in b9c9876

	hook.pytest_collection_modifyitems(
	session=self, config=self.config, items=items
	)

Thus, my initial comment holds:

Unfortunately, [pytest_collection_modifyitems()] will never be invoked in case the collection turns up empty.

[nicoddemus]

Hi @pmeier,

After the instantiate_tests() call, test_foo.py contains multiple test cases postfixed with the parameter (TestBarSpam, TestBarHam, ...).

Where are the test cases located? Do you inject them in the module's globals?

Just to make sure, by "test cases" you mean test classes?

Unfortunately, neither TestBar nor test_baz exist after the module is loaded.

Not sure I follow, I thought instantiate_tests would create the classes and inject them in the module. To exemplify, I thought this would hold true:

>>> import test_foo
>>> hasattr(test_foo, "TestBarSpam")
True

Thus, I thought I write a small pytest plugin to achieve this.

By "this" I assume you mean "run the test from the IDE", but I fail how you can do that without changing the IDE itself... perhaps I'm misunderstanding what you want to do?

[pmeier]

Hi @nicoddemus

After the instantiate_tests() call, test_foo.py contains multiple test cases postfixed with the parameter (TestBarSpam, TestBarHam, ...).

Where are the test cases located? Do you inject them in the module's globals?

Just to make sure, by "test cases" you mean test classes?

Correct and correct.

Not sure I follow, I thought instantiate_tests would create the classes and inject them in the module. To exemplify, I thought this would hold true:

>>> import test_foo
>>> hasattr(test_foo, "TestBarSpam")
True

Also correct, but kinda misleading. While TestBarSpam, TestBarHam, and so on are now part of the module, the original TestBar is no longer present. Thus running pytest test_foo.py::TestBar will fail. What I need is that call to run the instantiated tests TestBarSpam and so on.

Thus, I thought I write a small pytest plugin to achieve this.

By "this" I assume you mean "run the test from the IDE"

Not quite. After re-reading I should have been more clear. I want to write a plugin that basically collects TestBarSpam, TestBarHam and so on if TestBar is requested. The parameter names (Spam, Ham, ...) are known, so my idea was to modify the collection based on a simple name matching.

[nicoddemus]

Not quite. After re-reading I should have been more clear. I want to write a plugin that basically collects TestBarSpam, TestBarHam and so on if TestBar is requested. The parameter names (Spam, Ham, ...) are known, so my idea was to modify the collection based on a simple name matching.

Ahh I see, thanks for the clarifications.

Unfortunately I'm not sure that's possible, node matching from the command-line is engrained inside the command-line handling code:

pytest/src/_pytest/main.py

Lines 864 to 866 in 41a90cd

	def resolve_collection_argument(
	invocation_path: Path, arg: str, *, as_pypath: bool = False
	) -> Tuple[Path, List[str]]:

And that's not exposed to plugins.

[pmeier]

Hm, that is unfortunate. If I find a hacky workaround I'll post it here. Still, thanks a lot for your input.

[pmeier]

@nicoddemus I found a workaround albeit a hacky one:

I'm using the pytest_collect_makeitem hook to make custom UnitTestCase's and TestCaseFunction's. I simply reuse all of the functionality

class PyTorchTestCase(UnitTestCase):
    @classmethod
    def from_parent(cls, parent, *, name, obj=None):
        return super().from_parent(parent, name=TemplateName(name), obj=obj)

and only replace the name kwarg upon instantiation with TemplateName(name):

class TemplateName(str):
    def __eq__(self, other):
        return self.startswith(str(other))

    def __hash__(self) -> int:
        return super().__hash__()

With this they can keep their name and comparisons such as "TestBar" == TemplateName("TestBarSpam") work.

[nicoddemus]

Cool, thanks for sharing!