metafunc parametrization with session scope causes multiple fixture teardowns

[Idanjc]

Hi,

I'm not sure if this is something I'm doing wrong, a bug or known behaviour in recent versions of pytest, so trying to post here first.
Apologies in advanced if it's not the right place for it.

When I'm using metafunc.parametrize on session scoped fixtures it causes a teardown after each test function, instead of after the entire session.
This causes a heavy setup I have that should only run once per configuration (could have several configs in a single run) to run after every test function.

It doesn't happen in pytest 5.3.5, but does happen in pytest 5.4+ versions (reproduced on 5.4.3 & 6.1.2).

Reproduction code example:

# conftest.py

def pytest_addoption(parser):
    parser.addoption('--config', action='store', type=str,
                     help='load configuration for tests from some file (yaml/json etc)')

@pytest.fixture(scope='session')
def my_fixture(setup):
    # would yield a specific test object , e.g. db
    yield 'my_fixture'

@pytest.fixture(scope='session')
def setup(config):
    # do some setup to use in tests
    # actually yield an orchestration holding all the needed objects in tests after setup (server, db etc)
    yield 

@pytest.fixture(scope='session', autouse=True)
def config(request):
    conf = request.param
    # build final config
    yield conf

def pytest_generate_tests(metafunc):
    if metafunc.config.getoption('--config'):
        # load configurations from some file
        confs = [{'key1': 'val1'}, {'key2': 'val2'}]
        confs_ids = ['conf1', 'conf2']
        metafunc.parametrize('config', confs, ids=confs_ids, indirect=True)

# test_param.py

def test_foo(my_fixture):
    print('\n test_foo \n')


def test_bar(my_fixture):
    print('\n test_bar \n')

Running with: pytest -s --setup-show --config 'path/to/file'

Output from pytest version 5.3.5:

test_param.py 
SETUP    S config[{'key1': 'val1'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_foo[conf1] (fixtures used: config, my_fixture, request, setup)
 test_foo 

.
        test_param.py::test_bar[conf1] (fixtures used: config, my_fixture, request, setup)
 test_bar 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key1': 'val1'}]
SETUP    S config[{'key2': 'val2'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_foo[conf2] (fixtures used: config, my_fixture, request, setup)
 test_foo 

.
        test_param.py::test_bar[conf2] (fixtures used: config, my_fixture, request, setup)
 test_bar 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key2': 'val2'}]

Output from pytest versions 5.4.3 & 6.1.2:

test_param.py 
SETUP    S config[{'key1': 'val1'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_foo[conf1] (fixtures used: config, my_fixture, request, setup)
 test_foo 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key1': 'val1'}]
SETUP    S config[{'key1': 'val1'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_bar[conf1] (fixtures used: config, my_fixture, request, setup)
 test_bar 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key1': 'val1'}]
SETUP    S config[{'key2': 'val2'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_foo[conf2] (fixtures used: config, my_fixture, request, setup)
 test_foo 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key2': 'val2'}]
SETUP    S config[{'key2': 'val2'}]
SETUP    S setup (fixtures used: config)
SETUP    S my_fixture (fixtures used: setup)
        test_param.py::test_bar[conf2] (fixtures used: config, my_fixture, request, setup)
 test_bar 

.
TEARDOWN S my_fixture
TEARDOWN S setup
TEARDOWN S config[{'key2': 'val2'}]

Thanks, appreciate the help,
Idanjc

[Idanjc]

Hi,

Following up, I now noticed this seems to happen only when passing mutable objects to metafunc paramertize.

If we take my example above and change in pytest_generate_tests confs to:
confs = [('key1', 'val1'), ('key2', 'val2')]

Then it runs properly (same output as 5.3.5 version), but passing mutable object would cause the issue I mentioned.
Originally I passed a dict (real configuration is also a dict), but changing the above working example from tuple to list would also cause the issue, due to list being mutable:
confs = [['key1', 'val1'], ['key2', 'val2']]

Do you think it's a bug in newer pytest versions (since it worked before), or is it the expected behaviour for mutable object in paramertization and I should change how we load the configuration?

Thanks,
Idanjc

[bluetech]

I agree that the pytest 5.3.5 seems like the better one. The mutable thing is very odd. This is probably a bug, I'll try to look into it when I get a chance. I'll open an issue referencing this so we can track it.

Edit: #8102

[Idanjc]

Good to hear (for me at least :) ) that it's probably a bug and not an issue with our configuration loading.
Thanks