Support multiple test contacts in the ReportPortal plugin

[seberm]

Add the possibility of exporting multiple test contacts into the report portal instance.

Closes: #3407

TODO:

• Fix the local reportportal "contact" attribute test

Pull Request Checklist

• implement the feature
• extend the test coverage
• include a release note

[seberm]

Hello,
under which "key" do we want to save these contacts in the ReportPortal?

The contacts are always saved under the contact field of a specific test item with this patch. The attributes attached to the test item look the following:

[
  ...
  { "contact": "[email protected]" },
  { "contact": "[email protected]" },
  ...
]

Another option is to enumerate each contact key:

[
  ...
  { "contact_0": "[email protected]" },
  { "contact_1": "[email protected]" },
  ...
]

Any preferences? Thanks!

[seberm]

We decided we will go with the first option:

[
  ...
  { "contact": "[email protected]" },
  { "contact": "[email protected]" },
  ...
]

This PR is ready for review.

[psss]

Looks good, thanks!

[psss]

A short release note would be nice.

[seberm]

A short release note would be nice.

@psss Added, thanks for the review!

[psss]

Hm, /tests/finish/prune started to fail for no obvious reason with:

:: [ 09:13:23 ] :: [   FAIL   ] :: File /tmp/tmp.5qqARmNv2o/plan/execute/data/guest/default-0/default-2/write/test-data-3/data/out-test.txt should exist 

This does not seem to be related to this change. When executed locally, everything is just fine 😵‍💫

[lukaszachy]

@psss fails for me locally as well. I make rpm current content, installed (just tmt main package) into fedora:41 container and executed the tests/finish/prune/test.sh

It fails because that asserted file exists as test-data-4/data/out-test.txt, not test-data-3

Fastest way to check is (failing version)

$ cd tests/finish/prune/data
$ tmt run discover -vvv
        summary: 4 tests selected
            /default-0/test/test1
            /default-1/test/test2
            /default-2/write/plan-data
            /default-2/write/test-data

Passing version has the last two tests in switched order.

[happz]

@psss fails for me locally as well. I make rpm current content, installed (just tmt main package) into fedora:41 container and executed the tests/finish/prune/test.sh

It fails because that asserted file exists as test-data-4/data/out-test.txt, not test-data-3

Fastest way to check is (failing version)

$ cd tests/finish/prune/data
$ tmt run discover -vvv
        summary: 4 tests selected
            /default-0/test/test1
            /default-1/test/test2
            /default-2/write/plan-data
            /default-2/write/test-data

Passing version has the last two tests in switched order.

The problem with the changed order of tests has been mentioned this evening on tmt Slack channel, this seems like the same issue. I suspect fmf library is a culprit here, but I didn't follow fmf development closely enough for me to point a finger at a commit (and no tmt change rings a bell). But it should be easy to check, the test would pass with tmt & an older fmf version.

shell plugin takes the list of tests, constructs an fmf tree out of them, and populates internal Test instances from this tree - and if there was a change sorting of fmf nodes, items, keys, that might have affected this process

[lukaszachy]

@happz @psss Yes indeed, fmf-1.4.1 vs 1.5.0 return different order of tests.

EDIT: fmf behavior changed in teemtee/fmf#260, it's the 'sort' added in https://github.com/teemtee/fmf/blob/95c9745db00b0e2b545a8e93021544c10d8665ac/fmf/base.py#L714

[happz]

@happz @psss Yes indeed, fmf-1.4.1 vs 1.5.0 return different order of tests.

Which sounds like a breaking change in case of lists, as demonstrated by the shell disover plugin, where the order of tests was apparently given by their order in the list of tests.

[psss]

So the problem is that previously the data key sorting happened when the fmf.Tree was being built and now we sort the keys when the Tree.climb() method is called.

https://github.com/teemtee/fmf/blob/95c9745db00b0e2b545a8e93021544c10d8665ac/fmf/base.py#L710-L716

The change does not affect trees which are built directly from fmf files on disk because those were always sorted by node name. However, when the Tree.child() method is used for creating child nodes manually, those nodes were kept in the original order as they were created. This is what the shell discover plugin uses:

tmt/tmt/steps/discover/shell.py

Lines 389 to 394 in 01d38b8

	test_fmf_keys: dict[str, Any] = {
	key: value
	for key, value in data.to_spec().items()
	if key != 'name' and (key == 'duration' or value != data.default(key))
	}
	tests.child(data.name, test_fmf_keys)

So these nodes are now sorted as well, which for this specific use case probably does not make sense: User provided the tests in the form of a list and they most probably expect the order should be preserved.

I see two possible ways to fix this:

• do not sort keys when climbing the Tree
• make the sorting optional / configurable

If we go the first way, it would break tests which rely on the sorting. And I guess this approach can be quite widely used. So, let's make it configurable? Or any other/better ideas?

[lukaszachy]

make the sorting optional / configurable

I'd say it should be configurable. Caller of climb method should know if they want to keep the order in which they created the data (eg. as this discover.shell) or sorted (majority of other use cases).