Save the subresults for tmt-report-result

[seberm]

All the results generated by tmt-report-results become objects of tmt.result.SubResult and they get assigned under a parent tmt.Result. Also, set the TESTID and BEAKERLIB_COMMAND_REPORT_RESULT test environment variables, so every call of beakerlib's rlPhaseEnd macro becomes a subresult.

Related to:

• Add support for test phases #2826

TODOs:

• Try to find all the possible documentation places where this needs to be modified/mentioned.
• Add tests
• Fix existing tests / fix the code to pass tests

Pull Request Checklist

• implement the feature
• write the documentation
• extend the test coverage
• update the specification
• adjust plugin docstring
• modify the json schema
• mention the version
• include a release note

[seberm]

@happz Aren't we missing the data-path in the result and subresult schemas? Otherwise, I'm still getting the "result format violation" errors.

[happz]

Result schema is definitely missing data-path key, subresults shouldn't have one, IIUIC, data path is a path in which tmt stores test data and info and stuff, so there is no dedicated subpath for a subresult.

[seberm]

Are you sure about the data-path in the subresult? I do not have much insight, but it seems the subresults generated by tmt-report-result are using this field when the tmt-report-results.yaml gets created:

• tmt/tmt/steps/execute/scripts/tmt-report-result

  Line 20 in 7b16db6

  echo " data-path: ${TMT_TEST_DATA/$TMT_TEST_EXE}" >> "$REPORT_RESULT_OUTPUTFILE"

[happz]

Hmmm, that does not look right. TMT_TEST_EXE is ${TMT_TREE/tree/execute}, TMT_TEST_DATA is said to be "Path to the directory where test can store logs and other artifacts generated during its execution." and I can't imagine that ${TMT_TEST_DATA/$TMT_TREE/tree/execute} could be anything sane.

@psss WDYT? There is "test data" for the test, TMT_TEST_DATA points to it, but data paths for Beakerlib phases? tmt does not manage those, in custom results we do not even have any idea whether the test used any subresult data paths, so we don't touch them, this seems like a mistake to me.

[seberm]

Also, do you think we should somehow limit the data-path using a pattern in the schema?

For a parent result, the value is always a relative path (data/guest/default-0/tests/example-1/data). For subresults, the data-path seems to always start with pattern: "^/" - it's not relative, but absolute where root starts at 'execute' directory, e.g. /data/guest/default-0/tests/example-1/data.

Isn't it a bug? Also, the subresult's data-path seems to be always the same as the parent data-path (without / prefix).

[seberm]

I've tried to change the method docstrings, releases.rst and report-result.fmf story to reflect the subresults feature. Are there any other places where I should document this feature?

Let me know if you need any further adjustments / tests. Thanks.

[seberm]

/packit build