orch/run: Add unit test xml scanner

[VallariAg]

This PR adds following changes:

1. add run_unit_test to Remote
2. create util/scanner.py: add Scanner, UnitTestScanner, ValgrindScanner
3. new exception: UnitTestError
4. add new lxml dependency in setup.cfg

reason: to throw better exception messages for failures in unit tests and valgrind errors.

[kshtsk]

Why do you think this feature should be embedded in orchestra/run?

[VallariAg]

Why do you think this feature should be embedded in orchestra/run?

@kshtsk
Since the unit tests are executed when the execution-command is passed by ceph qa to orchestra's run function, so this feature scans xml when that execution fails. The flow of code for unit test execution does not go outside orchestra/run so that's why I included this in there.
I'm working on more changes in which I've wrapping these function in a class.
I didn't define the class somewhere else and import it here, since orchestra/run is the only place the scanner is used so it makes sense to just define it in run.py itself.

[kshtsk]

The run method is pretty basic method to run a remote command, it should not be overloaded with extra functionality, no specific parsing code should be there. If you want some kind of parsing of specific xml, you might add another method making subcall of run. For example, if you need some shortcut for running test which can produce java specific test, probably it makes sense to have a dedicated function that gives a clue to a user about what is it, for example, remote.nosetest. As an example, take a look at remote.sh method to have an idea of how the run can be reused.

[zmc]

I do think it's worth considering @kshtsk's view that this functionality ought to be placed outside the orchestra package.

It could be as simple as creating a new module e.g. teuthology.util.unit_tests, and moving find_unittest_error and parse_xml there. Tasks could then use the feature by doing something like this, or calling a wrapper function which does this:

try:
    remote.run(
        args=args,
        label="s3 tests against rgw",
    )
except CommandFailedError:
    find_unittest_error(remote, path=unittest_xml)
    raise

[zmc]

Suggested change

	raise Exception("Somthing went wrong while searching for error in XML file: " + repr(exc))
	log.exception("Something went wrong while searching for errors in XML file")
	raise

[zmc]

Suggested change

	except Exception as exc:
	except Exception:

[VallariAg]

@kshtsk @zmc
Thank you for explaining that! :)
I'll add an extra method to remote for this feature and import scanner functions from utils.

[VallariAg]

Outputs:

1. unit test using run_unit_test and UnitTestScanner with vallariag/ceph@xml-scanner ceph branch:

   • (s3 test): https://pulpito.ceph.com/vallariag-2023-09-19_18:37:01-smoke-main-distro-default-smithi/
   • (workunit test): https://pulpito.ceph.com/vallariag-2023-09-21_11:10:03-rados:monthrash-wip-vallariag-testing-distro-default-smithi/

2. valgrind error scans using ValgrindScanner using vallariag/ceph@test-valgrind-xml ceph branch:
   https://pulpito.ceph.com/vallariag-2023-09-19_20:10:42-rados-main-distro-default-smithi/

[kamoltat]

I went through the logic and it looks good, just have very minor formatting requests.

[zmc]

Looking great!

I'm avoiding merging immediately because I won't have much time to observe today; I can merge tomorrow, or feel free to do so yourself!