Is this format meant to report times for non test parts?

[sk-]

When debugging test execution times it's important to monitor different parts of the test system.

For example how long it took:

• to compile code (golang)
• to import the code (ruby, python)
• to collect the tests
• to setup/teardown the test (ideally with the name of each setup teardown when available)

I imagine one could do it with the current format by emitting more events, however, there will be no way to tell whether a measurement is from an actual test or from something else.

[marcphilipp]

That's a good question!

• to compile code (golang)

• to import the code (ruby, python)

Would it suffice to include them as language-specific metadata of an execution meaning one duration for compilation or importing?

• to collect the tests

In JUnit such a phase also exists and is called "test discovery". I had not intended it to be included but we can certainly discuss ways of doing so. One way would be to include an aggregate number. Alternatively, it could be a separate node for each "collector". In JUnit that would be each test engine.

• to setup/teardown the test (ideally with the name of each setup teardown when available)

I imagine one could do it with the current format by emitting more events, however, there will be no way to tell whether a measurement is from an actual test or from something else.

This could be done by reporting each test as a "container" rather than a "leaf" in the hierarchy, e.g.

• container
  • test
    • setup1
    • setup2
    • ...
    • execution (the actual test)
    • teardown1
    • teardown2
    • ...

Alternatively, we could think about introducing a separate data-structure to collect such measurments for each node.

Thoughts?