Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support discovery of suites with failed initialization by UniqueId #109

Open
pshevche opened this issue May 19, 2023 · 4 comments · May be fixed by #110
Open

Support discovery of suites with failed initialization by UniqueId #109

pshevche opened this issue May 19, 2023 · 4 comments · May be fixed by #110

Comments

@pshevche
Copy link

Describe the bug
Currently, the engine will create a custom TestDescriptor if the test suite initialization fails (ref). This descriptor will have a UniqueId of form [engine:scalatest]/[failed:<SUITE_NAME>]. However, the engine does not support the suite discovery based on this ID. This limits the ways to interact with the engine, since the consumer can use the discovery result to further limit the tests to be executed and then pass a smaller set of UniqueIdDescriptors to execute method.

To Reproduce
This branch has a test that shows the empty discovery result if the UniqueId of this synthetic TestDescriptor is used as input.

Expected behavior
The engine detects that its synthetic descriptor is used and returns a valid discovery result.

@pshevche
Copy link
Author

I also prepared a PR that should fix the issue, as it seemed pretty straight-forward. But let me know if that's not how you see it :).

@pshevche
Copy link
Author

I was also wondering, why did you decide to report such failures during the execution phase instead of making it a hard failure during the discovery phase?

@giurim
Copy link
Contributor

giurim commented May 19, 2023

Hi @pshevche, thank you for the report and especially for the PR. As far as I remember I did that weird workaround, because if it failed, the exception was somehow lost up in the Junit5 system during the discovery. As the result, no tests were "discovered" and the test run was "passing". I should investigate if it is still the issue.

@pshevche
Copy link
Author

@giurim , I see, it makes sense. Another benefit of this approach is that you could still execute as many test suites as possible and get at least some feedback instead of failing early, which might be a good thing.

If you decide to stick with the current strategy, then we would need to handle these custom descriptors one way or another, as it is documented as an engine requirement here: https://junit.org/junit5/docs/current/user-guide/#test-engines-requirements-mandatory (see the second point).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants