Extending RerunFormatter

[hovi]

🤔 What's the problem you've observed?

I am making my own version of RerunFormatter. I want it to save only failed scenarios with certain tag.

When trying to copy the class:

• NiceAppendable is not public (understandable, don't have to care myself as I use kotlin and don't have to worry about checked exceptions, replacing with regular PrintWriter works just fine for me )
• UTF8OutputStreamWriter is not public (no big problem, I can just use OutputStreamWriter(out, Charsets.UTF_8) )
• PrettyFormatter.relativize(URI) is not public (I have to duplicate this method in my code, the method doesn't call anything non-public so it's simple)

✨ Do you have a proposal for making it better?

I suggest 2 possibilities:

• Extract filter condition for relevant testcases to be rerun into another protected method + remove final keyword from the class. That way I can simply extend this class and modify the functionality.
• Make PrettyFormatter.relativize public so that I can make my own implementation less duplication.

📚 Any additional context?

Our tests are very unreliable because of the test environment, which is quite non-deterministic at times (videogame). We are for example expecting rock to just fall down and stay on the ground in the hole, but in very rare cases, the rock gets "lucky" and jumps away causing the test to fail.
Since the failure is quite rare, we tagged those tests and want to consider them failed only if they fail X times in a row. So first time we run all tests, then only failed tagged tests, etc.

Seems like "filtered rerun" is perfect fit. If you think rerun is not the good way to do this, I welcome suggestions.

[mpkorstanje]

Instead of using the rerun formatter you may want to consider using a programmatic API to drive your entire test execution including reruns. A programmatic API allows for much more control about what to rerun and when to do so.

Unfortunately Cucumber doesn't have a great API. However, JUnit 5 does, and Cucumber integrates with JUnit 5. It also maps most of Cucumbers features decently well including tags.

You can read about it specifically in this section of the readme, though if you choose to use JUnit 5 you should read it in it's entirety.

https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-junit-platform-engine#rerunning-failed-scenarios

[mpkorstanje]

Ah. I didn't recognize the generic Github Avatar so I forgot about some context.

Unfortunately #2637 does not work with JUnit 5. Where the CLI executes the pickles as a single list, JUnit 5 hierarchically works down a tree. All nodes with the same parent are ordered implicitly in the order they are discovered. Roughly speaking this means that you can only order feature files and within those only scenarios.

[hovi]

Thank you for all the time you took for the answer :-) I am playing with different options and exploring.
I found rerun option nice because it is one more way to "log" and "reuse" when trying to be expressive for testers so the file can be useful mostly at times when something goes wrong and then is "manually" checked.
For automation, doing this in the code itself seems better, currently based on io.cucumber.core.cli.Main, that way I have full control over plugins for each run.

[mpkorstanje]

For automation, doing this in the code itself seems better, currently based on io.cucumber.core.cli.Main, that way I have full control over plugins for each run.

The JUnit API should give you equivalent control over the plugins. Unfortunately you'll have to configure them all via a single property. So something like cucumber.plugin=pretty:target/pretty.txt,html:target/cucumber.html,ect,ect,ect.