#1706 - WIP - add feature retries

[nddipiazza]

Summary

Add a retry feature.

Details

Should be able to retry using a --retry parameter, or a Retry annotation.

Motivation and Context

#1706

How Has This Been Tested?

Pending test coverage.

Screenshots (if appropriate):

Types of changes

• New feature (non-breaking change which adds functionality).

Checklist:

• I've added tests for my code. TODO
• My change requires a change to the documentation.
• I have updated the documentation accordingly. TODO

[mpkorstanje]

I appreciate that you're creating a PR for a new feature. However we request that you open an issues to discuss significant contributions prior to making them.

This would have allowed me to point out that the reason IRetryListener doesn't work in TestNG due to testng-team/testng#891. And I would prefer that you work on fixing this issue rather then build it into the TestNG and JUNit integrations.

Additionally Cucumber-JVM currently doesn't support reporting on retried tests at all. So merely adding functionality to rerun tests until passed will not be sufficient to accept a PR. There is functionality coming up to make it possible to report on retried scenarios but that won't be available until the new Message protocol has been implemented (#1841).

Finally it would also have allowed me to point out that if you are using TestNG and aren't interested in fixing testng-team/testng#891 nor willing to wait for #1841 you can also extend AbstractTestNGCucumberTests and invoke runScenario until the tests passes.

[nddipiazza]

Cool your final comment will be sufficient for my purposes.
I will work on testng-team/testng#891 as time allows.
I'll also keep an eye on #1841

[nddipiazza]

Finally it would also have allowed me to point out that if you are using TestNG and aren't interested in fixing testng-team/testng#891 nor willing to wait for #1841 you can also extend AbstractTestNGCucumberTests and invoke runScenario until the tests passes.

@mpkorstanje question for you about this comment... let's say I do what you suggested here and then it passes on the 2nd+ try. Will the cucumber-report have a single successful attempt? or will the cucumber-report get messed up?

[mpkorstanje]

I honestly don't know. It will very much depend on the way a particular reporting plugin was written. However given that currently there is no way to differentiate between the events from the original execution and the events from any subsequent rerun, I suspect it won't go well.

[nddipiazza]

@mpkorstanje I'm currently using https://mvnrepository.com/artifact/net.masterthought/cucumber-reporting/4.11.2

What I can try to do if necessary is run test, some errors occur. we then parse the testng test output and initiate retry on each failing test where appropriate. and if the test passes after the 2nd try, inject the cucumber output from the initial test with the passing test to replace the previously failing test and regenerate the report html. Meh?

[mpkorstanje]

Ideally your reporter would understand that a test was rerun and report it as such. Currently only the native TestNG and JUnit reporters can do that. So you'd have to rely on those reports rather then anything generated by Cucumber or based on Cucumbers output.

So if you ask me what you can try: I'd spend my time making tests more stable.

[nddipiazza]

I'd spend my time making tests more stable.

this is the classic response to the flaky test retry situation. and in my case, there is a mainframe system that causes intermittent test failures completely outside of my control.

[mpkorstanje]

Been there done that.

#1120

In hindsight it was the biggest waste of my time. 😁

[nddipiazza]

@mpkorstanje this works great. I noticed that it still has a json record of the failures prior to the passing retry however. Is there some way to "erase" the previous failed attempts prior to letting the scenario pass? I am looking into the source code of the cucumber-reporting github project as well to see if I can do this during the generation of the pretty HTML report.

Because right now the test report looks like there are {num times each test failed before succeeding on a retry} failing tests. But actually the test is overall a pass.

This isn't that big of a deal, except for the fact that now the real failures if there are any are going to be lumped in with ones that were successfully retried later.

[nddipiazza]

@mpkorstanje never mind my previous question. i went ahead and added this ability into the cucumber-reporting in a PR.
see
damianszczepanik/cucumber-reporting#914

[mpkorstanje]

Ooh. Well done.