OrderWith in parameterized tests

[phoenix384]

The ordering introduced with #1130 is a great feature. Also the fact that the annotation is inherited is really helpful in some cases.

But when a parameterized test inherits an OrderWith annotation, its parameters get reordered as well.
Another case where this behaviour could be a problem would be a parameterized test with more than one test method, where you add an OrderWith annotation in order to sort the test methods but not the parameters.

As long as the parameters name value starts with "{index}" and the amount of parameters is <10 there is no problem, because the order remains the same (at least with the provided alphanumeric ordering). If there are >10 parameters, the alphabetical order is 1, 10, 11, 2, 3, ...

I could think of the following possibilites to solve this:

• OrderWith should only sort test methods - not parameters. You have enough control over the parameter order in the parameters method and that order should not get changed.
• Add leading zeros to the {index} placeholder value or add an additional placeholder that outputs the index with leading zeros.
• Add an ordering to use with the OrderWith annotation using the org.junit.runner.manipulation.Sorter.NULL sorter to be able to override an inherited ordering (only solves case 1)

[kcooney]

Sorry for the delay. We aren't actively planning another major release of JUnit, so I don't often review bugs.

OrderWith should only sort test methods - not parameters

The ordering is based on the Description values. The code doing the ordering has no knowledge that the description represents a parameter, so I don't see how we could do this.

Add leading zeros to the {index} placeholder value

This could be done, but has a small chance of breaking current users (then again, I suspect having more than nine parameters may be uncommon). I might be willing to approve a pull request for this proposal, but, again, we don't currently plan on releasing another major version of JUnit, so the changes might never be released.

add an additional placeholder that outputs the index with leading zeros.

I'm not thrilled with this approach. It would complicate the code and the surface area of the API, and would only solve the problem for people that knew about the the placeholders and understood when they should be used.

Add an ordering to use with the OrderWith annotation ... to be able to override an inherited ordering

In my experience, base test classes are over-used. One of the reasons we introduced Rules was to provide code reuse by composition instead of inheritance. Having a special way to "undo" inherited behavior seems like a step backwards.