Tests with same coverage are not minimized

[alisevych]

Description

There are generated two similar tests throwing StackOverflowError generated by Fuzzing.
Their coverage of the user code is the same.
But they are not minimized and both present in resulting test class.

To Reproduce

1. Install UnitTestBot plugin built from main in IntelliJ IDEA
2. Open UTBotJava project
3. Set Fuzzing 100% mode
4. Generate tests for Recursion$factorial
5. Find ///region FUZZER: ERROR SUITE for method factorial(int)

Expected behavior

There should be one execution left after minimization with same coverage and same exception class.

Actual behavior

There are two tests on StackOverflowError with the same coverage.
They both cover lines 9, 12, 15 of Recursion.java

Screenshots, logs

    ///region FUZZER: ERROR SUITE for method factorial(int)

    /**
     * @utbot.classUnderTest {@link Recursion}
     * @utbot.methodUnderTest {@link Recursion#factorial(int)}
     */
    @Test
    @DisplayName("factorial: n = 262144 (mutated from zero) -> throw StackOverflowError")
    public void testFactorialThrowsSOE() {
        Recursion recursion = new Recursion();

        assertThrows(StackOverflowError.class, () -> recursion.factorial(262144));
    }

    /**
     * @utbot.classUnderTest {@link Recursion}
     * @utbot.methodUnderTest {@link Recursion#factorial(int)}
     */
    @Test
    @DisplayName("factorial: n = 65537 (mutated from positive) -> throw StackOverflowError")
    public void testFactorialThrowsSOE1() {
        Recursion recursion = new Recursion();

        assertThrows(StackOverflowError.class, () -> recursion.factorial(65537));
    }
    ///endregion

Environment

IntelliJ IDEA version - Ultimate 2023.2
Project - Gradle
JDK - 17

Additional context

Reproducing with Symbolic + Fuzzing mode.
Reproducing with Recursion class copy-pasted into spring-petclinic project.

Also there are many (now - 7) almost identical tests on timeout, with n = 1056, 36, 16416, 36, 2049, 133, 69
Two values are quite the same - 36:

    ///region FUZZER: TIMEOUTS for method fib(int)

    /**
     * @utbot.classUnderTest {@link Recursion}
     * @utbot.methodUnderTest {@link Recursion#fib(int)}
     */
    @Test
    @DisplayName("fib: n = 1056 (mutated from 32)")
    @Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testFib1() {
        Recursion recursion = new Recursion();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> recursion.fib(1056));
    }

[IlyaMuravjov]

@alisevych

Minimization decides to keep both executions, because one StackOverflowError is thrown from the line public int factorial(int n) {, while the other one is thrown from the line return n * factorial(n - 1);.

Should we not consider the line which StackOverflowError is thrown from?

[alisevych]

@alisevych

Minimization decides to keep both executions, because one StackOverflowError is thrown from the line public int factorial(int n) {, while the other one is thrown from the line return n * factorial(n - 1);.

Should we not consider the line which StackOverflowError is thrown from?

@Markoutte Please advise

[alisevych]

related issue:

• Hanging tests duplicates are not minimized #2130

[alisevych]

Summarizing: currently minimization is designed to check the traces. And if they are a bit different, both executions are kept.