Optimizing Speed in Shards for Cargo Mutants through Caching

[ASuciuX]

Introduction

The purpose of this discussion is to explore potential strategies for optimizing the speed of cargo build and cargo test processes in the context of Cargo Mutants. Currently, these commands are run multiple times across different shards, potentially 8-16 times, which can be inefficient. The goal is to reduce this to a single run and then utilize the built and tested artifacts across all shards. This discussion outlines two proposed implementations for this optimization and seeks to understand the factors contributing to varying build times.

Key Points for Discussion

1. Single cargo build and cargo test Execution:

• Current Scenario: Presently, cargo build and cargo test are executed multiple times (8-16 times) across different shards.
• Proposed Change: Run cargo build and cargo test only once and use the resulting artifacts across all shards.

2. Caching and Accessing Build/Test Artifacts:

• Challenge: Efficiently caching the build and test artifacts so they can be accessed by subsequent shards.
• Possible Solutions:
  1. Static Location Approach: Create a static location for the initial cargo build and cargo test. This location's path would be passed as an argument when running cargo mutants for the shards.
  2. Dynamic Integration with Temp Jobs: Incorporate the build/test artifacts dynamically into the temporary jobs created by the mutant processes.

3. Understanding Variability in Build Times:

• Observation: Significant variability in build times has been noted. For instance, on the same package, build times varied from 130 to 600 seconds. Incremental Mutants Testing(0) -> Run filtering pr mutants from action lines 2595-2602

Goals of the Discussion

I appreciate all your work and effort you've already done, please let me know whether additional information might be helpful in our quest to optimize our CI workflow. Specifically, I'm interested in understanding if and how we could implement caching for cargo build and test to enable the use of just one CI workflow instance instead of multiple shards. The goal is to minimize the redundant use of parallel computing power. Any insights or advice on how to achieve this more streamlined and efficient process would be highly valuable.

[sourcefrog]

Proposed Change: Run cargo build and cargo test only once and use the resulting artifacts across all shards.

The difficulty here is that the mutated code is by definition different for each mutant, just as if you were editing it yourself.

Changing the code requires a new build to reflect the changes. (It is a new incremental build, so some information is cached.)

Similarly, when we change the code, we expect different results, that's the whole point, so we have to run the tests again.

If you want to look at it as a cache, then:

• A cache of build products (as in Bazel) has a key that includes the source hash, and therefore will be invalidated when the source is mutated
• A cache of test results (again as you can see in Bazel) depends on the binary hash, and so indirectly on the source contents, and so also has to be recalculated when the source changes, of course

Another way to look at this is that cargo build and test results are cached at the granularity of crates. If you split off some code into a smaller crate then both builds and tests will be faster.

If you did not already, you should read https://mutants.rs/how-it-works.html.

I would say that https://github.com/llogiq/mutagen does something like what you're saying about building a binary only once:

In order to be fast, it is necessary that the compilation of the test suite is performed only once. To achieve this, all mutations are baked into the code once and are selected at runtime via an environment variable. This means the mutations are not allowed to produce code that fails to compile.

However, it still needs to run the tests for each mutant, and it seems to me that for most trees, including yours, the test time will be longer than incremental builds.