Optional parameter to increase the number of simultaneous jobs

[coder543]

One feature I've wondered about for awhile would be to have parallel update tasks running. I have a very powerful desktop, with a 8-core/16-thread R7 1700X processor. When there are half a dozen packages to be updated, a large portion of the update process is spent in a single thread during the final stages of each package. Since only one update happens at a time, these final stage processes are sequential, making the update process very slow compared to what it could be.

If I could choose to run multiple package updates in parallel, then their final, sequential stage would run in parallel, greatly reducing the time to do the updates.

My preferred approach would be to naively run the specified number of package updates in parallel, allowing the operating system to sort out who gets access to the processor, but an approach that you might find more reasonable would be to artificially restrict each package update to a fraction of the processor. With cargo install-update -a -j4, we could have 4 update jobs running in parallel. On a 16-thread machine, each of these 4 jobs could be run with cargo install -j4 ... to equally divide the processor threads among the update tasks.

If this were to be implemented, it would be nice to have an option or sentinel value to just start all jobs simultaneously, so I don't have to count up how many updates need to be done and supply that number to the parameter.

[nabijaczleweli]

The problem with that is, as always, synchronising console output. But good idea, yes.

[shepmaster]

As I understand it, Cargo spins up a job server (I hope Makefile-compatible) that it uses to coordinate the parallel invocations of the compiler and the threads that each compiler starts. If it didn't do this, then you'd actually have NxM total threads, overwhelming the system.

It's possible that cargo-update could spin up a parent jobserver and then spin up parallel invocations for all the projects. Since they all share the same jobserver, the number of total threads would be automatically shared between all the compilers.

a single thread during the final stages of each package

This final step is often a memory-intensive one, as the linker is crunching lots of data. If there's many of these in parallel, I could see getting close to hitting swap.

synchronising console output

Always a very useful thing. I'd vote for something akin to what cargo test does — each parallel thing is captured and only output from a single thread (perhaps only when there's a failure).

[hcpl]

I'd vote for something akin to what cargo test does — each parallel thing is captured and only output from a single thread (perhaps only when there's a failure).

Unlike running tests though it can be useful to see which crates are building in real time, not after the fact.

I propose per-line buffering of outputs from each cargo. When output to the user, all lines get prefixes based on the name of the crate being built. For example,

   ...
   [cargo-bloat] Compiling version_check v0.1.5
   [twiggy] Compiling serde v1.0.80
   [twiggy] Compiling lazy_static v1.2.0
   [cargo-bloat] Compiling cc v1.0.25
   ...

[shepmaster]

Today's Cargo does this somehow, as you can see in the progress bar it shows when compiling many crates in parallel.

[hcpl]

Good thing you mentioned that, cargo-update could even have a progress bar per crate that update in parallel.

[nabijaczleweli]

I'm willing to provide assistance to someone who wants to implement this, but it seems like a lot of work in what is already an abysmal codebase.