rustup redownloads already installed toolchain

[matthiaskrgr]

I'm currently on rustc nightly-x86_64-unknown-linux-gnu - Up to date : 1.50.0-nightly (bb1fbbf84 2020-12-22).

When I use a rust-toolchain file and specify nightly as a compiler, this works fine.
But when I pin the version to the current nightly in particular

[toolchain]
channel = "nightly-2020-12-22"

rustup will redownload and reinstall that nightly although it is already installed.

rustup --version

rustup 1.23.1 (3df2264a9 2020-11-30)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.50.0-nightly (bb1fbbf84 2020-12-22)`

rustup show

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/matthias/.rustup

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu
beta-x86_64-unknown-linux-gnu
nightly-2020-10-25-x86_64-unknown-linux-gnu
nightly-2020-11-29-x86_64-unknown-linux-gnu
nightly-2020-12-09-x86_64-unknown-linux-gnu
nightly-2020-12-11-x86_64-unknown-linux-gnu
nightly-2020-12-14-x86_64-unknown-linux-gnu
nightly-2020-12-20-x86_64-unknown-linux-gnu
nightly-2020-12-22-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu (default)
1.20-x86_64-unknown-linux-gnu
1.40-x86_64-unknown-linux-gnu
1.41-x86_64-unknown-linux-gnu
1.42-x86_64-unknown-linux-gnu
1.43-x86_64-unknown-linux-gnu
1.44-x86_64-unknown-linux-gnu
1.45-x86_64-unknown-linux-gnu
1.46-x86_64-unknown-linux-gnu
1.47-x86_64-unknown-linux-gnu
master

active toolchain
----------------

nightly-x86_64-unknown-linux-gnu (default)
rustc 1.50.0-nightly (bb1fbbf84 2020-12-22)

[kinnison]

This is expected behaviour, rustup does not deduplicate toolchain installations because you could potentially encounter confusing behaviour if it did.

Consider if it worked the way you suggest it should:

day 1, nightly-2020-12-14 and nightly are the same thing, you run a build in your project whose rust-toolchain states nightly-2020-12-14 and it builds and runs.
day 2 you rustup update nightly because you wanted to for whatever reason
day 3 you go offline (perhaps on a plane) and you return to the project with the pinned toolchain and run cargo build and lo and behold you now lack the toolchain you previously had used.

This is the same reason that despite stable currently being 1.48.0, stable 1.48.0 and 1.48 are all considered to be different toolchains as far as rustup is concerned.

If you can suggest a mechanism to achieve what you're hoping for (reduced downloads) without introducing confusions when less-specific toolchains happened to match and now happen not to when dealing with a project with a more specific toolchain specifier, then I'd be interested.

In the meantime, this is rustup working as expected and thus not a bug.

[matthiaskrgr]

But could rustup simply copy the files from one folder to another instead of downloading them from a server, in case of stable, 1.48.0 and 1.48 all being equal?

[kinnison]

In theory doing that for stable/numbered toolchains would be possible, however extending it to beta/nightly would be near impossible to do reliably since the channel names and the compiler versions don't line up and are not even guaranteed to be offset in the same way over time. As such, rustup would be unable to know what the current nightly is in terms of nightly-YYYY-MM-DD thus it'd be unable to predict what/when to copy.

I'm loathe to introduce something for stable/numbered channels which cannot extend to beta/nightly in this way. I fear it'd just introduce even more confusion. At least for now it's consistent if not necessarily as helpful as people might hope.

There's a discussion of a similar point here - #2611 though that's more about 1.XX vs. 1.XX.Y and CLI overrides.

[workingjubilee]

Note that this is an ongoing pain point, because rust-toolchain.toml doesn't just get set for one's own projects, it also gets set in other people's projects, that someone might contribute to, and different ways of writing the same thing thus consume gratuitous disk space, leading to requests like:

• Support restricting core/std to a particular version (for msrv purposes) rust#108406

[smheidrich]

Maybe a stupid idea but: Instead of trying to associate version numbers or any of that, couldn't at least a subset of these situations be solved using the hashes rustup already stores and consults to check whether a channel is up to date?

Detailed explanation

As far as I understand it, how it currently works is that rustup downloads https://static.rust-lang.org/dist/channel-rust-{channel}.toml.sha256 (where {channel} is something like stable or 1.90), then stores that hash in ~/.rustup/update-hashes/{channel}-{arch} (where {arch} in my case is e.g. x86_64-unknown-linux-gnu). The hash is over the contents of the https://static.rust-lang.org/dist/channel-rust-{channel}.toml file ("manifest v2") which, in turn, contains hashes over all the individual component archives.

Notably, after doing rustup toolchain install stable && rustup toolchain install 1.90 just now (with 1.90 being the current stable version as of the time of writing), the hashes stored under ~/.rustup/update-hashes/ for stable and 1.90 are identical.

So couldn't it just briefly check after downloading this hash whether the toolchain for another channel in ~/.rustup/update-hashes/ has an identical one and, if so, copy already-installed components from that toolchain's folder?

A drawback over version-number-matching approaches is that this would only prevent redundant downloads for installations made in short succession, because I expect that the manifest file changes frequently. So I don't know if it's worth the effort / added complexity.

[rami3l]

Maybe a stupid idea but: Instead of trying to associate version numbers or any of that, couldn't at least a subset of these situations be solved using the hashes rustup already stores and consults to check whether a channel is up to date?

Detailed explanation

As far as I understand it, how it currently works is that rustup downloads https://static.rust-lang.org/dist/channel-rust-{channel}.toml.sha256 (where {channel} is something like stable or 1.90), then stores that hash in ~/.rustup/update-hashes/{channel}-{arch} (where {arch} in my case is e.g. x86_64-unknown-linux-gnu). The hash is over the contents of the https://static.rust-lang.org/dist/channel-rust-{channel}.toml file ("manifest v2") which, in turn, contains hashes over all the individual component archives.

Notably, after doing rustup toolchain install stable && rustup toolchain install 1.90 just now (with 1.90 being the current stable version as of the time of writing), the hashes stored under ~/.rustup/update-hashes/ for stable and 1.90 are identical.

So couldn't it just briefly check after downloading this hash whether the toolchain for another channel in ~/.rustup/update-hashes/ has an identical one and, if so, copy already-installed components from that toolchain's folder?

A drawback over version-number-matching approaches is that this would only prevent redundant downloads for installations made in short succession, because I expect that the manifest file changes frequently. So I don't know if it's worth the effort / added complexity.

Thanks for the suggestion! We do have plans for that, as already discussed with @Kobzol. I will put it on the roadmap.