Do not lock the artifact-dir for check builds

[ranger-ross]

What does this PR try to resolve?

This PR modifies the locking logic to avoid locking artifact-dir for check builds.
Part of #4282

Note: This change is not behind -Zbuild-dir-new-layout or -Zfine-grain-locking unstable flags.

How to test and review this PR?

See the tests included in the PR.
You can run cargo check to verify.

r? @epage

[weihanglo]

This should be in check command test file maybe?
https://github.com/rust-lang/cargo/blob/master/tests/testsuite/check.rs

[ranger-ross]

ah good idea. I was thinking about putting it in https://github.com/rust-lang/cargo/blob/master/tests/testsuite/artifact_dep.rs but noticed that file only has test for the unstable --artifact-dir flag.

master/tests/testsuite/check.rs seems like a good place for this

[weihanglo]

Should we also add one test asserting that no files are generated to the artifact directory from cargo check?

(Maybe we already have one though)

[ranger-ross]

Added a test in da0dfd3

[epage]

Instead of conditionally grabbing the lock, can we put the entire artifact layout in an Option? That has the potential to help us find bugs but unsure how messy that would be.

[ranger-ross]

I looked into this and it got pretty hairy. Switching from T to Option<T> is not too bad but there quiet a few places during compilation where the paths are used but not written to.

For example, in build_runner::prepare we build up the root_output

cargo/src/cargo/core/compiler/build_runner/mod.rs

Lines 395 to 397 in 8c4a25c

	self.compilation
	.root_output
	.insert(kind, layout.artifact_dir().dest().to_path_buf());

and we use this in fill_env()

cargo/src/cargo/core/compiler/compilation.rs

Lines 308 to 311 in 8c4a25c

	search_path.extend(super::filter_dynamic_search_path(
	self.native_dirs.iter(),
	&self.root_output[&kind],
	));

[epage]

Doesn't that make the case even stronger? We are trying to read from a location we do not write to and do not have locked.

[ranger-ross]

I looked into this and I think this is a good call. I ended up missing that we still need to lock the artifact-dir for cargo check --timings. Making this Option caused a test to fail instead of silently write files without locking.

I split out this change into a refactor commit before changing the behavior.

[epage]

Instead of intent, can we check if artifacts are produced? Do we know that?

[ranger-ross]

I checked and I think it may be possible to use some of the logic from CompilationFiles output but we would likely need to duplicate it as we need to construct Layout before CompilationFiles.

See https://github.com/rust-lang/cargo/blob/master/src/cargo/core/compiler/build_runner/compilation_files.rs#L405

Let me know if there is a better way to get this data.

[epage]

Hmm, that can get messy. We could create an enum for the base path and calculate the full path later but unsure how well that will work out.

However, this does highlight a need for either this or conditionally grabbing the lock (#16230 (comment))

cargo/src/cargo/core/compiler/build_runner/compilation_files.rs

Lines 529 to 540 in 8c4a25c

	if bcx.build_config.sbom && bcx.gctx.cli_unstable().sbom {
	let sbom_files: Vec<_> = outputs
	.iter()
	.filter(|o| matches!(o.flavor, FileFlavor::Normal | FileFlavor::Linkable))
	.map(|output| OutputFile {
	path: Self::append_sbom_suffix(&output.path),
	hardlink: output.hardlink.as_ref().map(Self::append_sbom_suffix),
	export_path: output.export_path.as_ref().map(Self::append_sbom_suffix),
	flavor: FileFlavor::Sbom,
	})
	.collect();
	outputs.extend(sbom_files.into_iter());

We would have been writing out SBOMs without holding the lock. If we don't feel SBOMs are relevant for cargo check then making the layout optional (to catch problems like this) is sufficient. If not, then we need to somehow be aware of this for locking and then that starts to look like needing to resolve this item.

[epage]

Note: if we change SBOMs, lets do that in its own PR and make a note against rust-lang/rfcs#3553

CC @arlosi

[epage]

Yikes, that is a pretty big exception for us to make that can negate this.

[ranger-ross]

yeah, its unfortunate.

I think we still need to lock the artifact-dir since it could be read during compile time like

let tool_bytes = include_bytes!(env!("CARGO_BIN_EXE_foo"));

Though, I'd imagine this would be a pretty rare usecase?

[epage]

The case where this could come up is with artifact deps though I don't know if those are based on uplifted binaries or not

[epage]

Also, I've wondered about moving CARGO_BIN_EXE_* to also work at runtime for tests. Encouraging a migration to that would make it clearer what the intent is with these.

#t-cargo > cargo_bin_exe and tests @ 💬