Tracking Issue for `-Z doctest-in-workspace`

[ehuss]

Summary

Original issue: #8097 #8993
Implementation: #9105
Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#doctest-in-workspace

The -Z doctest-in-workspace flag changes the behavior of the current working directory used when running doctests. Historically, Cargo has run rustdoc --test relative to the root of the package, with paths relative from that root. However, this is inconsistent with how rustc and rustdoc are normally run in a workspace, where they are run relative to the workspace root. This inconsistency causes problems in various ways, such as when passing RUSTDOCFLAGS with relative paths, or dealing with diagnostic output.

The -Z doctest-in-workspace flag causes cargo to switch to running rustdoc from the root of the workspace. It also passes the --test-run-directory to rustdoc so that when running the tests, they are run from the root of the package. This preserves backwards compatibility and is consistent with how normal unittests are run.

Unresolved issues

None at this time.

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

cc @Swatinem

[weihanglo]

I wonder what's the current status of this feature. It seems that there is nothing left but stabilizing rustdoc --test-run-directory flag. Should we propose a stabilization on rustdoc's side?

[Swatinem]

I would love to!

… and even go a step further and maybe make this the default in a future edition?

[weihanglo]

Out of curious. What's the consequence of making it the default in terms of compatibility from users' perspective. I can see one that all relative paths of test output format switch to workspace-relative.

[Swatinem]

That is the most obvious user visible change indeed. Otherwise it is mostly implementation details of how the doctests are being built, which I believe is mostly a positive change, as the original motivation for this flag is that doctests are built the same way as other tests.

But as there is some kind of change, making that opt-in via an edition is IMO a good way forward.

[Swatinem]

I opened up a PR to stabilize rustdoc --test-run-directory and wrote a stabilization report.

Doing so I realized that the rustdoc flag itself is just an implementation detail of the doctest-in-workspace cargo flag.

The cargo flag itself is used to opt into fixing an inconsistency in how different categories of tests are compiled and executed.

@ehuss would you agree to push for stabilization of the cargo flag? And to eventually change the default from opt-in to opt-out? If there is backwards compatibility concerns, that can be done as an edition change.

I believe the only user visible change here is the command line output when running doctests, otherwise this should only be implementation details.

Also, can you think of any way to stabilize this cargo feature without necessarily having to stabilize the underlying rustdoc feature?

[ehuss]

I would lean towards just changing it without an opt-in/out. I don't expect there to be widespread breakage, and it is clearly not working as it should, and thus I would categorize this as a bug fix.

Unfortunately using new rustdoc flags can be difficult, so I typically wait until they hit stable to switch Cargo over. That means if --test-run-directory can be stabilized in 1.67, then Cargo can start using it in 1.69. That means landing in Cargo's master branch some time late January and reaching stable in late April.

Also, can you think of any way to stabilize this cargo feature without necessarily having to stabilize the underlying rustdoc feature?

I think --test-run-directory is essentially required. An alternate solution could be --persist-doctests and have cargo handle running the tests, but I would prefer to not do that.