Finding the remote of Cargo’s checkouts of git dependencies

[SimonSapin]

Using pernosco-submit with Servo resulted in 14 lines like:

No remotes found for Git repo /home/simon/.cargo/git/checkouts/rust-mozjs-8611526964119dd6/c2896b9, packaging files instead...

In that directory:

$ git remote -v
origin	file:///home/simon/.cargo/git/db/rust-mozjs-8611526964119dd6 (fetch)
origin	file:///home/simon/.cargo/git/db/rust-mozjs-8611526964119dd6 (push)

That directory in turn is a bare repository that doesn’t have any remote configured.

Unfortunately I don’t know if there’s a solution to this without modifying Cargo. Even with a special case for $CARGO_HOME/git/checkouts/*/*, the hex part of the directory name is probably a non-reversible hash.

Maybe Cargo would be open to accepting changes as long as they’re not hard to achieve with libgit2, such as making git/checkouts/*/* be work trees rather than full clones, and configuring the remote in git/db/*

[rocallahan]

If you add /home/simon/.cargo to your pernosco-submit source dir whitelist, at least those files will be uploaded and be available. It will increase the size of your uploaded trace of course.

[rocallahan]

OK actually the situation is better (and worse) than it appears.

We have code running on our server that knows how to obtain the sources of crates and make them available. It decodes the cargo registry crate paths. So in fact crates under the cargo registry and also the Rust stdlib should have sources in Pernosco. Unfortunately they don't show up. What's happening is that because you didn't whitelist the source dirs in pernosco-submit, pernosco-submit uploads placeholder files with the "not whitelisted" message, which then override the files that our Rust tooling provides :-(.

[rocallahan]

I think we should probably change pernosco-submit so that the placeholder files are moved to a separate sources._user_placeholders and associated ZIP file. That will sort them before any other rules, so our Rust tooling will override them.

[rocallahan]

However, that won't fit this particular issue, because we don't handle cargo checkouts. In fact I don't think we can without the Cargo changes you mentioned.

[rocallahan]

Note that for our purposes we don't need the checkout directories to be actual git checkouts. We just need some way for Cargo to expose a URL to the remote that pernosco-submit can discover.

[rocallahan]

Actually it's probably better to create a way to explicitly prioritize mount rules in the sources JSON.

[rocallahan]

@SimonSapin if you could resubmit with this pernosco-submit branch: https://github.com/Pernosco/pernosco-submit/tree/placeholders things should be better. Let me know if it works, so if it does I'll merge it to master. It won't fix this bug, but it should mean e.g. the source to the backtrace crate is present in Pernosco.

[rocallahan]

For now I think if you submit with /home/simon/.cargo/git/checkouts whitelisted on the pernosco-submit command line, then in combination with that fix you should probably have a fairly good experience.

[SimonSapin]

I uploaded the same trace again with the placeholders branch but the output looks the same:

Running 'rr pack'...
rr: Packed trace directory `/home/simon/.local/share/rr/servo-2'.
Copying /usr/lib/x86_64-linux-gnu/libthread_db.so into trace...
Obtaining source file list...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/devices-1d074a12853a87bb/cb28c47, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c/src/ots, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c/src/ots/third_party/brotli, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c/src/ots/third_party/woff2, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/media-c23a3cd5aa97076a/220ed13, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/mio-4258ee83d0750d9a/846242c, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/mozjs-fa11ffc7d4f1cc2d/5906588, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/rust-azure-984e17605d95e219/1dbd223, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/rust-mozjs-8611526964119dd6/c2896b9, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/signpost-6572985c57aad28c/7ed7125, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/webrender-c3596abe1cf4f320/4f2d78a, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/webxr-702fa5244eec1e69/e44552d, packaging files instead...
No remotes found for Git repo /home/simon/.cargo/git/checkouts/wgpu-231d39f9490416dd/128a16b, packaging files instead...
Git repo at /home/simon/projects/rr: Checking for source files changed since revision 221ae405b2c69e01b8042f268561fa466e940e9d in remote origin (access via https://raw.githubusercontent.com/mozilla/rr/221ae405b2c69e01b8042f268561fa466e940e9d/)... no changes
Packaging 3297 modified and 3236 non-repository files...
Not uploading source file /home/simon/.cargo/git/checkouts/devices-1d074a12853a87bb/cb28c47/src/bluetooth.rs (add an allowed source directory to the command line?)
Not uploading source file /home/simon/.cargo/git/checkouts/devices-1d074a12853a87bb/cb28c47/src/lib.rs (add an allowed source directory to the command line?)
Not uploading source file /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c/src/fake-zlib/zlib.h (add an allowed source directory to the command line?)
Not uploading source file /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c/src/ffi.rs (add an allowed source directory to the command line?)
Not uploading source file /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c/src/lib.rs (add an allowed source directory to the command line?)
Not uploading source file /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c/src/ots_glue.cc (add an allowed source directory to the command line?)
Not uploading source file /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c/src/ots/include/opentype-sanitiser.h (add an allowed source directory to the command line?)
Not uploading source file /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c/src/ots/src/cff.cc (add an allowed source directory to the command line?)
Not uploading source file /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c/src/ots/src/cff.h (add an allowed source directory to the command line?)
Not uploading source file /home/simon/.cargo/git/checkouts/fontsan-75864180be477bc9/29e879c/src/ots/src/cff_type2_charstring.cc (add an allowed source directory to the command line?)
(too many disallowed-source-file warnings, suppressing the rest)
Compressing to /tmp/tmpt59iyywe...

Maybe that’s expected?

Anyway, this debugging session did not happen to lead to a missing source file, and I understand that allowing more directories would help if a future one does. I filed this since it seems that making Cargo dependencies in public git repos "just work" would improve the tool and reduce resource usage.

[rocallahan]

I got the pernosco-submit change all wrong :-(

[rocallahan]

OK I fixed up pernosco-submit and actually tested it, and it worked so I pushed to pernosco-submit master. You'll still get the same pernosco-submit output I guess, but the sources for published crates and Rust stdlib should work now.

[rocallahan]

I suppose pernosco-submit could suppress warnings about not being authorized to upload cargo registry sources, since we're going to find them anyway.

[rocallahan]

Done that.