Research, understand and document Nix build directory Permission denied

[vorburger]

While attempting to re-run #1873 on another new machine, while it had:

export HOME=/build/home
mkdir -p $HOME

I've (again) run into:

nix run --no-sandbox . -- help
error: builder for '/nix/store/039ff1qq9j7qfs52j3mvj76rik9j01sc-enola-31fd088.drv' failed with exit code 1;
       last 9 log lines:
       > Running phase: unpackPhase
       > unpacking source archive /nix/store/nwy42i2f4p1h280qmgzla4vgn9jmnmgs-smcvfhzk2i208iz3kjhb8b658xczqbhv-source
       > source root is smcvfhzk2i208iz3kjhb8b658xczqbhv-source
       > Running phase: patchPhase
       > Running phase: updateAutotoolsGnuConfigScriptsPhase
       > Running phase: configurePhase
       > no configure script, doing nothing
       > Running phase: buildPhase
       > mkdir: cannot create directory '/build': Permission denied
       For full logs, run:
         nix log /nix/store/039ff1qq9j7qfs52j3mvj76rik9j01sc-enola-31fd088.drv

It's interesting that this worked on another machine yesterday, but today doesn't (for me).

Removing the export HOME and its mkdir entirely just leads to:

  > tar: VENDOR/rules_jvm_external+: time stamp 2080-02-01 00:00:00 is 1713785808.208839943 s in the future
      > tar: VENDOR: time stamp 2080-02-01 00:00:00 is 1713785808.20883202 s in the future
      > /tmp/nix-build-enola-31fd088-dirty.drv-0/kpraxdzmb1fm9k2npf8926d4fmdg98r9-source
      > + protoc --version
      > libprotoc 32.0
      > ++ which protoc-gen-grpc-java
      > + GRPC_PLUGIN=/nix/store/ncrmvgi9chxr488syvl34gy5ccvz9gkc-protoc-gen-grpc-java-1.75.0/bin/protoc-gen-grpc-java
      > + rm -rf generated/protoc/java/dev
      > + mkdir -p generated/protoc/java
      > + find java -name '*.proto' -exec protoc --java_out=generated/protoc/java --plugin=/nix/store/ncrmvgi9chxr488syvl34gy5ccvz9gkc-protoc-gen-grpc-java-1.75.0/bin/protoc-gen-grpc-java --grpc-java_out=generated/protoc/java '{}' +
      > FATAL: mkdir('/homeless-shelter/.cache/bazel/_bazel_nixbld1'): (error: 13): Permission denied

I'd like to understand how Nix manages persmission in its build sandbox better (some day, not today anymore),

and come up with a correct solution for this problem in general.

@dotdoom FYI

[vorburger]

#1877 (comment) is helpful...

... this is a "Bazel in Nix" specific issue - because Bazel uses $HOME/.cache ... and $HOME is (apparently) intentionally not writeable in a Nix build sandbox.

So we should just override Nix's $HOME with our own. Let's just set it to a directory literally called HOME, under the CWD (pwd), which will be wherever Nix wants it to be, but definitely writeable.

This is better than /build, and also better than /tmp (because of that other Bazel vendor related bug with temp; see bazelbuild/bazel#27156.)

That HOME/ should better be on .gitignore and .bazelignore.

[vorburger]

Upon further reflection, instead of all of those top-level directories, like VENDOR/ and now HOME/, let's stuff all of that into the .built directory... I've just amended #1877 accordingly. But because that fails, see #1875 (comment), I've extracted the $HOME related change into a separate PR (unrelated to vendor) into #1878. That PR will "fix" and close this issue.