build-support: fix nix-prefetch-* on macOS #358685

jdelStrother · 2024-11-24T11:45:54Z

Since nix 2.20, nix-store --add-fixed doesn't accept paths where the parent directory is a symlink. On macOS, /tmp is a symlink to /private/tmp, which causes a '/tmp' is a symlink error:

$ nix run github:nixos/nixpkgs/24.11-beta#nix-prefetch-git -- --url https://github.com/IFTTT/polo.git --rev 316aa2ac210a45a7fc400ab921831493d5dd21b8 --hash sha256
Initialized empty Git repository in /private/tmp/git-checkout-tmp-1Bf9bIv7/polo-316aa2a/.git/
remote: Enumerating objects: 51, done.
remote: Counting objects: 100% (51/51), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 51 (delta 8), reused 19 (delta 5), pack-reused 0 (from 0)
Unpacking objects: 100% (51/51), 19.57 KiB | 541.00 KiB/s, done.
From https://github.com/IFTTT/polo
 * branch            HEAD       -> FETCH_HEAD
Switched to a new branch 'fetchgit'
removing `.git'...
error: path '/tmp' is a symlink

Avoid this by resolving /tmp to a real directory in all the prefetch scripts

Fixes #358113

I don't see mention of the symlink-change in the nix 2.20 release notes, but I tested with this to narrow down the point where things started breaking:

nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/8b27c1239e5c421a2bbc2c65d52e4a6fbf2ff296.tar.gz \
  --packages 'nix-prefetch-git.override { nix = pkgs.nixVersions.nix_2_20; }' \
  --command 'nix-prefetch-git --url https://github.com/IFTTT/polo.git --rev 316aa2ac210a45a7fc400ab921831493d5dd21b8 --hash sha256'

Substitute nix_2_20 with nix_2_19 to see the old behavior which happily accepted symlinked directories.

Things done

Add a 👍 reaction to pull requests you find important.

emilazy

I think this is just a Nix bug. We can work around it, but I’d like to see a comment next to the occurrences pointing to the bug so we can remove it later when it’s fixed.

emilazy · 2024-11-24T19:44:58Z

pkgs/build-support/docker/nix-prefetch-docker

@@ -121,7 +121,7 @@ fi

 sourceUrl="docker://$imageName@$imageDigest"

-tmpPath="$(mktemp -d "${TMPDIR:-/tmp}/skopeo-copy-tmp-XXXXXXXX")"
+tmpPath="$(mktemp -d "$(realpath "${TMPDIR:-/tmp}/skopeo-copy-tmp-XXXXXXXX")")"


How about:

Suggested change

tmpPath="$(mktemp -d "$(realpath "${TMPDIR:-/tmp}/skopeo-copy-tmp-XXXXXXXX")")"

tmpPath="$(realpath "$(mktemp -d --tmpdir skopeo-copy-tmp-XXXXXXXX)")"

and analoguously for all the others?

This actually used to use --tmpdir over 10 years ago, but was changed to use TMPDIR in 5e7a1cf.

It does look like my /usr/bin/mktemp does support --tmpdir on macOS 15.1.1, but I'm not sure how far back that goes. I could add coreutils to all the prefetch scripts to try & make sure we've always got a usable mktemp --tempdir ..?

Ah, I knew I kept my trusty 10.12 VM around for something:

So this would be fine as‐is for 25.05 I think but not for 24.11. However, yes, we probably ought to just add coreutils to the list of common dependencies in pkgs/tools/package-management/nix-prefetch-scripts/default.nix; all our other Bash scripts assume GNU coreutils anyway.

jdelStrother · 2024-11-24T21:30:47Z

I think this is just a Nix bug. We can work around it, but I’d like to see a comment next to the occurrences pointing to the bug so we can remove it later when it’s fixed.

OK, will try and get the nix devs to confirm whether it's a bug or feature so at least we have something concrete to link to.

Since nix 2.20, `nix-store --add-fixed` doesn't accept paths where the parent directory is a symlink. On macOS, /tmp is a symlink to /private/tmp, which causes a "'/tmp' is a symlink" error: ``` $ nix run github:nixos/nixpkgs/24.11-beta#nix-prefetch-git -- --url https://github.com/IFTTT/polo.git --rev 316aa2ac210a45a7fc400ab921831493d5dd21b8 --hash sha256 Initialized empty Git repository in /private/tmp/git-checkout-tmp-1Bf9bIv7/polo-316aa2a/.git/ remote: Enumerating objects: 51, done. remote: Counting objects: 100% (51/51), done. remote: Compressing objects: 100% (42/42), done. remote: Total 51 (delta 8), reused 19 (delta 5), pack-reused 0 (from 0) Unpacking objects: 100% (51/51), 19.57 KiB | 541.00 KiB/s, done. From https://github.com/IFTTT/polo * branch HEAD -> FETCH_HEAD Switched to a new branch 'fetchgit' removing `.git'... error: path '/tmp' is a symlink ``` Avoid this by resolving /tmp to a real directory in all the prefetch scripts

macOS 10.12 doesn't have a usable --tmpdir flag on the builtin mktemp, but we can make use of coreutil's mktemp instead.

jdelStrother · 2024-11-25T18:16:33Z

I think this is an intentional change rather than bug, judging by NixOS/nix@83c067c. Shall I just update the comments to point to that commit, rather than the issue at NixOS/nix#11941 ?

emilazy · 2024-11-25T18:20:54Z

I think that rather that’s just an API change that they didn’t properly update all the callers of. I still think it is a Nix bug.

NobbZ · 2024-11-25T18:23:46Z

I agree with emilazy. This breaks something that previously worked. And adding something that is behind a symlink (but not necessarily the symlink itself!) seems a valid usecase to me.

will · 2024-11-29T11:25:21Z

I think this PR would fix what is currently breaking crystal2nix, but I haven't yet verified.

It starts doing its work, then gets error: path '/tmp' is a symlink when calling nix-prefetch-git , then fails to parse the empty stdout response as json

❯ crystal2nix
Initialized empty Git repository in /private/tmp/git-checkout-tmp-UCUG3iTb/backtracer.cr/.git/
From https://github.com/sija/backtracer.cr
 * tag               v1.2.2     -> FETCH_HEAD
Switched to a new branch 'fetchgit'
removing `.git'...
error: path '/tmp' is a symlink
Unhandled exception: Expected BeginObject but was EOF at line 1, column 1
  parsing Crystal2Nix::PrefetchJSON at line 0, column 0 (JSON::SerializableError)
  from /nix/store/fpnllrcmwbhpj2l3njin35jjb2f7jfdb-crystal2nix-0.3.0/bin/.crystal2nix-wrapped in 'raise<JSON::SerializableError>:NoReturn'
  from /nix/store/fpnllrcmwbhpj2l3njin35jjb2f7jfdb-crystal2nix-0.3.0/bin/.crystal2nix-wrapped in 'Crystal2Nix::Worker#run:Nil'
  from /nix/store/fpnllrcmwbhpj2l3njin35jjb2f7jfdb-crystal2nix-0.3.0/bin/.crystal2nix-wrapped in '__crystal_main'
  from /nix/store/fpnllrcmwbhpj2l3njin35jjb2f7jfdb-crystal2nix-0.3.0/bin/.crystal2nix-wrapped in 'main'
Caused by: Expected BeginObject but was EOF at line 1, column 1 (JSON::ParseException)
  from /nix/store/fpnllrcmwbhpj2l3njin35jjb2f7jfdb-crystal2nix-0.3.0/bin/.crystal2nix-wrapped in 'raise<JSON::ParseException>:NoReturn'
  from /nix/store/fpnllrcmwbhpj2l3njin35jjb2f7jfdb-crystal2nix-0.3.0/bin/.crystal2nix-wrapped in 'JSON::PullParser#raise<String>:NoReturn'
  from /nix/store/fpnllrcmwbhpj2l3njin35jjb2f7jfdb-crystal2nix-0.3.0/bin/.crystal2nix-wrapped in 'Crystal2Nix::Worker#run:Nil'
  from /nix/store/fpnllrcmwbhpj2l3njin35jjb2f7jfdb-crystal2nix-0.3.0/bin/.crystal2nix-wrapped in '__crystal_main'
  from /nix/store/fpnllrcmwbhpj2l3njin35jjb2f7jfdb-crystal2nix-0.3.0/bin/.crystal2nix-wrapped in 'main'

jdelStrother · 2024-11-29T11:43:41Z

Yeah, sounds like the same issue.
I've re-written NixOS/nix#11941 to try & clarify it's not just some macOS weirdness.

Requested comment was added.

Mic92 · 2024-12-13T13:23:48Z

Tested those prefetchers and they seem to work. Will bring up the bug in Nix again in the next meeting.

nix-backports · 2024-12-13T13:24:28Z

Successfully created backport PR for release-24.11:

[Backport release-24.11] build-support: fix nix-prefetch-* on macOS #364921

will · 2024-12-13T13:55:00Z

Thanks!

Mic92 · 2024-12-15T17:40:31Z

Work in progress pull request in nix that attempts to fix this: NixOS/nix#12046

github-actions bot added the 6.topic: fetch label Nov 24, 2024

nix-owners bot requested review from philiptaron and roberth November 24, 2024 11:47

jdelStrother mentioned this pull request Nov 24, 2024

nix-store --add-fixed no longer accepts symlinks NixOS/nix#11941

Open

2 tasks

philiptaron requested a review from a team November 24, 2024 15:57

emilazy previously requested changes Nov 24, 2024

View reviewed changes

ofborg bot added 6.topic: darwin Running or building packages on Darwin 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Nov 25, 2024

jdelStrother force-pushed the prefetch-symlink-errors branch from b5af7fc to b81ee0e Compare November 25, 2024 09:38

build-support: Simplify tmpdir creation with coreutils

6cb8c5e

macOS 10.12 doesn't have a usable --tmpdir flag on the builtin mktemp, but we can make use of coreutil's mktemp instead.

jdelStrother force-pushed the prefetch-symlink-errors branch from b81ee0e to 6cb8c5e Compare November 25, 2024 09:41

jdelStrother requested a review from emilazy November 25, 2024 09:52

ofborg bot requested review from bennofs and offlinehacker November 26, 2024 02:16

will mentioned this pull request Nov 29, 2024

nix-prefetch-git broken (maybe macos only) nix-community/crystal2nix#24

Open

paparodeo mentioned this pull request Dec 13, 2024

nix-prefetch-git: Fix on macOS #364757

Closed

13 tasks

Mic92 added the backport release-24.11 Backport PR automatically label Dec 13, 2024

Mic92 merged commit 3dc8ff1 into NixOS:master Dec 13, 2024
31 checks passed

nix-backports bot mentioned this pull request Dec 13, 2024

[Backport release-24.11] build-support: fix nix-prefetch-* on macOS #364921

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build-support: fix nix-prefetch-* on macOS #358685

build-support: fix nix-prefetch-* on macOS #358685

jdelStrother commented Nov 24, 2024 •

edited

Loading

emilazy left a comment

emilazy Nov 24, 2024

jdelStrother Nov 24, 2024

emilazy Nov 24, 2024

jdelStrother commented Nov 24, 2024

jdelStrother commented Nov 25, 2024 •

edited

Loading

emilazy commented Nov 25, 2024

NobbZ commented Nov 25, 2024

will commented Nov 29, 2024

jdelStrother commented Nov 29, 2024

Mic92 commented Dec 13, 2024

nix-backports bot commented Dec 13, 2024

will commented Dec 13, 2024

Mic92 commented Dec 15, 2024

	tmpPath="$(mktemp -d "$(realpath "${TMPDIR:-/tmp}/skopeo-copy-tmp-XXXXXXXX")")"
	tmpPath="$(realpath "$(mktemp -d --tmpdir skopeo-copy-tmp-XXXXXXXX)")"

build-support: fix nix-prefetch-* on macOS #358685

build-support: fix nix-prefetch-* on macOS #358685

Conversation

jdelStrother commented Nov 24, 2024 • edited Loading

Things done

emilazy left a comment

Choose a reason for hiding this comment

emilazy Nov 24, 2024

Choose a reason for hiding this comment

jdelStrother Nov 24, 2024

Choose a reason for hiding this comment

emilazy Nov 24, 2024

Choose a reason for hiding this comment

jdelStrother commented Nov 24, 2024

jdelStrother commented Nov 25, 2024 • edited Loading

emilazy commented Nov 25, 2024

NobbZ commented Nov 25, 2024

will commented Nov 29, 2024

jdelStrother commented Nov 29, 2024

Mic92 commented Dec 13, 2024

nix-backports bot commented Dec 13, 2024

will commented Dec 13, 2024

Mic92 commented Dec 15, 2024

jdelStrother commented Nov 24, 2024 •

edited

Loading

jdelStrother commented Nov 25, 2024 •

edited

Loading