buildGoModule: Fix overriding with overlay-style stdenv #225051

ShamrockLee · 2023-04-06T21:32:47Z

Description of changes

The Go package builder buildGoModule is refactored to adopt the overlay-style stdenv.mkDerivation (#119942) and to fix overriding. One can now override vendorHash using overrideAttrs. This addresses #86349 (comment)

The goModules and vendorHash attributes are moved out of passthru as they are already exposed as stdenv.mkDerivation attributes. They can still be accessed as pet.vendorHash, but no longer pet.passthru.vendorHash. The builder now only adds go to passthru.

To address the issue of vendorHash = null through overrideAttrs may break the overriding of goModules, this PR also provides an overridable passthru.overrideModAttrs that will be applied to goModules when vendorHash is not null.

Closes #86349

Comparison to similar PRs:

In this PR, the diff of pkgs/build-support/go/module.nix is shorter than #171586.

Things done

zowoq

It may not be a breaking change but I think we're to close to branch off for this so I'm a very strongly against including it in 23.05.

Also, I am planning to make a few changes to buildGoModule after the branch, e.g. drop vendorSha256 and only have vendorHash, it'll be easier and make this PR simpler if that happens first.

cc @NixOS/golang

ShamrockLee · 2023-04-06T23:39:10Z

It may not be a breaking change but I think we're too close to branch off for this so I'm strongly against including it in 23.05.

Agree. Should I drop change to the the release note?

I place them there just because the new release note hasn't come out yet.

ShamrockLee · 2023-04-07T00:09:11Z

Would it be okay to just remove those mod*Phase attributes? We alredy have overrideModAttrs, which itself will also become a syntax sugar after this chage. Exposing those mod*Phase would be confusing.

zowoq · 2023-04-07T00:17:03Z

Should I drop change to the the release note?

I place them there just because the new release note hasn't come out yet.

Fine to leave them there for now, can be moved once the new release note is created after branch off.

Would it be okay to just remove those mod*Phase attributes? We alredy have overrideModAttrs, which itself will also become a syntax sugar after this chage. Exposing those mod*Phase would be confusing.

Maybe, have to give it some thought first.

Also, I'd appreciate it if you would revert all of the unrelated formatting changes, it makes the diff much harder to review.

ShamrockLee · 2023-04-08T20:43:43Z

Also, I'd appreciate it if you would revert all of the unrelated formatting changes, it makes the diff much harder to review.

It turns out that the indentation difference is not just caused by nixpkgs-fmt, but the inherent difference of two structures.

Even then, I managed to make them align by making reasonable adjustment to the original expression (mainly by simplifying the go-modules part before running nixpkgs-fmt). The adjustment (packed inside the first 2 commits) doesn't bring any rebuild. Maybe I could file another PR containing them to make this one easier to review.

The last commit contains the actual change. The relation between the original implementation and the new implementation can be seen from the diff. The expected attributes are organized at the beginning of the expression, and the corresponding comment documentation are prefixed with ATTRIBUTES:.

zowoq · 2023-04-08T23:08:19Z

@ShamrockLee I'm switching this back to draft as it isn't going to be merged until after branch off.

pkgs/build-support/go/module.nix

zowoq · 2023-04-09T04:04:10Z

#225371

Still needs more testing but I've opened a PR to (mostly) get rid of vendorSha256.

ShamrockLee · 2023-04-09T20:20:33Z

Go packages built with this buildGoModule implementation seems to take noticable longer to evaluate. Not sure if it also happens to #171586.

How can we benchmark the evaluation time?

doronbehar

Looks good overall! Well done for all the work.

doronbehar · 2024-07-29T05:07:01Z

pkgs/build-support/go/module.nix

+      # Canonicallize `overrideModAttrs` as an attribute overlay.
+      # `passthru.overrideModAttrs` will be overridden
+      # when users want to override `goModules`.
+      overrideModAttrs = toExtension overrideModAttrs;


Since this is the only place that toExtension is used, perhaps you'd like to put it inline here? It might make the diff a bit smaller..

Quoting myself:

Since this is the only place that toExtension is used, perhaps you'd like to put it inline here? It might make the diff a bit smaller..

What do you think about this idea @ShamrockLee ?

If this function works well, I plan to move it into lib.fixedPoints, so I defined it separately.

If this function works well, I plan to move it into lib.fixedPoints, so I defined it separately.

OK sounds like a good idea to me.

pkgs/build-support/go/module.nix

doronbehar · 2024-07-29T05:10:45Z

doc/languages-frameworks/go.section.md

+
+```sh
+cd path/to/nixpkgs
+nix-prefetch -E "{ sha256 }: ((import ./. { }).my-package.overrideAttrs (_: { vendorHash = sha256; })).go-modules"


Suggested change

nix-prefetch -E "{ sha256 }: ((import ./. { }).my-package.overrideAttrs (_: { vendorHash = sha256; })).go-modules"

nix-prefetch -E "{ hash }: ((import ./. { }).my-package.overrideAttrs (_: { vendorHash = hash; })).go-modules"

This is not supported by nix-prefetch. I changed it back.

doc/languages-frameworks/go.section.md

ShamrockLee · 2024-07-30T03:30:28Z

@doronbehar Thank you for reviewing and giving feedback. I revised the Overriding goModules part of the documentation. Please take a look.

doronbehar

Looks much better!

doc/languages-frameworks/go.section.md

doronbehar

One last round I hope!

doc/languages-frameworks/go.section.md

doronbehar · 2024-07-31T04:42:08Z

doc/languages-frameworks/go.section.md

+      };
+    }
+  );
+  pet-vendored = pet-foo.overrideAttrs { vendorHash = null; };


This is a bit surprising, perhaps the paragraph above should mention why you set vendorHash = null;? Or perhaps explain this here in a a comment? (I think I understood correctly this is because you set overrideModAttrs...)

venhorHash = null is to prevent vendoring from the Nix side by goModules. People set it because the source is "vendored" (contains the vendor directory), and they would like to use such vendoring. For example, the Apptainer project provided vendored source tarballs in each release when the project was still known as Singularity.

If a user would like to use the goModules vendoring instead of the upstream one, they would set deleteVendor = true instead.

venhorHash = null is to prevent vendoring from the Nix side by goModules. People set it because the source is "vendored" (contains the vendor directory), and they would like to use such vendoring. For example, the Apptainer project provided vendored source tarballs in each release when the project was still known as Singularity.

If a user would like to use the goModules vendoring instead of the upstream one, they would set deleteVendor = true instead.

I'm sorry but I did not understand :). I'll write again what I wanted you to explain in a comment / what I felt is missing:

The purpose of this example is to demonstrate how to use overrideModAttrs, but what is peculiar to me, is that you perform an additional overrideAttrs of vendorHash = null;. I don't understand how that is related to overrdeModAttrs - if you set overrideModAttrs do you have to also set vendorHash = null; afterwards? If so, why (explain in a code comment)? If you don't have to set vendorHash = null; afterwards, please remove that, and finish with:

in pet-foo

Sorry for causing the confusion.

In my imagination, the first overriding targeting goModules might be buried somewhere inside the codebase, while another user made another overriding and set vendorHash = null.

The purpose of this documentation is to avoid the conflicts between these overridings.

doronbehar · 2024-07-31T04:43:00Z

doc/languages-frameworks/go.section.md

+  pet-vendored = pet-foo.overrideAttrs { vendorHash = null; };
+in
+pet-vendored


Perhaps (hopefully it is still nixfmt correct):

Suggested change

pet-vendored = pet-foo.overrideAttrs { vendorHash = null; };

in

pet-vendored

in

pet-foo.overrideAttrs { vendorHash = null; };

doronbehar · 2024-08-01T14:18:10Z

doc/languages-frameworks/go.section.md

+let
+  pet-foo = pkgs.overrideAttrs (
+    finalAttrs: previousAttrs: {
+      passthru = previousAttrs.passthru // {
+        overrideModAttrs = lib.composeExtensions previousAttrs.passthru.overrideModAttrs (
+          finalModAttrs: previousModAttrs: {
+            # goModules-specific overriding goes here
+            postBuild = ''
+              # Here you have access to the `vendor` directory.
+              substituteInPlace vendor/github.com/example/repo/file.go \
+                --replace-fail "panic(err)" ""
+            '';
+          }
+        );
+      };
+    }
+  );
+  pet-vendored = pet-foo.overrideAttrs { vendorHash = null; };
+in
+pet-vendored


Sorry for causing the confusion.

No worries :).

In my imagination, the first overriding targeting goModules might be buried somewhere inside the codebase, while another user made another overriding and set vendorHash = null.

The purpose of this documentation is to avoid the conflicts between these overridings.

Hmm this is a bit complicated for at this point, so I'm still not sure I understand. Let's ask this: Why not:

Suggested change

let

pet-foo = pkgs.overrideAttrs (

finalAttrs: previousAttrs: {

passthru = previousAttrs.passthru // {

overrideModAttrs = lib.composeExtensions previousAttrs.passthru.overrideModAttrs (

finalModAttrs: previousModAttrs: {

# goModules-specific overriding goes here

postBuild = ''

# Here you have access to the `vendor` directory.

substituteInPlace vendor/github.com/example/repo/file.go \

--replace-fail "panic(err)" ""

'';

}

);

};

}

);

pet-vendored = pet-foo.overrideAttrs { vendorHash = null; };

in

pet-vendored

let

pet-foo = pkgs.overrideAttrs (

finalAttrs: previousAttrs: {

# TODO: Explain this?

vendorHash = null;

passthru = previousAttrs.passthru // {

overrideModAttrs = lib.composeExtensions previousAttrs.passthru.overrideModAttrs (

finalModAttrs: previousModAttrs: {

# goModules-specific overriding goes here

postBuild = ''

# Here you have access to the `vendor` directory.

substituteInPlace vendor/github.com/example/repo/file.go \

--replace-fail "panic(err)" ""

'';

}

);

};

}

);

in

pet-foo

Let's avoid the confusion by removing vendorHash = null and leaving it as a simple example demonstrating the correct overriding approach.

doronbehar

Excellent work! Thanks for working on this @ShamrockLee . I will give a few days for @zowoq a few more days to respond to this (it seemed they didn't like it initially), and if there will be no reply I'll merge this.

Fix overriding of vendorHash and various attributes via the fixed point attribute support of stdenv.mkDerivation. Pass as derivation attributes goModules, modRoot, vendorHash, deleteVendor, and proxyVendor. Move goModules and vendorHash out of passthru. Co-authored-by: Doron Behar <[email protected]>

ShamrockLee · 2024-08-11T05:39:25Z

The package in the overriding example in go.section.md was still named pet-foo. I renamed it to pet-overridden.

Haven't responded to any of the new changes happened to the PR recently.

nixos/doc/manual/release-notes/rl-2411.section.md

rl-2411.section.md: fix typo from #225051

philiptaron · 2024-08-20T13:47:38Z

The nixpkgs-manual fails to build due to this change:

nixpkgs/doc/languages-frameworks/go.section.md

Line 94 in bbbd473

### Overriding `goModules` (#buildGoModule-goModules-override)

The parentheses ought to be curly braces. I've opened PR #336061 to fix it.

nixos-discourse · 2024-08-21T12:12:30Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/overrideattrs-and-go-query/50919/2

SuperSandro2000 · 2024-09-02T15:54:22Z

One of my overlays broke because of a mistake I did but I think we can avoid that. Fix in #339042

ShamrockLee · 2024-09-14T14:01:46Z

@doronbehar I proposed lib.toExtension in #336414, addressing the concern in #225051 (comment)

github-actions bot added 6.topic: golang 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog 8.has: documentation labels Apr 6, 2023

ShamrockLee changed the title ~~goModule: Fix overriding with overlay-style stdenv and add missing parentheses~~ goModule: Fix overriding with overlay-style stdenv Apr 6, 2023

ShamrockLee changed the title ~~goModule: Fix overriding with overlay-style stdenv~~ buildGoModule: Fix overriding with overlay-style stdenv Apr 6, 2023

ofborg bot added 10.rebuild-darwin: 501+ 10.rebuild-darwin: 1001-2500 10.rebuild-linux: 501+ 10.rebuild-linux: 1001-2500 labels Apr 6, 2023

zowoq previously requested changes Apr 6, 2023

View reviewed changes

ShamrockLee force-pushed the go-module-overlay-stdenv branch 2 times, most recently from eeb5844 to 0e87a61 Compare April 8, 2023 20:40

ShamrockLee marked this pull request as ready for review April 8, 2023 20:49

ShamrockLee requested review from kalbasit and Mic92 as code owners April 8, 2023 20:49

ShamrockLee force-pushed the go-module-overlay-stdenv branch from 0e87a61 to bd74d74 Compare April 8, 2023 21:11

ShamrockLee requested a review from zowoq April 8, 2023 21:13

zowoq marked this pull request as draft April 8, 2023 23:08

zowoq reviewed Apr 9, 2023

View reviewed changes

pkgs/build-support/go/module.nix Show resolved Hide resolved

pkgs/build-support/go/module.nix Outdated Show resolved Hide resolved

zowoq added the 2.status: work-in-progress This PR isn't done label Apr 9, 2023

zowoq self-assigned this Apr 9, 2023

ShamrockLee force-pushed the go-module-overlay-stdenv branch from bd74d74 to fee7aae Compare April 10, 2023 19:41

doronbehar reviewed Jul 29, 2024

View reviewed changes

ShamrockLee force-pushed the go-module-overlay-stdenv branch 2 times, most recently from 115a1a8 to 7352540 Compare July 30, 2024 03:27

doronbehar reviewed Jul 30, 2024

View reviewed changes

doc/languages-frameworks/go.section.md Outdated Show resolved Hide resolved

doc/languages-frameworks/go.section.md Show resolved Hide resolved

ShamrockLee force-pushed the go-module-overlay-stdenv branch from 1fcaf0d to 8ca991e Compare July 31, 2024 01:45

ShamrockLee requested a review from doronbehar July 31, 2024 01:48

doronbehar reviewed Jul 31, 2024

View reviewed changes

ShamrockLee force-pushed the go-module-overlay-stdenv branch from c931b65 to 85c6d07 Compare July 31, 2024 16:18

doronbehar reviewed Aug 1, 2024

View reviewed changes

ShamrockLee force-pushed the go-module-overlay-stdenv branch from bea7e1a to 1d618a3 Compare August 10, 2024 13:01

ShamrockLee requested a review from doronbehar August 10, 2024 13:05

doronbehar approved these changes Aug 10, 2024

View reviewed changes

ShamrockLee force-pushed the go-module-overlay-stdenv branch from 1d618a3 to eed069a Compare August 11, 2024 05:38

doronbehar merged commit 0d920a9 into NixOS:staging Aug 16, 2024
23 checks passed

jian-lin reviewed Aug 16, 2024

View reviewed changes

nixos/doc/manual/release-notes/rl-2411.section.md Show resolved Hide resolved

ShamrockLee mentioned this pull request Aug 16, 2024

rl-2411.section.md: fix typo from #225051 #335200

Merged

13 tasks

jian-lin added a commit that referenced this pull request Aug 16, 2024

Merge pull request #335200 from ShamrockLee/release-note-typo

04d11e1

rl-2411.section.md: fix typo from #225051

qezz mentioned this pull request Aug 18, 2024

Go replace take 2 qezz/nixpkgs#1

Draft

philiptaron mentioned this pull request Aug 20, 2024

nixpkgs-manual: fix build #336061

Merged

2 tasks

Aleksanaa mentioned this pull request Aug 21, 2024

go-modules: simplify overriding #171586

Closed

13 tasks

ShamrockLee mentioned this pull request Aug 21, 2024

lib.fixedPoints.toExtension: init #336414

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

buildGoModule: Fix overriding with overlay-style stdenv #225051

buildGoModule: Fix overriding with overlay-style stdenv #225051

ShamrockLee commented Apr 6, 2023 •

edited

Loading

zowoq left a comment •

edited

Loading

ShamrockLee commented Apr 6, 2023

ShamrockLee commented Apr 7, 2023

zowoq commented Apr 7, 2023 •

edited

Loading

ShamrockLee commented Apr 8, 2023 •

edited

Loading

zowoq commented Apr 8, 2023

zowoq commented Apr 9, 2023

ShamrockLee commented Apr 9, 2023 •

edited

Loading

doronbehar left a comment

doronbehar Jul 29, 2024

doronbehar Jul 30, 2024

ShamrockLee Jul 30, 2024

doronbehar Jul 30, 2024

doronbehar Jul 29, 2024

ShamrockLee Aug 10, 2024

ShamrockLee commented Jul 30, 2024

doronbehar left a comment

doronbehar left a comment

doronbehar Jul 31, 2024

ShamrockLee Jul 31, 2024 •

edited

Loading

doronbehar Aug 1, 2024

ShamrockLee Aug 1, 2024 •

edited

Loading

doronbehar Jul 31, 2024

doronbehar Aug 1, 2024

ShamrockLee Aug 10, 2024

doronbehar left a comment

ShamrockLee commented Aug 11, 2024 •

edited

Loading

philiptaron commented Aug 20, 2024 •

edited

Loading

nixos-discourse commented Aug 21, 2024

SuperSandro2000 commented Sep 2, 2024

ShamrockLee commented Sep 14, 2024

	nix-prefetch -E "{ sha256 }: ((import ./. { }).my-package.overrideAttrs (_: { vendorHash = sha256; })).go-modules"
	nix-prefetch -E "{ hash }: ((import ./. { }).my-package.overrideAttrs (_: { vendorHash = hash; })).go-modules"

buildGoModule: Fix overriding with overlay-style stdenv #225051

buildGoModule: Fix overriding with overlay-style stdenv #225051

Conversation

ShamrockLee commented Apr 6, 2023 • edited Loading

Description of changes

Things done

zowoq left a comment • edited Loading

Choose a reason for hiding this comment

ShamrockLee commented Apr 6, 2023

ShamrockLee commented Apr 7, 2023

zowoq commented Apr 7, 2023 • edited Loading

ShamrockLee commented Apr 8, 2023 • edited Loading

zowoq commented Apr 8, 2023

zowoq commented Apr 9, 2023

ShamrockLee commented Apr 9, 2023 • edited Loading

doronbehar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShamrockLee commented Jul 30, 2024

doronbehar left a comment

Choose a reason for hiding this comment

doronbehar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShamrockLee Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ShamrockLee Aug 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

doronbehar left a comment

Choose a reason for hiding this comment

ShamrockLee commented Aug 11, 2024 • edited Loading

philiptaron commented Aug 20, 2024 • edited Loading

nixos-discourse commented Aug 21, 2024

SuperSandro2000 commented Sep 2, 2024

ShamrockLee commented Sep 14, 2024

ShamrockLee commented Apr 6, 2023 •

edited

Loading

zowoq left a comment •

edited

Loading

zowoq commented Apr 7, 2023 •

edited

Loading

ShamrockLee commented Apr 8, 2023 •

edited

Loading

ShamrockLee commented Apr 9, 2023 •

edited

Loading

ShamrockLee Jul 31, 2024 •

edited

Loading

ShamrockLee Aug 1, 2024 •

edited

Loading

ShamrockLee commented Aug 11, 2024 •

edited

Loading

philiptaron commented Aug 20, 2024 •

edited

Loading