Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot cross-build images for a foreign architecture #138

Open
szlend opened this issue Jun 22, 2024 · 7 comments · May be fixed by #153
Open

Cannot cross-build images for a foreign architecture #138

szlend opened this issue Jun 22, 2024 · 7 comments · May be fixed by #153

Comments

@szlend
Copy link

szlend commented Jun 22, 2024

It's currently not possible to cross-build images for a foreign architecture. Using nix2container on nixpkgs with a foreign crossSystem will attempt to build the nix2container-bin and jq binaries for the foreign system and then try to execute them at build time. You can work around this issue by using nix2container-bin based on pkgs.pkgsBuildHost. However this doesn't work correctly either, because arch is hardcoded to runtime.GOARCH.

You can work around both of these issues with a small wrapper:

{ go, jq, pkgsBuildHost, runCommand }:

# ...

buildImage = args:
  let
    arch = go.GOARCH;
    image = pkgsBuildHost.nix2container.buildImage args;
    nativeBuildInputs = [ jq ];
  in
  runCommand image.name { inherit arch nativeBuildInputs; } ''
    jq --arg ARCH "$arch" '.arch = $ARCH' ${image} > $out
  '';

Ideally this workaround wouldn't be necessary. We could fix this by:

  1. Making nix2container-bin based on pkgs.pkgsBuildHost (this is equivalent to pkgs when crossSystem is not set).
  2. Have buildImage pass a (new) --arch argument to nix2container-bin based on pkgs.go.GOARCH.
@nlewo
Copy link
Owner

nlewo commented Jun 29, 2024

Thank you for the report.

I agree with your proposal and contributions would be welcome ;)

@szlend
Copy link
Author

szlend commented Jun 29, 2024

After reviewing the source code, I can see that this is an issue all across the board and not just isolated to buildImage. I feel like we could restructure this a bit to make cross-compilation support easier to manage.

We could do the following:

  1. Wrap nix2container with makeScopeWithSplicing'. This gives us a way to express different platform variants of our builders/derivations (e.g. buildHost, hostTarget, etc).
  2. Wrap all builders/derivations into the callPackage pattern. This enables package splicing so nativeBuildInputs/buildInputs automatically pick the correct platform variants of our derivations when cross-compiling.
  3. (Optional) Since the callPackage pattern can be pretty noisy, I would suggest extracting individual builders/derivations to their own nix files where it makes sense. Similar to how crane does it here: https://github.com/ipetkov/crane/tree/master/lib

For reference, I implemented proper splicing/cross-compilation support in crane recently: ipetkov/crane#652

The alternative approach would be reviewing the source code and replacing certain instances of pkgs.<package> with pkgs.pkgsBuildHost.<package> and manually defining pkgsBuildHost variants of nix2container-bin/skopeo-nix2container. This is a more straightforward approach, but at the cost of a lot of cross-compilation specific noise.

@nlewo Which approach would you prefer? Both of these would be fully backwards compatible. I think the splicing approach would be nicer to work with, but it would be a larger change, cause big merge conflicts with open PRs and more difficult to review. Though the project has a pretty decent test coverage so we can be fairly certain we haven't broken something.

@szlend
Copy link
Author

szlend commented Jun 29, 2024

I opened a PoC PR in #139 to show what I mean.

@szlend szlend linked a pull request Oct 27, 2024 that will close this issue
@szlend
Copy link
Author

szlend commented Oct 27, 2024

@nlewo I added an alternative implementation that should be easier to review in #153

@guskovd
Copy link

guskovd commented Dec 18, 2024

Why not just make it an architecture parameter? like dockerTools does: https://ryantm.github.io/nixpkgs/builders/images/dockertools :

architecture is optional and used to specify the image architecture, this 
is useful for multi-architecture builds that don't need cross compiling. If 
not specified it will default to hostPlatform

@guskovd
Copy link

guskovd commented Dec 18, 2024

Why not just make it an architecture parameter? like dockerTools does: https://ryantm.github.io/nixpkgs/builders/images/dockertools :

architecture is optional and used to specify the image architecture, this 
is useful for multi-architecture builds that don't need cross compiling. If 
not specified it will default to hostPlatform

I prepared another PR. It's so stupid, but it works. And it works similarly to dockerTools

#160

@szlend
Copy link
Author

szlend commented Dec 19, 2024

Why not just make it an architecture parameter? like dockerTools does: https://ryantm.github.io/nixpkgs/builders/images/dockertools

It's a good start, but not really cross-compilation aware. That's fine though since it at least allows you to wire that up yourself. With dockerTools you do not need to explicitly set the architecture if you're on a cross-compiling nixpkgs instance.

But yeah I think it makes sense to start with being able to at least override the architecture.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants