`CROSS_CONTAINER_OPTS` does not replace `--platform linux/amd64` #1498

leet4tari · 2024-05-21T12:21:15Z

Checklist

I've looked through the issues and pull requests for similar reports

Describe your issue

I don't think that the CROSS_CONTAINER_OPTS env is replacing the --platform linux/amd64 command, at least not on Apple Silicon.

What target(s) are you cross-compiling for?

aarch64-unknown-linux-gnu, x86_64-unknown-linux-gnu

Which operating system is the host (e.g computer cross is on) running?

macOS
Windows
Linux / BSD
other OS (specify in description)

What architecture is the host?

x86_64 / AMD64
arm32
arm64 (including Mac M1)

What container engine is cross using?

docker
podman
other container engine (specify in description)

cross version

cross 0.2.5 (19be834 2024-05-17)

Example

Running

export CROSS_CONTAINER_OPTS="--platform linux/arm64"
export CROSS_BUILD_OPTS="--platform linux/arm64"
export CROSS_DEBUG=1

then trying run

cross build \
  --target aarch64-unknown-linux-gnu \
  --bin minotari_miner -v
[src/cargo.rs:156] ->
+ cargo metadata --format-version 1 --filter-platform aarch64-unknown-linux-gnu
[src/rustc.rs:370] ->
+ rustc --print sysroot
[src/docker/engine.rs:220] ->
+ /usr/local/bin/docker
[src/docker/engine.rs:295] ->
+ /usr/local/bin/docker version -f '{{ .Server.Os }},,,{{ .Server.Arch }}'
[src/docker/engine.rs:163] ->
+ /usr/local/bin/docker info -f {{.SecurityOptions}}
[src/rustup.rs:89] ->
+ rustup toolchain list
[src/rustup.rs:115] ->
+ rustup target list --toolchain nightly-2024-03-01-x86_64-unknown-linux-gnu
[src/rustup.rs:242] ->
+ rustup component list --toolchain nightly-2024-03-01-x86_64-unknown-linux-gnu
[src/docker/custom.rs:177] ->
+ /usr/local/bin/docker buildx build --platform linux/amd64 --progress auto --label 'org.cross-rs.for-cross-target=aarch64-unknown-linux-gnu' --label 'org.cross-rs.runs-with=x86_64-unknown-linux-gnu' --label 'org.cross-rs.workspace_root=/Users/leet/src/tari/tari-projects/tari' --tag localhost/cross-rs/cross-custom-tari:aarch64-unknown-linux-gnu-6c732-pre-build --build-arg 'CROSS_SCRIPT=./scripts/cross_compile_hack.sh' --build-arg 'CROSS_TARGET=aarch64-unknown-linux-gnu' --build-arg 'CROSS_DEB_ARCH=arm64' --file /Users/leet/src/tari/tari-projects/tari/target/aarch64-unknown-linux-gnu/Dockerfile.aarch64-unknown-linux-gnu-custom --platform linux/arm64 --output 'type=docker' /Users/leet/src/tari/tari-projects/tari

You might notice docker buildx build --platform linux/amd64 with --platform linux/arm64 much later in the command, which I think force running x86 images, even thou I would like to use an aarch64/arm64 image.

Reason been, that I was looking to run a CPU native docker image as my base to cross-compile, I was expecting/hoping to use aarch64-unknown-linux-gnu image (Ubuntu 18.04), but kept running into rosetta errors.

Additional information / notes

No response

The text was updated successfully, but these errors were encountered:

Emilgardis · 2024-05-21T14:22:06Z

to use a custom image with another platform you need to specify the platforms it can/should run with. See https://github.com/cross-rs/cross/blob/main/docs/config_file.md#targettargetimage

[target.aarch64-unknown-linux-gnu]
image.name = "alpine:edge"
image.toolchain = ["x86_64-unknown-linux-musl", "linux/arm64=aarch64-unknown-linux-musl"]

so in this case, you want

[target.aarch64-unknown-linux-gnu]
image.name = "my-image"
image.toolchain = ["aarch64-unknown-linux-gnu"]

leet4tari · 2024-05-22T07:10:49Z

Thanks for the reply @Emilgardis.

I had tried quite a few option and I might not have listed trying this:

[target.aarch64-unknown-linux-gnu]
image = "ubuntu:18.04"
pre-build = "./scripts/cross_compile_hack.sh"

[target.aarch64-unknown-linux-gnu.env]
passthrough = [
  "CARGO_BUILD_TARGET",
  "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc",
  "BINDGEN_EXTRA_CLANG_ARGS=--sysroot /usr/aarch64-linux-gnu/include/",
]

and my script does a uname -m which is listing x86 and not aarch64 as I had expected. Basic idea was that on MacOS, could use cross-rs and docker to cross-compile Linux binaries, with minimal fuss and effect.

The problem I am seeing, is that cross-rs is not letting me choose the docker platform to run/build on, even if I try and overload with either of the two env OPTS.

Emilgardis · 2024-05-22T07:21:47Z

You need to tell cross that the image can run on aarch64 platform, we assume x86_64 if nothing is specified.

To choose the platform, you need to add it as a valid platform for that image

[target.aarch64-unknown-linux-gnu]
image.name = "ubuntu:18.04"
image.toolchain = ["aarch64-unknown-linux-gnu"]

There is however no way to tell cross "of all the possible platforms, please use x", we should probably make that possible

leet4tari · 2024-05-22T07:28:55Z

Thanks again @Emilgardis , can we not use the env CROSS_CONTAINER_OPTS to force docker to use a platform? That is something I saw suggested in an issue while looking through this.

What I was raising is that the docker run command, thou it does add the env to the run but at the end, does not replace what looks like the default of docker buildx build --platform linux/amd64 vs replacing the --platform part if put into the env.

leet4tari · 2024-05-22T07:31:43Z

What my original expectation was that the docker would be running with local CPU arch, unless overridden or forced, might that be better user expectation?

Emilgardis · 2024-05-22T07:43:59Z

To not have cross insert --platform if it's already been specified with CROSS_CONTAINER_OPTS you'd have to modify

cross/src/docker/image.rs

Line 437 in 19be834

    
           pub fn specify_platform(&self, engine: &Engine, cmd: &mut std::process::Command) {

I think we can do that, it's consistent with what we do for --load on CROSS_BUILD_OPTS. There might however be other issues with that due to the expectation in the code that we've determined the platform and are using it.

What my original expectation was that the docker would be running with local CPU arch, unless overridden or forced, might that be better user expectation?

That's an interesting insight, it might be worth doing. My only concern is how moby/buildx now attests image provenence with unknown/unknown, which together with the issue presented in #1214 makes it kinda confusing.

leet4tari · 2024-05-22T07:59:48Z

That is the issue where I got the idea to try and force the docker platform, when I was able to debug that the arm64 was defaulting to using the x86_64 docker engine.

The idea I was raising and really hoping would be easy to fix or even test, would be to stop the defaulting to x86_64 or be able to use the --platform override, but my rust is not good enough to come up with sort of patch. Happy to try and test anything, if you or anybody else has some suggestions,.

Interestingly, somebody might want to force building with qemu or rosetta that is not the default docker target, but that is a different problem, I would just like to be able to use the native CPU arch, but been able to target a different OS, ie osx -> linux.

As for the docker images having unknown/unknown, I still don't understand this behaviour and have disable for now in my own projects.

Emilgardis · 2024-05-22T10:12:59Z

It's unclear to me if you've been able to solve your problem, the way to immediately solve your issue of not being able to specify --platform=linux/arm64 is the config I shared earlier, does that work for now?

With that config it's even possible to specify precisely what platform to use

[target.aarch64-unknown-linux-gnu]
image.name = "ubuntu:18.04"
# syntax = <platform>=<toolchain>
image.toolchain = ["linux/arm64=aarch64-unknown-linux-gnu"]

leet4tari · 2024-05-22T13:36:33Z

I missed your suggestion on how to possible target the platform, let me test and report back. Thanks @Emilgardis for all the support.

leet4tari · 2024-05-23T09:02:12Z

Thanks @Emilgardis , I was able to come right. I can cross-compile our project on both x86_64 and aarch64 OSX computers to Linux x86_64 and aarch64.

I am not sure I would have tried your suggestion from the documentation, thou that might just be me trying to get something do in a very specific kind of way.

Still not sure that it is right, but it works.

The docker command line might still be a bug, as it does not only issue one set of args as overrides, just duplicates what you have put into the env.

leet4tari · 2024-05-23T12:22:32Z

Adding some info and possible see which way this goes

[target.aarch64-unknown-linux-gnu]
image = "ubuntu:18.04"
# Need to use native aarch64 on Apple silicon
#image.name = "ubuntu:18.04"
#image.toolchain = ["linux/arm64=aarch64-unknown-linux-gnu"]
pre-build = "./scripts/cross_compile_ubuntu_18-pre-build.sh"

[target.x86_64-unknown-linux-gnu]
image = "ubuntu:18.04"
pre-build = "./scripts/cross_compile_ubuntu_18-pre-build.sh"

maybe better for me to open another issue regarding how the Apple Silicon does not seem to use the native platform and need to use the commented out targeting using the toolchain.

Using the selection as is, looks like it works on Windows and Linux, but they both x86_64 systems, vs the Apple machine which is native aarch64. I have to use the commented section instead to force docker image to use aarch64 image, else the build does not work.

Emilgardis · 2024-05-23T16:56:04Z

The image.toolchain config is an array, you can specify multiple platforms/toolchains

It works like that by design, as I've described earlier

Emilgardis · 2024-05-23T17:00:19Z

So, you would do

image.toolchain = ["linux/arm64=aarch64-unknown-linux-gnu", "linux/amd64=x86_64-unknown-linux-gnu"]

KarstenB · 2024-05-27T07:29:08Z

It's unclear to me if you've been able to solve your problem, the way to immediately solve your issue of not being able to specify --platform=linux/arm64 is the config I shared earlier, does that work for now?

With that config it's even possible to specify precisely what platform to use
[target.aarch64-unknown-linux-gnu]
image.name = "ubuntu:18.04"
# syntax = <platform>=<toolchain>
image.toolchain = ["linux/arm64=aarch64-unknown-linux-gnu"]

When I try this, I get:

❯ cross --version
cross 0.2.5
Error: 
   0: failed to parse file `"bla/Cross.toml"` as TOML
   1: invalid type: map, expected a string for key `target.aarch64-unknown-linux-gnu.image` at line 2 column 14

Emilgardis · 2024-05-27T08:59:19Z

@KarstenB I should clarify, above only works on the main branch, image.toolchain has not been released yet

so, you'll need to install cross as cargo install cross --git https://github.com/cross-rs/cross

Emilgardis mentioned this issue May 22, 2024

Make it possible to choose platform #1499

Open

Emilgardis added the enhancement label May 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`CROSS_CONTAINER_OPTS` does not replace `--platform linux/amd64` #1498

`CROSS_CONTAINER_OPTS` does not replace `--platform linux/amd64` #1498

leet4tari commented May 21, 2024

Emilgardis commented May 21, 2024

leet4tari commented May 22, 2024

Emilgardis commented May 22, 2024 •

edited

Loading

leet4tari commented May 22, 2024

leet4tari commented May 22, 2024

Emilgardis commented May 22, 2024 •

edited

Loading

leet4tari commented May 22, 2024

Emilgardis commented May 22, 2024

leet4tari commented May 22, 2024

leet4tari commented May 23, 2024

leet4tari commented May 23, 2024 •

edited

Loading

Emilgardis commented May 23, 2024 •

edited

Loading

Emilgardis commented May 23, 2024

KarstenB commented May 27, 2024

Emilgardis commented May 27, 2024

CROSS_CONTAINER_OPTS does not replace --platform linux/amd64 #1498

CROSS_CONTAINER_OPTS does not replace --platform linux/amd64 #1498

Comments

leet4tari commented May 21, 2024

Checklist

Describe your issue

What target(s) are you cross-compiling for?

Which operating system is the host (e.g computer cross is on) running?

What architecture is the host?

What container engine is cross using?

cross version

Example

Additional information / notes

Emilgardis commented May 21, 2024

leet4tari commented May 22, 2024

Emilgardis commented May 22, 2024 • edited Loading

leet4tari commented May 22, 2024

leet4tari commented May 22, 2024

Emilgardis commented May 22, 2024 • edited Loading

leet4tari commented May 22, 2024

Emilgardis commented May 22, 2024

leet4tari commented May 22, 2024

leet4tari commented May 23, 2024

leet4tari commented May 23, 2024 • edited Loading

Emilgardis commented May 23, 2024 • edited Loading

Emilgardis commented May 23, 2024

KarstenB commented May 27, 2024

Emilgardis commented May 27, 2024

`CROSS_CONTAINER_OPTS` does not replace `--platform linux/amd64` #1498

`CROSS_CONTAINER_OPTS` does not replace `--platform linux/amd64` #1498

Emilgardis commented May 22, 2024 •

edited

Loading

Emilgardis commented May 22, 2024 •

edited

Loading

leet4tari commented May 23, 2024 •

edited

Loading

Emilgardis commented May 23, 2024 •

edited

Loading