How to cross compile for ARM with MSVC?

[mstorsjo]

Normally when compiling with MSVC, you select the cross compilation target ("host" in the cross file) by running a script that sets up the environment (PATH, LIB) according to the desired host environment; the cl.exe in path is then the one for the chosen architecture, and LIB points to the directories containing libraries for this architecture.

If trying to run meson in such an environment, the default native compiler is also detected as cl from the path, a dialog box saying "The image file c:...\sanitycheckc.exe is valid, but is for a machine type other than the current machine" appears, and meson fails with the following error message:

meson.build:25:0: ERROR:  Could not invoke sanity test executable: [WinError 216] This version of %1 is not compatible with the version of Windows you're running. Check your computer's system information and then contact the software publisher.

How is one supposed to set up cross compilation with Meson with the MSVC compiler, when you can only have one compiler in PATH and one LIB set of directories in the environment at a time?

[jon-turney]

I think you must use a cross-file to specify the cross-toolchain.

[mstorsjo]

Ok, how do I specify a different LIB env var for the cross compiler vs the native one then?

[jpakkane]

Sigh. The main problem here is that Visual Studio is, shall we say, questionably designed and you can't run two compilers from the same build terminal.

On the other hand Meson's cross compilation support has been designed from the ground up to require that. The reason for this is that then you can compile code generators and run them transparently during the build in a single pass. Currently Visual Studio's limitations make this impossible. It would be awesome if Microsoft fixed this (as described in the link) but it is unlikely to happen soon (possibly ever).

However there are two horrible hack to get around this

• install MinGW or LLVM and use that as the native compiler and VS as the cross compiler (this works surprisingly well)
• create wrapper bat files for cl and linker that first invoke the cross toolchain's vcvarsall.bat and then the actual cross compiler and put those in the cross file

Yes, this is terrible on many different levels but that is sadly the current limitation we have.

[dcbaker]

It might be worth it to just create the awful wrappers ourselves and ship them with meson to avoid this coming up again and again?

[jon-turney]

Having a fix for #2392 would help, if we don't actually need to use the host compiler...

[mstorsjo]

Yes, this is terrible on many different levels but that is sadly the current limitation we have.

Ok, thanks for spelling it out!

Btw, a minor addition to your blogpost about awful things, on the topic of spawning processes. In addition to the quoting mess, the exec() functions are highly misleading, as they look like their posix counterparts, but doesn't behave the same (as you can't replace the current process, exec() spawns a new child process, but doesn't wait for it to exit, and instead exits the current process with a zero return code).

Having a fix for #2392 would help, if we don't actually need to use the host compiler...

Yup, for at least my current case, this would be enough.

[mstorsjo]

• create wrapper bat files for cl and linker that first invoke the cross toolchain's vcvarsall.bat and then the actual cross compiler and put those in the cross file

How do I specify to use this wrapper linker bat file as linker for the cross compilation in the cross file? As far as I can see, VisualStudioCCompiler is hardcoded to just use link.

[jon-turney]

How do I specify to use this wrapper linker bat file as linker for the cross compilation in the cross file? As far as I can see, VisualStudioCCompiler is hardcoded to just use link.

This is a defect :(

meson/mesonbuild/compilers/c.py

Lines 1340 to 1341 in 4a9fd0e

	def get_linker_exelist(self):
	return ['link'] # FIXME, should have same path as compiler.

[mstorsjo]

Also, to elaborate on the FIXME comment: Picking link from the same directory as the compiler would be doable, but also quite cumbersome.

Consider if you have one directory that you add to the path, with wrapper bat files like x86-cl.bat, arm64-cl.bat etc - you can't have it just pick link from this directory. Making that work would require one separate directory for each architecture again, with e.g. arm64-cl.bat and link.bat in that one, x86-cl.bat and link.bat in another one, etc. And that directory would have to be moved to the end of %PATH% so that the default native one still is first.

Being able to override the linker name via the cross file would be much more straightforward.

[jon-turney]

Trying to write a test case for #4463/#4040, it seems that lib defaults /machine to the native architecture, so this problem is only going to show up when cross-compiling.

@mstorsjo I wonder if you could share your setup for cross-compiling to Windows ARM? I'd like to try to add something like that to our CI...

[mstorsjo]

Trying to write a test case for #4463/#4040, it seems that lib defaults /machine to the native architecture, so this problem is only going to show up when cross-compiling.

The issue in #4463 should manifest itself also if the automatically picked architecture is of different bitness than the one linked - I can easily reproduce the same issue standalone with lib and rc e.g. like this:

$ lib -out:test.lib test.res test.obj
Microsoft (R) Library Manager Version 14.16.27024.1
Copyright (C) Microsoft Corporation.  All rights reserved.

LINK : warning LNK4068: /MACHINE not specified; defaulting to X86
test.obj : fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'x86'

So if you test both x86 + x64, you should be able to trigger it. But cross compilation for ARM(64) certainly is the best way to be sure to hit the error.

@mstorsjo I wonder if you could share your setup for cross-compiling to Windows ARM? I'd like to try to add something like that to our CI...

Generally for cross compiling for Windows ARM, I use https://github.com/mstorsjo/llvm-mingw, but that's not relevant here.

For MSVC, I actually mostly build on linux with MSVC wrapped in wine. My setup should of it should reproducible within Docker with the things I've published at https://github.com/mstorsjo/msvc-wine. (Since MSVC is nonredistributable, it requires you to zip up a folder from a real MSVC installation on windows though. If you want to give it a go I can also give you nondistributable urls to suitable zip files ready to run with this, in private.) With MSVC on top of linux, cross compiling with meson is just a matter of using a cross file e.g. like

[binaries]
c = 'cl'
cpp = 'cl'
ar = 'lib'
windres = 'rc'

[properties]
c_args = ['-DWINAPI_FAMILY=WINAPI_FAMILY_APP']

[host_machine]
system = 'windows'
cpu_family = 'arm'
cpu = 'armv7'
endian = 'little'

The c_args is necessary because Windows on ARM32 is generally limited to WinPhone/RT/Store/UWP/name-of-the-week. ARM64 doesn't have that limitation.

However, on linux, ninja doesn't support the MSVC details by default. For these setups, I use a custom patched ninja, with patches available at https://github.com/mstorsjo/ninja/commits/cross-msvc. I haven't tried upstreaming them, not yet at least.

To cross compile with MSVC for ARM running on actual Windows, things actually are rather messy, as discussed in this issue. I haven't tested the suggestion to have GCC/Clang available as default native compiler instead of MSVC yet though.

For the little testing I've done on real Windows, I just kept the ARM MSVC cl.exe as the main "native" compiler and hacked Meson to ignore the fact that it couldn't run the output of that compiler - that worked fine for the Meson project I built, as it didn't actually do any native compilation.

I've got a full example of cross compiling with Meson with MSVC for ARM of a real project, together with a bunch of other hacks, in the branch https://github.com/mstorsjo/msvc-wine/commits/dav1d. (That also includes a bunch of hacks around issue #4366, to allow building standalone .S ARM assembly with one standalone tool, while compiling C code with MSVC.)

[jon-turney]

• create wrapper bat files for cl and linker that first invoke the cross toolchain's vcvarsall.bat and then the actual cross compiler and put those in the cross file

It did occur to me that we could perhaps simplify this a bit by adding an [env] section to the cross/native file, which defines variables that are added to the environment when running anything from it's [binaries] section. This could be used to set LIB, INCLUDE etc. for MSVC tools, thus at least avoiding the need for wrapper .bat files as well.

[jpakkane]

No. No environment variables! They are of the devil. In addition to being conceptually terrible they can only be added by spawning executables via a Python wrapper binary. This doubles the amount of processes to run, which is sloooooooow on Windows.

[Neumann-A]

please merge #7190 so that we can actually compile for arm(64)-windows in vcpkg

On the other hand Meson's cross compilation support has been designed from the ground up to require that. The reason for this is that then you can compile code generators and run them transparently during the build in a single pass

That should be more an opt-in than an requirement.
It should always be possible to have a two step process or supply the code generators in another way. (e.g. prebuilt). Why? If you have to build for a lot of different architectures it seems unnecessary to recompile the code generator every time.
As such please fix mesons behavior ....

[jon-turney]

So, as of #7718, the headline issue here should be fixed, and we should have testing to cover it.

However, as mentioned above, the setup needed to successfully build something from a meson.build which uses both the native and cross-compiler using MS tools is complex.

It might be worth it to just create the awful wrappers ourselves and ship them with meson to avoid this coming up again and again?

Now we have [constants] in machine files, perhaps something like the following would work? (untested):

crossfile

[constants]
envwrapper = ['envwrapper.bat', 'amd64_arm64']

[binaries]
c = envwrapper + ['cl']
etc.

nativefile

[constants]
envwrapper = ['envwrapper.bat', 'amd64']

[binaries]
c = envwrapper + ['cl']
etc.

where envwrapper.bat is a script which invokes vcvarsall.bat to set the specified compilation environment, then invokes the given command in that environment.