MSVC first-class support for producing ARM64EC binaries

[jcbrill]

Preliminary research notes and thoughts for producing ARM64EC compatible binaries (ARM64X) with MSVC.

Example assuming a Visual Studio 2022 installation with the latest ARM64 toolkit version and latest Windows SDK:

• myprog.c

  #include <stdio.h>
  int main() {
      printf("Hello\n");
      return 0; 
  }
  

• vcvarsalls.bat invocation:

  • ARM64 host:
    "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" arm64
  • amd64 host:
    "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64_arm64

• compile:

  cl /c /nologo /arm64EC myprog.c
  

• link:

  link /SUBSYSTEM:CONSOLE /MACHINE:ARM64EC myprog.obj
  

• dump headers:

  link -dump -headers myprog.exe
  

  ...
            8664 machine (x64) (ARM64X)
  ...
  

ARM64EC is an alternate ABI (application binary interface) on Windows ARM64 computers. ARM64EC is not implemented via WoW redirection on ARM64 similar to x86 and arm binary architectures.

Challenges for directly supporting ARM64EC binaries in SCons:

• With respect to setting up MSVC, ARM64EC is not a host/target toolset architecture. The call to host/target architecture argument for vcvarsall.bat is the same for targeting ARM64 and ARM64EC (e.g., arm64, amd64_arm64, etc.).

  Adding ARM64EC as a "virtual architecture" is a bit challenging given the current implementation as the native host architecture of a Windows ARM64 box is ARM64 and is only supported by certain MSVC toolset versions and Windows SDK versions.

• The requirements for ARM64EC are based on the MSVC toolset version and the Windows SDK version.

  The minimum toolset version appears to be at least 14.31. The minimum Windows SDK version appears to be at least 10.0.22621.0. However, there appear to be combinations of MSVC toolset version and Windows SDK version that may result in build errors.

  In quick-and-dirty testing, attempting to build with MSVC toolset 14.31 and the latest Windows SDK failed. Attempting to build with MSVC toolset 14.31 and Windows SDK 10.0.22621.0 succeeded.

  Note: MSVC documentation may indicate a minimum MSVC toolset version of 14.34.

• When using modern Visual Studio versions, the MSVC toolset version and the windows SDK version should be known when validating MSVC arguments and detecting the the installed toolsets and Windows SDKs.

  Aside: there is an undocumented internal environment variable that forces the MSVC toolset version and Windows SDK version as command line arguments when running the vcvars batch files. This can be useful when the msvc cache is enabled in the presence of updates to the MSVC toolsets and/or Windows SDK versions in which the "older" versions are not uninstalled to prevent inadvertently using cached paths that may still exist but are not the latest.

  The MSVC toolset version and Windows SDK version are not saved in the SCons environment.

• Adding /arm64EC to the CCFLAGS construction variable probably should be done in Tool\msvc.py.

• Adding /MACHINE:ARM64EC to the LINKFLAGS construction variable probably should be done in Tool\mslink.py

• One possible way to implement support for ARM64EC would be to add a new construction variable for MSVC that indicates a desire to target the ARM64EC ABI. For example, MSVC_ARM64EC_ABI=True.

  There might be enough information in Tool\MSCommon\*.py to determine if targeting the ARM64EC is likely to succeed based on the target architecture (AMD64), MSVC toolset version, and Windows SDK version.

  Adding an internal-use only variable to the SCons environment, for example _MSVC_ARM64EC_ABI, that is either True or False following validation could be used by the msvc and mslink tools to determine if the compiler and linker flags for ARM64EC should be added or not.

• Validating arguments is somewhat complicated and reasonably necessary. Debugging vcvarsall batch file failures and MSVC command-line tool errors due to unsupported arguments is usually a challenge.

• The MSVC toolset version and Windows SDK version have first-class support via the MSVC_TOOLSET_VERSION and MSVC_SDK_VERSION construction variables, respectively.

It is not sufficient to use MSVC_VERSION in determining if ARM64EC is supported. Attempting to add ARM64EC as an architecture could be more difficult than it appears on the surface.

Edits:

1. Adding a construction variable and then setting an internal environment variable as dissussed above is problematic if msvc auto-detection is bypassed.

2. Interesting external ARM64EC discussion: http://www.emulators.com/docs/abc_arm64ec_explained.htm

3. Revised possible implementation:

   Add a function (e.g., msvc_is_arm64ec(env)) indicating if the build is for the ARM64EC ABI that is imported from SCons.Tool.MSCommon in the msvc tools (e.g., msvc and mslink, etc) which is used to add the relevant command-line options if necessary.

   The implementation of the function could be in SCons\MSCommon\Tool\vc.py. The implementation could then use the known information from the environment to decide if building with ARM64EC is likely to succeed or issue a warning (unsure) or an error (known to be invalid) based on the environment configuration.

   If the msvc batch file was invoked when setting up msvc, additional information could be stored in a "private" key in the environment. If the msvc batch file processing was bypassed, limited checks would be available and may have to assume the user is sure that the build will succeed.

   One additional construction variable (e.g., MSVC_ARM64EC_ABI=True) is required and all of the "heavy lifting" is in the appropriate source file under MSCommon. The msvc tools are already importing functions from MSCommon anyway.

[LeonarddeR]

Many thanks for your great outline.
I wonder whether we should take arm64x into account here as well. That's a sort of binary that can contain both arm64ec and arm64 code, as far as I now understand.
You can pass either /machine:arm64ec or /machine:arm64x to the linker, but I don't think it makes that much difference when you're only providing arm64ec objects to the linker. Basically when creating an arm64x binary, you should create both arm64 and arm64ec objects and then link them together in one arm64x binary.

[jcbrill]

Additional research results for ARM64EC and ARM64X.

Please report any errors and misinformation.

MSVC setup:

• "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" x64_arm64 [ARM64]

Compile combinations:

• C1: cl /c /nologo /Fo:myprog-C1.obj myprog.c [ARM64]
• C2: cl /c /nologo /arm64EC /Fo:myprog-C2.obj myprog.c [ARM64EC]

Link combinations:

• L1: link /SUBSYSTEM:console /MACHINE:ARM64 ... [ARM64]
• L2: link /SUBSYSTEM:console /MACHINE:ARM64EC ... [ARM64EC]
• L3: link /SUBSYSTEM:console /MACHINE:ARM64X ... [ARM64EC]

Compile/link combinations:

Status	C1.obj	C2.obj	L1	L2	L3	Exe Machine Type
	ARM64	ARM64EC	ARM64	ARM64EC	ARM64X
Pass	*		*			AA64 machine (ARM64)
Fail	*			*
Fail					*
Fail		*	*
Pass		*		*		8664 machine (x64) (ARM64X)
Fail		*			*
Fail	*	*	*
Pass	*	*		*		8664 machine (x64) (ARM64X)
Pass	*	*			*	AA64 machine (ARM64) (ARM64X)

The compiler and linker flags specification and inclusion needs to be done independently. In the presence of linking external libraries, there is no reasonable way for SCons to know the architecture/ABI of the linked binaries.

A construction variable for the compiler could be one/two valued: MSVC_ARM64_ABI=ARM64EC|ARM64?.

A construction variable for the linker could be two/three valued: MSVC_ARM64_MACHINE=ARM64EC|ARM64X|ARM64?.

At this point, I'm unsure of the effectiveness of validation of the options. Even validating the TARGET_ARCH, MSVC toolset version, Windows SDK version becomes difficult/unreliable if SCons MSVC auto-detection was bypassed. Perhaps just the "obviously* wrong can be caught.

Diagnosis of MSVC compiler/linker errors can sometimes be challenging and time consuming especially when run on remote computers. Catching as many errors as possible during configuration is far easier than trying to diagnose where and when everything went wrong.

[jcbrill]

For those following along, in order to build one's own AA64 (ARM64) (ARM64X) DLL/EXE, all of the source files have to be built twice: once as ARM64 and once as ARM64EC; and then both sets of object files are linked with machine type ARM64X.

Unless I'm missing something, and showing my lack of knowledge of how to actually use SCons, this would require at least two SCons environments as the CCFLAGS are different for the two sets of object file compilations and possibly a third to handle linking both sets of object files.

Feel free to weigh in on how this could be accomplished.

[bdbaddog]

Actually in this case you could just add another Action() with the different compiler flags to the sharedObject builder (and static object), and program. Potentially using a generator for the Builder()

And likely a different suffix (-1.obj and-2.obj for example instead of .obj)

[jcbrill]

Thanks!

I'm starting to wonder if it's easier to "get it right" simply using the compiler and linker flags rather than implementing a mechanism that automatically adds them based on some condition possibly involving additional construction variables or virtual target architectures.

Will have to think about it some more...

[bdbaddog]

So seems like if TARGET_HOST is set to x64_arm64 then this logic should be activated?

[jcbrill]

There is no way around needing at least one more piece of information.

For the compiler, when the TARGET_ARCH is ARM64 (x64_arm64, arm64, etc) then the possible ARM64 ABI's are:

• ARM64 (no flags ==> ARM64)
• ARM64EC (/arm64EC ==> ARM64EC).

For the linker, when the TARGET_ARCH is ARM64 then the possible generated exe/dll binary machine ABI's are:

• AA64 (ARM64) (only objects compiled as ARM64 are linked with the default /MACHINE:ARM64),
• 8664 (x64) (ARM64x) (objects compiled as ARM64EC plus zero or more objects compiled as ARM64 are linked with /MACHINE:ARM64EC),
• AA64 (ARM64) (ARM64X) (objects compiled as ARM64 plus objects compiled as ARM64EC are linked with /MACHINE:ARM64X)

If MSVC auto-detection is bypassed, I'm not sure if the TARGET_ARCH is even known. I would have to review the source code. The user's environment may be unreliable with respect to figuring out how MSVC was configured.

The setup for MSVC has one target architecture option (ARM64).

The compiler has two output target ABI options (default ARM64 and ARM64EC).

The linker has three output machine ABI options (default AA64 (ARM64), 8664 (x64) (ARM64X), and AA64 (ARM64) (ARM64X).

The crux of the problem is the minimum amount of information required to determine uniquely a combination of compiler and linker flags, if possible.

The logic above might work when the desired output exe/dll format is AA64 (ARM64) (ARM64X) which would need to compile for both ARM64 and ARM64EC objects and link with /MACHINE:ARM64X (i.e., MSVC target ARM64, compiler default and /arm64EC, linker /MACHINE:ARM64X).

A desired output exe/dll format of 8664 (x64) (ARM64) would need to compile ARM64EC objects and link with /MACHINE:ARM64EC (i.e., MSVC target ARM64, compiler '/arm64EC, linker /MACHINE:ARM64X`).

The default ("do nothing") output exe/dll format of AA64 (ARM64) is a traditional build with no additional compiler or linker flags.

One new construction variable (e.g., MSVC_ARM64_MACHINE) could take on three values:

• ARM64: default compiler (no added) flags and default linker (no added) flags
• ARM64EC: compiler /arm64EC and linker /MACHINE:ARM64EC
• ARM64X: compile objects with both default and /arm64EC flags and link with /MACHINE:ARM64X (as described above)

There is something about this that makes me wary, but I can't put my finger on it yet.

If it is determined that automatically compiling both object ABI's at the same time is a bad idea, then it would probably require two construction variables so the compiler and linker flags could be specified independently at which point it might be just as easy to use the compiler and linker flags directly.

[LeonarddeR]

I wrote a very small example that I needed to test how windows object hooks behave on the several systems. You can find the code at https://github.com/LeonarddeR/scons_arm64_poc

In short, relevant findings are:

1. There's no point in creating an arm64x .exe. It will just run the native arm64 code. arm64x is therefore only relevant for libraries.
2. It is actually pretty easy to setup the environments necessary and a combination of 3 environments should be enough to produce whatever you need.
3. I hoped that hooking with a arm64x library would work both in arm64 and arm64ec/x64 applications, but unfortunately that's not the case. That makes it less interesting for me personally to generate arm64x libraries.

[bdbaddog]

A couple notes on your example SConstruct.
You shouldn't need to specify tools=.. at all in your example.
msvc is part of the default list on windows.
default is the default list so none of that is needed.

[jcbrill]

1. There's no point in creating an arm64x .exe. It will just run the native arm64 code. arm64x is therefore only relevant for libraries.

There does not appear to be much utility in producing arm64x executables instead of arm64 executables with regard to arm64x libraries.

The only thing that comes to mind is that an arm64x executable would indicate during linking if all of the symbols for both arm64 and arm64ec were defined instead of individually linking arm64ec and arm64 executables.

2. It is actually pretty easy to setup the environments necessary and a combination of 3 environments should be enough to produce whatever you need.

Thanks!

I heavily modified your example to produce more dlls and executables. I downloaded the example directly as a source archive rather than cloning the repository.

There are a number of arm64ec executable and library combinations possible.

It is also possible to create an Arm64X pure forwarder DLL that has no implementation and automatically forwards to an Arm64 DLL or an x64|Arm64EC DLL.

See https://learn.microsoft.com/en-us/windows/arm/arm64x-build for details.

Creating the forwarding DLLs for your examples was pretty straightforward and involved a number of manual steps:

• create the def files from the dlls:

  minwgw64\bin\gendef.exe libx64.dll
  minwgw64\bin\gendef.exe libarm64ec.dll
  minwgw64\bin\gendef.exe libarm64.dll
  

• rename libx64.def to libx64-forwarder.def and manually edit the contents:

  EXPORTS
  injection_initialize = libx64.injection_initialize
  injection_terminate = libx64.injection_terminate
  

• rename libarm64ec.def to libx64ec-forwarder.def and manually edit the contents:

  EXPORTS
  injection_initialize = libarm64ec.injection_initialize
  injection_terminate = libarm64ec.injection_terminate
  

• rename libarm64.def to libarm64->forwarder.def and manually edit the contents:

  EXPORTS
  injection_initialize=libarm64.injection_initialize
  injection_terminate=libarm64.injection_terminate
  

The manual steps could probably be automated. It could also be problematic based on name mangling etc.

SCons was then used to create the import libraries and build the forwarding dlls.

Two pure arm64x forwarding dlls were built:

1. libarm64x-arm64ec.dll which forwards to libarm64ec.dll and libarm64.dll.
   See item 3 in the list below.

2. libarm64x-x64.dll which forwards to libx64.dll and libarm64.dll.
   See item 5 in the list below.

The lib.cpp and app.cpp source files were modified to print the ABI type string passed as a define when compiling. When run, the ABI of the executable and ABI of the DLL are printed to the console as a prefix of the existing printed messages.

The arm64ec executable and library combinations produced are:

1. ARM64EC exe -> ARM64EC [libarm64ec.dll]

   ...
   ARM64EC Initializing injection...
   ARM64EC injection_initialize
   ...
   ARM64EC injection_terminate
   ARM64EC Injection terminated.
   ...
   

2. ARM64EC exe -> ARM64EC [libarm64x.dll(arm64, arm64ec)]

   ...
   ARM64EC Initializing injection...
   ARM64EC injection_initialize
   ...
   ARM64EC injection_terminate
   ARM64EC Injection terminated.
   ...
   

3. ARM64EC exe -> libarm64x-arm64ec.dll -> ARM64EC [libarm64x-arm64ec.dll(arm64, arm64ec) to libarm64ec.dll]

   ...
   ARM64EC Initializing injection...
   ARM64EC injection_initialize
   ...
   ARM64EC injection_terminate
   ARM64EC Injection terminated.
   ...
   

4. ARM64EC exe -> x64 [libx64.dll]

   ...
   ARM64EC Initializing injection...
   x64 injection_initialize
   ...
   x64 injection_terminate
   ARM64EC Injection terminated.
   ...
   

5. ARM64EC exe -> libarm64x-x64.dll -> x64 [libarm64x-x64.dll(arm64, x64) to libx64.dll]

   ...
   ARM64EC Initializing injection...
   x64 injection_initialize
   ...
   x64 injection_terminate
   ARM64EC Injection terminated.
   ...
   

The arm64x executable and library combinations with the arm64x forwarding dlls are:

1. ARM64 -> libarm64x-arm64ec.dll -> ARM64 [libarm64x-arm64ec.dll(arm64, arm64ec) to libarm64.dll]

   ...
   ARM64 Initializing injection...
   ARM64 injection_initialize
   ...
   ARM64 injection_terminate
   ARM64 Injection terminated.
   ...
   

2. ARM64 -> libarm64x-x64.dll -> ARM64 [libarm64x-x64.dll(arm64, x64) to libarm65.dll]

   ...
   ARM64 Initializing injection...
   ARM64 injection_initialize
   ...
   ARM64 injection_terminate
   ARM64 Injection terminated.
   

3. I hoped that hooking with a arm64x library would work both in arm64 and arm64ec/x64 applications, but unfortunately that's not the case. That makes it less interesting for me personally to generate arm64x libraries.

Perhaps the forwarding examples (items 3 and 5 above) are closer to what you were hoping to achieve.

Personal note: I am time limited for the next three weeks. Reponses might take longer than usual.

[gexgd0419]

3. I hoped that hooking with a arm64x library would work both in arm64 and arm64ec/x64 applications, but unfortunately that's not the case. That makes it less interesting for me personally to generate arm64x libraries.

This is probably referring to a DLL injection technique that uses Windows hooks, such as SetWindowsHookEx.

Most of the hooks are in-process hooks, which means that the system will load (inject) the DLL that contains the hook function into other processes, so that the hook function can run in the same process where the specific event is happening.

Of course, the DLL has to have the same architecture and bitness as the process being injected, or the injection will fail, and the system will fall back to running the hook function in the hook owner's process instead.

Although an Arm64X DLL can be loaded by both x64/Arm64 processes and Arm64 processes, if you try to use an Arm64X DLL as the hook DLL, only the part of the DLL that can be loaded into your process can be injected. For example, if an Arm64 process installs an Arm64X hook DLL, then the Arm64EC part of the Arm64X DLL will be discarded, and it won't be injected into Arm64EC processes, although technically they are compatible.

[LeonarddeR]

Thank you for your extensive overview.
It is new to me that you can create a forwarder to arm64 and x64. Will that forwarder also load on x64 Windows?

[gexgd0419]

The forwarder was new to me as well. I'm not exactly sure of what real-world benefits might be realized.

Unlike x86 binaries where file system paths and registry paths are redirected, X64 binaries and ARM64 binaries are not getting any redirection. This means that you cannot have separate file paths for an X64 version and an ARM64 version of a COM component DLL, because they share the same registry key. So if you are writing an in-process COM DLL, and want the component to be able to be loaded by both x64/ARM64EC and ARM64 processes, you will have to use an ARM64X DLL.

[gexgd0419]

Perhaps, it might be useful if there was divergence later down the arm64 native and x64 native dependency chains.

If some library your DLL depends on only have X64 and ARM64 versions but no ARM64EC versions (maybe some closed-source SDK), you will not be able to compile your DLL in ARM64X. You will either have to load that library dynamically, or use a forwarder DLL and separate your DLL into X64 and ARM64 parts.

[jcbrill]

Thank you for the additional context. It is appreciated.

Typically, when all of the source is available and an ARM64X DLL is needed, would the ARM64EC DLL be built as well even if it is not necessarily required?

For example, does it make sense to build DLLS for ARM64, ARM64EC, and ARM64X where the ARM64EC DLL might not be necessary but since the object files have to be compiled to produce the ARM64X DLL the ARM64EC DLL it might as well be generated?

It almost seems easier just to produce all three libraries (ARM64, ARM64EC, and ARM64X) if ARM64X is the goal.

[gexgd0419]

Both Arm64EC and Arm64 will be compiled, but I think that whether you get an Arm64X DLL or an Arm64EC DLL depends on the linker parameters.

An Arm64X DLL is like a combination of an Arm64EC DLL and an Arm64 DLL, and it can be used as either of them. It's even possible with some #ifdefs to make the Arm64EC part and the Arm64 part behave differently.

For standalone software, usually either x64 plus Arm64, or x64 plus Arm64EC if pure Arm64 is not possible, would be enough. Arm64X would be a little overkill.

For plugin DLLs, or DLLs that can be loaded by any other process, you would want to make it as compatible as possible, so Arm64X DLLs may be useful.

[jcbrill]

I didn't really ask the question very well.

Assuming an ARM64X DLL is the goal, one set of steps could be:

1. compile source as ARM64 objects (compiler default)
2. link ARM64 objects to produce native ARM64 DLL (linker default)
3. compile source as ARM64EC objects (compiler: /arm64ec)
4. link ARM64EC objects to produce ARM64EC DLL (linker: /MACHINE:ARM64EC)
5. link ARM64 objects and ARM64EC objects to produce ARM64X DLL (linker: /MACHINE:ARM64X)

The earlier question was: does Step 4 make sense if the objects are being compiled anyway and end-goal target is ARM64X?

If generating the ARM64EC DLL is not a big deal:

1. a SharedLibary(...) for ARM64 could be built (steps 1 and 2),
2. a SharedLibrary(CCFLAGS="/arm64ec", LINKFLAGS="/MACHINE:ARM64EC") for ARM64EC could be built (steps 3 and 4), and
3. a SharedLibrary(LINKFLAGS="/MACHINE:ARM64X") for ARM64X could be built (step 5) using the source objects/artifacts (directly, processed, and/or configured by the user) from the previous two library builds.

The ultimate question though is: Is there any benefit in attempting to have SCons automatically add the flags for ARM64EC (CCFLAGS=/arm64ec, LINKFLAGS=/MACHINE:ARM64EC) and for ARM64X (LINKFLAGS=/MACHINE:ARM64X) to a user's environment?

In a perfect world, when the msvc and mslink tools are setup, it can be determined that the compiler and linker flags should be added and are likely to succeed. Unfortunately, real-world usage is messy. One thing that makes this harder is that SCons may not be calling the MSVC batch files (i.e., the user requested bypassing calling the batch files).

From an MSVC standpoint, there are minimum toolset version and windows sdk version restrictions. What is successful on one computer may not be successful on another computer depending on the environment.

Remote diagnosis of build failures due to issues with the build environment can be challenging and time consuming.

Edit: Apologies for the typographical errors and looking like it was typed by a 100 monkeys.

[jcbrill]

Attached archive of my modified files used for testing:
scons_arm64_poc_jbrill.zip

It should just work (I hope).

Notes:

• Compiled from an x64 VM so the host architectures are all amd64.
• All source files have minor modifications including adding the ABI strings to the console messages and adding a sleep time instead of "press enter key" so that all executables could be run from a windows batch file unattended.
• There are way more SCons environments than necessary which is the result of piecemeal additions that would not interfere with already "working" combinations.
• As the SConstruct file grew in size, an unfortunate side effect is that the environment definitions may have moved farther away from their uses.
• Personal development/testing configuration:
  • Source directory on a Windows 10 host.
  • Compiled in the host source directory from a Windows 11 VM with a mapped drive to a parent directory on the Windows 10 host.
  • ARM64 testing in the host source directory from a physical ARM64 box with a mapped drive to the Windows 10 host.