driver-driver and FAT builds

[iains]

We now have several scenarios in which a FAT (multiple architecture slices in a binary) build is relevant;

1. Darwin8 [i686, ppc] - I think that the 64b support is incomplete here (at least for ppc), I'd say it's a small case we'd want to use it.
2. Darwin9 [i686, ppc, x86_64, ppc64]
3. Darwin10 [i686, ppc, x86_64]
4. Darwin11-17 [i686, x86_64]
5. Darwin20-21 [x86_64, aarch64]

Three avenues I've partially explored:

A* Apple's driverdriver.c: sadly, this became incompatible with the GCC codebase around GCC-4.X when the handling of command line options was shifted to a new scheme.

B* Somewhere I have a "poor mans driver-driver" (basically a shell script that does the same kind of things).

C* I have also considered using the clang driver code but arranging for it to use GCC instances as the toolchains (this should be possible).

A needs a bit of a re-write and then a check on reintegration.
B is, obviously, slow but easy to hack on
C means dragging in a huge codebase, and maintaining something out-of-tree, since I don't see that being something upstream will want.

So they are all possible ..

[kencu]

fred has updated driverdriver.c somewhat already, here:

https://github.com/devernay/macportsGCCfixup/blob/master/driverdriver.c

[iains]

There are a few parts to the rewrite.

1. update the codebase to use the current mechanism for options handling (which is nicer than the one used in the old code)
2. update the actual options handling to reflect the differences in the built X86 toolchains that come after gcc-4.2
3. update the actual options handling to add the arm64 toolchains.

(maybe 3 will need us to update some options handling in the actual compiler, I have begun the work to recognise the lew linker version option .. )

The key to this work is to be highly organised - because there are a lot of differences across the years/archs/supporting tools and sometimes mistakes can cause rather subtle problems ....

Note that this also has to work (for GCC) for fortran, Ada, D etc. which Apple have never supported in either driver driver or clang - but those are important to FSF GCC users (in some cases more important than even the C-family).

One thing would be to start by tabulating exactly what the clang driver does (for C-family) in response to the various inputs.

e.g. supposing we had the driver driver for gcc-10 ...

gcc-10 -arch ppc -arch x86_64 -fsome-ppc-only-option -fsome-x86-only-option ... ..

[iains]

Note that this also has to work (for GCC) for fortran, Ada, D etc. which Apple have never supported in either driver driver or clang - but those are important to FSF GCC users (in some cases more important than even the C-family).

Actually, this is probably a relevant point against using the clang driver code, since it seems that flang is a separate driver ... so we probably want to factor the operations into those that are common across languages (i.e. more or less the backend ones) and those that are language-specific (where there are already separate drivers for c-family, fortran, ada and D .. and it looks like modula-2 has it's own too).

the point is it seems like it should be an easy job .. but that's not the case, it's actually quite tricky which is why no-one has knocked up a replacement over some spare weekend ;)

[barracuda156]

the point is it seems like it should be an easy job .. but that's not the case, it's actually quite tricky which is why no-one has knocked up a replacement over some spare weekend ;)

@iains I get the complexity now. Initially I kinda wondered why not just adapt a bit already existing code from gcc42. But the task is indeed far from trivial.

[barracuda156]

Note that this also has to work (for GCC) for fortran, Ada, D etc. which Apple have never supported in either driver driver or clang - but those are important to FSF GCC users (in some cases more important than even the C-family).

@iains By the way, does D work on PPC? I remember your link to libphobos, but not sure if that was completed.

[iains]

D is currently broken on 32bit Darwin (i686 and ppc), there is some bad code-gen which upsets the linker and breaks bootstrap. I don't know if @ibuclaw is intending to fix that, or if it's down to someone from the Darwin side (in which case, I don't have any spare cycles at present, as you know). Once 32b is working again, there were some additional things needed for support of long double on PPC - but I had draft patches.

[kencu]

@devernay looping you in

[barracuda156]

We now have several scenarios in which a FAT (multiple architecture slices in a binary) build is relevant;

1. Darwin8 [i686, ppc] - I think that the 64b support is incomplete here (at least for ppc), I'd say it's a small case we'd want to use it.
2. Darwin9 [i686, ppc, x86_64, ppc64]
3. Darwin10 [i686, ppc, x86_64]

I would add ppc64 to the list of Darwin10 – not necessarily for Darwin10 PPC (which at the moment looks like a lost case for ppc64), but at least for Darwin10 Intel, in order to cross-compile binaries that would run on 10.6 PPC as 32-bit and on 10.5 as 64-bit.

@Blzut3 mentions he could cross-build ppc64 binaries.
I am not sure if example from here works as is: https://github.com/Blzut3/xcodelegacy

xcodebuild MACOSX_DEPLOYMENT_TARGET=10.6 SDKROOT=macosx10.6 GCC_VERSION=4.2 CLANG_CXX_LIBRARY=libstdc++ \
 ARCHS="ppc7400 ppc64 i386 x86_64"

But is should work with SDKROOT=macosx10.5 which contains ppc64 code.

[Blzut3]

To be clear, I always use CMake and Makefiles to build. Not sure if there's any limitations in Xcode that would cause trouble, although I suspect it can be made to work.

As an aside I've personally found building universal binaries in one pass to be more hassle than it's worth. You're mileage may vary, but I've had enough instances of builds working fine built as a single arch and then failing when built together (due to the CMake project generating different flags for different arches I assume). Certainly solvable problem, but my preferred method of building universal binaries is to simply do separate builds and use the lipo tool to merge them. Then you can mix and match SDKs and library versions with ease to optimize for different relevant OS versions. Particularly useful for me since it seems Apple will apparently notarize a binary, even one with PowerPC parts, as long as the arm64 (or x86_64) part meets the requirements.

[barracuda156]

but my preferred method of building universal binaries is to simply do separate builds and use the lipo tool to merge them.

Yeah, that is what @kencu suggested to do on a number of instances.

[kencu]

For low-volume situations, just building manually and lipoing yourself is very practical, and much much simpler. I have done this many times too (and which is in fact exactly what I have recommended to barracuda a few times when he is very stuck trying to make something happen in MacPorts that just won't happen transparently because driverdriver.c doesn't exist for newer gccs at present).

But for a high-volume distro like MacPorts that is up into the millions of universal builds of 20,000+ ports on many thousands of different systems, something more standardized is needed.

The muniversal PortGroup basically mimics the manual build-separately-and-lipo workflow, and certainly works in some cases. It is not presently set up to use different compilers in different build folders, so breaks down with gcc cross-processor builds at present.

Having one compiler driver that can build either the separate archs individually, or both archs at once transparently, becomes essential to making this manageable. And it is also how clang works, so all the tooling in MacPorts and all the experience of the members and the contributors is down that path.

So for MacPorts, for them to make this work, they would need to resuscitate driverdriver.c.

[barracuda156]

So for MacPorts, for them to make this work, they would need to resuscitate driverdriver.c.

@kencu Have you kept a copy of your repo, by the way? https://github.com/kencu/macportsGCCfixup (link is dead atm)
Ref from https://trac.macports.org/ticket/62332#comment:5

[kencu]

no, deleted it. it was identical to upstream.

[barracuda156]

no, deleted it. it was identical to upstream.

Got it, thanks.

[iains]

I can also dig out my driverdriver.sh (although that has no support for arm64 at present)
will probably not be until after llvm-15 branches, too much to do until then.

[barracuda156]

I can also dig out my driverdriver.sh (although that has no support for arm64 at present)

@iains That will be definitely helpful, whenever you get time to find it.

[kencu]

for a powerful starting point, possibly easier, just the ability to support one multlib gcc building i386/x86_64 or ppc/ppc64 would go a very long way to getting us going.

[barracuda156]

@iains Given that now modern Apple uses two different archs, does it imply that the interest in fixing FAT for gcc is gonna be revived? It looks like it is not a one-person project to accomplish, given all constraints.

[iains]

very definitely, I've already had several enquiries .. but interest != available time :) ..
... once the current $dayjob stuff gets quieter, then maybe we can explore some options.

I'd expect the script to work for i686/x86_64 and ppc/ppc64 and x86_64 aarch64 (providing that the toolchains are built and installed correctly .. hacks in positioning or other feats of gymnastics will almost certainly break the script)

[barracuda156]

providing that the toolchains are built and installed correctly

@iains What is the recommended pattern? (I mean, outside of Macports, of course.)
It would be easier indeed to use same assumptions as existing code has rather than going through all sorts of pain every time to fix paths.

[iains]

actually, the default layout was used by the Apple gcc-4.x compilers (it is academic for clang except for the issue of multiple runtimes or SDKs for cross cases).

 prefix/
   bin/
   lib/
  libexec/
  etc....
  some-other-triple/ (i.e. a cross compiler)
    bin/
    lib/
    libexec/
    etc.
  another-non-native-triple/
   bin/
   lib/
  libexec/
   etc.

You do not need to do anything with a standard GCC configure if --target=some-other-triple --host=... --build=... are set.

The way to make this work seamlessly is to install (or symlink) the 'binutils' (ar, as, dsymutil, ld, lipo, libtool, nm, objdump, otool, ranlib, strip, strings, size) into some-other-triple/bin/

but here is not the place for a tutorial on building cross-compilers ...

[barracuda156]

but here is not the place for a tutorial on building cross-compilers ...

@iains I have moved respective topic into cross-compilation thread: #12 (comment)

[kencu]

for fun, here’s the last such script from libsdl that supported ppc:

https://github.com/SDL-mirror/SDL_ttf/blob/cc835c11e199995f228b3cf0e3f4d0a2760f616d/gcc-fat.sh

[barracuda156]

for fun, here’s the last such script from libsdl that supported ppc:
https://github.com/SDL-mirror/SDL_ttf/blob/cc835c11e199995f228b3cf0e3f4d0a2760f616d/gcc-fat.sh

@kencu Is it something similar to what Macports does?

[iains]

mine is more like the driver-driver code..

(this is a patch that adds the base script to contrib/ - which is the appropriate place for this - I think this probably works for simple cases, but I expect it's easy to break and likely is missing stuff)

0001-contrib-fat-gcc.sh.patch.txt

[iains]

One tedious, but necessary part of the work is to figure out the installation to make sure that tool names and suffixes agree and to install the "FAT" driver in place of the non-canonically-named driver.

I am thinking of making a cmake script to wrap around the build process to automate cross compiler builds and make sure that what was intended to happen does happen.

[iains]

having just re-read the code, I realise why it's not been touched since 2015 .. it's horribly inefficient .. lots of spawning of processes to find things out .. even as a script it could be better .. but probably better to check viability of the other options first (the script is very much a "poor man's fat compiler driver") ..

[iains]

there are a couple of actual bugs, but I tried this on i686-darwin9 with a powerpc-darwin9 cross compiler and it produced usable 4-arch binaries. I also tried it on x86_64-darwin20 with an aarch64-darwin20 cross-compiler and that worked ..
.. so, maybe I'll take this an turn it into c++ and see if we can minimise the process launches and general number of syscalls - possibly with an early exit in the zero/one arch case, so that we do not penalise the most frequent compilations too much.