PPC assembler and Darwin calling conventions vs ELF: advice needed

[barracuda156]

@iains Could you please refer me to some helpful materials to read beyond Apple documentation (which is far from being comprehensive) and IBM AIX docs (which do not exactly apply to Darwin)?

I have been in the process of fixing several compilers/langs for MacOS PPC: ruby, ghc, sbcl, ocaml. While certain progress has been achieved, there are issues with PPC assembler which are not trivial, at least for me :)

It is frequently the case that we have needed PPC code, but for Linux (ELF and/or ELFv2), sometimes LE-only, more often BE & LE. In many cases only ppc64 code is present, while ppc32 is ether dropped or never existed.

We can recall jit in TFF discussed in other thread here, v8 which I need very much for Stan language (AIX port of v8 exists, but not Darwin), go (not gcc-go, which is also broken anyway, but Google version) etc. There are also numerous small instances of similar problems, few of which I have fixed on my own or with others’ help, many of alike still remain unfixed.

To sum up the problem, everything above falls into either of the two cases:

1. There is an existing modern PPC code for Linux (rarely AIX), which has to be amended for Darwin ABI (cases of ruby, v8, go).
2. There is an old PPC code for Darwin which has to be somewhat changed to match the evolved structure of a compiler or a program (cases of ghc, sbcl, ocaml).

Could you suggest some reading and/or guidelines on how to go about such tasks? Beyond trivialities like slight differences between 32 and 64 bit instructions.

Few specific points to mention, which often pose a problem (I do not expect explanations here, but rather just wanna give some context):
– Apparently there are differences with respect to how certain registers are used between Darwin and ELF, but documentation offers little help. Are there some patterns here? There are some very old examples of a code that existed for both ABIs, but it is often not clear if register choice follows from ABI differences or just personal choices of those who wrote the code (likely at different time), or even errors.
– What should be done with code dealing with TOC, which is not used on MacOS?
– What is to be done with .cfi_ directives? Apparently they aren’t supported on Darwin (or at least I got complaints re invalid code with them). Like here: http://stffrdhrn.github.io/software/toolchain/openrisc/2020/12/13/cxx-exception-unwinding.html
Throwing them out together with other unsupported directives like .type removes build errors, but I am not sure that’s the way.
– Often there is .align statement in beginning of a file (usually .align 2 or .align 3), but while initially I assumed they simply correspond to 32 and 64 bit, respectively, I was wrong. This should be something simple, so I beg a pardon for a silly question.
– How do symbols of a form L+number work (apparently used for branching)? When I compared Darwin PPC vs Linux PPC code in the last version of ocaml which still had the latter, three-digit numbers after L looked arbitrary and did not fully match between the two.
– Sometimes there are numerical constants defined which differ for ELF 32 bit, ELF 64 bit and ELFv2 64 bit. It is not always clear which are to be used for Darwin. Example pulled from ocaml:

/* Stack layout */
#if defined(MODEL_ppc)
#define RESERVED_STACK 16
#define PARAM_SAVE_AREA 0
#define LR_SAVE 4
#define TRAP_SIZE 16
#define TRAP_HANDLER_OFFSET 0
#define TRAP_PREVIOUS_OFFSET 4
#define CALLBACK_LINK_SIZE 16
#define CALLBACK_LINK_OFFSET 0
#endif
#if defined(MODEL_ppc64)
#define RESERVED_STACK 48
#define PARAM_SAVE_AREA (8*8)
#define LR_SAVE 16
#define TOC_SAVE 40
#define TOC_SAVE_PARENT 8
#define TRAP_SIZE 32
#define TRAP_HANDLER_OFFSET 56
#define TRAP_PREVIOUS_OFFSET 64
#define CALLBACK_LINK_SIZE 32
#define CALLBACK_LINK_OFFSET 48
#endif
#if defined(MODEL_ppc64le)
#define RESERVED_STACK 32
#define PARAM_SAVE_AREA 0
#define LR_SAVE 16
#define TOC_SAVE_PARENT 8
#define TOC_SAVE 24
#define TRAP_SIZE 32
#define TRAP_HANDLER_OFFSET 40
#define TRAP_PREVIOUS_OFFSET 48
#define CALLBACK_LINK_SIZE 32
#define CALLBACK_LINK_OFFSET 32
#endif

https://github.com/ocaml/ocaml/blob/trunk/runtime/power.S

[iains]

@iains Could you please refer me to some helpful materials to read beyond Apple documentation (which is far from being comprehensive) and IBM AIX docs (which do not exactly apply to Darwin)?

Humm... yes, really it would be nice if we found some time to write down the differences between 'documented' and 'reality'.

The situation with any system that is "in the wild" is that the ABI is defined by the compiler(s) that built it - independent of what the documented ABI was supposed to be.

However, you should always start with the Apple ABI description - that is probably 80-90% of the story - and then consider adding notes where that is not clear (or was not implemented in reality).

So, over the years, I've gradually found such differences - and usually incorporated a suitable change in GCC to try and match the [Darwin9] system ABIs for ppc32 and 64.

This is not written down anywhere in a useful text form but, the test cases here:
iains/LLVM-7-branch@36548ee

and the ABI description coded here:
iains/LLVM-7-branch@d6671d5

collect things together somewhat.

Actually there was a pretty good match between GCC-4.0 and that branch (i.e. the two compilers had equivalent ABIs when running the compat testsuites in GCC).

The ppc64 ABI description:
iains/LLVM-7-branch@c448fc8
is less secure, the argument passing rules are _ completely_ different and much more complex.

I wish I had time to work on updating this stuff .. but not in the near future.

[iains]

edit: splits some of the answer up.

Could you suggest some reading and/or guidelines on how to go about such tasks? Beyond trivialities like slight differences between 32 and 64 bit instructions.

The Darwin ppc32 ABI is completely different from the ppc64 one.

Few specific points to mention, which often pose a problem (I do not expect explanations here, but rather just wanna give some context): – Apparently there are differences with respect to how certain registers are used between Darwin and ELF, but documentation offers little help.

if you look in gcc/config/rs6000/rs6000-logue.cc there are ascii art stack layouts for the supported targets, which are correct AFAIR.

Are there some patterns here? There are some very old examples of a code that existed for both ABIs, but it is often not clear if register choice follows from ABI differences or just personal choices of those who wrote the code (likely at different time), or even errors.

• Some directives (like .type) are ELF-only and you will always need to make adjustments for Mach-O assembler

• Hand-written asm can do anything with registers it likes, providing that it obeys the ABI w.r.t calling conventions, callee/caller saves, red zone etc.

• ELF != Mach-O PPC32 != PPC64

– What should be done with code dealing with TOC, which is not used on MacOS?

Nothing; R2 is a usable register - you can use it any way that fits with the ABI.

– What is to be done with .cfi_ directives? Apparently they aren’t supported on Darwin (or at least I got complaints re invalid code with them).

They are not supported (yet) because of
(a) no support in earlier assemblers,
(b) problems with llvm and compact unwind handling with .s file inputs
(c) problems, apparently, in ld64 handling of mixed compact and DWARF unwinding.

Often there is .align statement in beginning of a file (usually .align 2 or .align 3), but while initially I assumed they simply correspond to 32 and 64 bit, respectively, I was wrong. _This should be something simple, so I beg a pardon for a silly question.

.align just tells the assembler to align the next item accordingly - it has nothing to do with what the arch pointer size is. Quite often items will need to be aligned according to their size (e.g. int to 4 bytes, double to 8 and long double / vector to 16) ..

– How do symbols of a form L+number work (apparently used for branching)?

those are local symbols, they do not appear in the assembler's symbol table,

The bottom line is that it will take you some time, but you need to grok the differences between Mach-O (read the Apple doc) and sysV ABIs (read the baseline and then the ps [processor-specific] ABI) for x86 and ppc/ppc64.

no short cuts .. unfortunately ..

[barracuda156]

Possibly off-topic, but would it be useful for the poking around if we could run ppc64 Mac OS X in QEMU? I tried it a while back but I don't think it's supported at all yet.

Qemu is apparently quite broken for PPC. For instance, ppc32 host code is missing altogether in the latest version. I think we still have a PR hanging in Macports where I changed assembler instructions to match ppc32 ISA, but could not test that, away from native hardware since June.

Yeah, this: macports/macports-ports#14807
Disclaimer: it apparently builds with my patch (link there in comments), but no one knows if it works correctly.

[hamishmb]

I have an iBook G4 - maybe I could test it. I thought they removed PowerPC support for older PowerPC processor but not the G4 aka 750?

G4 == 74xx/75xx

Edit: Oops the G3 is the 750. I think the G3 is meant to be supported still, though. I think it was the 603 (?) that was dropped.

However, the point of QEMU is that we should be able to run powerpc code on x86 too.

G5 (ppc970) support is missing.

Also potentially of note is this from https://qemu-project.gitlab.io/qemu/about/removed-features.html:

ppc64abi32 CPUs (removed in 7.0)

The ppc64abi32 architecture has a number of issues which regularly tripped up the CI testing and was suspected to be quite broken. For that reason the maintainers strongly suspected no one actually used it.

It is possible, of course, to run ppc32 code on ppc64 hardware and to use ppc64 features in a limited manner .. quite a different thing from running ppc64 code.

[hamishmb]

Looks like someone other than me (I tagged my edit with Edit:) editing my comment above. That's fine, but it means I only noticed the nee information because I happened to check today - not ideal.

[iains]

hmm .. I intended to do a "quote reply" .. if it came out as an edit .. apologies (I added the comments about G4 == 74xx/75xx, and G5 (ppc970) support is missing.)

[hamishmb]

No worries :) I still need to actually file these issues.

[barracuda156]

@iains Thank you very much for detailed reply and references! I will go through your links first.

[iains]

additional observation;

non-"c-family" languages are, of course, free to choose a different ABI - and most likely do so (so much of what is written above might not apply) .. the only constraint being that they need to adhere to the c-family CC when calling system routines. Sometimes such languages use libffi to do that.

There are some constraints on vectors, however; if the VMX (altivec) registers are used, then the VSAVE register must be updated to reflect the registers in use, or they will not be preserved across context switches. This applies regardless of the language ABI in use.