"Deallocation of a pointer not malloced," or: Pointers crossing the libstdc++ chasm

[evanmiller]

I would like some clarity on the bug reported here:

https://trac.macports.org/ticket/63551

@kencu indicated that it's a known issue introduced between GCC 7.4 and GCC 7.5 – as I understand it, the system's libgcc and the compiler's libgcc have some disagreement when it comes to keeping track of allocated pointers. I understand the warning is harmless, but it is rather annoying. Is this something that can be fixed in GCC 7.5? Is there another bug report where I can read and learn more about it? Do I have any options besides the DYLD_LIBRARY_PATH / wrapper script workaround described in the ticket? Thanks in advance for any light you can shine on this dark corner of the run-time.

[kencu]

Not harmless! Causes frequent outright crashes.

Iain has a fix in mind, involving namespaced symbols (just like llvm uses in libcxx).

All of it takes time, interest, debugging, etc and if you haven't yet noticed, pretty much everything to do with gcc/libgcc on darwin/macOS is Iain's one-man-show here (with helpers at times)!

[kencu]

By the way, the place I seem to see it most often is involving software that uses facets. I read that there are two different symbols in libgcc for facets, one for pre-c++11 and one for post-c++11, and I have surmised that might be why that is where the error shows up (cmake, many others), as the pre-c++11 symbol might clash with the ancient one in the system's implementation.

I have no illusions though -- facets are just the tip of the iceberg here. It needs a proper fix if things are to go on. Apparently the only really good fix is to replace some parts of the system software on 10.4 and 10.5 with newer parts -- hah hah hah. Try selling that one ;>

[kencu]

this MacPorts ticket is quite related:

https://trac.macports.org/ticket/59832

and less so perhaps this ticket:

https://trac.macports.org/ticket/59706

[iains]

perhaps the rosetta dyld is the same on 10.6 and 10.5 - it makes some kind of sense, since rosetta was intended to be a backwards compatibility function, not a new development platform.

... anyway, we need to solve this, it is (as always) made complex by the fact that we have a broken system in the wild, but still people will no doubt not want the fix to require existing exes to be re-built ...

[barracuda156]

... anyway, we need to solve this, it is (as always) made complex by the fact that we have a broken system in the wild, but still people will no doubt not want the fix to require existing exes to be re-built ...

@iains I am actually very much willing to re-build system components, since that can a) substantially improve 10.6 PPC, b) implement ppc64 support there and in Rosetta (I saw your old comment in some thread considering ppc64 in Rosetta).
I just do not know how to do it in a correct way :)

Apple Opensource has sources for many (most?) OS components, but Darwinbuild is hopelessly broken, and I assume we cannot build anything for MacOS itself in Macports, linking will turn a disaster.
Is there a procedure for re-building? I have your GCCs built outside of Macports, but no specific build environment otherwise.

[iains]

nothing so complicated as that :)

What I mean is that we might make a fix that means existing macports exes would need to be rebuilt .. but obviously that is not the best solution (I managed to find a way to work around a similar problem with libgcc_s .. so .. let's see)

[barracuda156]

What I mean is that we might make a fix that means existing macports exes would need to be rebuilt .. but obviously that is not the best solution (I managed to find a way to work around a similar problem with libgcc_s .. so .. let's see)

@iains I doubt that tweaking OS itself will be acceptable in Macports (after all, immediate issue is that it is not universally portable, and we end up with different versions and no one to maintain ports for those).

What I meant is I am eager to work on that myself – and interested to contribute, if you, @catap or @kencu will do something to that end for 10.5/10.6.

[iains]

@iains I doubt that tweaking OS itself will be acceptable in Macports (after all, immediate issue is that it is not universally portable, and we end up with different versions and no one to maintain ports for those).

That is a long-term interest - basically to see if a PPC system with a newer set of OS components [incl kernel] is viable (but it's loooow on the TODO at the moment). Of course, we would still be tied to the last version of the closed-source Frameworks, so limited in what can be achieved.

[iains]

I would like some clarity on the bug reported here:

https://trac.macports.org/ticket/63551

@kencu indicated that it's a known issue introduced between GCC 7.4 and GCC 7.5 – as I understand it, the system's libgcc and the compiler's libgcc have some disagreement when it comes to keeping track of allocated pointers. I understand the warning is harmless, but it is rather annoying. Is this something that can be fixed in GCC 7.5? Is there another bug report where I can read and learn more about it? Do I have any options besides the DYLD_LIBRARY_PATH / wrapper script workaround described in the ticket? Thanks in advance for any light you can shine on this dark corner of the run-time.

The problem is that we can have a situation in which we:

• compile C++ code with "new gcc" (any version), which then refers to "new libstdc++" in the shared library dependencies.
• The code also has dependencies on system libraries and/or frameworks which, in turn, have dependencies on /usr/lib/libstdc++.

So now we have two copies of libstdc++ in the same process. In principle, dyld can tell the difference "two level namespace" because the two libraries are installed at different paths. However, at least on 10.5 (darwin9) and earlier it seems that dyld is not that smart and only allocates one slot for each symbol. So we can have the situation in which and object is constructed in one libstdc++ and then we try to destroy it in the other (which obviously has no idea about the object, and then we get reasonable complaints about double free()s when the memory is deallocated). Hence the sub-title about 'pointers crossing library boundaries'.

There are two painful solutions:

1. Use a DYLD_LIBRARY_PATH pointing to "new libstdc++" installation path. Because of the way that DYLD_LIBRARY_PATH works, the two-level namespacing is discarded and all dependencies on libstdc++ will end up using the new one. Problem solved.
2. Rebuild all the dependencies to use "new libstdc++"; this is not always feasible, since we do not have source code for all the libraries or frameworks in the system - but it can be feasible sometimes (as @barracuda156 references, I've done this [for example] with newer ld64 versions that use libxar).

The most likely proper solutions:

1. I think the 'best correct' solution is to use an inline namespace for 'new libstdc++' so that it has distinct symbols from /usr/lib/libstdc++. That does mean rebuilding stuff that you want to work properly
2. IFF we had someone with huge amounts of time on their hands, then they could make a way to rebuild "new libstdc++" so that only the new symbols were present and the old ones were reexported from /usr/lib/libstdc++.

For my part, I have test versions of (1) for master, 11, 10 and 7.5 .. but they are not ready yet,

and I am going to repeat:
in order to fix the broken cases, you will have to rebuild everything using libstdc++ (but at least you will be able to drop the DYLD_LIBRARY_PATH wrappers).

Additional notes:

FWIW, it seems that dyld on 10.6+ is much better behaved, at least, my simple reproducer for the crash we see on 10.5 works without problems there.

between 10.8 (there are not many uses in 10.7, but ISTR one or two) and 10.13, the only users of /usr/lib/libstdc++ would be third party code (no system uses) - so the chances of clashes are small ---- UNLESS you start using clang -stdlib=libstdc++, and shoot yourself in the foot that way ;)

After 10.14 this is academic - since nothing will use /usr/libstdc++ (clang does not work with -stdlib=libstdc++ because the headers are not installed), After 11, the library is omitted as well.

So, sorry, once again, 'watch this space' - I have now fixed libgcc_s on master, and those fixes will be backported, but the libstdc++ fix is still pending a few nanoseconds to work on it.

edit:
It is not really "introduced by later versions of GCC" - the issue is with two libraries with some shared symbols in the same process (and, at least to some extent, is a bug with dyld not honouring the two-level namespaces). We see it more on later stuff, primarily because we are now compiling more c++11 code and that is pulling in the C++11 symbols [so actually from gcc-4.8+ if one wants to associate a version].

[iains]

I case you care .. ;) ...

To see this "happening" .. you can make a simple exe that exhibits the problem and then:

DYLD_PRINT_LIBRARIES=1 DYLD_PRINT_BINDINGS=1 ./exe

that produces a lot of output - but I expect that you will see some DTOR call that triggers the behaviour.

[barracuda156]

FWIW, it seems that dyld on 10.6+ is much better behaved, at least, my simple reproducer for the crash we see on 10.5 works without problems there.

@iains 10.6.8 Rosetta is quite problematic here.

Doxygen basically does not work, and many ports using it fail with these malloc errors, example: https://trac.macports.org/ticket/64610

Just now ran into the same kind of error with py-shiboken: https://trac.macports.org/ticket/65708

[ 96%] Running generator for 'shiboken'...
cd /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-shiboken/py310-shiboken/work/Shiboken-1.2.4/shibokenmodule && /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-shiboken/py310-shiboken/work/build/generator/shiboken --project-file=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-shiboken/py310-shiboken/work/build/shibokenmodule/shibokenmodule.txt
shiboken(63722,0xa0bcc620) malloc: *** error for object 0xa0f15754: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
shiboken(63722,0xa0bcc620) malloc: *** error for object 0xa0f157c4: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Detecting inconsistencies in typesystem...   [OK]
/bin/sh: line 1: 63722 Bus error               /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-shiboken/py310-shiboken/work/build/generator/shiboken --project-file=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-shiboken/py310-shiboken/work/build/shibokenmodule/shibokenmodule.txt
make[2]: *** [shibokenmodule/shiboken/shiboken_module_wrapper.cpp] Error 138
make[2]: Leaving directory `/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_python_py-shiboken/py310-shiboken/work/build'
make[1]: *** [shibokenmodule/CMakeFiles/shibokenmodule.dir/all] Error 2

@kencu confirmed that on Tiger with gcc7 Doxygen works fine though: macports/macports-ports#15763 (comment)
But on 10.6 with gcc11/gcc12 we got a problem.

[barracuda156]

and I am going to repeat: in order to fix the broken cases, you will have to rebuild everything using libstdc++ (but at least you will be able to drop the DYLD_LIBRARY_PATH wrappers).

@iains Does this imply rebuilding dependencies of a failing port with a new GCC (which are otherwise built with gcc-4.2)?

P. S. @kencu This cannot be a solution for Macports, so I am not gonna try introducing such fixes :) But I need to fix some ports locally, otherwise I cannot work with their dependents.

[kencu]

we never saw this issue until MacPorts updated to libgcc 7.5. Then we saw the same facet error in many different ports.

rolling back to libgcc 7.4 made those errors disappear.

But eventually too many ports, esp ICU, would not build with gcc 7.4/libgcc 7.4 so we had to move on.

Academic, probably, as we are not going back now to change it, but something changed between gcc 7.4 and 7.5 to make this underlying latent issue start to show up in many different software builds, and disappeared when gcc was rolled back.

I tried to find the commit that did it a few years ago, but the bisecting was too time consuming for me to finish, knowing it wouldn’t matter anyway.

[iains]

interesting - a quick check here says that the library exports the same symbols for 7.4 and 7.5 (so, presumably, not an issue with symbol visibility). I'd guess that the change must be in some static CTOR - since the clash seems to happen very early.

Maybe a change in headers (I didn't compare those so far) or in handling of initialisation.

In some ways, it would be nice to know what triggered it (I made myself a very simple reproducer) .. unfortunately, I don't have enough machine bandwidth to bisect either (unless I can reduce the commits to test to a small number).

However, as I think you noted elsewhere, the problem was/is latent, so a solution is needed going forward.

[kencu]

I would be interested to see your simple reproducer.

I did all this two years ago or so -- I should try to repro it again now on 10.5 Intel to see if what I said still holds up. Other things have changed too (the cctools we use, in particular, has been updated a few times.

[iains]

A "hello world" using iostream (which has a global init).

#include <iostream>
using namespace std;

int main()
{
    string msg = "Hello, C++ world. from ";
    cout << msg << __VERSION__ << " using iostream " << endl;
}

then link with libxar (even though we have no dependency on it - ld64 will add it if we say so - so long as we do not 'dead strip dylibs').

$   /opt/iains/powerpc-apple-darwin9/gcc-5-5r0/bin/g++ /source/test/hello.cc -o hc -std=c++11 -lxar

$ ./hc
hc(83885) malloc: *** error for object 0x8a33dc: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
hc(83885) malloc: *** error for object 0x8a336c: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
Hello, C++ world. from 5.5.0 using iostream

and, as you can see, the problem manifests on earlier versions - and later
(5.5 yes, 6.5 apparently no, 7.4 fails here for me, 7.5 yes, 8.5 yes).

Not sure why 6.5 is apparently OK - it might be "dumb luck" with the ordering of initialisers.

[iains]

and the output of
DYLD_PRINT_LIBRARIES=1 DYLD_PRINT_BINDINGS=1 ./hc

contains (a lot of output) .. but the salient lines are:

dyld: bind: libstdc++.6.dylib:__ZdlPv$lazy_ptr = libstdc++.6.dylib:__ZdlPv, *0x0036b6c8 = 0x0025c2d4
dyld: bind: libstdc++.6.dylib:_free$lazy_ptr = libSystem.B.dylib:_free, *0x0036b668 = 0x9009aa4c
hc(419) malloc: *** error for object 0x8c73dc: Non-aligned pointer being freed

so it seems that lazy symbol resolution is somehow getting the wrong edition of libstdc++ to find operator delete(void*)

[iains]

having said that, the operator delete() might be secondary to some earlier call that ends up with the wrong pointer value (which is then passed to operator delete() )