corelibgen: Generate stub libraries with CFI to unwind ELF corefiles
   --------------------------------------------------------------------

If you have a core dump from a system with different shared library
versions then your debugger is likely to show corrupted backtraces for
stacks calling into those libraries.  You can fix this by giving a copy
of the libraries to the debugger (set solib-search-path in gdb).
Unfortunately this may not be possible, typically if they are on
inaccessible customer systems.

Corelibgen provides an alternative by locating the DWARF Call Frame
Information (CFI) information required by the debugger in the core file
and creating stub libraries.

The only requirement is that the core files must be created including
the library segments from the process's memory image.  This is not the
default on Linux but

  echo 127 > /proc/<pid>/coredump_filter

or a programmatic equivalent will tell the kernel to dump all of the
memory mapping types for process <pid>.  See core(5) for more details.

I frequently use corelibgen to generate stub libraries containing CFI
that gdb can then use to obtain correct stack traces for a core file
without having the appropriate libraries.  It works for glibc and other
libraries including the example C program and i386 and x86_64 asm
libraries below.

(This is extended from a program called wxcoretool which I wrote for my
employer, Kognitio (http://www.kognitio.com/), and posted to the gdb
mailing list at <https://sourceware.org/ml/gdb/2013-05/msg00134.html>.)


Implementation
--------------

The CFI is present in the library in the process's memory image, because
C++ uses it to unwind when processing exceptions.  If these segments are
present in the core file then corelibgen can see them.

Identifying the CFI in a library file is easy: it is in the .eh_frame
section, whose offset is given in the ELF library header.  But that
header is not part of the process's memory image so is not available in
the core file; so instead corelibgen looks for ELF sections of type
PT_GNU_EH_FRAME.  (Luckily this section is required to implement C++
unwinding for exceptions so it tends to be present for modern
libraries.)

Another difficulty is that corelibgen has to guess appropriate names for
the stub libraries it creates.  It does that by looking for the string
"GLIBC_" in the library's ELF section of type PT_LOAD and walking
backwards until it finds a string beginning "lib" or "ld-linux.so".

It is enough, at least for gdb, to create a very minimal stub library of
the appropriate name with a header and sections .shstrtab and .eh_frame.


Limitations
-----------

1. I have only tested this for i386 and x86-64 core files.

2. This doesn't provide a stub (or genuine) libthread_db, so
thread-local variables can't be resolved by the debugger.

3. This has to be done manually.  It would be nice if gcc or lldb could
do it automatically!  (This would presumably be a bit like using
__jit_debug_descriptor for JIT/runtime-generated code in gdb.)

4. It is somewhat fragile, partly because it tries to guess the library
names.

5. This doesn't work for libraries that don't have CFI, in particular
really old library versions.

6. coredump_filter is not available on Linux kernels prior to version
2.6.28.

7. This only fixes the backtrace addresses and doesn't provide *symbols*
within the shared libraries.  So your backtrace will still give question
marks for the addresses in the shared libraries, but those addresses
will now be correct!

8. I have only tested this with gdb.  I haven't tested it on a very wide
variety of programs and libraries.


Example session: x86-64
-----------------------

  [ubuntu corelibgen]$ gcc -o corelibgen corelibgen.c -Wall -ggdb

  [ubuntu corelibgen]$ gcc -Wl,-soname,library-x86_64.so -shared -fPIC -o library-x86_64.so library-x86_64.S

  [ubuntu corelibgen]$ gcc -ggdb -o frametest-x86_64 frametest.c library-x86_64.so

  [ubuntu corelibgen]$ echo 127 > /proc/self/coredump_filter

  [ubuntu corelibgen]$ ulimit -c unlimited

  [ubuntu corelibgen]$ LD_LIBRARY_PATH=. ./frametest-x86_64
  Trace/breakpoint trap (core dumped)

  [ubuntu corelibgen]$ gdb frametest-x86_64 core --ex backtrace --ex quit
  GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
  Copyright (C) 2014 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-linux-gnu".
  Type "show configuration" for configuration details.
  For bug reporting instructions, please see:
  <http://www.gnu.org/software/gdb/bugs/>.
  Find the GDB manual and other documentation resources online at:
  <http://www.gnu.org/software/gdb/documentation/>.
  For help, type "help".
  Type "apropos word" to search for commands related to "word"...
  Reading symbols from frametest-x86_64...done.

  warning: core file may not match specified executable file.
  [New LWP 10482]
  Core was generated by `./frametest-x86_64'.
  Program terminated with signal SIGTRAP, Trace/breakpoint trap.
  #0  0x00007f1adfad96a5 in library_function () from ./library-x86_64.so
  #0  0x00007f1adfad96a5 in library_function () from ./library-x86_64.so
  #1  0x00007fffdbd14be0 in ?? ()
  #2  0x00000000004006d6 in f2 (x=2, y=98 'b') at frametest.c:11
  #3  0x0000000000400701 in f1 (x=1, y=97 'a') at frametest.c:16
  #4  0x0000000000400716 in main () at frametest.c:21

  [ubuntu corelibgen]$ rm library-x86_64.so

  [ubuntu corelibgen]$ gdb frametest-x86_64 core --ex backtrace --ex quit
  GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
  Copyright (C) 2014 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-linux-gnu".
  Type "show configuration" for configuration details.
  For bug reporting instructions, please see:
  <http://www.gnu.org/software/gdb/bugs/>.
  Find the GDB manual and other documentation resources online at:
  <http://www.gnu.org/software/gdb/documentation/>.
  For help, type "help".
  Type "apropos word" to search for commands related to "word"...
  Reading symbols from frametest-x86_64...done.

  warning: core file may not match specified executable file.
  [New LWP 10482]

  warning: Could not load shared library symbols for ./library-x86_64.so.
  Do you need "set solib-search-path" or "set sysroot"?
  Core was generated by `./frametest-x86_64'.
  Program terminated with signal SIGTRAP, Trace/breakpoint trap.
  #0  0x00007f1adfad96a5 in ?? ()
  Python Exception <class 'gdb.MemoryError'> Cannot access memory at address 0x7fffc99cf548:
  #0  0x00007f1adfad96a5 in ?? ()
  Cannot access memory at address 0x7fffc99cf548

  [ubuntu corelibgen]$ ./corelibgen -f. core
  Found 64-bit core file
  Creating stub library libc.so.6
  Creating stub library library-x86_64.so
  Creating stub library library-x86_64.so
  Failed to open file
  Creating stub library libunknown_13
  Creating stub library libunknown_20

  [ubuntu corelibgen]$ gdb --ex 'set solib-search-path lib' \
  >                        --ex 'set sysroot .' \
  >                        --ex 'file frametest-x86_64' \
  >                        --ex 'core core' \
  >                        --ex backtrace \
  >                        --ex quit
  GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
  Copyright (C) 2014 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-linux-gnu".
  Type "show configuration" for configuration details.
  For bug reporting instructions, please see:
  <http://www.gnu.org/software/gdb/bugs/>.
  Find the GDB manual and other documentation resources online at:
  <http://www.gnu.org/software/gdb/documentation/>.
  For help, type "help".
  Type "apropos word" to search for commands related to "word".
  Reading symbols from frametest-x86_64...done.
  warning: core file may not match specified executable file.
  [New LWP 10482]
  warning: Could not load shared library symbols for /lib64/ld-linux-x86-64.so.2.
  Do you need "set solib-search-path" or "set sysroot"?
  Core was generated by `./frametest-x86_64'.
  Program terminated with signal SIGTRAP, Trace/breakpoint trap.
  #0  0x00007f1adfad96a5 in ?? ()
  #0  0x00007f1adfad96a5 in ?? ()
  #1  0x00007fffdbd14be0 in ?? ()
  #2  0x00000000004006d6 in f2 (x=2, y=98 'b') at frametest.c:11
  #3  0x0000000000400701 in f1 (x=1, y=97 'a') at frametest.c:16
  #4  0x0000000000400716 in main () at frametest.c:21


Example session: i386
---------------------

  [ubuntu corelibgen]$ gcc -o corelibgen corelibgen.c -Wall -ggdb

  [ubuntu corelibgen]$ gcc -m32 -Wl,-soname,library-i386.so -shared -fPIC -o library-i386.so library-i386.S

  [ubuntu corelibgen]$ gcc -m32 -ggdb -o frametest-i386 frametest.c library-i386.so

  [ubuntu corelibgen]$ echo 127 > /proc/self/coredump_filter

  [ubuntu corelibgen]$ ulimit -c unlimited

  [ubuntu corelibgen]$ LD_LIBRARY_PATH=. ./frametest-i386
  Trace/breakpoint trap (core dumped)

  [ubuntu corelibgen]$ gdb frametest-i386 core --ex backtrace --ex quit
  GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
  Copyright (C) 2014 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-linux-gnu".
  Type "show configuration" for configuration details.
  For bug reporting instructions, please see:
  <http://www.gnu.org/software/gdb/bugs/>.
  Find the GDB manual and other documentation resources online at:
  <http://www.gnu.org/software/gdb/documentation/>.
  For help, type "help".
  Type "apropos word" to search for commands related to "word"...
  Reading symbols from frametest-i386...done.
  [New LWP 10442]
  Core was generated by `./frametest-i386'.
  Program terminated with signal SIGTRAP, Trace/breakpoint trap.
  #0  0xf7773518 in library_function () from ./library-i386.so
  #0  0xf7773518 in library_function () from ./library-i386.so
  #1  0x08048575 in f2 (x=2, y=98 'b') at frametest.c:11
  #2  0x0804859f in f1 (x=1, y=97 'a') at frametest.c:16
  #3  0x080485be in main () at frametest.c:21

  [ubuntu corelibgen]$ rm library-i386.so

  [ubuntu corelibgen]$ gdb frametest-i386 core --ex backtrace --ex quit
  GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
  Copyright (C) 2014 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-linux-gnu".
  Type "show configuration" for configuration details.
  For bug reporting instructions, please see:
  <http://www.gnu.org/software/gdb/bugs/>.
  Find the GDB manual and other documentation resources online at:
  <http://www.gnu.org/software/gdb/documentation/>.
  For help, type "help".
  Type "apropos word" to search for commands related to "word"...
  Reading symbols from frametest-i386...done.
  [New LWP 10442]

  warning: Could not load shared library symbols for ./library-i386.so.
  Do you need "set solib-search-path" or "set sysroot"?
  Core was generated by `./frametest-i386'.
  Program terminated with signal SIGTRAP, Trace/breakpoint trap.
  #0  0xf7773518 in ?? ()
  #0  0xf7773518 in ?? ()

  [ubuntu corelibgen]$ ./corelibgen -f. core
  Found 32-bit core file
  Creating stub library libc.so.6
  Creating stub library library-i386.so
  Creating stub library library-i386.so
  Failed to open file
  Creating stub library libunknown_13

  [ubuntu corelibgen]$ gdb --ex 'set solib-search-path lib' \
  >                        --ex 'set sysroot .' \
  >                        --ex 'file frametest-i386' \
  >                        --ex 'core core' \
  >                        --ex backtrace \
  >                        --ex quit
  GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.3) 7.7.1
  Copyright (C) 2014 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-linux-gnu".
  Type "show configuration" for configuration details.
  For bug reporting instructions, please see:
  <http://www.gnu.org/software/gdb/bugs/>.
  Find the GDB manual and other documentation resources online at:
  <http://www.gnu.org/software/gdb/documentation/>.
  For help, type "help".
  Type "apropos word" to search for commands related to "word".
  Reading symbols from frametest-i386...done.
  [New LWP 10442]
  warning: Could not load shared library symbols for /lib/ld-linux.so.2.
  Do you need "set solib-search-path" or "set sysroot"?
  Core was generated by `./frametest-i386'.
  Program terminated with signal SIGTRAP, Trace/breakpoint trap.
  #0  0xf7773518 in ?? ()
  #0  0xf7773518 in ?? ()
  #1  0x08048575 in f2 (x=2, y=98 'b') at frametest.c:11
  #2  0x0804859f in f1 (x=1, y=97 'a') at frametest.c:16
  #3  0x080485be in main () at frametest.c:21