cmd/compile: consider using DWARF 5

[aclements]

DWARF 5 has several advantages over previous versions of DWARF. Notably,

1. It supports position-independent representations, which significantly reduces the number of relocations in object files and hence the size of object files and the load on the linker. In the go binary, 49% of the 503,361 total relocations are in the DWARF.

2. It supports much more compact location and range list formats. The location and range list sections are 6% of the 12MiB of the go binary, even when zlib compressed.

3. It has an official language code for Go. :)

DWARF 5 is quite new, and I don't think the rest of the ecosystem is ready yet, but I wanted to get the idea floating. It is supported by the GNU and LLVM toolchains and some debuggers. Support was added in GCC 7.1 (May 2017) and GDB 8.0 (June 2017). It appears to be in the latest LLVM, which covers most of the Xcode tools, though I can't find when it was added.

It is currently not supported by LLDB or the macOS linker. We could potentially get around the macOS linker by leaving out the Go DWARF from the objects we pass to the system linker and then merging it in to the final binary (we already do a merge step). This is more feasible with DWARF5 because it's mostly position-independent, so we wouldn't need dsymutil to relocate it for us.

/cc @cherrymui @heschik @dr2chase @randall77 @ianlancetaylor

[heschi]

Strongly in favor of doing this the moment we think we can get away with it. I'm happy to do the work for location lists.

[bradfitz]

Un-proposaling this per discussion with @ianlancetaylor.

[aarzilli]

For reference, the debug info for the linux_amd64/go1.11.1 version of cmd/compile is 15MB of which 8MB are in debug_loc and debug_ranges, specifically:

total size 20752567

section             compressed    uncompressed
.zdebug_abbrev             274 B         467 B
.zdebug_line            739610 B     1931456 B
.zdebug_frame           158093 B      559020 B
.zdebug_pubnames         38451 B      251078 B
.zdebug_pubtypes         45270 B      211390 B
.debug_gdb_scripts                        42 B
.zdebug_info           1257828 B     3694829 B
.zdebug_loc             989048 B     5914085 B
.zdebug_ranges          442419 B     2461408 B

Rewriting debug_loc and debug_ranges in the new format specified by DWARF 5 will reduce the size of debug_ranges to 774460 B (31% of DWARF 4 size) and of debug_loc to 2032960 B (34% of DWARF 4 size).

Edit: debug_ranges and debug_loc were accidentally swapped in the last sentence.

[dr2chase]

Is compression still doing a better job at reducing binary size than DWARF 5, or can D5 also be compressed? I did just mostly finish a DWARF splitter for OSX that recreates the expected file in the expected place, and adds the expected UUID for matching the two files. I suspect I ought to be using a hash of the binary contents excluding debugging information, so that identical files will remain identical after UUIDs are added.

[aclements]

The compression isn't actually part of the DWARF spec, so it's orthogonal to the DWARF version and can be used with DWARF 5.

@aarzilli, thanks for doing that experiment. Can you get the numbers for DWARF 5 if it's also zlib compressed?

[aarzilli]

@aarzilli, thanks for doing that experiment. Can you get the numbers for DWARF 5 if it's also zlib compressed?

After compression the size is basically the same, 98% (of compressed DWARF4) for debug_ranges and 77% (of compressed DWARF4) for debug_loc. The compression time however is reduced by 40%.

[dr2chase]

Is that compression time+space comparing current low-effort (higher speed) versus future low-effort? Sorry to be so picky, it's just that we've made mistakes here before.

[aarzilli]

I'm comparing compression using zlib.BestSpeed for both, like the linker. Since measuring how much time it actually took inside the linker is hard I measured how much time it takes to recompress the compressed section.

Because I have send the whole section to the compressor and the linker doesn't, I get slightly better compression than the linker. But the difference is small (around 1%) and since I'm doing it for both DWARF 4 and DWARF 5 the result should be valid.

[joeyjiaojg]

Parsed a DWARF5 linux kernel module, the below statement of index 0 is not nil. It returns the real file path.

<html>
<body>
<!--StartFragment-->

// Files returns the file name table of this compilation unit as of
--
  | // the current position in the line table. The file name table may be
  | // referenced from attributes in this compilation unit such as
  | // AttrDeclFile.
  | //
  **| // Entry 0 is always nil, since file index 0 represents "no file".**
  | //
  | // The file name table of a compilation unit is not fixed. Files
  | // returns the file table as of the current position in the line
  | // table. This may contain more entries than the file table at an
  | // earlier position in the line table, though existing entries never
  | // change.
  | func (r *LineReader) Files() []*LineFile {
  | return r.fileEntries
  | }

<!--EndFragment-->
</body>
</html>

[mdempsky]

FWIW, GCC 11.1 was released on April 27, 2021. From https://gcc.gnu.org/gcc-11/changes.html:

For targets that produce DWARF debugging information GCC now defaults to DWARF version 5 (with the exception of VxWorks and Darwin/Mac OS X which default to version 2 and AIX which defaults to version 4).

Also, Clang 14 was released on March 25, 2022. From https://releases.llvm.org/14.0.0/tools/clang/docs/ReleaseNotes.html#dwarf-support-in-clang:

The default DWARF version has increased from DWARFv4 to DWARFv5. You can opt back in to the old behavior with -gdwarf-4 or -fdebug-default-version=4. Some platforms (Darwin, Android, and SCE for instance) already opt out of this version bump as is suitable for the platform

[thanm]

Hi all, I'm working on a set of patches to switch the Go compiler/linker over to DWARF 5, just FYI.

[gopherbot]

Change https://go.dev/cl/633878 mentions this issue: cmdinternal/dwarf: add DW_LNCT and DW_UT constant definitions

[gopherbot]

Change https://go.dev/cl/633880 mentions this issue: cmd/internal/objabi: add new R_DWTXTADDR_* relocation types

[gopherbot]

Change https://go.dev/cl/635345 mentions this issue: cmd/compile,cmd/link: move to DWARF5-style range lists

[gopherbot]

Change https://go.dev/cl/635337 mentions this issue: cmd: initial compiler+linker support for DWARF5 .debug_addr

[gopherbot]

Change https://go.dev/cl/633879 mentions this issue: cmd/link, cmd/internal/dwarf: add DWARF5 line table support

[gopherbot]

Change https://go.dev/cl/633877 mentions this issue: internal/goexperiment: add a new experiment to gate DWARF version 5

[gopherbot]

Change https://go.dev/cl/634415 mentions this issue: cmd/internal/objabi,cmd/link/internal/sym: add SDWARFADDR symbol type

[gopherbot]

Change https://go.dev/cl/635836 mentions this issue: cmd/compile,cmd/link: move to DWARF5-style location lists

[gopherbot]

Change https://go.dev/cl/636518 mentions this issue: dwtest: fixes for DWARF5 support

[thanm]

For those following along, the patch stack I've been working on is now functional; if you check out the stack at https://go-review.googlesource.com/c/go/+/635836 and then build with GOEXPERIMENT=dwarf5 you'll get a DWARF5 binary. Please feel free to try it out and kick the tires if you like.

What works and what doesn't at the moment:

• all.bash runs cleanly with GOEXPERIMENT=dwarf5
• x/debug/dwtest tests are failing with GOEXPERIMENT=dwarf5, I need to dig into this a bit (suspect is has something to do with .debug_rnglists )

[gopherbot]

Change https://go.dev/cl/639177 mentions this issue: internal/gocore: fixes for DWARF5 support

[thanm]

What works and what doesn't at the moment:

Update: still some Delve issues to work through, but things are getting closer; Alessandro has been helping me test this.

It looks like this work is also dependent on the fix for [delve issue 3861(https://github.com/go-delve/delve/issues/3861) (e.g. Go CL 628876) going in, so hopefully we can come to some sort of resolution as to what to do with that Go CL.

[thanm]

Here are compilebench numbers for the changes (tip of master vs master + DWARF5).

                         │ out.base.txt │           out.dwarf5.txt            │
                         │    sec/op    │   sec/op     vs base                │
Template                    61.56m ± 1%   61.28m ± 1%        ~ (p=0.136 n=50)
Unicode                     45.54m ± 1%   45.32m ± 1%        ~ (p=0.145 n=50)
GoTypes                     348.9m ± 0%   347.1m ± 1%   -0.52% (p=0.014 n=50)
Compiler                    47.94m ± 1%   47.04m ± 2%        ~ (p=0.061 n=50)
SSA                          2.880 ± 1%    2.863 ± 0%   -0.61% (p=0.003 n=50)
Flate                       36.76m ± 1%   36.28m ± 1%   -1.31% (p=0.001 n=50)
GoParser                    70.02m ± 1%   69.50m ± 1%   -0.75% (p=0.028 n=50)
Reflect                     178.6m ± 1%   175.6m ± 1%   -1.67% (p=0.000 n=50)
Tar                         75.51m ± 1%   74.51m ± 1%   -1.33% (p=0.001 n=50)
XML                         86.60m ± 1%   86.48m ± 1%        ~ (p=0.688 n=50)
LinkCompiler                297.5m ± 3%   295.8m ± 4%        ~ (p=1.000 n=50)
ExternalLinkCompiler       1035.6m ± 1%   887.2m ± 1%  -14.33% (p=0.000 n=50)
LinkWithoutDebugCompiler    168.0m ± 4%   166.8m ± 3%        ~ (p=0.571 n=50)
StdCmd                       14.27 ± 1%    14.16 ± 0%   -0.78% (p=0.000 n=50)
geomean                     208.8m        204.9m        -1.89%

                         │ out.base.txt │           out.dwarf5.txt            │
                         │ user-sec/op  │ user-sec/op  vs base                │
Template                    210.8m ± 4%   208.5m ± 3%        ~ (p=0.304 n=50)
Unicode                     59.32m ± 9%   56.72m ± 6%        ~ (p=0.363 n=50)
GoTypes                      1.544 ± 1%    1.519 ± 1%        ~ (p=0.111 n=50)
Compiler                    85.86m ± 4%   84.66m ± 6%        ~ (p=0.270 n=50)
SSA                          13.31 ± 1%    13.20 ± 1%        ~ (p=0.053 n=50)
Flate                       123.1m ± 3%   118.1m ± 4%   -4.05% (p=0.029 n=50)
GoParser                    232.4m ± 2%   232.7m ± 2%        ~ (p=0.856 n=50)
Reflect                     701.0m ± 1%   691.7m ± 1%        ~ (p=0.085 n=50)
Tar                         262.1m ± 2%   258.9m ± 5%        ~ (p=0.926 n=50)
XML                         301.3m ± 2%   290.5m ± 2%   -3.60% (p=0.026 n=50)
LinkCompiler                518.2m ± 5%   512.1m ± 5%        ~ (p=0.808 n=50)
ExternalLinkCompiler       1118.4m ± 1%   991.9m ± 2%  -11.31% (p=0.000 n=50)
LinkWithoutDebugCompiler    204.2m ± 4%   199.6m ± 5%        ~ (p=0.163 n=50)
geomean                     385.4m        375.1m        -2.67%

          │ out.base.txt │           out.dwarf5.txt            │
          │  text-bytes  │  text-bytes   vs base               │
HelloSize   957.6Ki ± 0%   957.7Ki ± 0%  +0.01% (p=0.000 n=50)
CmdGoSize   12.21Mi ± 0%   12.20Mi ± 0%  -0.05% (n=50)
geomean     3.378Mi        3.378Mi       -0.02%

          │ out.base.txt │            out.dwarf5.txt             │
          │  data-bytes  │  data-bytes   vs base                 │
HelloSize   14.99Ki ± 0%   14.99Ki ± 0%       ~ (p=1.000 n=50) ¹
CmdGoSize   407.1Ki ± 0%   407.1Ki ± 0%       ~ (p=1.000 n=50) ¹
geomean     78.11Ki        78.11Ki       +0.00%
¹ all samples are equal

          │ out.base.txt │            out.dwarf5.txt             │
          │  bss-bytes   │  bss-bytes    vs base                 │
HelloSize   141.9Ki ± 0%   141.9Ki ± 0%       ~ (p=1.000 n=50) ¹
CmdGoSize   206.2Ki ± 0%   206.2Ki ± 0%       ~ (p=1.000 n=50) ¹
geomean     171.0Ki        171.0Ki       +0.00%
¹ all samples are equal

          │ out.base.txt │       out.dwarf5.txt        │
          │  exe-bytes   │  exe-bytes    vs base       │
HelloSize   1.491Mi ± 0%   1.444Mi ± 0%  -3.18% (n=50)
CmdGoSize   18.51Mi ± 0%   17.99Mi ± 0%  -2.84% (n=50)
geomean     5.253Mi        5.096Mi       -3.01%