Ideally we should rename this file, but for now I'm not willing to take that on

Previously, `get_absfilename()` was replacing '/..' for '/', without
deleting the parent directory string.

Add LGTM code quality badges

Clarify that the usual usage pattern for icecc (adding its symlinks to $PATH) should not be followed when using ccache's own symlinks.

Improve ccache + icecc advice

* Add unit tests for assembler options

This is to ensure we don't break existing functionality
in subsequent patches.

* Clean up assembly argument handling to process all arguments

This patch should have no functional changes besides handling
a few more esoteric cases of assembler arguments.

The -Wa arguments are split by comma and analyzed one by one with
a new helper function.

* Check for assembler options that have two arguments

According to the gas help file there are two options that
have two arguments where the second argument does not start
with a hyphen (--debug-prefix-map OLD=NEW and --defsym SYM=VAL).
With the current code, these arguments will be forced to run
locally, even though there's no good reason for that.

* Ensure assembler argument files run locally

Any assembler option beginning with @ specifies more options that
must be in a file. Instead of trying to parse all of those arguments,
we just force that file to run locally.

* Stop forcing the use of -pipe in the compiler

The -pipe flag does not work when passing additional assembly
files with -Wa. GCC is not smart enough to add an additional '-'
argument to ensure the assembler still reads from stdin when there are
additional files passed through -Wa.

Additionally, as a matter of policy, I think it's wrong for icecc
to enforce using -pipe and should be left to the individual build
systems to make this decision.

Besides that, in this day and age, the performance benefits of
-pipe are dubious at best.

* Add support for additional assembly files passed through gcc

The Linux kernel has started passing an extra assembly file to all
compilation steps in order to add globally available inline macros.
It does this by passing an extra file using -Wa. Currently this causes
icecc to force the build locally which means compiling the kernel
cannot be distributed.

In order to support this in icecc, we add any such files to
the extra files in the environment.

…icecc#426)

ICECC_REMOTE_CPP is typically used to avoid remote (clang) compilation
errors, so we should avoid disabling it where possible.

The test case in icecc#410 passes if you specify an official language
standard rather than using GCC's default (which allows GNU extensions).
Let's check this before deciding to disable ICECC_REMOTE_CPP.

Followup to icecc#411.

* Also include base libraries for AVX2 and AVX512 builds

glibc supports searching for libraries in subdirs according to the CPU
features. On x86-64, those are "haswell" and "avx512_1", as in:

$ ldd /usr/bin/icecc | grep haswell
        libstdc++.so.6 => /usr/lib64/haswell/avx512_1/libstdc++.so.6
        libc.so.6 => /usr/lib64/haswell/libc.so.6
        libm.so.6 => /usr/lib64/haswell/avx512_1/libm.so.6

With this change, we'll include the base library or libraries in the
environment. In the system above, we'll include /usr/lib64/libm.so.6,
/usr/lib64/haswell/libm.so.6 in addition to the libm listed in ldd's
output.

* Fix finding ld.so.conf on Clear Linux

Clear Linux has a "stateless" /etc, meaning that all files there are
supposed to belong to the user. The ld.so.cache file is instead in
/var/cache/ldconfig.

* Fix warning printed when /lib or /lib64 are symlinks

icecc-create-env was printing:
[...]
adding file /usr/bin/objcopy
adding file /etc/ld.so.conf=/tmp/icecc_ld_so_confTsSNqs
cp: -r not specified; omitting directory '/lib'
cp: -r not specified; omitting directory '/lib64'
creating d7801039695fbb794e026bdaf287276c.tar.gz

The -p option means to preserve ownership, timestamps, etc., but doesn't
tell cp to operate on the symlink itself. The -P option does.

Detect xz and zst compressions too. This allows teams that know all
their iceccd servers have been updated to this version to package the
environment using more efficient file formats.

The test is updated to detect different file extensions, should
icecc-create-env produce them.

* Add Cirrus-CI config for FreeBSD builds

* Add build status badge

It's apparently making Clang slower (presumably -include-pch is handled
even in the -frewrite-includes mode where it's useless). Gcc's -include .gch
is handled the same way already.

filter out also -include-pch when preprocessing for the remote

Update instructions for gentoo

Fix ignoring objective c/c++ language option as valid.

Keep -Werror=missing-include-dirs local

…cecc#447)

* Removed dependency on native tar command and used libarchive library instead
* Updated NEWS and README.

6756498 added 'make dist' to the end of the travis build commands,
effectively hiding exit code from 'make test'

It'd insert a pragma that'd make the remote compilation break.
And ccache recommends it as one way to make ccache work with PCH.

For whatever sick reason the pch test fails locally with gcc9 because
"not used because `__FLT_MIN__' defined as
` 1.17549435082228750796873653722224568e-38F' not ` 1.1'".

compileserver.cpp uses it too

It may be because of clang's -fdebug-compilation-dir, but gdb here
now shows the path only as debug/debug2.cpp and not the full path.
As long as the actual gdb output works (and thus gdb can find the file),
this doesn't actually matter, so do not require the full path.

They are profiling options for the compiler itself.

Always compile cmake tests locally

'make dist' just creates a tarball, it doesn't build the sources

'Server' is just too vague.

bzip2 is kind of obsolete these days.

the mach codepath doesn't obtain `MemTotal`, which will always cause the
a pessimistic 100MB memory limit. as later on, we'll just return 0.

the best way on macos to obtain the total memory is reading `hw.memsize`

Zstandard (https://zstd.net) is a modern compression library also of the
Lempel-Ziv family of compressors. Compared to LZO, if given the same CPU
time, zstd will compress better. Testing with the "lzop" and "zstd"
command-line tools indicates that the same-CPU-time level compression
for a 7241 kB preprocessed source lies between zstd levels 2 and 3 (LZO
takes 916 µs; zstd level 2 takes 753 µs and level 3 takes 1024 µs).
Level 2 compresses this file to 1028 kB, while level 3 reaches 929 kB,
compared to LZO's 1581 kB.

However, when testing a full compilation of a large-ish source code (not
including linking), we find that CPU time is the bottleneck, shared
between the compressor and the preprocessors. The results of a best-of-5
run on this source are:

	lzo:    22.989s
	zstd1:  20.421s
	zstd2:  20.554s
	zstd3:  20.630s
	zstd4:  20.762s
	zstd5:  21.355s
	zstd6:  21.341s

Therefore, this commit chooses level 1 as the default. The 11% gain in
build time is caused the reduction in both CPU contention and the
network transmission -- zstd level 1 compresses the above source to 1029
kB, which is almost 35% smaller than the LZO content.

And this 'where possible' is everythere except for the address/leak
sanitizer, which is broken for whatever mysterious reason. We already
had to explicitly force an older version of Ubuntu Trusty
for the asan/lsan build, so keep it that way. In a summary:
- Normal builds work fine with newer Ubuntu (if 16.04 can be called "newer").
- Sanitizers generally work with -fuse-ld=gold, so maybe the BFD ld is
  old/broken on these old Ubuntu versions.
- Asan/lsan breaks when tests try to terminate the iceccd daemon. Either
  the signal handling is broken, or the leak cleanup is broken, it simply
  exists and the only thing it says is 'fatal error'.
- I have no idea how to fix that. Even when using gcc9
  from the ubuntu-toolchain-r-test it's still broken.
- So stay on the ancient Trusty version.
- However, there's no official zstd package for Trusty, so grab one
  from a "random" unsupported repository. It works *shrug*.
- Also, asan contains lsan, so just use them together, so that the special
  handling is at least done only once.

There are cases where a remote compilation is failing due to a missing
file, that we can't detect with limited amount of work before kicking
off the compile. One such occasion is the Linux kernel including a file
in a inline ASM statement in a C file via the incbin directive.

If we detect a remote error with a missing file, force a local recompile.
This should only be hit in a very limited number of cases, so there is no
need for a more clever workaround.

There are some other issues, which require a local rebuild as a workaround.
Rename the stderrfix to localrebuild and also honor it when building with
clang to make it reusable.

When fdirectives-only is used the local cpp can fail on some macro constructs
that are incompatible with this mode of preprocessing. Currently this fails
the build, as it's not a remote error, which would get recovered by a local
rebuild, but the local side failing.

Instead of failing the build, trigger a local rebuild. The failure rate is
so small that any more sophisticated recovery seems to be wasted effort. This
leads to an additional local recompile in case of broken code, that fails in
the cpp, but I don't think that matters, as performance of failing builds is
probably not as important.

The tests need to account for the local fallback being triggered by a
failing cpp invocation.

More fixes for Linux kernel build with -fdirectives-only. Some of the
kernel tools are passing the above options directly to the preprocessor
via -Wp. Need to make sure they are applied on the local cpp side.

One clang binary can be used to compile code for different architectures.
One way to select the target architecture is the name of the compiler:
A symlink with the target triple, e.g. armv7-unknown-linux-gnueabihf-clang.

Add support for this in icecc-create-env by calling the clang binary as
specified on the command-line when querying other tools and headers.
This way the correct tools and headers for the desired architecture are
added to the environment.

It doesn't seem to be actually needed (tested with Clang 8.0.0 and 4.0.1).
I have no idea why I added it, the commit doesn't say. Possibly
initially -frewrite-includes had a bug that necessitated this.

As far as I can tell, this is mostly just needless trouble, and the purpose
of the code is unclear and hard to understand. AFAICT the extension check
means that it's mostly skipped anyway (because most headers have
the extension starting with 'h'). And there are several commits
in the history that try to fix this somehow only breaking something else.

Without handling these options in any special way this should work in all
cases, except for the case when the -include(-pch) option is required
(i.e. the source does not include the headers from which the PCH has been
built, so the PCH is the only source of that code). Since we do not
ship PCHs to remote nodes and assume the code will build fine even without
the PCH, this will obviously break. But although that mode of operation is
supported by GCC, I consider it to be weird. And if this actually breaks
things for somebody who uses it, they can complain so that it actually
becomes clear what the problem is and how it can possibly be handled.

do not package Clang internal headers in icecc-create-env

remove special PCH handling with -include and -include-pch

Since we strip the PCH for the remote build.

This is old code for backwards compatibility, so just silence the warning
somehow in a way that works and be done with it.

zstd clamps the value to the allowed range, and recent zstd versions
allow even negative values for even faster compression

find_last_of() searches for any of the given characters, which is not
used in these cases -> using find_last_of() is a needless confusion.

…tion

This removes the last cases of hardcoded paths like /usr/bin/gcc. Now
icecc can be invoked with compiler in any location and the environment
sent to the remote host will include the proper compiler. Supported
compilers (binary names) are gcc/g++, clang/clang++ and cc/c++ (the first
two including prefixes and postfixes, e.g. aarch64-suse-linux-gcc-6.

There's a small chance that icecc-create-env will get invoked again
for the same result (e.g. because different wrapper scripts point
to the same real compiler), and so the existing tarball would
be temporarily broken.

This is based on pull requests icecc#425. Both gcc and clang
have their peculiarities when handling -gsplit-dwarf depending
on the exact order of this and other -g arguments. So keep the order
the same and do not handle any of those arguments specially.

This technically changes the protocol, but let's keep it as part
of the recent version 41 change.

keep order of remote arguments, especially -gsplit-dwarf (icecc#435)

Fix get_absfilename()