Make test suite builder-agnostic

[eschaton]

On my Mac using a very recent macOS and Xcode, I can easily build fcode-utils with just a quick make, but make tests unfortunately fails with a huge set of diffs.

It looks like the test suite works by comparing the test result logs with a static set of logs created in specific build environments, and doesn’t take differences in the file headers or whitespace into account.

For example, here’s the first page of results in my terminal:

CygTestLogs=/Users/cmh/Documents/Projects/OpenFirmware/fcode-utils/testlogs/testlogs-ppc-linux csh AutoCompare

177
177 TokCondl/TokExstCondTstY.Log files differ.
3,4c3,4
< Welcome to toke - OpenBIOS tokenizer v1.0.2
< (C) Copyright 2001-2006 Stefan Reinauer.
---
> Welcome to toke - FCode tokenizer v1.0.3
> (C) Copyright 2001-2010 Stefan Reinauer.
10,11d9
< 	Tokenizer Compiled on PPC under GNU_Linux
< 		Mon, 23 Oct 2006 at 13:20:18 CDT
TokCondl/TokExstCondTstY.DeTok files differ.
1,2c1,2
< \  Welcome to detok - OpenBIOS detokenizer v1.0.2
< \  (C) Copyright 2001-2006 Stefan Reinauer.
---
> \  Welcome to detok - FCode detokenizer v1.0.3
> \  (C) Copyright 2001-2010 Stefan Reinauer
16c16
<                 " Begin Nest Test Test"
---
>         " Begin Nest Test Test"
19c19
<                 " Exists, level 1"
---
>         " Exists, level 1"
22c22
<                 " Exists and exists, level 2"
---
>         " Exists and exists, level 2"
25c25

and so on.

It seems like the test suite could be improved in a couple of ways:

• Have one standard set of known-good results to compare against, rather than per-build-environment sets.
• Output the results into a separate directory (specifiable via a make variable) rather than the same directory as the test sources.
• Ignore whitespace in result comparisons when that doesn’t matter. (Does it ever? I don’t know the code or tests…)

That would make it much easier to find meaningful test failures, and to do repeatable CI builds.

[reinauer]

This needs a lot more work. running the release test suite on release 1.0.2 fails on modern systems (e.g. my Ubuntu 20 workstation). If you have time to help with this, it's highly appreciated

[eschaton]

Unfortunately I can’t directly help due to my employment, I try to compensate for that by offering as much detail as I can. Sorry. :(

[joevt]

I used the following to update most of the test logs:

cd "/Volumes/Updates/Docs/Open_Firmware_and_Forth/OpenBIOS/openbios-fcode-utils"

# Do this only once
IFS=$'\n'
for thefile in $(find testlogs -name '*.DeTok'); do
	perl -pE '
		s|\\  Welcome to detok - OpenBIOS detokenizer v1\.0\.2|\\  Welcome to detok - FCode detokenizer v1.0.3|;
		s|\\  \(C\) Copyright 2001-2006 Stefan Reinauer\.|\\  (C) Copyright 2001-2010 Stefan Reinauer|;
		s|^        ( *")|$1|;
	' \
	-i "$thefile"
done

IFS=$'\n'
for thefile in $(find testlogs -name '*.Log'); do
	perl -0777 -pE '
		s|Welcome to toke - OpenBIOS tokenizer v1\.0\.2|Welcome to toke - FCode tokenizer v1.0.3|;
		s|\(C\) Copyright 2001-2006 Stefan Reinauer\.|(C) Copyright 2001-2010 Stefan Reinauer.|;
		s|	Tokenizer Compiled on PPC under GNU_Linux\n||;
		s|		Mon, 23 Oct 2006 at 13:20:18 CDT\n||;
		s/(\t(?:No|  )ABORT-Quote-Throw\n)(\t(?:No|  )String-remark-escape\n)/$1\tNoSun-Style-Checksum\n$2/g;
	' \
	-i "$thefile"
done

The diffs are easier to see using an app like BBEdit.app:

cd "/Volumes/Updates/Docs/Open_Firmware_and_Forth/OpenBIOS/openbios-fcode-utils"
bbdiff testsuite testlogs/testlogs-ppc-linux