Need help with initial meson test

[hyenias]

Description of problem:
After going through all the instructions I could find on the site, some test cases have either failed or been skipped. meson-logs/testlog-malloc.txt indicates that various programs along its path cannot be found.

Ksh version:
GNU/Linux x86_64 Ubuntu 18.04
2020.0.0-beta1-192-g0d76ee4b

How reproducible:
Run meson with --prefix ~/kshdev and meson test.

Steps to reproduce:

1. From within $HOME directory

• mkdir git
• mkdir kshdev

2. cd git
3. git clone https://github.com/att/ast.git
4. cd ast
5. meson --prefix ~/kshdev build
6. ninja -C build
7. cd build
8. meson test --setup=malloc

• Full log written to .../git/ast/build/meson-logs/testlog-malloc.txt
• This log file is full of test cases that cannot find programs along its path.

Actual results:
OK: 277
FAIL: 1
SKIP: 15
TIMEOUT: 0

123/293 API/ksh/ksh_debug FAIL 0.13 s

Within the testlog-malloc.txt file under a test case between the stderr section:
stderr:
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh: line 219: mkfifo: not found
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh: line 220: mkfifo: not found
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh: line 239: mkdir: not found
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh: line 240: mkdir: not found
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh[464]: run_api: line 270: diff: not found
run_test[272]: Stderr for ksh_debug had unexpected differences:
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh[464]: run_api: line 273: diff: not found

Expected results:
No previous experience to compare with, not sure. Overall, I would hope that all the possible tests would run and pass. All of the not found programs have been installed and exist along the PATH.

Additional info:
I am a new github user and new to this ksh repo. I believe that I have installed all of the required programs that were mentioned within the CONTRIBUTING.md and wiki section Building and Debugging (meson). I look forward to getting my development environment up and running so that I can contribute to this ksh project.

[krader1961]

That's a great first issue. Thanks for contributing. Those errors should not occur. Are you perhaps running this in a chroot or other sandboxed environment? The fact the unit test harness can't find essential commands like diff, mkfifo and mkdir in $PATH should cause many more failures than you reported. Note that we've been building and testing on multiple Linux, BSD, Cygwin, and other platforms (e.g., Solaris) for a long time. None of those environments exhibit the behavior you describe. So something is unusual about your build environment. What that is and whether we can work around it is unclear.

Are you seeing those "not found" errors for all unit tests? What do you see in the log if you add set -x near the top of src/cmd/ksh93/tests/util/run_test.sh and run a single unit test such as `meson test b_alias.sh*.

P.S., One of the test environments I use to verify all changes to this project is a Ubuntu 16.04 32-bit VM. So it is definitely possible to build and successfully pass all unit tests on Ubuntu (as well as several other Linux distros).

[krader1961]

some test cases have either failed or been skipped.

FWIW, Skipping some tests is expected due to deficiencies of specific platforms. For example, not having a /proc pseudo-filesystem. Those deficiencies either mean the unit test itself cannot run successfully or the behavior of the ksh binary depends on features that the platform does not provide. If any skipped tests should have been run, @hyenias, because your platform supports the necessary features we will be happy to enable those tests on your platform and deal with any test failures that result from that change.

[hyenias]

Bare metal, booted up an old PC x86_64 and installed Linux Mint 19.3 on it. Some of the required tooling was not available as packages on my ARM single board computers. Just a few hours ago, I successfully ran the TravisCI and all tests/builds past. :) Then, I moved to creating a local test environment.

Within the testlog-malloc.txt file, the first test that has an issue is 123/293 API/ksh/ksh_debug with a FAIL. All prior tests above execute fine with an OK status. The immediate test above ksh_debug is 122/293 API/tm/tvgettime. From the ksh_debug point and below, the stderr sections have not founds for cat, rm, and chmod as well.

After editing the run_test.sh with set -x statement, running "meson test b_alias.sh"; the various OLD_PATH, SAFE_PATH, FULL_PATH, and PATH all have the correct system directories. This individual test completes successfully.

[krader1961]

the first test that has an issue is 123/293 API/ksh/ksh_debug with a FAIL. All prior tests above execute fine with an OK status.

That is not surprising. The API tests (excluding the API/ksh/ tests) do not depend on the ksh scripting language. That the non-API ksh scripting tests fail with errors such as cat and rm not found tells me you either

1. don't have those programs installed (which is unlikely), or

2. there is something unusual about your test environment. For example, on my Ubuntu server cat is /bin/cat and it is found when running meson test.

That API/ksh/ksh_debug fails is not particularly surprising since it depends on features, such as a working addr2line program, which may not be available on your system. In that case the unit tests should be skipped. If that is not happening we need to modify the test framework to skip the affected tests if something like addr2line isn't available.

[krader1961]

@hyenias You wrote

Just a few hours ago, I successfully ran the TravisCI and all tests/builds past. :) Then, I moved to creating a local test environment.

Are you trying to cross-compile? That isn't something we have spent a single second thinking about. So if that is what you are trying to do I am not surprised it results in failures.

[hyenias]

Programs do exist:
$ whence mkfifo cat chmod rm diff mkdir
/usr/bin/mkfifo
/bin/cat
/bin/chmod
/bin/rm
/usr/bin/diff
/bin/mkdir
$
No, I am not trying to cross-compile. I am new to all of this and do not currently possess those skills. My plans are to load up and compile ksh on each SBC.

I am just learning and attempting for the first time to contribute to an open source project that I believe in. I have attempted to follow the provided web docs to the best of my abilities. At this time, I do not know how to fix the situation of having error statements of not finding files.

Any more suggestions, hints? I will try.

[hyenias]

FYI, I just spot checked other tests down the line that passed but had the not founds in them and after running them singularly no issues were noted in the log files.

[hyenias]

$ which addr2line
/usr/bin/addr2line
$

[hyenias]

I just repeated my above steps on one of my ARM boxes running Ubuntu 18.04 and the log file has the same not founds in them. However, on this system it has an additional failure at:
20/293 API/misc/debug FAIL 0.29 s
OK: 276
FAIL: 2
SKIP: 15
TIMEOUT: 0

Starting at the "123/293 API/ksh/ksh_debug FAIL 0.30 s" part of the log file and below the not founds show up on all remaining tests to end of file.

[krader1961]

Bare metal, booted up an old PC x86_64 and installed Linux Mint 19.3 on it. Some of the required tooling was not available as packages on my ARM single board computers.

Is it x86_64 or ARM? I get the sense you're trying to build ksh on both architectures and having problems on both. In which case I'd like to suggest focusing on one of them, preferably x86_64, until you have a clean build and all tests pass. Then we can deal with other architectures.

the various OLD_PATH, SAFE_PATH, FULL_PATH, and PATH all have the correct system directories.

Please attach the entire build/meson-logs/testlog.txt log file to this issue. It's not that I don't believe you but in my four decades working with computers it has been my experience that people see what they expect to see. Thus sometimes overlooking an important detail (e.g., semi-colons where colons should be present).

Please also attach the build/meson-logs/meson-log.txt log file.

P.S., I see your original comment makes no mention of doing ninja install before running meson test --setup=malloc. This will cause one test in b_typeset.sh to fail because it won't find the expected autoloaded functions. I'll update the documentation to make that clearer. Other than that I ran your sequence of commands on my Ubuntu system without seeing any of the "not found" errors you're seeing.

P.P.S., Note that meson thinks nearly all the unit tests are passing. So either those "not found" errors are either irrelevant or they're keeping the unit tests from even being run. But even if irrelevant they should not be present in the test log so we need to determine the root cause of those messages.

[hyenias]

After review of my notes and history log, I made some mistakes in the information I provided yesterday. I actually ended up editing two separate files with the set -x command at the top. With all the new, flipping back-and-forth from web browser and command prompts, as well as paging through 120k lines of a log file; I focused on the wrong run_test.sh file. My mind no longer possessed the awareness needed to see the differences. I apologize.

By having a break and spending more time with the log file, I have become more familiar with it and as such noticed the details today. I can now express the problem to a better degree.

The meson test commands create two separate log files depending on how you run them:

• meson test: meson-logs/testlog.txt
• meson test --setup=malloc: meson-logs/testlog-malloc.txt

Each of these log files have their content created from two separate run_test.sh scripts. The libast run script executes first for tests 1-122 ending with "122/293 API/tm/tvgettime". The cmd run_test.sh scripts picks up at "123/293 API/ksh/ksh_debug" to end of file "293/293 special-dev-paths.sh/shcomp".

• 1-122: ast/src/lib/libast/tests/run_test.sh
• 123-293: ast/src/cmd/ksh93/tests/util/run_test.sh

I wanted a little easier way for my untrained eyes to see the PATHing problems in each of these run_test.sh scripts. So, I removed the set -x commands from the top of the files and adding in the following.

In ast/src/lib/libast/tests/run_test.sh after:

readonly test_name=${x%.c}
readonly api_binary=$BUILD_DIR/$1
readonly test_src_dir=$2

I added:

if [[ -z "$PATH" ]]; then
    echo '=============>>>> ERROR: PATH was empty!'
    PATH=/home/xxx/oclint-0.13.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
fi

Then skip down a little after:

export OLD_PATH="$PATH"
export SAFE_PATH="$TEST_DIR:$TEST_DIR/space dir:$test_src_dir:$BUILD_DIR/src/cmd/builtin"
export FULL_PATH="$SAFE_PATH:$OLD_PATH"
export PATH="$FULL_PATH"

I added:

echo " OLD_PATH-lib: $OLD_PATH"
echo "SAFE_PATH-lib: $SAFE_PATH"
echo "FULL_PATH-lib: $FULL_PATH"
echo "     PATH-lib: $PATH"

Then I repeated the process for ast/src/cmd/ksh93/tests/util/run_test.sh:

function log_error {
    typeset lineno=$1
    print -u2 -r "<E> run_test[$lineno]: ${*:2}"
}
alias log_error='log_error $LINENO'

I inserted the following right after the above function:

if [[ -z "$PATH" ]]; then
    echo '=============>>>> ERROR: PATH was empty!'
    PATH=/home/xxx/oclint-0.13.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
fi

Skip down a little again after

export OLD_PATH=$PATH
export SAFE_PATH="$TEST_DIR:$TEST_DIR/space dir:$TEST_ROOT:$BUILD_DIR/src/cmd/builtin"
export FULL_PATH=$SAFE_PATH:/xxx/bsd:/usr/gnu/bin:/usr/xpg4/bin:$OLD_PATH
export PATH=$FULL_PATH

Added this

echo " OLD_PATH-cmd: $OLD_PATH"
echo "SAFE_PATH-cmd: $SAFE_PATH"
echo "FULL_PATH-cmd: $FULL_PATH"
echo "     PATH-cmd: $PATH"

Finally, I executed the meson test --setup=malloc command. When I examined its meson-logs/testlog-malloc.txt file, I noticed several things:

1. It completed successfully with only skipping 3 tests.
2. The first part of the log file that was created by ast/src/lib/libast/tests/run_test.sh had an OLD_PATH that was similar to my user PATH but possessed only the default system directories, OLD_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin. This path was missing my oclint bin directory.
3. The second part of the log file that was created by ast/src/cmd/ksh93/tests/util/run_test.sh did not have an OLD_PATH defined. Thanks to my condition check, the script received a hard coded path to utilize allowing all the viable tests to run.

Now with success and knowledge of what is going wrong with meson test --setup=malloc command, I wondered why running just meson test works. If you run just meson test for all tests or just a single test, such as meson test b_alias.sh, the correct user path is available for use for OLD_PATH.

Problem Summary

meson test --setup=malloc command fails to incorporate path from user environment. In particular, ast/src/lib/libast/tests/run_test.sh uses another source for PATH other than user's perhaps even hard coded. ast/src/cmd/ksh93/tests/util/run_test.sh somehow fails to inherit PATH from user environment. Note: Running a meson test without the malloc option executes successfully by utilizing user's PATH for all tests as well as single test.

time meson test --setup=malloc

Tests	Time
OK: 290	real 3m43.417s
FAIL: 0	user 2m2.064s
SKIP: 3	sys 0m20.158s
TIMEOUT: 0

time meson test

Tests	Time
OK: 290	real 3m44.155s
FAIL: 0	user 2m1.803s
SKIP: 3	sys 0m20.488s
TIMEOUT: 0

[hyenias]

krader1961, I did not notice your updated comment within my web browser. I had been editing my comment for quite some time.
Yes, my old PC is an x86_64 and do plan on focusing with that. Yes, I repeated my outlined steps on other boxes/platforms.
You are right. I have never run a "ninja install" command. However, the b_typeset.sh test passes.
I confirm your 4 decades of experience as I am human and my mind did see what it wanted to see or a least did not have the ability to determine the differences.

[krader1961]

It appears that enabling malloc debugging causes your platform to unset, or otherwise mangle, $PATH. Nothing in meson or the ksh test framework would cause what you observed. Similarly, nothing in this project would cause your oclint bin directory to be removed from $PATH for a non-debug malloc test run. Which suggests that something in the configuration of your /bin/sh, /bin/bash, or whatever shell you normally use, is mangling $PATH. When I display $PATH in src/cmd/ksh93/tests/util/run_test.sh it includes all the entries found in my $PATH. From my testlog-malloc.txt after adding set -x just before the section of run_test.sh that sets up the PATH vars:

+ OLD_PATH=/Users/krader/symlinks:/Users/krader/bin:/Users/krader/sbin:/usr/local/sbin:/usr/local/bin:/Users/krader/go/bin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/opt/gettext/bin

We can certainly test for $PATH being unset/empty and provide a reasonable default value (e.g., by using the output of getconf PATH) but that really shouldn't be necessary.

@hyenias What is your interactive shell? Bash? Zsh? Csh? Something else? Have you checked your shell config files to see if they are the reason $PATH is being mangled this way? It might be interesting to see what the output of a /usr/bin/env command at the top of src/cmd/ksh93/tests/util/run_test.sh looks like on your system.

[krader1961]

Also, again, please attach the files in the build/meson-logs/ directory. We know what those logs should contain and can examine them for unexpected content much more easily than someone new to the project 😄

[krader1961]

@hyenias Does https://github.com/krader1961/ast/commit/8faaac480e8a9733a50c08ae972a3cc9b9797d96, without your edits to the test scripts, fix this issue?