Need help with initial meson test

[hyenias]

Description of problem:
After going through all the instructions I could find on the site, some test cases have either failed or been skipped. meson-logs/testlog-malloc.txt indicates that various programs along its path cannot be found.

Ksh version:
GNU/Linux x86_64 Ubuntu 18.04
2020.0.0-beta1-192-g0d76ee4b

How reproducible:
Run meson with --prefix ~/kshdev and meson test.

Steps to reproduce:

1. From within $HOME directory

• mkdir git
• mkdir kshdev

2. cd git
3. git clone https://github.com/att/ast.git
4. cd ast
5. meson --prefix ~/kshdev build
6. ninja -C build
7. cd build
8. meson test --setup=malloc

• Full log written to .../git/ast/build/meson-logs/testlog-malloc.txt
• This log file is full of test cases that cannot find programs along its path.

Actual results:
OK: 277
FAIL: 1
SKIP: 15
TIMEOUT: 0

123/293 API/ksh/ksh_debug FAIL 0.13 s

Within the testlog-malloc.txt file under a test case between the stderr section:
stderr:
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh: line 219: mkfifo: not found
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh: line 220: mkfifo: not found
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh: line 239: mkdir: not found
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh: line 240: mkdir: not found
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh[464]: run_api: line 270: diff: not found
run_test[272]: Stderr for ksh_debug had unexpected differences:
xxx/git/ast/src/cmd/ksh93/tests/util/run_test.sh[464]: run_api: line 273: diff: not found

Expected results:
No previous experience to compare with, not sure. Overall, I would hope that all the possible tests would run and pass. All of the not found programs have been installed and exist along the PATH.

Additional info:
I am a new github user and new to this ksh repo. I believe that I have installed all of the required programs that were mentioned within the CONTRIBUTING.md and wiki section Building and Debugging (meson). I look forward to getting my development environment up and running so that I can contribute to this ksh project.

[krader1961]

That's a great first issue. Thanks for contributing. Those errors should not occur. Are you perhaps running this in a chroot or other sandboxed environment? The fact the unit test harness can't find essential commands like diff, mkfifo and mkdir in $PATH should cause many more failures than you reported. Note that we've been building and testing on multiple Linux, BSD, Cygwin, and other platforms (e.g., Solaris) for a long time. None of those environments exhibit the behavior you describe. So something is unusual about your build environment. What that is and whether we can work around it is unclear.

Are you seeing those "not found" errors for all unit tests? What do you see in the log if you add set -x near the top of src/cmd/ksh93/tests/util/run_test.sh and run a single unit test such as `meson test b_alias.sh*.

P.S., One of the test environments I use to verify all changes to this project is a Ubuntu 16.04 32-bit VM. So it is definitely possible to build and successfully pass all unit tests on Ubuntu (as well as several other Linux distros).

[krader1961]

some test cases have either failed or been skipped.

FWIW, Skipping some tests is expected due to deficiencies of specific platforms. For example, not having a /proc pseudo-filesystem. Those deficiencies either mean the unit test itself cannot run successfully or the behavior of the ksh binary depends on features that the platform does not provide. If any skipped tests should have been run, @hyenias, because your platform supports the necessary features we will be happy to enable those tests on your platform and deal with any test failures that result from that change.

[hyenias]

Bare metal, booted up an old PC x86_64 and installed Linux Mint 19.3 on it. Some of the required tooling was not available as packages on my ARM single board computers. Just a few hours ago, I successfully ran the TravisCI and all tests/builds past. :) Then, I moved to creating a local test environment.

Within the testlog-malloc.txt file, the first test that has an issue is 123/293 API/ksh/ksh_debug with a FAIL. All prior tests above execute fine with an OK status. The immediate test above ksh_debug is 122/293 API/tm/tvgettime. From the ksh_debug point and below, the stderr sections have not founds for cat, rm, and chmod as well.

After editing the run_test.sh with set -x statement, running "meson test b_alias.sh"; the various OLD_PATH, SAFE_PATH, FULL_PATH, and PATH all have the correct system directories. This individual test completes successfully.

[krader1961]

the first test that has an issue is 123/293 API/ksh/ksh_debug with a FAIL. All prior tests above execute fine with an OK status.

That is not surprising. The API tests (excluding the API/ksh/ tests) do not depend on the ksh scripting language. That the non-API ksh scripting tests fail with errors such as cat and rm not found tells me you either

1. don't have those programs installed (which is unlikely), or

2. there is something unusual about your test environment. For example, on my Ubuntu server cat is /bin/cat and it is found when running meson test.

That API/ksh/ksh_debug fails is not particularly surprising since it depends on features, such as a working addr2line program, which may not be available on your system. In that case the unit tests should be skipped. If that is not happening we need to modify the test framework to skip the affected tests if something like addr2line isn't available.

[krader1961]

@hyenias You wrote

Just a few hours ago, I successfully ran the TravisCI and all tests/builds past. :) Then, I moved to creating a local test environment.

Are you trying to cross-compile? That isn't something we have spent a single second thinking about. So if that is what you are trying to do I am not surprised it results in failures.

[hyenias]

Programs do exist:
$ whence mkfifo cat chmod rm diff mkdir
/usr/bin/mkfifo
/bin/cat
/bin/chmod
/bin/rm
/usr/bin/diff
/bin/mkdir
$
No, I am not trying to cross-compile. I am new to all of this and do not currently possess those skills. My plans are to load up and compile ksh on each SBC.

I am just learning and attempting for the first time to contribute to an open source project that I believe in. I have attempted to follow the provided web docs to the best of my abilities. At this time, I do not know how to fix the situation of having error statements of not finding files.

Any more suggestions, hints? I will try.

[hyenias]

FYI, I just spot checked other tests down the line that passed but had the not founds in them and after running them singularly no issues were noted in the log files.

[hyenias]

$ which addr2line
/usr/bin/addr2line
$

[hyenias]

I just repeated my above steps on one of my ARM boxes running Ubuntu 18.04 and the log file has the same not founds in them. However, on this system it has an additional failure at:
20/293 API/misc/debug FAIL 0.29 s
OK: 276
FAIL: 2
SKIP: 15
TIMEOUT: 0

Starting at the "123/293 API/ksh/ksh_debug FAIL 0.30 s" part of the log file and below the not founds show up on all remaining tests to end of file.

[krader1961]

Bare metal, booted up an old PC x86_64 and installed Linux Mint 19.3 on it. Some of the required tooling was not available as packages on my ARM single board computers.

Is it x86_64 or ARM? I get the sense you're trying to build ksh on both architectures and having problems on both. In which case I'd like to suggest focusing on one of them, preferably x86_64, until you have a clean build and all tests pass. Then we can deal with other architectures.

the various OLD_PATH, SAFE_PATH, FULL_PATH, and PATH all have the correct system directories.

Please attach the entire build/meson-logs/testlog.txt log file to this issue. It's not that I don't believe you but in my four decades working with computers it has been my experience that people see what they expect to see. Thus sometimes overlooking an important detail (e.g., semi-colons where colons should be present).

Please also attach the build/meson-logs/meson-log.txt log file.

P.S., I see your original comment makes no mention of doing ninja install before running meson test --setup=malloc. This will cause one test in b_typeset.sh to fail because it won't find the expected autoloaded functions. I'll update the documentation to make that clearer. Other than that I ran your sequence of commands on my Ubuntu system without seeing any of the "not found" errors you're seeing.

P.P.S., Note that meson thinks nearly all the unit tests are passing. So either those "not found" errors are either irrelevant or they're keeping the unit tests from even being run. But even if irrelevant they should not be present in the test log so we need to determine the root cause of those messages.

[hyenias]

After review of my notes and history log, I made some mistakes in the information I provided yesterday. I actually ended up editing two separate files with the set -x command at the top. With all the new, flipping back-and-forth from web browser and command prompts, as well as paging through 120k lines of a log file; I focused on the wrong run_test.sh file. My mind no longer possessed the awareness needed to see the differences. I apologize.

By having a break and spending more time with the log file, I have become more familiar with it and as such noticed the details today. I can now express the problem to a better degree.

The meson test commands create two separate log files depending on how you run them:

• meson test: meson-logs/testlog.txt
• meson test --setup=malloc: meson-logs/testlog-malloc.txt

Each of these log files have their content created from two separate run_test.sh scripts. The libast run script executes first for tests 1-122 ending with "122/293 API/tm/tvgettime". The cmd run_test.sh scripts picks up at "123/293 API/ksh/ksh_debug" to end of file "293/293 special-dev-paths.sh/shcomp".

• 1-122: ast/src/lib/libast/tests/run_test.sh
• 123-293: ast/src/cmd/ksh93/tests/util/run_test.sh

I wanted a little easier way for my untrained eyes to see the PATHing problems in each of these run_test.sh scripts. So, I removed the set -x commands from the top of the files and adding in the following.

In ast/src/lib/libast/tests/run_test.sh after:

readonly test_name=${x%.c}
readonly api_binary=$BUILD_DIR/$1
readonly test_src_dir=$2

I added:

if [[ -z "$PATH" ]]; then
    echo '=============>>>> ERROR: PATH was empty!'
    PATH=/home/xxx/oclint-0.13.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
fi

Then skip down a little after:

export OLD_PATH="$PATH"
export SAFE_PATH="$TEST_DIR:$TEST_DIR/space dir:$test_src_dir:$BUILD_DIR/src/cmd/builtin"
export FULL_PATH="$SAFE_PATH:$OLD_PATH"
export PATH="$FULL_PATH"

I added:

echo " OLD_PATH-lib: $OLD_PATH"
echo "SAFE_PATH-lib: $SAFE_PATH"
echo "FULL_PATH-lib: $FULL_PATH"
echo "     PATH-lib: $PATH"

Then I repeated the process for ast/src/cmd/ksh93/tests/util/run_test.sh:

function log_error {
    typeset lineno=$1
    print -u2 -r "<E> run_test[$lineno]: ${*:2}"
}
alias log_error='log_error $LINENO'

I inserted the following right after the above function:

if [[ -z "$PATH" ]]; then
    echo '=============>>>> ERROR: PATH was empty!'
    PATH=/home/xxx/oclint-0.13.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
fi

Skip down a little again after

export OLD_PATH=$PATH
export SAFE_PATH="$TEST_DIR:$TEST_DIR/space dir:$TEST_ROOT:$BUILD_DIR/src/cmd/builtin"
export FULL_PATH=$SAFE_PATH:/xxx/bsd:/usr/gnu/bin:/usr/xpg4/bin:$OLD_PATH
export PATH=$FULL_PATH

Added this

echo " OLD_PATH-cmd: $OLD_PATH"
echo "SAFE_PATH-cmd: $SAFE_PATH"
echo "FULL_PATH-cmd: $FULL_PATH"
echo "     PATH-cmd: $PATH"

Finally, I executed the meson test --setup=malloc command. When I examined its meson-logs/testlog-malloc.txt file, I noticed several things:

1. It completed successfully with only skipping 3 tests.
2. The first part of the log file that was created by ast/src/lib/libast/tests/run_test.sh had an OLD_PATH that was similar to my user PATH but possessed only the default system directories, OLD_PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin. This path was missing my oclint bin directory.
3. The second part of the log file that was created by ast/src/cmd/ksh93/tests/util/run_test.sh did not have an OLD_PATH defined. Thanks to my condition check, the script received a hard coded path to utilize allowing all the viable tests to run.

Now with success and knowledge of what is going wrong with meson test --setup=malloc command, I wondered why running just meson test works. If you run just meson test for all tests or just a single test, such as meson test b_alias.sh, the correct user path is available for use for OLD_PATH.

Problem Summary

meson test --setup=malloc command fails to incorporate path from user environment. In particular, ast/src/lib/libast/tests/run_test.sh uses another source for PATH other than user's perhaps even hard coded. ast/src/cmd/ksh93/tests/util/run_test.sh somehow fails to inherit PATH from user environment. Note: Running a meson test without the malloc option executes successfully by utilizing user's PATH for all tests as well as single test.

time meson test --setup=malloc

Tests	Time
OK: 290	real 3m43.417s
FAIL: 0	user 2m2.064s
SKIP: 3	sys 0m20.158s
TIMEOUT: 0

time meson test

Tests	Time
OK: 290	real 3m44.155s
FAIL: 0	user 2m1.803s
SKIP: 3	sys 0m20.488s
TIMEOUT: 0

[hyenias]

krader1961, I did not notice your updated comment within my web browser. I had been editing my comment for quite some time.
Yes, my old PC is an x86_64 and do plan on focusing with that. Yes, I repeated my outlined steps on other boxes/platforms.
You are right. I have never run a "ninja install" command. However, the b_typeset.sh test passes.
I confirm your 4 decades of experience as I am human and my mind did see what it wanted to see or a least did not have the ability to determine the differences.

[krader1961]

It appears that enabling malloc debugging causes your platform to unset, or otherwise mangle, $PATH. Nothing in meson or the ksh test framework would cause what you observed. Similarly, nothing in this project would cause your oclint bin directory to be removed from $PATH for a non-debug malloc test run. Which suggests that something in the configuration of your /bin/sh, /bin/bash, or whatever shell you normally use, is mangling $PATH. When I display $PATH in src/cmd/ksh93/tests/util/run_test.sh it includes all the entries found in my $PATH. From my testlog-malloc.txt after adding set -x just before the section of run_test.sh that sets up the PATH vars:

+ OLD_PATH=/Users/krader/symlinks:/Users/krader/bin:/Users/krader/sbin:/usr/local/sbin:/usr/local/bin:/Users/krader/go/bin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/opt/gettext/bin

We can certainly test for $PATH being unset/empty and provide a reasonable default value (e.g., by using the output of getconf PATH) but that really shouldn't be necessary.

@hyenias What is your interactive shell? Bash? Zsh? Csh? Something else? Have you checked your shell config files to see if they are the reason $PATH is being mangled this way? It might be interesting to see what the output of a /usr/bin/env command at the top of src/cmd/ksh93/tests/util/run_test.sh looks like on your system.

[krader1961]

Also, again, please attach the files in the build/meson-logs/ directory. We know what those logs should contain and can examine them for unexpected content much more easily than someone new to the project 😄

[krader1961]

@hyenias Does https://github.com/krader1961/ast/commit/8faaac480e8a9733a50c08ae972a3cc9b9797d96, without your edits to the test scripts, fix this issue?

[hyenias]

I had to do some research on the various shells and then investigate them. Currently, my default login shell is set to bash. I knew about the /etc/environment file that only contains the default path for my system; in the past, I had removed /usr/games:/usr/local/games from it.

As for your interactive shell influencing the mangling of the PATH environment variable train-of-thought, I decided to create a new test user account on my system and remove as much startup script loading for each of the interactive shells I could before running the various meson tests again. This test user does not have the oclint program available to it. Unfortunately, no matter which interactive shell I used; the not founds persisted when malloc was used. I have selected to run only the single tests for comparison.

Updated steps for test build

1. From within $HOME directory

• mkdir git
• mkdir kshdev

2. cd git
3. git clone https://github.com/att/ast.git
4. cd ast
5. meson --prefix ~/kshdev build
6. ninja -C build
7. cd build
8. ninja install
9. MESON options:

• meson test --setup=malloc
• meson test
• meson test --setup=malloc test_script
• meson test test_script

$ uname -a
Linux testbox 5.0.0-37-generic #40~18.04.1-Ubuntu SMP Thu Nov 14 12:06:39 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ echo $SHELL
/bin/bash
$ bash --version
GNU bash, version 4.4.20(1)-release (x86_64-pc-linux-gnu)
$ which sh
/bin/sh
$ ls -l /bin/sh
lrwxrwxrwx 1 root root 4 Dec 23  2018 /bin/sh -> dash
$ apt list --installed dash
Listing... Done
dash/bionic,now 0.5.8-2.10 amd64 [installed]
$ zsh --version
zsh 5.4.2 (x86_64-ubuntu-linux-gnu)

SHELL	COMMAND	NOTES
bash	exec bash	replace current login shell with normal bash instance thus skipping most of the startup code. I have renamed the .bshrc file so it could have not loaded.
_	unset -f command_not_found_handle	only one function was left within namespace after new bash instance
_	_	meson test --setup=malloc b_alias.sh: meson-logs/testlog-malloc.txt: not founds persist
_	_	meson test b_alias.sh: meson-logs/testlog.txt: successful
dash	exec dash	meson test --setup=malloc b_alias.sh: meson-logs/testlog-malloc.txt: not founds persist
_	_	meson test b_alias.sh: meson-logs/testlog.txt: successful
zsh	exec zsh	select q to Quit and do nothing.
_	_	meson test --setup=malloc b_alias.sh: meson-logs/testlog-malloc.txt: not founds persist
_	_	meson test b_alias.sh: meson-logs/testlog.txt: successful

I will start working on the log files. Then, move towards testing your fix out.

[hyenias]

Here the files.

[hyenias]

Yes, your commit 071352e from your ast fork at krader1961 on github under the empty-path branch worked for the malloc tests. As before starting at the API/ksh/ksh_debug test, the path was empty/unset and your additions provided a minimal path.

--- stderr ---
<W> run_test[78]: PATH is empty/unset -- using $(/usr/bin/getconf PATH)
<W> run_test[81]: PATH is now /bin:/usr/bin
-------

[krader1961]

@hyenias I'd still like to know why your ksh test environment does not have a $PATH env var but, since your situation appears to be extremely atypical, I'm not willing to spend more time debugging this issue.

[hyenias]

@krader1961 Thank you for all of the hard work and time spent towards attempting to resolve this issue I opened up. As a new github user and with a different level of willingness to help the project, I was taken back to find my issue closed. From my viewpoint, by having my issue closed; it hides away an unresolved problem from others in the community that might wish to assist in resolving this issue. Since I do not see a reopen button on my web browser, this causes me to have a sad face. After all, my issue was closed without concurrence or a chance to reply after only 4 hours. It made me think, "Is this a team of 1?" Surely not, as I know there are others in the community. Or perhaps I am just too new to this all and in order to perform a commit to master branch you needed to close the associated issue along with it.

What were the results of your analysis of the log files I provided? It would be nice if you shared your efforts with the community. I, for one, am trying to learn.

At this time, I particularly do not wish your workaround to be incorporated into master as it masks problem(s). If the environment is troubled, let it be troubled and hopefully cause something to error out or fail in some way so it can be addressed. Perhaps a separate PATH test should be added at the beginning of the cases but I prefer to try to resolve the problem. As I already know how to manually set my complete path into the run scripts, I can do that on my test boxes or just not test with the malloc option until I figure out why.

[krader1961]

@hyenias The logs show, indirectly, that $PATH is set to the empty string. The ksh program, and libast, actually has a workaround for $PATH being unset:

ast/src/cmd/ksh93/data/msg.c

Line 163 in 2950ede

const char e_defpath[] = "/bin:/usr/bin:";

ast/src/lib/libast/path/pathbin.c

Line 36 in 2950ede

static const char *def_cs_path = "/bin:/usr/bin:/usr/local/bin";

However, that existing workaround only works if the var is unset:

KSH PROMPT:1: env -u PATH /bin/ksh -c 'echo "PATH |$PATH|"; type cat'
PATH ||
cat is a tracked alias for /bin/cat

If set to the empty string the workaround to have a sensible, minimal, $PATH isn't used:

KSH PROMPT:2: env PATH='' /bin/ksh -c 'echo "PATH |$PATH|"; type cat'
PATH ||
/bin/ksh: whence: cat: not found

The reason all of the ksh tests were passing for you, despite all the command "not found" errors in the test log, is that the resulting test harness script had nothing but the #! line because the cat command couldn't be found. I'm still pondering what, if anything, to do about this failure mode that indicates the unit tests pass when in fact they weren't even run.

And, after sleeping on the issue, I've decided to move the workaround, minus the warnings about PATH being empty, into the recently introduced global config file.

I closed this issue because there isn't anything else for us to do. Figuring out whatever is causing $PATH to be set to the empty string on your system(s) when the ksh unit tests are run isn't really our responsibility. I'm only implementing an improved workaround (compared to the one that has existed for at least 7 years) because it is generally useful in case someone does the equivalent of env PATH='' ksh. As I said earlier if you figure out what is causing that to happen on your system(s) I encourage you to add a comment to this issue. If you trace the behavior to a bug in ksh, which is extremely unlikely, we'll reopen the issue and fix the bug. If you want to continue investigating why this behavior occurs only on your system(s) the strace command might be useful. As might adding some strategic DPRINTF() calls to the ksh source.

[hyenias]

Nice reply! Nothing but gold. Thank you so much for educating me on the insights gained from my logs and more. Especially, I now know why the two run_test.sh's differed. That narrows the possibilities. Excellent, work.

I am glad that this issue has brought some light on the unit tests not failing when they were not even run. To me this is huge problem as tests should fail if they do not work that is why you have them. Hopefully, you or the team will find a way to alter the meson test scripts to account for it.

I did not know about the existence of a new global config file. I will have a look at this latter when I finally get my test environment up and running as I want to be able to contribute code changes to this project.

I will continue to investigate why this behavior occurs on my systems. I am not convinced that my issue only occurs with my setups as they are mostly fresh installed systems from various Ubuntu 18.04 based Linux distributions. I will circle back to the basics before attempting to learn the strace command or using some DPRINTF() calls. Thank you for making me aware of those commands.

[krader1961]

I am not convinced that my issue only occurs with my setups as they are mostly fresh installed systems from various Ubuntu 18.04 based Linux distributions.

Sure, but consider that is also true every time the unit tests are run in each Travis CI environment. I'm sure I, and anyone else, will be able to reproduce this once we know what it is about your environment that is triggering this behavior. But so far, in the past two years, only you, and no one else, has reported seeing this behavior. Which is a pretty strong indicator there is something unique about your environment. The question is, what is that unique characteristic?

One frequent source of unexpected behavior is incorrect handling of a locale that is not ASCII compatible (which includes the ISO 8859 and Unicode UTF-8 encodings). It's unlikely to be the cause of this issue but I'm curious what the output of locale is on your system if it isn't en_US.UTF-8 simply because an unusual locale is often the source of unexpected behavior -- for many programs, not just ksh. This is the type of thing that most people won't notice. So while your locale isn't likely the cause it's likely something of that sort.

[hyenias]

Took a little time but I found the root cause, a workaround, and a fix. While learning from this issue I have found other issues. One of which being the restricted shell issue that I believe you have already solved. I will work on collecting my notes and conveying the results for this issue then open up other issues for other things I found.

[krader1961]

@hyenias That's great. And, despite what some other individuals seem to think, I actually do care about all bugs in this project. Especially those introduced since the ksh93v- release; i.e., by the work done by myself and @siteshwar over the past two years.

@jghub Read this issue and think about how the O.P. acted when they found a problem. Then think about how it compares to your behavior.

[jghub]

@hyenias:
in case you wonder what krader's last remark refers to, please look at #1464(not so much the primary topic but what comes further down in the thread) and #1466 for an explanation of context (you were right with your "team of 1 guess", mostly, and rightly objected to your issue being closed w/o a resolution in my view) and possible future developments.