Spec Tests

Back to Contributing.

Spec Tests are written with the sh_spec.py framework. There are some comments at the top of that file.

Quick Start

$ ./spec.sh install-shells
$ ./spec.sh smoke   # a single file -- look at the list of functions
$ ./spec.sh all     # all in parallel

Test-Driven Development

The idea behind the spec tests to figure out how OSH should behave (the spec) by taking an automated survey of the behavior of other shells. I follow a test-driven process like this:

1. Write spec tests for a new feature.
2. Make the spec tests pass on every shell except OSH. If shells differ in behavior, this may require annotations on the expected results.
   1. A given shell may not implement a feature. For example, bash and zsh both implement the dirs builtin, but mksh and dash don't.
   2. Shells may implement the same feature differently. For example, pushd in bash prints the stack to stdout, but pushd in zsh doesn't.
3. Write code in OSH to make the tests pass.

After step 2, all columns should be green or yellow, except OSH. After step 3, the OSH column should be green or yellow as well.

Format of the Tests

The spec/foo.test.sh files are designed to be syntax-highlighted like normal shell scripts.

Each line is a "token". Lines with three hashes ### begin a test case. Lines with two hashes ## add metadata to the test case, e.g. assertions to make on status/stdout/stderr. (Legacy: sometimes we use one hash.)

Basic Test:

### test for echo
echo 1
## status: 0
## stdout: 1

Test with multiline assertion:

### multiline test
echo 1
echo 2
## status: 0
## STDOUT:
1
2
## END

You can also add qualifiers to tests, to account for the different behavior of different shells. Example:

### multiline test with qualifier
echo 1
echo 2
if test $SH = dash; then
  echo 3
fi
## status: 0
## STDOUT:
1
2
## OK dash STDOUT:
1
2
3
## END

You can also write assertions as JSON:

### test for echo with tab
echo -e '1\t2'
## status: 0
## stdout-json: "1\t2\n"

Notes

• Spec tests don't run in an isolated environment, but they should (issue 42). Right now I run them on Ubuntu 16.04.
• It's OK to check in tests that don't pass on OSH yet. This helps because it specifies the behavior we want to implement. However, spec tests should not be submitted until they are green/yellow on OTHER shells. (They can be disabled if the feature isn't implemented at all in a shell.)
  • To prevent failing test runs, adjust --allowed-failures in test/spec.sh. For example, --allowed-failures 3 will make the sh_spec.py framework exit 0 if there are exactly 3 red failures in the OSH column. We only releaes OSH when all spec tests exit 0.