Spec Tests

Back to Contributing.

Spec Tests are written with the sh_spec.py framework. There are some comments at the top of that file.

Quick Start

$ ./spec.sh install-shells
$ ./spec.sh smoke   # a single file -- look at the list of functions
$ ./spec.sh all     # all in parallel

Test-Driven Development

The idea behind the spec tests to figure out how OSH should behave (the spec) by taking an automated survey of the behavior of other shells. You can follow a test-driven process like this:

1. Write spec tests for a new feature.
2. Make the spec tests pass on every shell except OSH. If shells differ in behavior, this may require annotations on the expected results.
   1. A given shell may not implement a feature. For example, bash and zsh both implement the dirs builtin, but mksh and dash don't.
   2. Shells may implement the same feature differently. For example, pushd in bash prints the stack to stdout, but pushd in zsh doesn't.
3. Write code in OSH to make the tests pass.

Notes

1. Spec tests don't run in an isolated environment, but they should (issue 42). Right now I run them on Ubuntu 16.04.