Feature skip #59

hlangeveld · 2015-04-19T09:57:17Z

Adds a few things:

Counters for passed and skipped assertions.
A skip keyword that will cause the next assert to ignore any failures and increase the skipped counter instead.
The colour yellow for skipped tests.
Suppression of colour output if stdin is not a terminal.

Currently based on the POSIX update.

Remove quotes from command substitution when used in an assignment. Two assertions were reading file VERSION where once is sufficient.

Adds function `skip`. If a test invokes `skip` prior to an assertion, the assertion will be resolved, but failing match will not result in a failure, and instead count as a *skipped* test. This update adds counters for passed and skipped tests. This also changes the summary line, and color codes. A third color is introduced for skipped tests: yellow. The summary line will report numbers in black, with the words 'failures', 'passed', and 'skipped' in red, green, and yellow, respectively. Finally, tests are included for the presence of a terminal. If no terminal is present, color codes are replaced as follows: green OK passed red NO failed yellow ?? skipped

And that's 1 failure and 1 skipped assertion.

hlangeveld · 2015-04-19T11:25:12Z

I'm looking into adding support for an option -f <expected_failure_count> to make this produce its full output and to make it pass the full test in the case of expected failures.
Check my 'travis' branch from this feature for live updates.

Sometimes you want a test or assertion to Fail *visibly*. However, we also want to use the shell `-e`, or *exit on error* option. This extends the expected failure matcher with a temporary suspension of that option.

Except for an example test, `shpec` itself does not check the results of a sourced file.

shell options now enabled -e Exit on error. Can be suspended temporarily for some tests. -u Error on undefined variables. Checking for errors requires that we capture exit status for failing commands with `|| _r=$?` The `||` will prevent error exit, while the assignment preserves the exit status for the actual assertion.

POSIX case labels can have balanced parentheses now. Fixes a minor annoyance with editors like vi[m].

Using `skip_next_assert` is more explicit about what it does. It implies that normal processing is restored *after* the assertion.

The function name clarifies the meaning to the reader.

The second `case` block contained only one entry. Removed the case clutter. Integrate failure count into final report and `shpec` exit status/return value.

Indentation was one level to high for a command substitution. `-f` option had assignment and shift in one line

We renamed the keyword to `skip_next_assert` but did not change the name of the test. Fixed now.

hlangeveld · 2015-04-19T23:53:56Z

I would like to see some review before we even consider to pull this.
(For one thing, I forgot to bump the version. Again.)

In the shpec files, this just adds one keyword: skip_next_assert.

Skipping some assertions requires support for managing shell errors,
extra colours in output, and extra counters beyond the number of failed tests/assertions.
We now add counters for passed and skipped assertions.

In passing, this eliminated another external command by using a case statement
instead of grep. We also cater for tests without a terminal by prefixing output with
the two character codes 'NO', 'OK', and '??'.

An extra challenge was added by imposing the -e and -u options onto shpec.
It makes for more robust scripts, but potential errors are easily overlooked.

hlangeveld · 2015-04-19T23:57:24Z

.travis.yml

+script:
+  - time bash bin/shpec -f 1 shpec/shpec_shpec.sh
+  - time dash bin/shpec -f 1 shpec/shpec_shpec.sh
+  - time ksh  bin/shpec -f 1 shpec/shpec_shpec.sh


Now explicitly run the test suite in all known POSIX shells.

All three shells should produce the same output.

Any specific reason we are not testing against zsh?

(if it's about the end keyword, it can be disabled, cf https://github.com/AdrieanKhisbe/shpec/blob/dev/shpec.plugin.zsh#L15)

@AdrieanKhisbe @hlangeveld - see https://travis-ci.org/hlangeveld/shpec/builds/59189529
(ie. to allow failure on zsh - until you put a fix in)

@locochris indeed it's the end keyword. :)

Question is where the good place to disable it?.
So far I did it in a wrapping alias. (to turn it on and off). But I thinks it might better to get this inside the script itself.

rylnd · 2015-04-21T19:40:16Z

@hlangeveld let's focus on getting #58 merged before pushing this any further. My main concern is that without discussing and collaborating on these features, we can easily get away from shpec's base intentions.

hlangeveld · 2015-04-21T20:33:41Z

On 04/21/15 21:40, Ryland Herrick wrote:

@hlangeveld https://github.com/hlangeveld let's focus on getting #58 #58 merged before
pushing this any further. My main concern is that without discussing and collaborating on these features, we can easily get away
from shpec's base intentions.

That's fine. I did some not very clean refactoring in this bit, and it got interspersed with adding the feature.
I'd rather complete the POSIX PR first, then refactor, and then we can discuss the 'skip' feature.

Cheers all,
Henk

@locochris

Use advanced .travis.yml to generate separate jobs for each shell, by @locochris

rylnd · 2017-04-02T21:42:23Z

@hlangeveld what is the status of this PR?

hlangeveld · 2017-04-03T23:17:32Z

Good question. Let me have a look...
.
.
.
In its current state I believe I need to refactor the whole thing again before resubmitting.
Not tonight though.

hlangeveld added 6 commits April 19, 2015 11:48

Remove unnecessary quotes

fbaed9c

Remove quotes from command substitution when used in an assignment. Two assertions were reading file VERSION where once is sufficient.

Does travis support sh as a language?

4ce719d

Tell travis what to expect

934db77

And that's 1 failure and 1 skipped assertion.

Add option for expected number of failures

d35fa6a

add trace

8adb1cb

hlangeveld closed this Apr 19, 2015

hlangeveld added 13 commits April 19, 2015 17:04

Skip exitonerror set -e for expected fails

0bd8503

Sometimes you want a test or assertion to Fail *visibly*. However, we also want to use the shell `-e`, or *exit on error* option. This extends the expected failure matcher with a temporary suspension of that option.

Stop checking status of a shpec file.

8d133e4

Except for an example test, `shpec` itself does not check the results of a sourced file.

Add double parentheses for all case labels

08d362d

POSIX case labels can have balanced parentheses now. Fixes a minor annoyance with editors like vi[m].

Rename skip to skip_next_assert

e24c270

Using `skip_next_assert` is more explicit about what it does. It implies that normal processing is restored *after* the assertion.

Add function terminal_input()

ec3593c

The function name clarifies the meaning to the reader.

Remove redundant case block, add final report

4464d91

The second `case` block contained only one entry. Removed the case clutter. Integrate failure count into final report and `shpec` exit status/return value.

Restore function terminal_input()

9b1c1fb

Fix indentation, split -f option processing

8e43a9a

Indentation was one level to high for a command substitution. `-f` option had assignment and shift in one line

Rename var _v to _version

179b5d5

See if we can make travis run three shells

4de1f08

Add ksh

1bc3a77

Rename skip test to match new keyword.

19938db

We renamed the keyword to `skip_next_assert` but did not change the name of the test. Fixed now.

hlangeveld reopened this Apr 19, 2015

hlangeveld reviewed Apr 19, 2015
View reviewed changes

chris added 2 commits April 20, 2015 11:09

Split travis tests up

b09d8ec

Allow failures in zsh

816e577

rylnd mentioned this pull request Apr 21, 2015

Run CI against multiple shells #61

Closed

Merge pull request #1 from locochris/feature_skip-split_travis_tests

e697e6d

Use advanced .travis.yml to generate separate jobs for each shell, by @locochris

rylnd mentioned this pull request Apr 24, 2015

Stop checking status of a shpec file. #69

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature skip #59

Feature skip #59

hlangeveld commented Apr 19, 2015

hlangeveld commented Apr 19, 2015

hlangeveld commented Apr 19, 2015

hlangeveld Apr 19, 2015

AdrieanKhisbe Apr 20, 2015

locochris Apr 20, 2015

AdrieanKhisbe Apr 20, 2015

rylnd commented Apr 21, 2015

hlangeveld commented Apr 21, 2015

rylnd commented Apr 2, 2017

hlangeveld commented Apr 3, 2017 •

edited

Loading

Feature skip #59

Are you sure you want to change the base?

Feature skip #59

Conversation

hlangeveld commented Apr 19, 2015

hlangeveld commented Apr 19, 2015

hlangeveld commented Apr 19, 2015

hlangeveld Apr 19, 2015

Choose a reason for hiding this comment

AdrieanKhisbe Apr 20, 2015

Choose a reason for hiding this comment

locochris Apr 20, 2015

Choose a reason for hiding this comment

AdrieanKhisbe Apr 20, 2015

Choose a reason for hiding this comment

rylnd commented Apr 21, 2015

hlangeveld commented Apr 21, 2015

rylnd commented Apr 2, 2017

hlangeveld commented Apr 3, 2017 • edited Loading

hlangeveld commented Apr 3, 2017 •

edited

Loading