Logging capabilities

[lzap]

Hello,

I am evaluating bats for running installation and end-to-end cli testing of our project. So far so good, but one issue I have is I would like to be able to inspect output of the installation steps.

I use normal @test taks to do installation steps. It looks like something is logged as /tmp/bats.PID.out and then deleted.

What I would like to have would be something like:

• --verbose option to watch the progress of all tests (output + stderr)
• combined output log file of all stdout/stderr output

I will be happy to contribute one or both features, as long as you are fine with that.

[sstephenson]

Sounds good. Let's start with a way to log the stdout and stderr of every test case to a file.

[lzap]

Ok this sounds great. Do you prefer one log file per test, or combined log file for all tests with some separators?

Also, how would you like to be this enabled? I guess not by default, so new option?

[sstephenson]

Yes, with a command-line option. Seems like the simplest way to make this work is just to append the log output to the filename specified in the option's argument.

[algernon]

I wouldn't mind if there would be a way to have the output in different files, perhaps the specified filename + test file name + test number (or filename + PID or any combination of these). That'd make it easier to find which output belongs where, than having to split a larger file first.

[lzap]

I was thinking about this as well, but problem with that are filenames.
Plain number is useless since you do not usually know the number of
test, PID does not help at all too. The only useful way is to name files
using test names, but I wonder if that is feasible. Test names can be
quite long, they contain spaces. Best approach could be:

N_test_name_with_underscores_instead_spaces.log

But this can be problematic because we need to escape special
characters (? * /) etc.

For this reason, I will start with single log file, I will probably add
some separators with test names, so it is easy to find the output there.

On Mon, Nov 04, 2013 at 01:55:22PM -0800, Gergely Nagy wrote:

I wouldn't mind if there would be a way to have the output in different files, perhaps the specified filename + test file name + test number (or filename + PID or any combination of these). That'd make it easier to find which output belongs where, than having to split a larger file first.

---

Reply to this email directly or view it on GitHub:
#28 (comment)

Later,

Lukas "lzap" Zapletal
irc: lzap #theforeman

[sstephenson]

Suggestion: prefix each line of log output with a timestamp and the test name.

[henriquemoody]

May be useful to use logger to create these logs. What you think, guys?

[mislav]

@henriquemoody I think that would be overengineering it. Just dump the output to a file (with test name + timestamp) and Sam suggested. No need to use an external utility or split output to multiple files. Keep it simple.

@lzap Want to do a PR?

[lzap]

Sure thing! I was waiting for more recommendations, it looks like I will
be doing this today/tomorrow:

• simple option to get one file log output
• prefix with time+test number
• some separators between tests
• plain bash solution

LZ

On Tue, Nov 05, 2013 at 01:38:24PM -0800, Mislav Marohnić wrote:

@henriquemoody I think that would be overengineering it. Just dump the output to a file (with test name + timestamp) and Sam suggested. No need to use an external utility or split output to multiple files. Keep it simple.

@lzap Want to do a PR?

---

Reply to this email directly or view it on GitHub:
#28 (comment)

Later,

Lukas "lzap" Zapletal
irc: lzap #theforeman

[lzap]

I am wondering how to pass the new option --log into libexec scripts. I was thinking about environment variable. Opinions?

[lzap]

Ok got a first version, here it how it looks like:

[29059] running a suite with one test file
[29102] a passing test
[29102] ok a passing test
[29059] ok running a suite with one test file
[29126] counting tests in a suite
[29126] ok counting tests in a suite
[29228] aggregated output of multiple tests in a suite
[29291] truth
[29291] ok truth
[29310] more truth
[29310] ok more truth
[29318] quasi-truth
[29318] ok quasi-truth
[29228] ok 1 truth
[29228] ok 2 more truth
[29228] ok 3 quasi-truth
[29228] ok aggregated output of multiple tests in a suite
[29353] a failing test in a suite results in an error exit code
[29416] truth
[29416] ok truth
[29435] more truth
[29435] ok more truth
[29443] quasi-truth
[29443] not ok quasi-truth
[29353] not ok 3 quasi-truth
[29353] ok a failing test in a suite results in an error exit code
[29473] running an ad-hoc suite by specifying multiple test files
[29541] truth
[29541] ok truth
[29560] more truth
[29560] ok more truth
[29568] quasi-truth
[29568] ok quasi-truth
[29473] ok 1 truth
[29473] ok 2 more truth
[29473] ok 3 quasi-truth
[29473] ok running an ad-hoc suite by specifying multiple test files
[29603] extended syntax in suite
[29645] truth
[29645] ok truth
[29664] more truth
[29664] ok more truth
[29672] quasi-truth
[29672] not ok quasi-truth
[29603] ok extended syntax in suite

Frankly, not the best output. Tried to output test numbers but it resets every suite, so PID looks much nicer in this case. I tried various dividers but it looks quite ugly, so I ended up with this simple and plain output. Opinions?

[JigarJoshi]

We are using it as part of CI build for cookbooks, I would like to see here is the ability to produce some (xml may be) reports, so that there is one common parser to read the reports and puts it on pretty html view on build reports, like most of the CI server does for some test frameworks (junit for example)

[petecarapetyan]

Is this functionality going to be added? I tried using --verbose but the command barfed, neither did a man page indicate same. I have latest as of today.

[lzap]

Bats can be easily integrated with Jenkins. See the README.

@lzap Want to do a PR?

I still have a local patch that does not work, no time :-(

Later,

Lukas "lzap" Zapletal
irc: lzap #theforeman

[sstephenson]

#50

[lzap]

@JigarJoshi the default bats TAP output has plugin for Jenkins, it works nice for us: http://ci.theforeman.org/view/Installer/job/systest_foreman/1566/

@sstephenson on my radar

[aerickson]

Here's an evolution of #50 that sends the output to the log also.

aerickson@91d1fbb