Automated tests

[th-we]

I added some test files in my pull requests, but there's no automation for testing. Could we collect some ideas how to test? Not sure whether there is an existing testing framework that fits the needs of this project. (Sorry for the lengthy issue - IMO tests will be important for sustainability.)

I think creating individual test files for individual problems is sensible. There are different kinds of tests I can think of:

1. Assert that the file runs smoothly through conversion and Lilypond processing (e.g. for this commit)
2. Assert that the output of the conversion fulfills certain conditions (like in this commit where the output should contain the string testlyrics)
3. Assert that the XSLT conversion succeeds or fails (the latter if it receives some bogus input or something it does not support)
4. Possibly assert that the graphical output (either PDF, PS, or SVG) fulfills certain conditions (but usually the generated Lilypond code should be easier to analyze than graphics)

So, how could one organize these different kinds of checks? We could add one or multiple <annot> elements with type="assert-test" in the header of each test file. These <annot>s should describe the tests to be performed. For case 2, it might look like this:

<annot type="assert-test" subtype="ly-analyze">
    grep 'testlyrics' "$ly"
</annot>

subtype="ly-analyze" would indicate that the script supplied as content will analyze the generated Lilypond file. If that command exits with code 0 (which grep does if it finds testlyrics), the test passes, otherwise it fails.

The different @types for the different kinds of tests I listed above could be:

1. ly-success
2. ly-analyze
3. xsl-success, xsl-fail
4. svg-analyze, ps-analyze, pdf-analyze

For xsl-success and xsl-fail, only the XSL transformation would have to be performed. For all others, Lilypond would have to be run as well.

A script for testing could look like this:

#!/bin/bash

# Generates a test script from the MEI test files
# and immediately executes it by piping to `source`.

saxonb-xslt \
  -xsl:generate-test-script.xsl \
  -s:generate-test-script.xsl \
  testfiles="$(ls -1 test/*.mei)" \
  | source /dev/stdin

We'd have to write an XSLT that takes all the test MEI files and turns the <annot> elements into commands for testing.

Should I give it a go?

[rettinghaus]

Sounds like a fine idea!
I started off with a very basic script just converting all MEI files and running LilyPond.
Please do whatever you think is appropriate.
Just the <annot> wouldn't make me happy. How 'bout this:
We use the <seriesStmt> (see Guidelines 2.1.5) for basic information on those files (title: MEILER test suite; using the <identifier> and stuff.
For more detailed test commands the <extMeta> should be used (as those are indeed some kind of extended metadata on what the file is for).
Could you do that?

[th-we]

I find <annot>s more convenient than <extMeta> for two reasons:

• <annot> is a much more powerful element. The @type/@subtype system is useful for test categories. <extMeta> only has @label.
• <annot>s can be put in different places. I did that as an example in my color pull request to make clear which test is responsible for which piece of MEI code.

What's your thought behind using <seriesStmt>? My tendency as the lazy programmer that I am is to keep everything as minimal as possible, considering these tests aren't actually musicologically useful data.

[rettinghaus]

The idea of <extMeta> is to fill it up with non-MEI markup. For example:

<extMeta>
  <test xmlns="http://www.meil.er/xml">
    <command system="linux">grep 'testlyrics' "$ly"</command>
  </test>
</extMeta>

<seriesStmt> should just state that this is a file for testing purposes (from the MEILER project) and not actually musicologically useful data.

[th-we]

Ah, I didn't notice <extMeta> could contain arbitrary markup.

My idea was to just use bash which should work on Mac, Linux and also Windows 10 (though I have no Windows 10 to test whether Lilypond and Java are already working properly). I don't think we should distinguish between system="linux", system="windows" and system="mac".

Bash has the disadvantage that it might not work as flawlessly cross-platform as e.g. Python. There are some minute differences between certain commands like sed (not bash itself) in Mac and Linux/Windows environments. But with bash, it's more straightforward to glue XSLT, Lilypond and tests together.