Skip to content

Latest commit

 

History

History
161 lines (141 loc) · 7.03 KB

TESTCASES.md

File metadata and controls

161 lines (141 loc) · 7.03 KB

Creating and Documenting Test Cases

Principles

  • Test cases should be based on a single requirement, although there may be dependencies on other requirements.
  • Metadata test cases should not generally represent invalid METS documents, that is issues caught by METS schema validation
  • A test case incorporates a set of rules that are the fine grained tests needed to completely validate the requirement.

Atomicity

Every test case should be atomic and minimal. Atomic means that it violates only a single aspect of the specification. Minimal means that it contains only the data and metadata required to ensure only a single validator error is produced.

A good starting principle is that there should be one test case per requirement. In practise this might not always be possible.

Complex requirements

More complex requirements might require multiple test cases, or appear to. If this feels necessary then first discuss with the A3 group, as a comment on the test case thread. It might be that the requirement is too complex and could be broken down. If this isn't the case then there will be a one to many relationship between specification requirements and test cases. The test cases provide fine-grained, detailed demonstrations of pass and fail conditions for the requirements.

There should be pass and fail corpus packages for each test case, although it's possible that multiple test cases share a single "pass" case for brevity.

The E-ARK information package validation process has been broken into 3 parts, described here.

NOTE: To simplify this we want to introduce the concepts of "well-formed", "valid", and "audited" (what's a good word for passing integrity checks?), to allow minimal files that pass only the well formed structural checks possible, without the need for content

Style Guide

Paths

When quoting XML XPaths or file system paths please state the full path for clarity.

Rule error messages

Error messages assigned to rules should be concise and specific. Consider the CSIP requirement description for the mets@TYPE attribute.

Mandatory in this specification. The TYPE attribute must be used for identifying the type of the package (genre), for example ERMS, RDBMS, digitised construction plans. However, there is no fixed vocabulary and as such implementers are welcome to use values most suitable for their needs.

The error message for when mets/@TYPE attribute does not exist should be:

"mets/@TYPE attribute not present. The @TYPE attribute must be present with a value identifying the package type."

rather than quoting the full specification text.

Documenting Test Cases

  • Assign an ID
  • Provide reference to the specification
  • Define the error level, message and description expected from the validator
  • Document the minimal corpus package structure.

XML for structured test case description

Test case XML Notes

  • Using the METS element name doesn't work for non-METS requirements, e.g. folder structure requirements, use the requirement ID structure instead.
  • do we need to document the minimal test IP structure?
<!-- Root element for an individual test case, allows these to be wrapped into
XML lists -->
<testCase testable="TRUE">
  <!-- Unique ID for the test case -->
  <id specification="" version="" section="" requirementId="" />
  <!-- URL references to requirements for convenient lookup -->
  <references>
    <reference requirementId=""><!-- URL for reference here --></reference>
  </references>
  <!-- Textual description of the test case, may just be a repeat of the
  requirment text. -->
  <description></description>
  <!-- List of requirments that this test case depends on in addition to the
  main requirememt, e.g. general requirments on the form of IDs -->
  <dependencies>
    <!-- Individual test case dependencies listed by ID -->
    <dependency specification="" version="" section="" requirement="" />
  </dependencies>
  <!-- A list of the validation rules derived from the test case -->
  <rules>
    <!-- Individual validation rule entry, identified by a running number
    that's unique within the test case only -->
    <rule id="">
      <!-- Text description of the validation rule -->
      <description></description>
      <!-- Details of the validation error generated by the message, can be
      one of: [ ERROR | WARNING | INFO ]-->
      <error level="">
        <!-- Expected error message when the rule is violated. -->
        <message></message>
      </error>
      <corpusPackages>
        <!-- Details of corpus packages designed to test this validation rule. -->
  	    <package name="descriptive_package_name" isValid="TRUE if package is a valid, FALSE if an invalid package.">
  	      <path><!-- The relative path to the test corpus package root folder --></path>
  		    <description><!-- Full description of corpus package features. --></description>
        </package>
  	  </corpusPackages>
    </rule>
  </rules>
</testCase>

XML Examples

Taken from https://github.com/DILCISBoard/eark-ip-test-corpus/tree/testcase/mets-xml/mets-root-element/corpus/mets-xml/mets/CONTENTTYPESPECIFICATION

<testCase>
  <id specification="" version="" section="" requirement="" />
  <reference>https://carlwilson.github.io/E-ARK-CSIP/specification/implementation/metadata/#CSIP4</reference>
  <description>
    @CONTENTTYPESPECIFICATION MUST be present and have a value
    that is one of [ SMURFERMS | SMURFSFSB | SIARD1 | SIARD2 | SIARDDK |
    GeoVectorGML | GeoRasterGeotiff | MIXED | OTHER ]
  </description>
  <dependencies />
  <rules>
    <rule id="1">
      <description>@CONTENTTYPESPECIFICATION attribute does not exist</description>
      <error level="ERROR">
        <message>
          mets/@CONTENTTYPESPECIFICATION attribute does not exist.
          MUST be: attribute exists and has a value from the fixed vocabulary of
          [ SMURFERMS | SMURFSFSB | SIARD1 | SIARD2 | SIARDDK | GeoVectorGML |
          GeoRasterGeotiff | MIXED | OTHER ].</message>
      </error>
    </rule>
    <rule id="2">
      <description>@CONTENTTYPESPECIFICATION attribute value is emptry string</description>
      <error level="ERROR">
        <message>
          mets/@CONTENTTYPESPECIFICATION attribute value is emptry string.
          MUST be: a value from the fixed vocabulary of [ SMURFERMS | SMURFSFSB
          | SIARD1 | SIARD2 | SIARDDK | GeoVectorGML | GeoRasterGeotiff
          |MIXED | OTHER ].
        </message>
      </error>
    </rule>
    <rule id="2">
      <description>
        @CONTENTTYPESPECIFICATION attribute value does not match any of the
        allowed values in the vocabulary.
      </description>
      <error level="ERROR">
        <message>
          mets/@CONTENTTYPESPECIFICATION attribute value does not match any of
          the allowed values in the vocabulary. MUST be: a value from the fixed
          vocabulary of [ SMURFERMS | SMURFSFSB | SIARD1 | SIARD2 | SIARDDK |
           GeoVectorGML | GeoRasterGeotiff | MIXED | OTHER ].
        </message>
      </error>
    </rule>
  </rules>
</testCase>