Test organization

Toolkit defines a test as a group of transactions that interact with the system under test (SUT). A transaction is defined as a parameterized template managed by a test-specific control script.

Tests have a highly organized structure. The key concepts are:

• Test Kit
• Test Collection
• Test
• Section
• Step

Test Kit

A Test Kit is a library of tests that can be run in Toolkit. Toolkit always ships with an internal Test Kit that is located in the WAR file. Optionally, the External Cache can hold multiple alternative Test Kits. There can be multiple copies of a test. There is a logical search path for finding a test definition.

Test Collection

A test collection is a list of test IDs. Test collections are created for many reasons and they are mostly hidden from the user's view. Any time you see a grouping of tests in the user interface there is a test collection behind it. The most visible test collection is the grouping of tests for an actor or option.

Test collections are managed indirectly by indexing tests with the collections they belong to. Each test definition has a file collections.txt which holds the name of all collections that test is part of. The master index is created each time Toolkit is started or when the Reindex Test Kits button is pressed in the Toolkit Configuration tool.

Test

A test is an identifiable unit of testing that is indexed in Gazelle and assigned to a known actor and option. A test can be run using the Conformance Test Tool. Sometimes tests depend on other tests. When they do they reference the log files produced by the other tests. Running a test when the dependency test has not been run leads to an error.

Section

A test is comprised of one or more sections. A section is the smallest independently run-able element of testing. A section can be run through the Conformance Test Tool. Sometimes sections depend on other sections within the test. A section is defined in a file named testplan.xml within the directory defining a test. The running of a section generates a single log file in the External Cache.

Step

A step generates a single transaction. A step cannot be run in isolation. When a section is run, all steps in the section are run in the order they are coded in the testplan.xml file. The execution of the section terminates when it runs out of steps to run or when a step fails.

A step can reference the output of a previously run step, in the same section, a different section of the same test or in a different test. The log file produced by testing, log.xml, is generated at the section level (a section generates a single log.xml file) but it is the step that references or is referenced across tests.