Testing Methodology

Testing Methodology

Table of Contents

• Static message validation
• Dynamic message exchange and validation
• Workflow testing
• Toolkit capabilities
• Relating test methodology to visible Toolkit features and tools

Toolkit focuses on Conformance Testing, the validation of systems and messages compared against a reference. In this case the reference is a combination of well defined XML Schemas and the Toolkit developers' interpretation of the relevant standards and IHE profiles. Toolkit also offers support for Interoperability Testing which is not described here.

The testing performed can be described in terms of three levels of scope and comprehensiveness. The three levels, from shallowest to deepest, are static message validation, dynamic message validation, and workflow testing.

Static message validation

The most basic form of conformance testing focuses on the evaluation of static messages, messages examined outside of the context in which they are used. We divide this into three parts:

• Schema validation

• Validation against rules of the profile (beyond Schema)

• Code system validation

Dynamic message exchange and validation

This level adds capabilities on top of static message validation. This is described via two scenarios which are different by the role played by the test harness.

We use the term test harness to describe Toolkit and SUT to label the system under test.

Scenario 1 - Harness sends request message. SUT sends reply message. Reply message is evaluated in the context of the request message.

Scenario 2 - SUT sends request message for a particular purpose. Harness evaluates technical correctness of the request (static validation). Harness also evaluates whether request message meets specified purpose.

Workflow testing

This level adds capabilities on top of dynamic message exchange and validation. This is described via two scenarios which are different by the role played by the test harness.

Scenario 1 - Harness builds test environment. This includes collection of clients and servers the SUT will interact with.

SUT and test environment actors (clients and servers) are fed messages to put them into a known state. Messages sent to the SUT this way are restricted to SUT features previously tested.

Once establised, the test environment actors are considered part of the test harness.

The test is initiated by one or more messages being sent from the test harness to the SUT. Its response(s) are evaluated.

Scenario 2 - Extended test management is established as in Scenario 1. The test is initiated by the SUT sending the first message.

Scenario 3 & 4 - These are versions of scenarios 1 & 2 but allowing for multiple SUT actors either of the same actor type or different.

Toolkit capabilities

Static message validation - this is embedded in the other testing styles.

Dynamic message exchange - Toolkit has coverage for both Scenario 1 & 2 type testing. In Toolkit documentation they are referred to as Server SUT and Client SUT. This naming is driven by the SUT relationship with the first message exchanged.

Workflow testing - Toolkit has coverage for both Scenario 1 & 2 type testing. At this level the Server SUT or Client SUT labels becomes interesting because the style of test operation must change to accomodate SUT operation - sending an initial message to the SUT vs. giving social instructions to a human operator that controls the SUT.

Relating test methodology to visible Toolkit features and tools

The central focus for the test methology is the Conformance Tool. Tests are run and results reviewed here. Much of the work is done by other identifiable tools which have their own presense and user interfaces.

System configuration manager - user to enter a system’s network endpoints and configuration parameters.

Environment - holds elements of the test harness configuration. This includes code systems used, TLS certificates used in message interactions. Note - configuring the environment includes configuration of the underlying Tomcat service.

Test Session - container for test elements and results. each SUT is managed within its own Test Session. A Test Session includes test server configurations (simulators), test client configurations, test appliance logs (test appliance refers to both the test client and simulators), and testing status.

Inspector - test client log viewer

Simulator log viewer

Simulator manager - for manually creating/destroying simulators

Client tools - collection of tools for probing a SUT server with manual requests.

Test Session manager - add/delete/select