Testing Guide

This guide is intended to layout the general automated test strategy for OpenLMIS.

Test Strategy

OpenLMIS, like many software projects, relies on testing to guide development and prevent regressions. To effect this we've adopted a standard set of tools to write and execute our tests, and categorize them to understand what types of tests we have, who writes them, when they're written, run, and where they live.

Types of Tests

The following test categories have been identified for use in OpenLMIS. As illustrated in this great slide deck, we expect the effort/number of tests in each category to reflect the test pyramid:

Unit
Integration
Component
Contract
End-to-End

Unit Tests

Who: written by code-author during implementation
What: the smallest unit (e.g. one piece of a model's behavior, a function, etc)
When: at build time, should be /fast/ and targeted - I can run just a portion of the test suite
Where: Reside inside a service, next to unit under test. Generally able to access package-private scope
Why: to test fundamental pieces/functionality, helps guide and document design and refactors, protects against regression

Unit Test Examples

Every single test should be independent and isolated. Unit test shouldn't depend on another unit test.

DO NOT:

List<Item> list = new ArrayList<>();

@Test
public void shouldContainOneElementWhenFirstElementisAdded() {
  Item item = new Item();
  list.add(item);
  assertEquals(1, list.size());
}

@Test
public void shouldContainTwoElementsWhenNextElementIsAdded() {
  Item item = new Item();
  list.add(item);
  assertEquals(2, list.size());
}

One behavior should be tested in just one unit test.

DO NOT:

@Test
public void shouldNotBeAdultAndShouldNotBeAbleToRunForPresidentWhenAgeBelow18() {
  int age = 17;
  boolean isAdult = ageService.isAdult(age);
  assertFalse(isAdult);

  boolean isAbleToRunForPresident = electionsService.isAbleToRunForPresident(age)
  assertFalse(isAbleToRunForPresident);
}

DO:

@Test
public void shouldNotBeAdultWhenAgeBelow18() {
  int age = 17;
  boolean isAdult = ageService.isAdult(age);
  assertFalse(isAdult);
}

@Test
public void shouldNotBeAbleToRunForPresidentWhenAgeBelow18() {
  int age = 17;
  boolean isAbleToRunForPresident = electionsService.isAbleToRunForPresident(age)
  assertFalse(isAbleToRunForPresident);
}

Every unit test should have at least one assertion.

DO NOT:

@Test
public void shouldNotBeAdultWhenAgeBelow18() {
  int age = 17;
  boolean isAdult = ageService.isAdult(age);
}

DO:

@Test
public void shouldNotBeAdultWhenAgeBelow18() {
  int age = 17;
  boolean isAdult = ageService.isAdult(age);
  assertFalse(isAdult);
}

Don't make unnecessary assertions. Don't assert mocked behavior, avoid assertions that check the exact same thing as another unit test.

DO NOT:

 @Test
 public void shouldNotBeAdultWhenAgeBelow18() {
   int age = 17;
   assertEquals(17, age);

   boolean isAdult = ageService.isAdult(age);
   assertFalse(isAdult);
 }

Unit test has to be independent from external resources (i.e. don't connect with databases or servers)

DO NOT:

 @Test
 public void shouldNotBeAdultWhenAgeBelow18() {
   String uri = String.format("http://127.0.0.1:8080/age/", HOST, PORT);
   HttpPost httpPost = new HttpPost(uri);
   HttpResponse response = getHttpClient().execute(httpPost);
   assertEquals(HttpStatus.ORDINAL_200_OK, response.getStatusLine().getStatusCode());
 }

Unit test shouldn't test Spring Contexts. Integration tests are better for this purpose.

DO NOT:

 @RunWith(SpringJUnit4ClassRunner.class)
 @ContextConfiguration(locations = {"/services-test-config.xml"})
 public class MyServiceTest implements ApplicationContextAware
 {

   @Autowired
   MyService service;
   ...
     @Override
     public void setApplicationContext(ApplicationContext context) throws BeansException
     {
         // something with the context here
     }
 }

Test method name should clearly indicate what is being tested and what is the expected output and condition. The "should - when" pattern should be used in the name.

DO:
```
@Test
public void shouldNotBeAdultWhenAgeBelow18() {
  ...
}
```
DO NOT:
```
@Test
public void firstTest() {
  ...
}

@Test
public void testIsNotAdult() {
  ...
}
```

Unit test should be repeatable - each run should yield the same result.

DO NOT:

@Test
public void shouldNotBeAdultWhenAgeBelow18() {
  int age = randomGenerator.nextInt(100);
  boolean isAdult = ageService.isAdult(age);
  assertFalse(isAdult);
}

You should remember about intializing and cleaning each global state between test runs.

DO:

@Mock
private AgeService ageService;
private age;

@Before
public void init() {
  age = 18;
  when(ageService.isAdult(age)).thenReturn(true);
}

@Test
public void shouldNotBeAdultWhenAgeBelow18() {
  boolean isAdult = ageService.isAdult(age);
  assertTrue(isAdult);
}

Test should run fast. When we have hundreds of tests we just don't want to wait several minutes till all tests pass.

DO NOT:

 @Test
 public void shouldNotBeAdultWhenAgeBelow18() {
   int age = 17;
   sleep(1000);
   boolean isAdult = ageService.isAdult(age);
   sleep(1000);
   assertFalse(isAdult);
 }

Integration Tests

Who: Code author during implementation
What: Test basic operation of a service to persistent storage or a service to another service. When another service is required, a test-double should be used, not the actual service.
When: As explicitly asked for, these tests are typically slower and therefore need to be kept separate from build to not slow development. Will be run in CI on every change.
Where: Reside inside a service, separated from other types of tests/code.
Why: Ensures that the basic pathways to a service's external run-time dependancies work. e.g. that a db schema supports the ORM, or a non-responsive service call is gracefully handled.

For testing controllers, they are divided up into unit and integration tests. The controller unit tests will be testing the logic in the controller, while the integration tests will be mostly testing serialization/deserialization (and therefore do not need to test all code paths). In both cases, the underlying services and repositories are mocked.

Component Tests

Who: Code author during implementation
What: Test more complex operations in a service. When another service is required, a test-double should be used, not the actual service.
When: As explicitly asked for, these tests are typically slower and therefore need to be kept separate from build to not slow development. Will be run in CI on every change.
Where: Reside inside a service, separated from other types of tests/code.
Why: Tests interactions between components in a service are working as expected.

These are not integration tests, which strictly test the integration between the service and an external dependency. These test the interactions between components in a service are working correctly. While integration tests just test the basic pathways are working, component tests verify that, based on input, the output matches what is expected.

These are not contract tests, which are more oriented towards business requirements, but are more technical in nature. The contract tests will make certain assumptions about components, and these tests make sure those assumptions are tested.

Contract Tests

Who: Code author during implementation, with input from BA/QA.
What: Enforces contracts between and to services.
When: Ran in CI.
Where: Reside inside separate repository: openlmis-contract-tests.
Why: Tests multiple services working together, testing contracts that a Service both provides as well as the requirements a dependant has.

The main difference between contract and integration tests: In contract tests, all the services under test are real, meaning that they will be processing requests and sending responses. Test doubles, mocking, stubbing should not be a part of contract tests.

Refer to this doc for examples of how to write contract tests.

End-to-End Tests

Who: QA / developer with input from BA.
What: Typical/core business scenarios.
When: Ran in CI.
Where: Resides in seperate repository.
Why: Ensures all the pieces are working together to carry-out a business scenario. Helps ensure end-users can achieve their goals.

Testing services dependent on external APIs

OpenLMIS is using WireMock for mocking web services. An example integration test can be found here: https://github.com/OpenLMIS/openlmis-example/blob/master/src/test/java/org/openlmis/example/WeatherServiceTest.java

The stub mappings which are served by WireMock's HTTP server are placed under src/test/resources/mappings and _src/test/resources/_files For instructions on how to create them please refer to http://wiremock.org/record-playback.html

Testing Tools

spring-boot-starter-test
- Spring Boot Test
- JUnit
- Mockito
- Hamcrest
WireMock
REST Assured
raml-tester

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TESTING.md

TESTING.md

Testing Guide

Test Strategy

Types of Tests

Unit Tests

Unit Test Examples

Integration Tests

Component Tests

Contract Tests

End-to-End Tests

Testing services dependent on external APIs

Testing Tools

Files

TESTING.md

Latest commit

History

TESTING.md

File metadata and controls

Testing Guide

Test Strategy

Types of Tests

Unit Tests

Unit Test Examples

Integration Tests

Component Tests

Contract Tests

End-to-End Tests

Testing services dependent on external APIs

Testing Tools