Trace based testing - System integration and microservices

[practicalli-johnny]

mulog tracing

Mulog provides a log of events that can be published to a wide range of log aggregators
Each event captures information to allow tracing of events

Trace-based testing — a new test paradigm

In the last few years, distributed tracing technologies have been gaining momentum. Standards like OpenTelemetry enable a different way to look at microservices: from a trace standpoint.

Distributed tracing allows overseeing everything that happens when one single operation is activated. By using this method, we could approach each operation and get a holistic view of everything that happens in my application triggered by this operation, instead of looking at it separately.

In OpenTelemetry terminology — the trace is built of spans. A span represents an operation — sending an HTTP request, making a DB query, handling an asynchronous event, etc. Ideally, using spans can help us review any attributes we want to validate for the test. Moreover, the trace also shows a continuous flow and the relation between operations, as opposed to a single operation, which can be powerful for test purposes.

How to Create Trace-Based Testing
When using traces for tests, you first need to deploy a tracing solution. As mentioned before, the common tracing solution in the industry is

OpenTelemetry, or OTeL. It also allows searching capabilities.

Let’s review the things you should do if you want to succeed in testing with OTeL:

Deploy the OTeL SDK.
Build processor services that will take OTEL data and transform it into assets you can work with.
Create an ELK service that can save all the traces.
Create a tests framework that will allow you to search for a specific span inside traces.
And that’s it. Note that we didn’t discuss any visualization tools that would help you investigate traces, such as Zipkin on Jaeger, but you can use them as well.

But at the end of the day — another tool was still missing.

Looking at the relevant open source tools in the domain that are based on OpenTelemetry, we observed that most of them require the implementation of OpenTelemetry as a prerequisite, which left us with the same headaches.

Helios
Helios is a free tool that instruments OpenTelemetry and allows developers to generate test code for each trace.

Through spans, Helios collects all the payloads of each request and response. Using this capability, developers can generate tests without changing a single code line.

It’s also built in a smart way that enables the creation of a test from a bug. Even better, using the trace visualization tool, a developer can visually create a test directly from a trace.

Bottom line: The way developers test microservices today is broken. It’s time consuming and inefficient. Using distributed tracing testing is the right answer — it allows developers to properly test their microservices, since it provides a holistic view of all microservices. Whichever tool you choose, make sure you take advantage of its abilities