Alerting

Alerting

PerfRepo provides a great feature for performance regression detection called alerting. The principle is very simple. You set up a condition for a specific test and every new test execution value must hold that condition. If it the condition evaluated to false, PerfRepo sends an email to all users that subscribed to the test. Let's see it in practice!

Subscription

First of all, you have to tell PerfRepo that you want to receive all alerts related to specific test. To do this, you simply search for the test you want and click "Subscribe for alerts". Now, every time PerfRepo discovers broken condition on this test, it will automatically send you an email to an email address you specified in your user profile. Of course, you can subscribe to as many tests as you want.

Entering condition

The key part is to set up the condition. PerfRepo comes with its own DSL (Domain Specific Language) for specifying condition. Its syntax looks somewhat similar to SQL, so it should not be a problem to use it effectively in just a few minutes.

Please note that when entering the condition, PerfRepo should be smart enough to tell you, if that condition makes sense. E.g. it should detect syntactic errors and let you know.

Single value alerting

This is the basic form of alerting. Single value alerting is unsurprisingly linked to single valued metrics. Any time you can boil down your problem to a comparison of two simple values, this is a way to go. Let's see an example!

CONDITION result > baseline DEFINE baseline = (SELECT WHERE id = 123)

Every condition has two parts: condition (expression) part and defining part. Every condition starts with keyword CONDITION, then everything until another keyword DEFINE is considered the expression. It can be any boolean expression you want, as complicated as you want. You can use arbitrary number of variables, whose values you define after the DEFINE keyword. There is only one thing to keep in mind. That there is a reserved variable result, which contains the value of the new test execution being entered. In the example above, we simple want to check that every new test execution value is higher that some baseline, some specific test execution. Here are more examples of conditions:

• CONDITION result > 2*version1 && result > 3*version2 DEFINE version1 = ..., version2 = ...
• CONDITION (result > 2*(version1-100) || version1 < version2) && (result > 3*version2) DEFINE version1 = ..., version2 = ...
• simply any expression that can be evaluated by e.g. JavaScript engine

After defining the condition, you have to assign value to every value that you used (except result obviously) and you do it in defining part, which starts with DEFINE keyword. Syntax is following:

CONDITION ... DEFINE variableName1 = [expression], variableName2 = [expression], ...

Where expression is a select query in our DSL. Selects can be two types, single select (always returning at most one test execution) or multiselect (possibly returning more than one test execution). In case of single select, it can be placed right after =, eg. (parenthesses are optional)

CONDITION ... DEFINE x = (SELECT WHERE id = 1), y = (SELECT WHERE id = 2)

In case of multiselect, we have to agreage the results into one number, because variable can contain only one value. To do so, we use agregate function AVG (average), MIN (minimum) or MAX (maximum). Example:

CONDITION ... DEFINE x = AVG(SELECT WHERE id IN (1,2))

The last thing you need to know is how to specify selects and what are the options. Possible select types:

• SELECT WHERE id = 1 - selects one specific test execution with specified ID (simple select)
• SELECT WHERE id IN (1,2) - selects specific test executions with ID in the list (multi select)
• SELECT WHERE tags = "tag1 tag2" - selects all test executions that have all the space-separated tags present (it may have more), e.g. it will select test execution with tags tag1 tag2 tag3, but won't select the one with tags tag1 tag3. (multi select)
• SELECT WHERE date >= "2015-01-01 00:00" - selects all test executions that have execution time after specified date. Accepted date format: YYYY-MM-DD HH:mm (multi select)
• SELECT WHERE date <= "2015-01-01 00:00" - selects all test executions that have execution time before specified date. Accepted date format: YYYY-MM-DD HH:mm (multi select)
• all of the above WHERE clauses joined with AND with obvious meaning, e.g. SELECT WHERE date >= "2015-01-01 00:00" AND WHERE date >= "2015-01-01 00:00" AND tags = "tag1 tag2" (multi select)
• SELECT LAST 1 - selects last test execution similarly to SQL LIMIT. (simple select)
• SELECT LAST 5 - selects last 5 test execution similarly to SQL LIMIT. (multi select)
• SELECT LAST 10, 5 - selects 5 test executions ordered by date ascending starting from the 10th from the end, similarly to SQL LIMIT. Syntax means LAST <lastFrom>, <howMany>, therefore LAST x is equivalent to LAST x,x
• any combination of LAST and WHERE from above in order SELECT WHERE ... LAST ..., here are some more complex complete examples

    CONDITION x == result DEFINE x = MAX(SELECT WHERE tags = "firstTag secondTag" AND date >= "2015-01-01 00:00" AND date <= "2015-02-01 00:00" LAST 2)

    CONDITION x == result DEFINE x = MAX(SELECT WHERE tags = "firstTag secondTag" LAST 3, 2)

    CONDITION x == result DEFINE x = MAX(SELECT WHERE id IN (1,2))

Multi value alerting

This is a more complex use case and assumes you understand what a multi value test is. To give you an idea - imagine you have a test, with approximately 40 iterations and in each of them you measure throughput. Now once you upload new test execution, you want the alerting to compare that in every iteration the throughput holds given condition against previous execution(s). You can do that with multivalue alerting ... and not just that!

Firstly let me stress out that multi value result can be seen as a graph and in that graph, the value you want to compare is on y-axis (in the above example the throughput). X-axis is considered to be fixed, e.g. same for all compared test executions. In the provided example, the iterations will always be a sequence of 1,2,3,... and so on. Note that the amount of point on x-axis can differ amongst compared execution, one execution can have 40 iteration and be compared to the other with 45 and it will still work.

Secondly, the syntax. It is very similar to that of single value alerting. The very basic formula goes like this: MULTIVALUE CONDITION ... DEFINE ... No big surprise there, right? But it is not that simple, let me cover one use case at a time...

Basic multi value alerting

This covers the use case I gave above.

Following rules and/or limitations apply to multi value alerting:

• you are only allowed to define one variable in the DEFINE statement
• multiple variables would too easily allow for comparison of pears and apples
• the variable you define can hold more than one test execution
• ... DEFINE x = SELECT LAST 5
• x will therefore contain 5 test executions
• all test executions have to be multi valued
• you are not allowed to use grouping functions
• there is a separate formula for that, see below

The actual formula for this use case is: MULTIVALUE [STRICT] CONDITION ... DEFINE ...

The keyword STRICT is in square brackets because it is optional. It allows you to specify whether you want newly added test execution to have exactly the same amount of x-axis points (iterations) as the one you compare it to. By default STRICT is not used and the condition is verified for only as long as both compared executions have that iteration.

How does the comparison work when my variable holds a result of several executions? PerfRepo will take a value of monitored metric in every iteration and verify that the condition holds by comparing it to relevant value in relevant iteration in each test execution the variable contains.

Apart from what was written here, it works the same way as with single value alerting. You are free to have any condition and any define statement (as long as there is no grouping).

Here are few inspirational alerts:

MULTIVALUE CONDITION x == result DEFINE x = SELECT WHERE id = 100
MULTIVALUE STRICT CONDITION result >= x DEFINE x = SELECT WHERE id IN (100, 101)
MULTIVALUE CONDITION result == x DEFINE x = SELECT WHERE tags = "epicTag legendaryTag" LAST 1
MULTIVALUE STRICT CONDITION result >= (0.95 * x) && result <= (1.05 * x) DEFINE x = SELECT WHERE id IN (110, 111)

Multi value alerting with grouping functions

What is the use case for grouping and multi value tests? Let me shed some light... Say we have a load test - it runs for a given amount of iterations and in each iteration we measure throughput. Now what if I want to say, that when adding new test execution, I want to take an average of its throughput (remember, it has n iterations) and say that it has to be equal to average of previously added test execution of this kind. Or even crazier - what if I want it to be equal to average of last five executions? That is where grouping comes into play.

Following rules and/or limitations apply to multi value alerting with grouping functions:

• you are allowed to have as many variables as you like
• all variable declarations must begin with grouping function
• x = AVG(...), y = AVG(...)
• tests selected within variable obviously need to be multi value tests
• only one grouping function (AVG, MIN, MAX) is allowed per alert
• that means all variables you define have to have the same grouping function applied
• remember the magical result variable? Well, PerfRepo needs to apply the same grouping function to this variable (holding one multi value test execution) and it needs to know which one

The actual formula for this use case is: MULTIVALUE GROUPING CONDITION ... DEFINE ...

So how is grouping actually applied on multivalue test? (e.g. on result for instance) Ok, say you use AVG as a grouping function in your alert. Then PerfRepo will look at the result (currently added execution) and sum up all values of desired metric, then divide it by number of points on x-axis (number of iterations in my example above). Just a plain simple average function. When a variable defined by you selects multiple test executions (e.g. using LAST 5) the process is done for each execution separately and then repeated for gathered results again resulting in a single value.

So in the end multi value grouping always results in a simple comparison of single values. Just keep in mind how it is done.

Once again, apart from what is written here, same rules as for single value alerting applies and you are free to write any alerts you like.

Few samples never hurt:

MULTIVALUE GROUPING CONDITION result == x DEFINE x = AVG(SELECT WHERE id IN (112, 113))
MULTIVALUE GROUPING CONDITION result == x && result < y DEFINE x = MIN(SELECT WHERE id IN (112, 113)), y = MIN(SELECT WHERE id IN(100, 101))

Feel free to check out the ConditionCheckerTest file for more examples. Enjoy PerfRepo alerting!