Skip to content
This repository has been archived by the owner on Feb 22, 2024. It is now read-only.

Alerting

Jump to bottom
Jiří Holuša edited this page Apr 16, 2015 · 5 revisions

Alerting

PerfRepo provides a great feature for detecting performance regression called alerting. The principle is very simple. You set up a condition to specific test and every new test execution's value must hold that condition. If it breaks the condition, PerfRepo sends an email to all users that subscribed to test. Let's see it in practise!

Subscription

First of all, you have to tell PerfRepo that you want to receive all alerts from specific test. To do this, you simple search for the test you want and click "Subscribe for alerts". Now, every time PerfRepo discovers broken condition on this test, it will automatically send you a mail to an email address you have specified in your user profile. Of course, you can subscribe to as many tests as you want.

Entering condition

The key part is to set up the conditions. PerfRepo comes with its own DSL (Domain Specific Language) for specifing condition. Its syntax looks a little like SQL, so it should not be a problem to use it effectively in a few minutes. Let's see an example!

CONDITION result > baseline DEFINE baseline = (SELECT WHERE id = 123)

Every condition has two parts: condition (expression) part and defining part. Every condition starts with keyword CONDITION, then everything until another keyword DEFINE is the expression. It can be any boolean expression you want, as complicated as you want. You can use arbitrary number of variables, whose values you definie after the DEFINE. There is only one thing you have to remember that there is reserved variable result, which contains the value of the new test execution being entered. In the example above, we simple want to check that every new test execution value is higher that some baseline, some specific test execution. Here are more examples of conditions:

  • CONDITION result > 2*version1 && result > 3*version2 DEFINE version1 = ..., version2 = ...
  • CONDITION (result > 2*(version1-100) || version1 < version2) && (result > 3*version2) DEFINE version1 = ..., version2 = ...
  • simply any expression that can be evaluated by e.g. JavaScript engine

After defining the condition, you have to assign value to every value that you used (except result obviously) and you do it in defining part, which starts with DEFINE keyword. Syntax is following:

CONDITION ... DEFINE variableName1 = [expression], variableName2 = [expression], ...

Where expression is a select query in our DSL. Selects can be two types, single select (always returning at most one test execution) or multiselect (possibly returning more than one test execution). If single select, it can be placed right after =, eg. (parenthesses are optional)

CONDITION ... DEFINE x = (SELECT WHERE id = 1), y = (SELECT WHERE id = 2)

If multiselect, we have to agreage the results into one number, because variable can contain only one value. To do so, we use agregate function AVG (average), MIN (minimum) or MAX (maximum). Example:

CONDITION ... DEFINE x = AVG(SELECT WHERE id IN (1,2))

The last thing you need to know is how to specify selects and what are the options. Possible select types:

  • SELECT WHERE id = 1 - selects one specific test execution with specified ID (simple select)
  • SELECT WHERE id IN (1,2) - selects specific test executions with ID in the list (multi select)
  • SELECT WHERE tags = "tag1 tag2" - selects all test executions that have all the space-separated tags present (it may have more), e.g. it will select test execution with tags tag1 tag2 tag3, but won't select the one with tags tag1 tag3. (multi select)
  • * SELECT WHERE date >= "2015-01-01 00:00" - selects all test executions that have execution time after specified date. Accepted date format: YYYY-MM-DD HH:mm (multi select) - selects all test executions that have execution time after specified date. Accepted date format: YYYY-MM-DD HH:mm (multi select)
  • SELECT WHERE date <= "2015-01-01 00:00" - selects all test executions that have execution time before specified date. Accepted date format: YYYY-MM-DD HH:mm (multi select)
  • all of the above WHERE clauses joined with AND with obvious meaning, e.g. SELECT WHERE date >= "2015-01-01 00:00" AND WHERE date >= "2015-01-01 00:00" AND tags = "tag1 tag2" (multi select)
  • SELECT LAST 1 - selects last test execution similarly to SQL LIMIT. (simple select)
  • SELECT LAST 5 - selects last 5 test execution similarly to SQL LIMIT. (multi select)
  • SELECT LAST 10, 5 - selects 5 test executions ordered by date ascending starting from the 10th from the end, similarly to SQL LIMIT. Syntax means LAST <lastFrom>, <howMany>, therefore LAST x is equivalent to LAST x,x
  • any combination of LAST and WHERE from above in order SELECT WHERE ... LAST ..., here are some more complex complete examples
    CONDITION x == result DEFINE x = MAX(SELECT WHERE tags = "firstTag secondTag" AND date >= "2015-01-01 00:00" AND date <= "2015-02-01 00:00" LAST 2)

    CONDITION x == result DEFINE x = MAX(SELECT WHERE tags = "firstTag secondTag" LAST 3, 2)

    CONDITION x == result DEFINE x = MAX(SELECT WHERE id IN (1,2))

Feel free to check out the ConditionCheckerTest file for more examples. Enjoy PerfRepo alerting!

Clone this wiki locally