This page provides a hands-on approach to learning how to configure pipeline aggregations
PipeCalculations performed on top of results of aggregations.
The steps below follow video presented by Zachary (Zach) Tong (@ZacharyTong, software developer at Elastic). We begin [17:18] into the 28 Oct 2015 video Introduction to Elastic 2.0 Overview.
-
Install the Elastic 2.0 ELK stack if you haven't already.
NOTE: Data in versions before Elastic 2.0 would need to be migrated up using the tool Elastic provided because its indexes are different.
Alternately, Bonsai on Heroku is also free.
-
Read Zach's book on the ELK stack (if you haven't already).
-
Obtain the sample data (bulk request). QUESTION: Zach?
GET ... {"index":{}} {"color":"red","price": 14.50} ```
-
PROTIP: Run to make sure the data can be ingested without error.
-
Specify standard aggregation calculations.
GET /test/test/_search { "size":0, "aggs": { "colors": { "terms": { "field": "color", "size": 10 } } } } ```
NOTE: These aggregations are run only on nodes.
Results are under buckets:
```
"aggregations": {
"colors": {
"doc_counter_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "red",
"doc_count": 6
},
{
"key": "green",
"doc_count": 4
}
]
}
}
```
-
Run the standard aggregations.
-
Add pipeline aggregations based on averages.
"mean_of_means":{ "avg_bucket":{ } }
NOTE: Pipeline aggregations are run only on the coordinating node after each shard calculates other buckets.
31 MB per node.
-
Add pipeline aggregations:
A sample time series:
<img width="688" alt="elastic20-timeseries" src="https://cloud.githubusercontent.com/assets/300046/11342759/2ae0ce00-91c6-11e5-8b67-e5a91083c4a4.png">
Derivities
Time-series predictions (moving averages)
Anomaly detection
Seasonality adjustments
In previous versions, an external application processes records.
## Timelines https://www.elastic.co/videos/time-series-analysis-using-timelionholt-winters season. The time period can be touchy.
"predict":60
PROTIP: alpha, beta, and gamma parameters can be added, but they are difficult to optiminze.
A https://www.elastic.co/blog/out-of-this-world-aggregations
Additional information is at https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggreations-pipeline.html
An example is calculating the rate of change*.
## Resources