Skip to content

Latest commit

 

History

History
148 lines (98 loc) · 3.42 KB

elk-config.md

File metadata and controls

148 lines (98 loc) · 3.42 KB
Raw

This page provides a hands-on approach to learning how to configure pipeline aggregations

PipeCalculations performed on top of results of aggregations.

The steps below follow video presented by Zachary (Zach) Tong (@ZacharyTong, software developer at Elastic). We begin [17:18] into the 28 Oct 2015 video Introduction to Elastic 2.0 Overview.

  1. Install the Elastic 2.0 ELK stack if you haven't already.

    NOTE: Data in versions before Elastic 2.0 would need to be migrated up using the tool Elastic provided because its indexes are different.

    Alternately, Bonsai on Heroku is also free.

  2. Read Zach's book on the ELK stack (if you haven't already).

  3. Obtain the sample data (bulk request). QUESTION: Zach?

GET ... {"index":{}} {"color":"red","price": 14.50} ```

  1. PROTIP: Run to make sure the data can be ingested without error.

  2. Specify standard aggregation calculations.

GET /test/test/_search { "size":0, "aggs": { "colors": { "terms": { "field": "color", "size": 10 } } } } ```

NOTE: These aggregations are run only on nodes.

Results are under buckets:

```
"aggregations": {
    "colors": {
        "doc_counter_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
            {
            "key": "red",
            "doc_count": 6
            },
            {
            "key": "green",
            "doc_count": 4
            }
        ]
    }
}
```

  1. Run the standard aggregations.

  2. Add pipeline aggregations based on averages.

    "mean_of_means":{
    "avg_bucket":{
        
    }
}

    NOTE: Pipeline aggregations are run only on the coordinating node after each shard calculates other buckets.

    31 MB per node.

  3. Add pipeline aggregations:

A sample time series:
<img width="688" alt="elastic20-timeseries" src="https://cloud.githubusercontent.com/assets/300046/11342759/2ae0ce00-91c6-11e5-8b67-e5a91083c4a4.png">

Derivities

Time-series predictions (moving averages)

Anomaly detection

Seasonality adjustments

In previous versions, an external application processes records.

## Timelines https://www.elastic.co/videos/time-series-analysis-using-timelion 
holt-winters season. The time period can be touchy.
## Forecasts 0. [37:50] Extend the holt_winters algorithm to make a forecast 60 days into the future. 
"predict":60

PROTIP: alpha, beta, and gamma parameters can be added, but they are difficult to optiminze.
## Aggregations

A https://www.elastic.co/blog/out-of-this-world-aggregations

Additional information is at https://www.elastic.co/guide/en/elasticsearch/reference/master/search-aggreations-pipeline.html

An example is calculating the rate of change*.

## Resources