Skip to content

Releases: cloudflare/alertmanager2es

v0.1.0

30 Jun 23:03
Compare
Choose a tag to compare

First release in a while, this is mostly a tidy up:

Moving the builds onto a modern go
Moving us to a go module
Running various gofmt/vet to clean stuff up

0.0.2

10 May 12:14
Compare
Choose a tag to compare

This release works with Alertmanager 0.6.x, older versions are not supported.

Initial release

13 Apr 13:52
Compare
Choose a tag to compare
Send AlertManager notifications to Elasticsearch

Add a new service, `alertmanager2es`, which receives [HTTP webhook
notifications][] from [AlertManager][] and inserts them into an
[Elasticsearch][] index for searching and analysis.

Notifications are received on the `/webhook` endpoint.

Prometheus metrics are exposed on `/metrics`.

Note that this service will not capture silenced or inhibited alerts
currently.

I used a custom type for `groupKey` so that the `groupKey` field can be
marshalled into JSON as a string, since Elasticsearch does not currently
support unsigned integers:

https://github.com/elastic/elasticsearch/issues/13951

I tested extensively to try to map the integer to a string using
Elasticsearch's mappings, which I succeeded in doing, however Kibana
then failed to display the documents because Elasticsearch's
highlighting could not handle the unsigned integer stored in `_source`:

    RemoteTransportException[[Eternal Brain][172.17.0.2:9300][indices:data/read/search[phase/query+fetch]]]; nested: FetchPhaseExecutionException[Fetch Failed [Failed to highlight field [alerts.labels.decoded_pem_md5]]]; nested: IllegalStateException[No matching token for number_type [BIG_INTEGER]];
    Caused by: FetchPhaseExecutionException[Fetch Failed [Failed to highlight field [alerts.labels.decoded_pem_md5]]]; nested: IllegalStateException[No matching token for number_type [BIG_INTEGER]];
            at org.elasticsearch.search.highlight.PlainHighlighter.highlight(PlainHighlighter.java:123)
            at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:140)
            at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:188)
            at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:490)
            at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:392)
            at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:389)
            at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
            at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
            at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
            at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)

The dependencies for this code are vendored using gvt.

[AlertManager]: https://github.com/prometheus/alertmanager
[Elasticsearch]: https://www.elastic.co/products/elasticsearch
[HTTP webhook]: https://prometheus.io/docs/alerting/configuration/#webhook-receiver-<webhook_config>