Skip to content

Latest commit

 

History

History

demo

README.md

es2loki demo

This folder demonstrates how you can use es2loki to transfer logs from Elasticsearch to Loki.

The included docker-compose.yml populates Elasticsearch with some "old" logs that needs to be transferred and some "new" logs that are already reside in Loki and we es2loki must not interfere with them.

Components

  1. Elasticsearch
  2. Kibana
  3. filebeat (imports "old" logs to Elasticsearch)
  4. Grafana (login: admin/admin)
  5. Loki
  6. Promtail (imports "new" logs to Loki)
  7. PostgreSQL (needed for es2loki)
  8. es2loki

Usage

In order to run a demo you may use:

docker compose up

Once you run it, all the components will spin up and after 180 seconds es2loki will transfer logs from Elasticsearch to Loki. You can validate that by accessing Grafana Explore and issue a following query for the December 3rd, 2022:

{job="logs"}

Or click the following link.

How it works

example.py is where magic happens. Elasticsearch sorting and filtering are specified there as well as mapping of documents to Loki labels.

You may also take a look at example-packetbeat.py to understand how to tweak the transfer even more.