This repository contain an observability stack using tools.
These one is deploying with Docker Compose to test all the stack's tools.
It is only a testing stack with no security, no production-ready configuration, ... This for testing purpose.
The observability stack is composed by Mimir, Loki and Tempo which are deployed in Monolithic mode with 3 instances of each.
It's also include a Grafana vizualisation tool.
The application stack is composed by a FastAPI application with some routes and PostgreSQL database with 1 table.
Logs are written in a log file, scrape by Grafana Promtail and send to Grafana Agent.
Metrics are exposed via Prometheus Client and scraped by Grafana Agent.
Traces are sent via Opentelemetry SDK to Grafana Agent.
Unit tests are performed by k6.io.
Mimir, Loki and Tempo metrics are also exposed througth Grafana Agent.
The metrics of all containers are exposed with cAdvisor.
This repository is run on Mac M1 (ARM architecture). If you want to use another architecture, be sure to modify the Dockerfiles which import arm binary (ex. promtail on the app Dockerfile).
- Install Docker.
- Create Docker network :
docker network create observability-network
-
Clone this repository :
git clone https://github.com/thomasdesplaces/observability_docker.git
-
Build & deploy observability stack :
docker-compose --profile grafana up --build
-
Access to Grafana for visualization : http://localhost:3000
-
Build & deploy application stack :
docker-compose --profile application up --build
Deploy an application or database or other on the same docker network with these parameters :
- Traces :
- Protocol : http
- Host : nginx
- Port : 3600
- Path : /v1/traces
- Logs :
- Protocol : http
- Host : nginx
- Port : 3500
- Path : /loki/api/v1/push
- Metrics :
- Expose with Prometheus (or based on Prometheus) on example port :
8000
and modify the targets value in agent.yaml (line 24).
-
Get Logs : Promtail --> Nginx : 3500 --> Agent : 3501 --> Nginx : 3502 --> Loki : 3503
-
Get Traces : oTel SDK --> Nginx : 3600 --> Agent : 3601 --> Nginx : 3602 --> Tempo : 3603
-
Get Metrics : Prometheus Exporter : 5050 (on path /metrics) <-- Agent --> Nginx : 3702 --> Mimir : 3703
-
Consult Logs on Grafana : Grafana --> Nginx : 4502 --> Loki : 3503
-
Consult Traces on Grafana : Grafana --> Nginx : 4602 --> Tempo : 4603
-
Consult Metrics on Grafana : Grafana --> Nginx : 4702 --> Mimir : 3703
-
Front port : 5055
-
Backend/API port : 5050
-
Database port : 5432
Logo/Link | Version | Usage |
---|---|---|
![]() |
main-6f9d397 | Used to scrape data from applications (front, back, databases, ...) and send to each specific storage (like OpenTelemetry Collector). Based on https://github.com/grafana/agent/blob/main/example/docker-compose/docker-compose.yaml |
![]() |
v1.5.0 | Used to store Traces (like Jaeger or Zipkin). Based on https://github.com/grafana/tempo/tree/main/example/docker-compose/scalable-single-binary |
![]() |
v2.4.0 | Used to store Metrics (like Prometheus or Cortex). Based on https://github.com/grafana/mimir/blob/main/docs/sources/mimir/tutorials/play-with-grafana-mimir/docker-compose.yml |
![]() |
v2.7.0 | Used to store Logs (like Elasticsearch). Based on https://github.com/grafana/loki/tree/main/examples/getting-started |
![]() |
v9.3.1 | Used to visualize data (like Kibana). Based on https://github.com/grafana/grafana |
![]() |
Latest | Used to load balance traffic between each instance (on Cloud, use S3, Google Cloud Storage or similar). Based on https://github.com/minio/minio |
![]() |
Latest | Used to load balance traffic between each instance. Based on https://github.com/nginx/nginx |
![]() |
Latest | Used to expose all containers metrics |
Thanks to @Blueswen for the FastAPI observability configuration.
Logo/Link | Version | Usage |
---|---|---|
![]() |
Latest | Used to scrape application logs from file. |
![]() |
0.88.0 | Used to expose API. Based on https://github.com/Blueswen/fastapi-observability |
![]() |
15-Bullseye | Used to store application data. |
![]() |
Latest | Used to export Postgres metrics. Based on https://hub.docker.com/r/wrouesnel/postgres_exporter |
![]() |
Latest | Used to performed unit tests. |
![]() |
1.15.0 | Used to generate traces. |