GitHub - matthew-hollick/kafka-playground

Name	Name	Last commit message	Last commit date
Latest commit Matthew Hollick Update client.sh Oct 7, 2019 6ddfc4b · Oct 7, 2019 History 17 Commits
Dockerfiles	Dockerfiles	Added haggar load tested	Oct 7, 2019
.env	.env	Now includes a demo log pipeline	Oct 7, 2019
.gitignore	.gitignore	improvements but still does not work right	Sep 16, 2019
README.md	README.md	Update README.md	Oct 7, 2019
carbon-clickhouse.conf	carbon-clickhouse.conf	iadded frontend	Oct 7, 2019
client.sh	client.sh	Update client.sh	Oct 7, 2019
collectd.conf	collectd.conf	Added haggar load tested	Oct 7, 2019
docker-compose-logs.yml	docker-compose-logs.yml	Now includes a demo log pipeline	Oct 7, 2019
docker-compose-metrics.yml	docker-compose-metrics.yml	moved haggar variables to a varibales file	Oct 7, 2019
docker-compose.yml.backup	docker-compose.yml.backup	Merge branch 'mjh'	Oct 7, 2019
graphite-clickhouse.conf	graphite-clickhouse.conf	iadded frontend	Oct 7, 2019
init.sql	init.sql	latest	Oct 6, 2019
logfile	logfile	Now includes a demo log pipeline	Oct 7, 2019
logger.sh	logger.sh	Now includes a demo log pipeline	Oct 7, 2019
logstashback.conf	logstashback.conf	Now includes a demo log pipeline	Oct 7, 2019
logstashfront.conf	logstashfront.conf	Now includes a demo log pipeline	Oct 7, 2019
riemann.config.logging	riemann.config.logging	improvements but still does not work right	Sep 16, 2019
riemannback.config	riemannback.config	iadded frontend	Oct 7, 2019
riemannfront.config	riemannfront.config	iadded frontend	Oct 7, 2019
rollup.xml	rollup.xml	improvements but still does not work right	Sep 16, 2019

Repository files navigation

First build a couple of containers

docker-compose -f docker-compose-logs.yml build

Then Metrics work:

docker-compose -f docker-compose-metrics.yml up

Expect loads of errors as it gets running. Then you have:

collectd -> riemann -> kafka -> riemann -> carbon-clickhouse -> clickhouse -> graphite-clickhouse -> carbonapi -> grafana

haggar -> riemann -> kafka -> riemann -> carbon-clickhouse -> clickhouse -> graphite-clickhouse -> carbonapi -> grafana

Grafana is available on port 3000

Logs kind of work:

docker-compose -f docker-compose-logs.yml up

Expect loads of errors as it gets running. Then you have:

logger -> logstash -> kafka -> logstash -> elasticsearch -> kibana

Kibana is available on port 5601

Log formatting is all screwed up..

Reference:

and others that I have mislaid.