This repository demonstrates how to use Prometheus and Grafana for monitoring an Apache Kafka cluster.
-
Kafka (Confluent)
-
Zookeeper
-
Kafka Schema Registry (Confluent)
-
KSQLDB
-
Kafka Connect
-
Prometheus
-
Grafana
-
AKHQ (https://akhq.io/)
-
ZooNavigator (https://zoonavigator.elkozmon.com/en/stable/)
$ ./up -h
--------------------------------------------------------------------------------
____ __ _____ __ _
/ __ \____ _/ /_____ _ / ___// /_________ ____ _____ ___ (_)___ ____ _
/ / / / __ `/ __/ __ `/ \__ \/ __/ ___/ _ \/ __ `/ __ `__ \/ / __ \/ __ `/
/ /_/ / /_/ / /_/ /_/ / ___/ / /_/ / / __/ /_/ / / / / / / / / / / /_/ /
/_____/\__,_/\__/\__,_/ /____/\__/_/ \___/\__,_/_/ /_/ /_/_/_/ /_/\__, /
/____/
Powered by Apache Kafka!
--------------------------------------------------------------------------------
Usage: ./up [options]
-n <[NAME]> : Name of the stack to deploy (required)
-s <[SERVICE]> : Service names to deploy (run all services if empty)
-h : Print this Help.
The table below lists the available docker-compose stacks:
NAME (./up -n <NAME>) | FILE (docker compose -f <FILE> up -d)
=================================== ==================================================
zk-kafka-multiple-nodes-sasl | zk-kafka-multiple-nodes-sasl-stack.yml
zk-kafka-multiple-nodes | zk-kafka-multiple-nodes-stack.yml
zk-kafka-single-node-full | zk-kafka-single-node-full-stack.yml
zk-kafka-single-node-sasl | zk-kafka-single-node-sasl-stack.yml
zk-kafka-single-node | zk-kafka-single-node-stack.yml
zkless-kafka-multiple-nodes | zkless-kafka-multiple-nodes-stack.yml1. Clone the Kafka Monitoring Suite repository.
$ git clone https://github.com/streamthoughts/kafka-monitoring-stack-docker-compose.git
$ cd kafka-monitoring-stack-docker-compose2. Start Confluent/Kafka cluster.
Deploy one of the provided docker-compose stack:
Note: Depending on your network speed, this may take few minutes to download all images.
Start/Stop with:
$ ./zk-kafka-single-node-stack-start.sh
$ ./zk-kafka-single-node-stack-stop.sh
# or directly
$ docker-compose -f zk-kafka-single-node-stack.yml up -d
$ docker-compose -f zk-kafka-single-node-stack.yml down$ docker-compose -f zk-kafka-multiple-nodes-stack.yml up -d
$ docker-compose -f zk-kafka-multiple-nodes-stack.yml down$ ./zk-kafka-single-node-sasl-stack-start.sh
$ ./zk-kafka-single-node-sasl-stack-stop.sh$ docker-compose -f zk-kafka-multiple-nodes-sasl-stack.yml up -d
$ docker-compose -f zk-kafka-multiple-nodes-sasl-stack.yml downStart/Stop with:
$ ./full-single-node-stack-start.sh
$ ./full-single-node-stack-stop.sh
# or directly
$ docker-compose -f zk-kafka-single-node-full-stack.yml up -d
$ docker-compose -f zk-kafka-single-node-full-stack.yml down3. Create Topic.
Create demo-topic with 6 partitions and 3 replicas.
$ docker exec -it kafka101 \
kafka-topics \
--create \
--partitions 6 \
--replication-factor 3 \
--topic demo-topic \
--bootstrap-server kafka101:290924. Produce messages.
Open a new terminal window, generate some message to simulate producer load.
$ docker exec -it kafka101 \
kafka-producer-perf-test \
--throughput 500 \
--num-records 100000000 \
--topic demo-topic \
--record-size 100 \
--producer-props bootstrap.servers=kafka101:290925. Consume messages.
Open a new terminal window, generate some message to simulate consumer load.
$ docker exec -it kafka101 \
kafka-consumer-perf-test \
--messages 100000000 \
--timeout 1000000 \
--topic demo-topic \
--reporting-interval 1000 \
--show-detailed-stats \
--bootstrap-server kafka101:290926. Open Grafana.
Open your favorite web browser and open one of the provided Grafana dashboards :
-
Kafka Cluster / Global Health Check
-
Kafka Cluster / Performance
-
Kafka Cluster / Zookeeper Connections
-
Kafka Cluster / JVM & OS
-
Kafka Cluster / Hard disk usage
-
Kafka Cluster / Topic Logs
Grafana is accessible at the address : http://localhost:3000
Security are :
-
user :
admin -
password :
kafka
Prometheus is accessible at the address : http://localhost:9090
