DEV Community

Cover image for Kafka UI in Action: Monitoring and Managing Kafka Like a Pro
Kneonix
Kneonix

Posted on

2

Kafka UI in Action: Monitoring and Managing Kafka Like a Pro

Why You Need to Monitor Your Kafka Broker

Recently, I was working with Apache Kafka as the message broker for a microservice architecture. I quickly realized that it can be difficult to verify if Kafka is running correctly, especially when the rest of your application hasn’t finished connecting. In scenarios like this, a monitoring tool can be incredibly helpful and that’s where Kafka UI shines.

Image description

What Can You Do With Kafka UI?

Kafka UI is a powerful, lightweight monitoring tool for Apache Kafka. Here's what it offers out of the box:

  • Multi-Cluster Management : Monitor and manage multiple Kafka clusters in one place.
  • Performance Monitoring with Metrics Dashboard : Track essential Kafka metrics through an intuitive dashboard.
  • View Kafka Brokers : Inspect topic and partition assignments, controller status, and broker metadata.
  • View Kafka Topics : See partition count, replication status, and custom topic configurations.
  • View Consumer Groups : Monitor per-partition parked offsets and consumer lag (combined and per partition).
  • Browse Messages : View Kafka messages in JSON, plain text, or Avro formats.
  • Dynamic Topic Configuration : Create and configure new topics dynamically.
  • Configurable Authentication : Secure the UI using GitHub, GitLab, or Google OAuth 2.0.
  • Custom Serialization/Deserialization Plugins : Use built-in serializers like AWS Glue or Smile, or implement your own.
  • Role-Based Access Control (RBAC) : Manage fine-grained access permissions to the UI.
  • Data Masking : Obfuscate sensitive data in topic messages.

How to Set Up a Kafka Broker in Your System

You can run Kafka and Zookeeper in two main ways: Docker and Kubernetes.

Option 1: Docker

Here’s a simple docker-compose.yml setup:

version: "3.8"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    ports:
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
Enter fullscreen mode Exit fullscreen mode

Run:

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

Option 2: Kubernetes

Create two YAML files: zookeeper.depl.yaml and kafka.depl.yaml.

zookeeper.depl.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: zookeeper-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zookeeper
  template:
    metadata:
      labels:
        app: zookeeper
    spec:
      containers:
        - name: zookeeper
          image: confluentinc/cp-zookeeper:latest
          ports:
            - containerPort: 2181
          env:
            - name: ZOOKEEPER_CLIENT_PORT
              value: "2181"
            - name: ZOOKEEPER_TICK_TIME
              value: "2000"
---
apiVersion: v1
kind: Service
metadata:
  name: zookeeper-srv
spec:
  selector:
    app: zookeeper
  ports:
    - name: zookeeper
      protocol: TCP
      port: 2181
      targetPort: 2181
Enter fullscreen mode Exit fullscreen mode

kafka.depl.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-broker-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-broker
  template:
    metadata:
      labels:
        app: kafka-broker
    spec:
      containers:
        - name: kafka-broker
          image: confluentinc/cp-kafka:latest
          ports:
            - containerPort: 9092
          env:
            - name: KAFKA_BROKER_ID
              value: "1"
            - name: KAFKA_ZOOKEEPER_CONNECT
              value: "zookeeper-srv:2181"
            - name: KAFKA_ADVERTISED_LISTENERS
              value: PLAINTEXT://kafka-broker-srv:9092
            - name: KAFKA_LISTENERS
              value: PLAINTEXT://0.0.0.0:9092
            - name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
              value: "1"
            - name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
              value: "false"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-broker-srv
spec:
  selector:
    app: kafka-broker
  ports:
    - name: kafka-broker
      protocol: TCP
      port: 9092
      targetPort: 9092
Enter fullscreen mode Exit fullscreen mode

Apply them by these command:

kubectl apply -f zookeeper.depl.yaml
kubectl apply -f kafka.depl.yaml
Enter fullscreen mode Exit fullscreen mode

How to Set Up Kafka UI

You can deploy Kafka UI via Docker or Kubernetes as well.

Option 1: Docker

Add this to your docker-compose.yml file:

kafka-ui:
  image: provectuslabs/kafka-ui:latest
  container_name: kafka-ui
  ports:
    - "8080:8080"
  depends_on:
    - kafka
  environment:
    KAFKA_CLUSTERS_0_NAME: "local"
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: "kafka:9092"
Enter fullscreen mode Exit fullscreen mode

Then run:

docker-compose up -d
Enter fullscreen mode Exit fullscreen mode

Option 2: Kubernetes

Create a kafkaUI.depl.yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-ui-depl
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kafka-ui
  template:
    metadata:
      labels:
        app: kafka-ui
    spec:
      containers:
        - name: kafka-ui
          image: provectuslabs/kafka-ui:latest
          ports:
            - containerPort: 8080
          env:
            - name: KAFKA_CLUSTERS_0_NAME
              value: "local"
            - name: KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS
              value: "kafka-broker-srv:9092"
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-ui-srv
spec:
  selector:
    app: kafka-ui
  ports:
    - name: kafka-ui
      protocol: TCP
      port: 8080
      targetPort: 8080
Enter fullscreen mode Exit fullscreen mode

Apply it:

kubectl apply -f kafkaUI.depl.yaml
Enter fullscreen mode Exit fullscreen mode

Viewing the UI

If you’re running this on Kubernetes, forward the port:

kubectl port-forward service/kafka-ui-srv 8080:8080
Enter fullscreen mode Exit fullscreen mode

Then open your browser and go to http://localhost:8080 to monitor and manage your Kafka broker, topics, and consumer groups.

Learn More

Modern auth, access control, and billing built for engineers

Modern auth, access control, and billing built for engineers

Skip the glue code. Launch faster with authentication, RBAC, and billing that live in your stack from day one.

Get a free account

Top comments (1)

Collapse
 
_mohamadmohamadi profile image
Mohamad

so helpful
thanks for sharing that❤️

Feature flag article image

Create a feature flag in your IDE in 5 minutes with LaunchDarkly’s MCP server 🏁

How to create, evaluate, and modify flags from within your IDE or AI client using natural language with LaunchDarkly's new MCP server. Follow along with this tutorial for step by step instructions.

Read full post

👋 Kindness is contagious

Discover fresh viewpoints in this insightful post, supported by our vibrant DEV Community. Every developer’s experience matters—add your thoughts and help us grow together.

A simple “thank you” can uplift the author and spark new discussions—leave yours below!

On DEV, knowledge-sharing connects us and drives innovation. Found this useful? A quick note of appreciation makes a real impact.

Okay