Forem: Arata

Building a Multi-Node Kubernetes Cluster with Vagrant

Arata — Mon, 10 Nov 2025 07:49:56 +0000

"In distributed systems, consistency isn’t just a property — it’s a promise."

Why This Article

Imagine you’re building a small banking application. Users can deposit and withdraw money, check their balances, and expect data accuracy every single time — even if multiple requests hit the system simultaneously. But the moment you deploy it across containers, networks, and replicas, one question starts haunting every architect:

How do we keep data consistent when everything is happening everywhere?

In this tutorial, we’ll explore that question through a hands-on story — from concept to infrastructure — and deploy a Spring Boot + PostgreSQL banking demo across a five-node Kubernetes lab, fully automated with Vagrant. Our goal isn’t to ship production code, but to understand the design thinking behind consistency, locking, and automation.

Most tutorials use Minikube or kind, which are great for learning but limited to single-node simulations.

What if you could spin up a full Kubernetes cluster — control plane, multiple worker nodes, real networking, storage, and ingress — entirely automated and reproducible?

It’s a perfect local lab for experimenting with deployments, storage, and load testing — without relying on cloud services.

What You’ll Learn

Build a 5-node Kubernetes cluster using Vagrant and VirtualBox
Automate provisioning with Bash scripts
Deploy a real Spring Boot + Postgres application
Test the rest endpoint using k6 load testing

Build, deploy, and test a real multi-node Kubernetes cluster from scratch — all on your local machine.

Design

The system provides RESTful endpoints for withdrawal and deposit operations, served by a Spring Boot–based API backend. When the API receives a client request, it updates the account balance in a PostgreSQL relational database. To ensure data consistency under concurrent transactions, the system supports both optimistic and pessimistic locking mechanisms.

Both the API backend and the PostgreSQL database are deployed on a Kubernetes cluster comprising five virtual machines:

One control plane node for cluster management
One edge node for network routing and ingress
Two worker nodes hosting the Spring Boot web applications
One database node running PostgreSQL

This is a visual representation of the cluster setup:

K8S Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    Kubernetes Cluster                       │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  Control Plane (k8s-cp-01)                                  │
│  └─ IP: 192.168.56.10                                       │
│  └─ Role: Master node, API server, scheduler, controller    │
│                                                             │
│  Worker Nodes:                                              │
│  ├─ k8s-node-01 (192.168.56.11) - tier: edge                │
│  │  └─ Ingress Controller, Local Path Provisioner, MetalLB  │
│  ├─ k8s-node-02 (192.168.56.12) - tier: backend             │
│  │  └─ Spring Boot Application Pods                         │
│  ├─ k8s-node-03 (192.168.56.13) - tier: backend             │
│  │  └─ Spring Boot Application Pods                         │
│  └─ k8s-node-04 (192.168.56.14) - tier: database            │
│     └─ PostgreSQL Database                                  │
│                                                             │
│  LoadBalancer IP Pool: 192.168.56.240-250                   │
└─────────────────────────────────────────────────────────────┘

Building the Complete Environment:

The environment will be provisioned using Vagrant, which automates the creation of virtual machines and the setup of the Kubernetes cluster. Once the infrastructure is ready, it will deploy the prebuilt cloud-native Spring Boot web application and the fully configured PostgreSQL database, assembling the complete application service environment.

Prerequisites

The following tools are required on your host machine:

Tool	Version	Install
VirtualBox	≥ 7.1.6	See install checklist
Vagrant	≥ 2.4.9	See install checklist
RAM	≥ 13 GB	3GB for cp + 2 GB per node
CPU	≥ 4 cores	Recommended
Network	192.168.56.0/24	VirtualBox Host-Only

Below is the installation command:

# Windows (Chocolatey, run admin PowerShell)
choco install virtualbox vagrant

# Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y virtualbox
# Get latest vagrant from HashiCorp website or apt repo
# Install HashiCorp GPG key
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
# Add HashiCorp repository
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get install -y vagrant

Step 1: Vagrant

Clone the project

git clone https://github.com/arata-x/vagrant-k8s-bank-demo.git

Project Structure

The outline of the project structure is as follows:

Root:
│  Vagrantfile
│
└─provision
    ├─deployment
    │  │
    │  └─standard
    │      ├─app
    │      │      10-config.yml
    │      │      20-rbac.yml
    │      │      30-db-deploy.yml
    │      │      40-app-deploy.yml
    │      │      50-services.yml
    │      │      60-network-policy.yml
    │      │      70-utilities.yml
    │      │
    │      └─infra
    │              10-storage-class.yml
    │              20-metallb.yaml
    │
    └─foundation
            10-common.sh
            20-node-network.sh
            30-control-panel.sh
            40-join-node.sh
            50-after-vagrant-setup.sh
            join-command.sh

A Vagrantfile is a configuration file written in Ruby syntax that defines how Vagrant should provision and manage a virtual machine (VM). It’s the heart of any Vagrant project—used to automate the setup of reproducible development environments.

Vagrantfile specifies:

-Base OS image (e.g., ubuntu/jammy64) -Resources (CPU, memory, disk) -Networking (port forwarding, private/public networks) -Provisioning scripts (e.g., install Java, Maven, Docker) -Shared folders between host and VM

Below is the content of Vagrantfile

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/jammy64"
  config.vm.synced_folder ".", "/vagrant"
  root_path = "provision/foundation/";
  # Common setup for all nodes
  config.vm.provision "shell", path: root_path + "10-common.sh"
  # Node definitions
  nodes = [
    { name: "k8s-cp-01",  ip: "192.168.56.10", script: "30-control-panel.sh", memory: 3072 },
    { name: "k8s-node-01", ip: "192.168.56.11", script: "40-join-node.sh",    memory: 2048 },
    { name: "k8s-node-02", ip: "192.168.56.12", script: "40-join-node.sh",    memory: 2048 },
    { name: "k8s-node-03", ip: "192.168.56.13", script: "40-join-node.sh",    memory: 2048 },
    { name: "k8s-node-04", ip: "192.168.56.14", script: "40-join-node.sh",    memory: 2048 },
  ]
  # Create VMs
  nodes.each do |node|
    config.vm.define node[:name] do |node_vm|
      node_vm.vm.provider "virtualbox" do |vb|
        vb.cpus = 2
        vb.memory = node[:memory]
      end
      node_vm.vm.hostname = node[:name]
      node_vm.vm.network "private_network", ip: node[:ip]
      node_vm.vm.provision "shell", path: root_path + "20-node-network.sh", args: node[:ip]      
      node_vm.vm.provision "shell", path: root_path + "#{node[:script]}"
    end
  end
end

1️⃣ Overview

This Vagrantfile provisions :

Creates 5 Ubuntu 22.04 VMs
Installs Docker, kubeadm, kubelet, kubectl
Initializes Kubernetes control plane
Joins 4 worker nodes
Configures Calico CNI networking

2️⃣ Global Configuration

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/jammy64"
  config.vm.synced_folder ".", "/vagrant"

Vagrant.configure("2") → Uses configuration syntax version 2.
config.vm.box → Every VM uses Ubuntu 22.04 LTS (“jammy64”).
config.vm.synced_folder → Shares your project folder on the host with each guest VM at /vagrant.

3️⃣ Common Provisioning

  root_path = "provision/foundation/"
  config.vm.provision "shell", path: root_path + "10-common.sh"

Runs once globally for all machines to install baseline packages, set host files, etc.

4️⃣ Cluster node setup

nodes = [
  { name: "k8s-cp-01",  ip: "192.168.56.10", script: "30-control-panel.sh", memory: 3072 },
  { name: "k8s-node-01", ip: "192.168.56.11", script: "40-join-node.sh",    memory: 2048 },
  { name: "k8s-node-02", ip: "192.168.56.12", script: "40-join-node.sh",    memory: 2048 },
  { name: "k8s-node-03", ip: "192.168.56.13", script: "40-join-node.sh",    memory: 2048 },
  { name: "k8s-node-04", ip: "192.168.56.14", script: "40-join-node.sh",    memory: 2048 },
]

Defines five nodes for node creation loop.

5️⃣ Node Creation Loop

nodes.each do |node|
  config.vm.define node[:name] do |node_vm|
    node_vm.vm.provider "virtualbox" do |vb|
      vb.cpus = 2
      vb.memory = node[:memory]
    end
    node_vm.vm.hostname = node[:name]
    node_vm.vm.network "private_network", ip: node[:ip]
    node_vm.vm.provision "shell", path: root_path + "20-node-network.sh", args: node[:ip]      
    node_vm.vm.provision "shell", path: root_path + "#{node[:script]}"
  end
end

For each node:

Allocates cpus = 2 and memory
Defines a named VM.
Sets hostname inside the guest.
Configures a private network on 192.168.56.0/24.
Runs setup-node-network.sh to configure IPs, /etc/hosts, etc.
Runs role-specific script (setup-controller.sh or setup-node.sh).

6️⃣ Build the Cluster

Run vagrant up to start provisioning.

cd k8s
vagrant up

🕒 Expected duration: 10–15 minutes.

Verify all VMs are running:

vagrant status

7️⃣ Provisioning Scripts Deep Dive

Let's take a closer look at the shell scripts used during provisioning.

10-common.sh

The script configures the operating system’s memory settings, installs the required Kubernetes components, and sets up the environment for Kubernetes networking.

# Disable Swap
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

# Install Core Dependencies
sudo apt-get install -y kubelet kubeadm containerd

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay
sudo modprobe br_netfilter

sudo cat <<'EOF' | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system

Purpose

1. Disable Swap Ensures Kubernetes can accurately manage memory resources.

2. Add Kubernetes Repository and Install Components

Configures the official Kubernetes APT repo and installs kubelet, kubeadm, and containerd.

3. Setup Kernel Modules

Enables overlay and br_netfilter modules required for container networking and storage layers.

4. Set Kernel Parameters

Adjusts sysctl settings to enable IP forwarding and proper packet handling between bridged interfaces.

20-common.sh

This script configures the Kubernetes node’s network identity by explicitly assigning its IP address to the kubelet service, ensuring proper communication and cluster registration.

NODE_IP=$(ip -4 addr show enp0s8 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')

DROPIN_FILE=/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

if ! grep -q -- "--node-ip=$NODE_IP" "$DROPIN_FILE"; then
  sudo sed -i "0,/^Environment=\"KUBELET_KUBECONFIG_ARGS=/s|\"$| --node-ip=$NODE_IP\"|" "$DROPIN_FILE"
fi

sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl restart kubelet

When using Vagrant, a default NAT interface (enp0s3) is created for outbound network access. A second, user-defined network interface (enp0s8) is typically added for internal cluster communication. However, Kubernetes may fail to correctly resolve the node’s IP address in this setup, requiring manual configuration.

After testing, the following approach proves effective: explicitly assign the node IP to the enp0s8 interface and configure the kubelet to use this IP. Once applied, the kubelet service starts with the correct node IP address, ensuring reliable communication between cluster components and accurate node registration within the Kubernetes control plane.

30-contol-panel.sh

This script automates the setup of a Kubernetes control plane node in a virtualized environment. It also takes care of installing and configuring essential tools like kubectl, the Container Network Interface (CNI), and a monitoring stack to give you full visibility into your cluster.

sudo apt-get install -y kubectl

# Initialize cluster
sudo kubeadm init --apiserver-advertise-address=192.168.56.10 --pod-network-cidr=10.224.0.0/16

# Setup kubeconfig
mkdir -p /home/vagrant/.kube
cp /etc/kubernetes/admin.conf /home/vagrant/.kube/config
chown vagrant:vagrant /home/vagrant/.kube/config
mkdir -p ~/.kube
cp /etc/kubernetes/admin.conf ~/.kube/config

# Create join command
sudo kubeadm token create --print-join-command > /vagrant/provision/foundation/join-command.sh

# Install Calico network plugin
AUTO_METHOD="cidr=192.168.56.0/24"
curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.30.3/manifests/calico.yaml
kubectl apply -f calico.yaml
kubectl set env daemonset/calico-node -n kube-system IP_AUTODETECTION_METHOD="$AUTO_METHOD"
sudo systemctl restart kubelet

# Install K9s
wget https://github.com/derailed/k9s/releases/latest/download/k9s_linux_amd64.deb
sudo apt install ./k9s_linux_amd64.deb

1. Installs kubectl

Initializes the Kubernetes cluster using kubeadm, specifying the API server advertise address (192.168.56.10) and the Pod network CIDR (10.224.0.0/16).

2. Kubeconfig Configuration

Sets up the Kubernetes configuration (admin.conf) for both the vagrant user and the root user, enabling access to cluster management commands.

3. Node Join Command Generation

Creates and stores the cluster join command in /vagrant/provision/foundation/join-command.sh for worker nodes to join the cluster.

3. Calico Network Plugin Setup

Downloads and applies the Calico manifest to enable networking between pods. Configures Calico’s IP autodetection method to use the local network (cidr=192.168.56.0/24).

4. Kubernetes Management Tool Installation

Installs k9s, a terminal-based Kubernetes cluster management tool

Step 2: Post-Init Setup

SSH into the control plane:

vagrant ssh k8s-cp-01

Run the post-initialization script:

sudo /vagrant/provision/foundation/50-after-vagrant-setup

50-after-vagrant-setup

This script configures functional labels to define node roles(edge, backend, database), and configures essential Kubernetes components with targeted scheduling.

NODES=(k8s-node-01 k8s-node-02 k8s-node-03 k8s-node-04)
kubectl label node "${NODES[0]}" node-role.kubernetes.io/worker-node="" tier=edge --overwrite
kubectl label node "${NODES[1]}" node-role.kubernetes.io/worker-node="" tier=backend --overwrite
kubectl label node "${NODES[2]}" node-role.kubernetes.io/worker-node="" tier=backend --overwrite
kubectl label node "${NODES[3]}" node-role.kubernetes.io/worker-node="" tier=database --overwrite
# Install Local Path Provisioner for dynamic storage provisioning
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
# Install NGINX Ingress Controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml
# Install MetalLB
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.15.2/config/manifests/metallb-native.yaml

After labeling, it deploys several core infrastructure components:

Local Path Provisioner – Enables dynamic storage provisioning
NGINX Ingress Controller – Provides ingress routing
MetalLB – Implements Layer 2 load balancing with controller deployment

Verify:

kubectl get nodes --show-labels
kubectl get pods -A

Step 3: Deploy Infrastructure

Apply storage and networking configuration.

Run Deployment

kubectl apply -f /vagrant/provision/deployment/standard/infra

What’s Inside

10-storage-class.yml — Local path dynamic PV provisioning
20-metallb.yaml — IP pool and L2Advertisement setup

Confiuration Review

10-storage-class.yml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: vm-storage
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

This manifest defines a StorageClass named vm-storage that uses the Rancher Local Path Provisioner to dynamically create node-local PersistentVolumes. It sets volumeBindingMode: WaitForFirstConsumer so volume provisioning is deferred until a pod is scheduled, ensuring the PV is created on the same node as the workload. The reclaimPolicy: Delete cleans up underlying storage when the PersistentVolumeClaim is removed.

20-metallb.yaml

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-address-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.168.56.240-192.168.56.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default-l2-advert
  namespace: metallb-system

In cloud environments, services like AWS or GCP automatically provide load balancers to expose your applications to the outside world. But on bare-metal or virtualized Kubernetes clusters, you don’t get that luxury out of the box — and that’s where MetalLB steps in.

The manifest configures MetalLB to handle external traffic just like a cloud load balancer would. It defines an IPAddressPool that allocates IPs from 192.168.56.240–192.168.56.250, and an L2Advertisement that announces those addresses at Layer 2 so other devices on the network can reach your services directly.

The result is seamless, cloud-like load balancing for your on-premises or Vagrant-based Kubernetes setups — giving your local cluster the same networking power as a managed one.

Verify:

kubectl get storageclass
kubectl get ipaddresspool -n metallb-system

Step 4: Deploy the Application

In this step, we’re bringing everything together — deploying a complete multi-tier application stack on Kubernetes. This manifests sets up dedicated namespaces, injects configuration data, and applies the necessary RBAC permissions for secure access control. It also provisions a PostgreSQL database backed by persistent storage, then deploys a Spring Boot application with multiple replicas for scalability and resilience.

To make the services accessible, it exposes them through ClusterIP and NodePort, and strengthens cluster security with NetworkPolicies that control how pods communicate. Optionally, it can also install monitoring and maintenance utilities, giving you full visibility and manageability of your application stack — all running seamlessly inside Kubernetes.

Run Deployment

kubectl apply -f /vagrant/provision/deployment/standard/app

What’s Inside

10-config.yml — Namespaces, ConfigMaps, Secrets
20-rbac.yml — RBAC setup
30-db-deploy.yml — PostgreSQL with PVC
40-app-deply.yml — Spring Boot app (2 replicas)
50-services.yml — ClusterIP and NodePort
60-network-policy.yml — Secure traffic rules
70-utilities.yml — Optional utilities

Confiuration Review

10-config.yml

apiVersion: v1
kind: Namespace
metadata: { name: demo }

Creates a demo namespace for isolating application resources.

20-rbac.yml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: database-rolebinding
  namespace: demo
subjects:
- kind: ServiceAccount
  name: postgres-sa
  namespace: demo
roleRef:
  kind: Role
  name: database-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-rolebinding
  namespace: demo
subjects:
- kind: ServiceAccount
  name: app-sa
  namespace: demo
roleRef:
  kind: Role
  name: app-role
  apiGroup: rbac.authorization.k8s.io

The manifest defines two service accounts: database-sa for PostgreSQL and app-sa for the Spring Boot service, enabling least-privilege access and clear separation of duties.

30-db-deploy.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: db-cm
  namespace: demo
data:
  POSTGRES_USER: postgres
  POSTGRES_DB: appdb
  APP_USER: appuser
---
apiVersion: v1
kind: Secret
metadata:
  name: db-secret
  namespace: demo
type: Opaque
stringData:
  POSTGRES_PASSWORD: strong-password
  APP_PASSWORD: strong-password
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: db-init
  namespace: dem
data:
  00-roles.sql: (skip)
  01-db.sql: (skip)
  02-schema.sql: (skip)
  03-comments.sql: (skip)
  04-table.sql:  (skip)
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-headless
  namespace: demo
  labels:
    app: postgres      
spec:
  clusterIP: None
  ports:
    - name: pg
      port: 5432
      targetPort: 5432
  selector:
    app: postgres
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: demo
  labels:
    app: postgres
spec:
  type: ClusterIP
  ports:
    - name: pg
      port: 5432
      targetPort: 5432
  selector:
    app: postgres
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: demo
spec:
  serviceName: postgres-headless
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      serviceAccountName: database-sa
      nodeSelector:
        tier: database
      containers:
        - name: postgres
          image: postgres:18
          ports:
            - containerPort: 5432
              name: pg
          envFrom:
            - secretRef:
                name: db-secret
            - configMapRef:
                name: db-cm
          volumeMounts:
            - name: db-data
              mountPath: /var/lib/postgresql/
            - name: run-socket
              mountPath: /var/run/postgresql
            - name: db-init
              mountPath: /docker-entrypoint-initdb.
      volumes:
        - name: run-socket
          emptyDir: {}
        - name: db-init
          configMap:
            name: db-init
  volumeClaimTemplates:
    - metadata:
        name: db-data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: vm-storage
        resources:
          requests:
            storage: 5Gi

The manifest provisions a single-replica PostgreSQL database as a StatefulSet on the database tier. It uses the database-sa service account, loads environment variables and credentials from ConfigMaps and Secret, runs optional init SQL from a ConfigMap, and persists data via a PersistentVolumeClaim using the vm-storage StorageClass, and exposes the database through two Kubernetes Services: a standard ClusterIP Service postgres for in-cluster access on port 5432, and a headless Service postgres-headless to enable direct pod-to-pod communication and stable DNS resolution—typically.

40-app-deply.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: springboot-cm
  namespace: demo
  labels:
    environment: demo
data:
  BPL_JVM_THREAD_COUNT: "100"
  JAVA_TOOL_OPTIONS: "-XX:InitialRAMPercentage=25.0 -XX:MaxRAMPercentage=75.0"
  LOGGING_LEVEL_ROOT: INFO
  SPRING_PROFILES_ACTIVE: prod
  SPRING_DATASOURCE_URL: "jdbc:postgresql://postgres.demo.svc.cluster.local:5432/appdb"
---
apiVersion: v1
kind: Secret
metadata:
  name: springboot-secret
  namespace: demo
  labels:
    environment: demo
type: Opaque  
stringData:
  spring.datasource.username: appuser
  spring.datasource.password: strong-password
---
apiVersion: v1
kind: Service
metadata:
  name: api-svc
  namespace: demo
  labels:
    environment: demo
spec:
  selector:
    app: api
  ports:
    - port: 80
      targetPort: 8080
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bank-account-demo
  namespace: demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api     
    spec:
      serviceAccountName: app-sa
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: tier
                operator: In
                values:
                - backend
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: api
      containers:
        - name: bank-account-demo
          image: docker.io/aratax/bank-account-demo:1.0
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: springboot-cm
            - secretRef:
                name: springboot-secret
      - name: wait-for-database
        image: busybox
        command: ['sh', '-c', 'until nc -z postgres.demo.svc.cluster.local 5432; do echo waiting; sleep 2; done;']

This manifest provisions a two-replica Spring Boot application deployment in the backend tier. It uses the app-sa service account, loads runtime configuration and credentials from the ConfigMap and Secret, and connects to the PostgreSQL database via the internal DNS endpoint postgres.demo.svc.cluster.local:5432. The deployment includes an init container wait-for-database to ensure the database is reachable before application startup. It exposes the application through a ClusterIP Service named api-svc on port 80. The deployment is set up to run only on backend nodes and to spread its pods evenly across different nodes for better reliability and load balance.

50-services.yml

apiVersion: v1
kind: Service
metadata:
  name: database-nodeport
  namespace: demo
  labels:
    environment: demo
spec:
  type: NodePort
  selector:
    app: postgres
  ports:
    - name: pg
      port: 5432         
      targetPort: 5432    
      nodePort: 30000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: webapp-ingress
  namespace: demo
  labels:
    environment: demo
spec:
  ingressClassName: nginx
  rules:
    - host: app.demo.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-svc
                port:
                  number: 80

The manifest enables both the application and the database to be accessed from inside and outside the cluster. It defines a NodePort Service named database-nodeport that exposes the PostgreSQL database on port 30000 for external access, typically used for development and debugging. It also creates an Ingress resource named webapp-ingress that routes web traffic for app.demo.local to the internal api-svc service, which runs the Spring Boot application on port 80.

Step 5: Review Application Design

To implement a simple banking system, two tables were designed:

accounts — stores core account information (owner, currency, balance, etc.).
ledger_entries — records all debit/credit transactions linked to each account for auditing and reconciliation.

This schema ensures data integrity, supports concurrent balance updates via versioning, and provides immutable transaction history.

Database Table Layout:

+------------------------------------------------------------+
|                         accounts                           |
+------------------------------------------------------------+
| Column       | Type           | Constraints / Default      |
|--------------|----------------|----------------------------|
| id           | UUID (PK)      | DEFAULT uuidv7()           |
| owner_name   | TEXT           | NOT NULL                   |
| currency     | CHAR(3)        | NOT NULL                   |
| balance      | NUMERIC(18,2)  | NOT NULL DEFAULT 0         |
| version      | BIGINT         | NOT NULL DEFAULT 0         |
| updated_at   | TIMESTAMPTZ    | NOT NULL DEFAULT NOW()     |
+------------------------------------------------------------+
| INDEX: idx_accounts_owner (owner_name)                     |
+------------------------------------------------------------+

                  1
     accounts ─────────────┐
                           │  (fk_ledger_account)
                           ▼

+------------------------------------------------------------+
|                     ledger_entries                         |
+------------------------------------------------------------+
| Column       | Type           | Constraints / Default      |
|--------------|----------------|----------------------------|
| id           | UUID (PK)      | DEFAULT uuidv7()           |
| account_id   | UUID (FK)      | REFERENCES accounts(id)    |
| direction    | TEXT           | NOT NULL                   |
| amount       | NUMERIC(18,2)  | NOT NULL CHECK (amount > 0)|
| reason       | TEXT           |                            |
| created_at   | TIMESTAMPTZ    | NOT NULL DEFAULT NOW()     |
+------------------------------------------------------------+

SpringBoot:

The Java application provide unified transaction endpoint processes both deposit and withdrawal operations, allowing clients to specify the locking strategy (OPTIMISTIC or PESSIMISTIC) per request.

Rest Endpoints

@RestController
@RequestMapping("/api/accounts")
public class AccountController {

    private final AccountService accountService;

    public AccountController(AccountService accountService) { this.accountService = accountService;}

    @PostMapping(value = "/{id}/transaction", produces = MediaType.APPLICATION_JSON_VALUE)
    public ResponseEntity<TransactionResponse> transaction(
            @PathVariable UUID id,
            @Valid @RequestBody TransactionRequest request) {
        TransactionResponse response = accountService.executeTransaction(id, request );
        return ResponseEntity.ok(response);
    }

}

Rest Example POST /api/accounts/3f93c1c2-1c52-4df5-8c6a-9b0c6d7c5c11/transaction

{
  "type": "DEPOSIT",
  "amount": 500,
  "lockingMode": "OPTIMISTIC",
  "reason": "API_DEPOSIT"
}
# or
{
  "type": "WITHDRAWAL",
  "amount": 300,
  "lockingMode": "PESSIMISTIC",
  "reason": "API_WITHDRAWAL"
}

Concurrency Control Strategy in JPA The Java application uses JPA (Java Persistence API) — an ORM framework — to interact with a PostgreSQL database while maintaining data integrity during concurrent transactions. It also explores two different locking strategies, described below, to demonstrate how JPA handles concurrency in real-world scenarios.

Optimistic Locking Strategy

The @Version field provides optimistic concurrency control — each update automatically increments the version. When two transactions modify the same Account, the second commit detects a version mismatch and throws an OptimisticLockException, preventing lost updates without requiring database locks. A retry strategy with controlled backoff (5 attempts) can be applied to gracefully handle these transient conflicts.

Entity

@Data
@Entity
@Table(name = "accounts", schema = "app")
public class Account {

  @Id @UuidGenerator
  private UUID id;

  @Column(name = "owner_name", nullable = false)
  private String ownerName;

  @Column(length = 3, nullable = false)
  private String currency;

  @Column(nullable = false, precision = 18, scale = 2)
  private BigDecimal balance = BigDecimal.ZERO;

  @Version
  @Column(nullable = false)
  private long version;

  @Column(name = "updated_at", columnDefinition = "timestamptz", nullable = false)
  private Instant updatedAt = Instant.now();

}

Service

@Transactional(isolation= Isolation.READ_COMMITTED, rollbackFor = Exception.class)
@Override
public TransactionResponse execute(UUID id, TransactionType type, BigDecimal amt, String reason) {
    var account = accountRepo.findById(id).orElseThrow();
    if (TransactionType.DEPOSIT.equals(type)) 
      account.deposit(amt); 
    else 
      account.withdraw(amt);
    var ledgerEntry = ledgerRepo.save(LedgerEntry.of(account, type, amt, reason));
    return TransactionResponse.success(account, ledgerEntry);
}

Pessimistic Locking Strategy

The @Lock(LockModeType.PESSIMISTIC_WRITE) annotation enforces pessimistic locking by issuing a database-level SELECT ... FOR UPDATE query. This explicitly locks the selected Account row until the current transaction completes.

public interface AccountRepository extends JpaRepository<Account, UUID> {

  // Pessimistic row lock (SELECT ... FOR UPDATE)
  @Lock(LockModeType.PESSIMISTIC_WRITE)
  @Query("select a from Account a where a.id = :id")
  Optional<Account> findForUpdate(@Param("id") UUID id);

}

Step 6: Access the App

Get the Ingress IP:

kubectl get ingress -n demo

Example output:

NAME             CLASS   HOSTS            ADDRESS          PORTS   AGE
webapp-ingress   nginx   app.demo.local   192.168.56.240   80      21h

Then test endpoints from your host:

curl -H "Host: app.demo.local"  http://192.168.56.240/actuator/health

Step 7: Testing with k6

In this step, the environment is fully prepared to simulate concurrent bank account transactions and observe how the locking mechanisms behave under load.

You can use the built-in k6 test script to run the simulation. For each test run, you’ll choose an account and a specific locking strategy (for example, optimistic or pessimistic locking). The script then launches 50 virtual users (VUs) running concurrently, using the shared-iterations executor — a total of 100 iterations distributed across all VUs. This setup effectively mimics concurrent access to the same account, allowing you to verify how data integrity is preserved during simultaneous transactions.

To get started, install k6 on your host system:

# Debian/Ubuntu
sudo gpg -k
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update
sudo apt-get install k6
# Windows
choco install k6

Run the load test:

# host
k6 run -e BASE_URL=`http://192.168.56.240` -e ACCOUNT_ID=`3f93c1c2-1c52-4df5-8c6a-9b0c6d7c5c11` -e MODE=`OPTIMISTIC` test/k6-load-test.js

You will get the output like this:


         /\      Grafana   /‾‾/  
    /\  /  \     |\  __   /  /   
   /  \/    \    | |/ /  /   ‾‾\ 
  /          \   |   (  |  (‾)  |
 / __________ \  |_|\_\  \_____/ 

     execution: local
        script: k6-load-test.js
        output: -

     scenarios: (100.00%) 1 scenario, 50 max VUs, 2m30s max duration (incl. graceful stop):
              * concurrent_load: 100 iterations shared among 50 VUs (maxDuration: 2m0s, gracefulStop: 30s)

INFO[0000] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  source=console
INFO[0000] ▶ K6 Load Test for AccountController          source=console
INFO[0000] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  source=console
INFO[0000] ▶ Target base URL : http://192.168.56.240:8080     source=console
INFO[0000] ▶ Account ID      : 3f93c1c2-1c52-4df5-8c6a-9b0c6d7c5c11  source=console
INFO[0000] ▶ Locking Mode    : OPTIMISTIC                source=console
INFO[0000] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  source=console
INFO[0000]                                               source=console
INFO[0000] 📊 Initial Account State:                      source=console
INFO[0000]    Balance: 10 USD | Version: 494 | Owner: Alice  source=console
INFO[0000] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  source=console
INFO[0000]                                               source=console
ERRO[0000] [ERROR 422] WITHDRAWAL 60 failed: Business rule violation - insufficient funds  source=console
INFO[0000] [2025-11-02T13:20:44.705Z] TX:a52d4700-e63a-463c-808e-66bf4132ba26 | DEPOSIT 24 USD | Balance: 34 USD (v494)  source=console                                                                                                                                               
INFO[0000] [2025-11-02T13:20:44.709Z] TX:8d001c33-c64a-44eb-b39f-93ed854c6e02 | DEPOSIT 46 USD | Balance: 159 USD (v496)  source=console
INFO[0000] [2025-11-02T13:20:44.709Z] TX:ba848747-f5ea-4ebf-b1b2-3c501c44cd2b | DEPOSIT 79 USD | Balance: 113 USD (v495)  source=console                                                                                                                                              
INFO[0000] [2025-11-02T13:20:44.723Z] TX:87d8874d-af6c-47a5-9901-823f616e8add | DEPOSIT 10 USD | Balance: 169 USD (v497)  source=console
...
skip
...
INFO[0004] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  
INFO[0004] 📊 Final Account State:                        source=console
INFO[0004]    Balance: 501 USD | Version: 582 | Owner: Alice                                                                                                                                                                                                              
INFO[0004] 📈 Changes:                                    source=console                                                                                                                                                                                                              
INFO[0004]    Balance Change: +491 USD                   source=console                                                                                                                                                                                                               
INFO[0004]    Version Change: +88                        
INFO[0004] ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  source=console                                                                                                                                                                                         
INFO[0004] ✅ Test completed successfully!                source=console                                                                                                                                                                                                              


  █ THRESHOLDS

    http_req_duration
    ✓ 'p(95)<2000' p(95)=1.02s

    http_req_failed
    ✗ 'rate<0.1' rate=11.76%

    version_conflicts
    ✓ 'rate<0.3' rate=0.00%


  █ TOTAL RESULTS

    checks_total.......: 300    74.544329/s
    checks_succeeded...: 88.00% 264 out of 300
    checks_failed......: 12.00% 36 out of 300

    ✗ status is 200
      ↳  88% — ✓ 88 / ✗ 12
    ✗ response has account data
      ↳  88% — ✓ 88 / ✗ 12
    ✗ response has transaction data
      ↳  88% — ✓ 88 / ✗ 12

    CUSTOM
    account_balance................: avg=499.056818 min=30       med=506     max=983   p(90)=767.9    p(95)=815.95
    deposits_total.................: 48     11.927093/s
    other_errors...................: 12     2.981773/s
    version_conflicts..............: 0.00%  0 out of 0
    withdraws_total................: 52     12.921017/s

    HTTP
    http_req_duration..............: avg=308.79ms   min=10.07ms  med=91.88ms max=2.11s p(90)=900.17ms p(95)=1.02s
      { expected_response:true }...: avg=332.89ms   min=10.07ms  med=99.69ms max=2.11s p(90)=932.24ms p(95)=1.24s
    http_req_failed................: 11.76% 12 out of 102
    http_reqs......................: 102    25.345072/s

    EXECUTION
    iteration_duration.............: avg=1.5s       min=578.54ms med=1.48s   max=3.79s p(90)=2.12s    p(95)=2.51s
    iterations.....................: 100    24.84811/s
    vus............................: 1      min=1         max=50
    vus_max........................: 50     min=50        max=50

    NETWORK
    data_received..................: 94 kB  23 kB/s
    data_sent......................: 28 kB  6.9 kB/s

running (0m04.0s), 00/50 VUs, 100 complete and 0 interrupted iterations                                                                                                                                                                                                               
concurrent_load ✓ [======================================] 50 VUs  0m04.0s/2m0s  100/100 shared iters

Step 8: Monitoring Kubernetes

Using k9s

k9s is a terminal-based UI for Kubernetes. Instead of typing dozens of kubectl commands, you get a fast, interactive dashboard right inside your terminal — perfect for developers, DevOps engineers, and operators who live in the CLI.

k9s

Useful views:

:node - View all nodes
:pod - View all pods
:deployment - View deployments
:service - View services
:ingress - View ingress rules
:pv - View persistent volumes
:pvc - View persistent volume claims
:event - View cluster events

Step 9: Cleanup

Stop all VMs but keep state:

vagrant halt

Destroy everything (full reset):

vagrant destroy -f

Key Takeaways

Infrastructure as Code made simple — Spin up a complete multi-node Kubernetes cluster with one vagrant up. No cloud required.
Realistic local lab — Simulate a production-like environment with control plane, workers, networking, storage, and ingress — all from your laptop.
Application + Infrastructure synergy — Deploy a real Spring Boot + PostgreSQL system to understand how app logic and cluster behavior interact under load.
Data consistency in action — Experiment hands-on with JPA’s optimistic and pessimistic locking strategies to see how concurrency control works in practice.
Performance validation — Use k6 to generate concurrent transactions and validate system reliability through real metrics and stress tests.
Full observability from the CLI — With k9s, monitor nodes, pods, and resources interactively — no GUI required.
Reproducibility and cleanup — Destroy and rebuild your environment anytime with vagrant destroy -f, ensuring consistent test conditions for every run.

Conclusion

We’ve built more than just a demo — we’ve created a fully automated multi-node Kubernetes lab that runs a real Spring Boot + PostgreSQL banking system with live networking, storage, and load testing. From Vagrant provisioning to JPA locking strategies and k6 concurrency simulations, every layer demonstrates how consistency and automation come together in modern systems.

This setup isn’t about production readiness — it’s about understanding. You now have a reproducible playground to experiment with distributed transactions, concurrency control, and cluster operations — all on your own machine. It’s a hands-on way to learn how reliability and scalability emerge when software, data, and infrastructure align.

Resources

🧡 “Build it. Break it. Rebuild it — that’s how real engineering insight is forged.”

— ArataX

Mastering Kafka: Concept, Architecture, and Deployment

Arata — Tue, 23 Sep 2025 13:03:09 +0000

Preface

Before diving into this deep-dive, I encourage you first to read the article “Kafka Made Simple: A Hands-On Quickstart with Docker and Spring Boot”

That piece serves as a practical gateway into the Kafka ecosystem, helping you set up a local cluster, publish your first events, and see how Kafka fits into a real Spring Boot project.

This article builds on that foundation. Instead of focusing only on the how, here we unpack the why and the what:

The concepts that make Kafka more than just a messaging system.
The architecture that ensures durability, scalability, and fault tolerance.
The design principles behind Kafka’s performance.
A systematic deep dive into partitions, logs, replication, producers, consumers, transactions, and rebalancing.
Practical deployment insights and configuration guidance.

👉 Think of this as the conceptual companion to your hands-on quickstart—helping you see the big picture, design production-ready systems, and apply Kafka confidently in real-world projects.

Outline

1. Core Design Principles
2. Partitions
3. Log
4. Key and Log Compaction
5. Replication
6. Controller
7. Producer
8. Consumer
9. Offset Tracking
10. Rebalance
11. Exactly-Once and Transactions
12. Deployment
13. Key Takeaways
14. Conclusion
Appendix: Demo Project

1. Core Design Principles

Distributed and Scalable Architecture

Kafka runs as a cluster of brokers, enabling horizontal scalability.
Topics are partitioned across brokers to support parallelism and high throughput.

Immutable, Append-Only Log

Each partition is a structured commit log with sequential message appends.
Simplifies replication, recovery, and stream processing.

Decoupled Producers and Consumers

Kafka uses a publish-subscribe model with loose coupling.
Consumers read independently without affecting producers.

Message Durability and Fault Tolerance

Messages are persisted to disk and replicated across brokers.
Leader-follower replication ensures durability during broker failures.

High Throughput and Low Latency

Kafka handles millions of messages per second with minimal latency.
Batching, compression, and efficient I/O optimize performance.

Stream-Oriented Processing

Kafka Streams and integrations (e.g., Flink, Spark) support real-time processing.
Enables event-driven architectures and stateful computations.

Consumer-Controlled Offset Management

Consumers manage their own offsets for replayability and fault recovery.
Supports exactly-once or at-least-once semantics based on configuration.

Pluggable and Extensible APIs

Kafka provides Producer, Consumer, Streams, and Connect APIs.
Kafka Connect simplifies integration with external systems like databases and Hadoop.

2. Partitions

Partitions are fundamental to Kafka’s ability to scale horizontally and maintain high availability across distributed systems.
Each topic is split into one or more partitions, which serve as independent, ordered logs.

What is a Partition?

An ordered, immutable log of records.
Each record has a unique offset (like a line number).
Ordering is guaranteed within a partition, but not across partitions.
Producers append sequentially, consumers read sequentially.

✅ Think of a partition as a “mini-log” that can be processed independently.

Partitioning Strategy

Round-robin → default if no key is provided; balances evenly.
Key-based hashing → same key always maps to the same partition; ensures per-key ordering.
Custom partitioner → user-supplied logic for specialized routing.

✅ Use a meaningful key (e.g., customer ID) for predictable ordering.

Ordering Guarantees

Records with the same key always land in the same partition.
Per-key ordering is guaranteed.
Global ordering across partitions is not provided.

⚠️ If you need total ordering, use a single partition (but this limits throughput).

Parallelism & Consumer Scaling

One consumer in a group reads from one or more partitions.
More partitions → more consumers can share the workload.
This enables Kafka to scale horizontally with consumer groups.

✅ Match partition count to expected parallelism (e.g., number of consumer instances).

Trade-offs

Adding partitions boosts throughput and enables horizontal scaling, but also increases metadata, file handles, and controller load—balance performance with operational overhead.

⚠️ Too many partitions per broker can hurt stability (common pitfall in large clusters).

Partition Reassignment & Expansion

Kafka supports rebalancing partitions across brokers for load balancing.
Adding partitions later increases capacity but may break key ordering (keys may re-hash to new partitions).

✅ Plan partition counts in advance. Increase only when unavoidable.

Summary

Partitions = scaling + ordering + parallelism.
They allow Kafka to distribute work across consumers and brokers.
The number of partitions directly impacts performance, cost, and design trade-offs.

💡 Pick partition counts carefully: balance parallelism vs overhead.

3. Log

At the core of Kafka is the log — an append-only data structure where each topic-partition maintains a sequential list of records. The log underpins durability, ordering, and replayability in Kafka.

Log Fundamentals

Append-only: Producers write new records only at the end.
Sequential reads: Consumers read messages by offset in order.
Immutability: Records are never modified once written.
Ordering: Within a partition, offsets guarantee strict ordering.
Durability: Backed by disk with efficient sequential writes and OS page cache.

✅ Simplifies recovery and replay by ensuring deterministic ordering.
⚠️ Updates or deletes are handled via compaction or tombstones, not in-place mutation.

Partition as a Folder

Each partition maps to a directory on disk (e.g., /var/lib/kafka/volumes/kafka_data/_data/order-0).

✅ Keeps partition data isolated for replication and recovery.

Inside a Partition Directory

File Name	Purpose
`*.log`	Stores Kafka records (key-value pairs).
`*.index`	Maps offsets to byte positions in the `.log` file.
`*.timeindex`	Maps timestamps to offsets for time-based lookups.
`leader-epoch-checkpoint`	Tracks leader epochs for replication consistency.
`partition.metadata`	Stores partition-level configuration or state.

Log Lifecycle

As data grows, Kafka rolls logs into segments.
Each segment has a .log, .index, and .timeindex file.
New messages go into the active segment (latest .log).
Old segments can be safely deleted or compacted based on retention rules.

Example (partition order-0):

00000000000000000000.log        → Log segment storing the actual messages
00000000000000000000.index      → Offset index for fast lookup of records
00000000000000000000.timeindex  → Timestamp index for time-based queries
leader-epoch-checkpoint         → Tracks changes in partition leadership
partition.metadata              → Metadata about the partition configuration

As more data arrives and the first segment grows beyond the configured segment size, Kafka rolls over to create new segments:

00000000000000000001.log
00000000000000000001.index
00000000000000000001.timeindex

Retention and Compaction

Kafka does not keep logs forever → policies determine retention.

Retention Policies:

Time-based: Delete records older than retention.ms.
Size-based: Delete when total log size exceeds retention.bytes.
Compaction: Retain only the latest value per key.

✅ Retention prevents unbounded disk usage.
⚠️ Aggressive retention can delete records needed for replay or lagging consumers.

Performance Considerations

Segment size and retention settings impact disk churn and log cleanup frequency.
Disk throughput and filesystem tuning (XFS recommended) directly affect performance.
Consumer lag → large replay windows may require higher retention to allow catch-up.

✅ SSDs improve latency, but sequential disk writes mean HDDs can still perform well.
⚠️ Misconfigured retention can either exhaust disk or delete needed data too quickly.

Summary

The Kafka log is:

Append-only → simple and efficient for writes.
Segmented → scalable and manageable on disk.
Retained or compacted → supports both replayability and bounded storage.

💡 Proper tuning of segment size, retention, and compaction ensures Kafka logs remain durable, performant, and aligned with application needs.

4. Key and Log Compaction

Kafka topics allow multiple messages with the same key, and Kafka provides log compaction to keep only the latest value per key. This design supports stateful stream processing, caching, and event sourcing use cases.

Keys in Kafka

Kafka does not enforce uniqueness of keys.
The key determines partition placement:
- Same key → always routed to the same partition.
- Ensures per-key ordering of events.

Common Use Cases:

Updates to the same entity (e.g., user profile changes).
Event streams per entity (e.g., customer actions).
Stateful stream processing (aggregates or reducers).
Materialized views (latest state per key).
Caching or event sourcing (replay per entity).

⚠️ Keys don’t guarantee global uniqueness — they only ensure ordering within a partition.

Log Compaction

Log compaction removes older records for a given key, retaining only the most recent value.
Enabled via cleanup.policy=compact.

✅ Benefits:

Keeps the latest value per key for stateful applications.
Reduces disk usage while preserving key-level history.

⚠️ Considerations:

Compaction is asynchronous → old versions may remain temporarily.
Offsets and order are preserved even after compaction.
Not a replacement for time/size-based retention.

Key Configurations:

cleanup.policy=compact → enable compaction.
min.cleanable.dirty.ratio → % of log dirtiness before cleaning triggers.
min.compaction.lag.ms / max.compaction.lag.ms → control delay before segments are compacted.
delete.retention.ms → how long tombstones are retained.

Tombstones

A tombstone is a message with a key and a null value.
Signals that all previous values for that key should be deleted during compaction.

Example:

{ "key": "user123", "value": null }

How Tombstones Work:

Marks the key for deletion → tells Kafka “forget this key.”
During compaction, Kafka removes earlier messages with that key.
The tombstone itself is later removed after delete.retention.ms.

✅ Enables explicit deletes in a compacted topic.
⚠️ Consumers must be designed to interpret null values correctly.

Summary

Keys define partitioning and enable ordered per-entity streams.
Log compaction ensures only the latest record per key is retained, reducing log size while preserving correctness.
Tombstones provide a mechanism for deleting keys in compacted topics.

💡 keys + compaction allow Kafka to serve as both a durable event log and a state store for real-time applications.

5. Replication

Replication in Kafka ensures resilience and fault tolerance by distributing partitions across multiple brokers. Each partition has one leader and one or more followers that maintain synchronized copies.

Leader and Followers

Leader → handles all reads and writes for the partition.
Followers → replicate the leader’s log asynchronously to stay in sync.

✅ Clients always interact with the leader, simplifying producer/consumer logic.

Replication Factor

Defines the number of copies per partition.
Common default: 3 (1 leader, 2 followers).

✅ Higher replication factor = stronger fault tolerance.
⚠️ Increases storage and network overhead.

In-Sync Replicas (ISR)

ISRs are replicas fully caught up with the leader.
Only ISRs are eligible for promotion during failover.

✅ Ensures safe and consistent recovery.
⚠️ Too many out-of-sync replicas weaken durability guarantees.

Leader Election and Failover

If the leader fails, a new one is chosen from the ISR set.
The Controller (see Section 8) coordinates this election.

✅ Enables fast recovery and high availability.

Consistency vs Latency Trade-offs

acks=all → strongest durability. Leader waits for all ISR acknowledgments.
acks=1 → leader-only acknowledgment. Faster writes, but less durable.

⚠️ More replicas = More safety, but also higher cost and latency.

Summary

Replication provides:

High availability through leader/follower design.
Durability via multiple replicas and ISRs.
Fault tolerance with automatic leader election.

💡 Balance safety and performance by adjusting replication and acknowledgments.

6. Controller

The Kafka Controller is a special broker role that manages cluster-wide metadata and coordination.

In modern KRaft mode (Kafka Raft), controllers form a quorum that replaces ZooKeeper, ensuring metadata consistency and high availability.

Metadata Management

Tracks topics, partitions, broker registrations, and configurations.
Persists updates in the internal metadata log __cluster_metadata.

✅ Ensures all brokers share a consistent view of the cluster.

Leader Election

Coordinates partition leader elections when brokers fail or join.
Relies on the ISR set maintained by replication (see Section 7).

✅ Keeps partitions highly available with minimal downtime.

Partition Assignment

Distributes partitions across brokers for load balancing.
Reassigns partitions during rebalances, broker failures, or cluster expansion.

⚠️ Frequent reassignments add overhead; prefer stable membership.

Quorum Coordination (KRaft)

Controllers form a Raft quorum:
- One acts as the active leader.
- Others are followers, replicating metadata changes.

✅ Provides fault tolerance without external ZooKeeper.

Cluster Health and Recovery

Detects broker failures and updates cluster state.
Removes failed brokers from the ISR (in coordination with replication).
Triggers leader re-election for affected partitions.

✅ Enables rapid self-healing and resilience.

Active vs. Follower Controllers

Active Controller (Leader)
- Makes cluster-wide decisions:
- Runs leader elections.
- Updates ISR lists.
- Tracks broker registrations and failures.
- Applies config changes (topics, ACLs, quotas).
- Persists changes in __cluster_metadata, replicated to followers.

👉 Functions as the “cluster brain.”

Follower Controllers
- Replicate metadata log entries from the active controller.
- Do not make independent decisions.
- Stay ready to take over if the active controller fails.

👉 Serve as “standby brains.”

Summary

The Controller is the control plane of Kafka:

Maintains metadata consistency.
Runs leader elections based on ISR information.
Coordinates partition assignment and cluster state changes.
In KRaft mode, controllers use Raft quorum replication, removing ZooKeeper.

💡 Together with Replication (7), the Controller ensures Kafka remains highly available, consistent, and fault-tolerant.

7. Producer

Producers are responsible for reliable, ordered, and efficient delivery of messages to Kafka topics. Their configuration balances durability, ordering, latency, and resource usage through several key mechanisms.

Durability and Acknowledgments (acks)

Producers control how many broker acknowledgments are required before a send is considered successful.
- acks=0 → fire-and-forget, lowest latency, no durability.
- acks=1 → leader acknowledgment only, balances latency and durability.
- acks=all → requires leader + ISR acknowledgment, strongest durability.

✅ Use acks=all for critical data.

Ordering and Retries

Kafka producers retry failed sends automatically.
Retries can break ordering if multiple requests are in flight.
Use max.in.flight.requests.per.connection=1 to strictly preserve order.
Idempotence (enable.idempotence=true) ensures retries don’t produce duplicates.

✅ Combine retries + idempotence to achieve exactly-once semantics.

Batching and Latency Trade-offs

Producers buffer messages into batches before sending.
batch.size controls max size of a batch in bytes.
linger.ms sets how long to wait before sending a partially full batch.
- Larger batches / higher linger → better throughput, higher latency.
- Smaller batches / lower linger → lower latency, reduced throughput.

✅ Tune for workload: real-time systems prefer low latency; batch pipelines prefer throughput.

Compression

Supported codecs: gzip, snappy, lz4, zstd.
Compression applies per batch, saving bandwidth and storage.
Default is none.
gzip costs higher CPU usage for compression/decompression.

✅ lz4 or zstd for good speed/ratio balance.

Resource Limits and Buffering

buffer.memory: max memory available for unsent records.
max.block.ms: how long send() will block when buffer is full.
max.request.size: prevents oversized requests.
These settings protect the producer and broker from overload.

✅ Monitor producer metrics (buffer exhaustion, errors) to detect bottlenecks.

Summary

Producer tuning is about balancing:

Durability vs. latency (acks).
Ordering vs. throughput (retries, in-flight requests).
CPU vs. I/O efficiency (compression, batching).

💡 With correct configuration, producers achieve high throughput without sacrificing reliability.

8. Consumer

Consumers are responsible for reading messages from topics, tracking their progress, and coordinating with other consumers in a group. Their configuration impacts delivery guarantees, throughput, latency, fault tolerance, and ordering.

Offset Management and Delivery Guarantees

Automatic commits (enable.auto.commit=true) → simple, but only at-least-once delivery since commits are decoupled from processing.
Manual commits (commitSync / commitAsync) → give precise control to commit only after successful processing.
For exactly-once semantics, bind offset commits to transactions or use manual synchronous commit.
auto.offset.reset determines startup behavior if no committed offset exists:
- earliest → start from the beginning (useful for replays).
- latest → only consume new records.

✅ Use manual commits or transactional commits in critical pipelines.

Partition Assignment and Rebalancing

Within one consumer group, each partition is assigned to at most one member at a time.
Multiple consumer groups can read the same partition independently.
Assignment strategies:
- Range → contiguous partition sets.
- RoundRobin → even distribution across members.
- Sticky → minimizes partition movement during rebalances.
Frequent join/leave events → trigger rebalances and pause consumption.

✅ Keep membership stable to reduce churn.

⚠️ Tune session.timeout.ms and heartbeat.interval.ms:

Higher values tolerate long GC pauses or transient work.
Lower values detect failures faster but may cause false positives.

Poll and Fetch Tuning

max.poll.records:
- Increase for higher throughput.
- Reduce to limit per-iteration processing and avoid long loops.
max.partition.fetch.bytes and fetch.max.wait.ms:
- Larger values → better for bulk processing.
- Smaller values → better for low-latency use cases.
fetch.min.bytes:
- Set higher to batch more data (throughput).
- Set to 1 for immediate returns (latency).
The poll loop must call poll() frequently:
- Long processing requires increasing max.poll.interval.ms.
- Handle rebalance callbacks to stay responsive.

✅Balance throughput vs latency depending on workload.

Summary

Consumer tuning balances:

Delivery guarantees vs. simplicity (auto vs manual commits).
Partition stability vs. flexibility (assignment and rebalance strategies).
Throughput vs. latency (poll/fetch tuning).

💡Use manual or transactional commits for critical pipelines, keep consumer group membership stable, and tune poll/fetch settings to balance throughput with latency.

9. Offset Tracking

An offset is a position marker that tells a consumer which record it has read up to in a partition, and where to resume on restart or after a failure. Kafka tracks offsets per partition, per consumer group, allowing multiple consumers to share work safely.

How Offset Tracking Works

Consumer Pull Model
- Consumers request data from partitions starting from a specific offset.
- They control whether to begin from earliest, latest, or a committed offset.
Offset Commitment
- Consumers save progress by committing offsets, either automatically or manually.
- Committed offsets are stored in Kafka’s internal topic __consumer_offsets, which is partitioned and replicated.

✅ Automatic commits are simple for at-least-once delivery.
⚠️ Manual commits are safer for critical processing, but require more application logic.

Consumer Position vs. Committed Offset

Consumer Position → the next record the consumer will read (held in memory).
Committed Offset → the last offset safely stored as a checkpoint.

[00][01][02][03][04][05][06][07][08][09][10][11]
                                      ^-- committed = 09 (resume here)
                                              ^-- position = 11 (next to read)

👉 If the consumer crashes, it restarts from the committed offset, not the in-memory position.

This means it may re-read some records but won’t skip any.

Summary

Offsets are per-partition position markers.
Kafka persists committed offsets in the __consumer_offsets topic.
The gap between position vs. committed offset provides fault tolerance, but may cause duplicates.

💡 Correct offset management is essential for delivery guarantees (at-least-once, at-most-once, exactly-once).

10. Rebalance

Rebalancing is the process where Kafka’s Group Coordinator redistributes partitions among consumers in a consumer group whenever the workload relationship changes.

When Rebalancing Happens

A new consumer joins the group (more parallelism).
An existing consumer leaves or fails (load must be reassigned).
A topic’s partitions increase (new partitions must be assigned).

How Rebalancing Works

Group Coordinator detects a change in group membership.
All consumers stop fetching temporarily.
Coordinator calculates a new partition assignment.
Each consumer receives its updated assignment.
Consumers resume reading from their assigned offsets.

💡 Minimize unnecessary group membership changes and control partition counts carefully to reduce rebalance frequency and consumer downtime.

11. Exactly Once and Transactions

Kafka’s Exactly-Once Semantics (EOS) ensures that messages are processed once and only once, even in the face of retries or failures. This combines idempotent production, transactions, and offset commits into a unified model for reliable stream processing.

Idempotent Producer

When enable.idempotence=true, the producer is assigned a Producer ID (PID) and per-partition sequence numbers.
Retries are deduplicated at the broker using these sequence numbers.

✅ Guarantees no duplicates in a single partition, even under retries.
⚠️ Does not guarantee atomicity across multiple partitions or topics by itself.

Transactional Producer

A transactional producer groups multiple writes and offset commits into a single atomic unit.
Either all messages + offset commits succeed, or none do.
Controlled via a stable transactional.id, which enables fencing (old producers with the same ID are invalidated).

✅ Provides atomic read → process → write semantics.

Transaction Coordinator

A special broker component that manages transaction state.
Persists transaction metadata in the internal topic __transaction_state.
Ensures commit/abort decisions are coordinated for each transactional.id.

⚠️ Coordinator bottlenecks can occur if too many producers use transactions with wide scope.

Consumer Isolation Levels

Consumers control visibility into transactional writes via isolation.level:
- read_uncommitted → sees all records (including aborted transactions).
- read_committed → sees only records from successfully committed transactions.

✅ Use read_committed in pipelines that require strict correctness.

Offsets in Transactions

The sendOffsetsToTransaction API binds offset commits to producer transactions.
Offsets are only committed if the producer transaction itself commits.

✅ Ensures exactly-once end-to-end semantics: messages are processed and offsets advanced atomically.

Summary

Idempotence removes duplicates per partition.
Transactions extend atomicity across topics + offsets.
Coordinators maintain transaction state.
Isolation levels let consumers choose between speed (read_uncommitted) and safety (read_committed).

💡 Enable enable.idempotence=true by default and use transactions (transactional.id + sendOffsetsToTransaction) only when strict exactly-once guarantees across topics and offsets are required.

12. Deployment

Cluster Topology and Roles

Separate controller and broker roles on dedicated nodes for production-scale clusters.
Run a controller-only quorum of 3 or 5 nodes.
- Three controllers are sufficient for moderate clusters.
- Five controllers are preferred for larger clusters or higher availability needs.
Use broker-only nodes for the data plane (producers and consumers).
Deploy at least three brokers and configure replication.factor ≥ 3 for critical topics.

Storage and Disks

Use JBOD (Just a Bunch of Disks) — no RAID. Present disks individually to brokers and let Kafka handle replication.
Prefer the XFS filesystem tuned for large files; mount broker volumes with noatime (or relatime if atime tracking is required).
Use HDDs on brokers for high sequential throughput and cost efficiency. Consider SSDs/NVMe for controller nodes (metadata logs) or if your workloads involve heavy random reads or strict latency SLAs.
Tune log.segment.bytes and retention policies to manage the number of segments and control mmap usage.

Memory, Heap, and OS Tuning

Keep broker JVM heap small and fixed (typically 4–8 GB). Leave the remaining RAM for the OS page cache.
Apply the RAM sizing rule: provision enough RAM to buffer approximately 30 seconds of peak ingest throughput in the page cache.

Example
If ingest is 300 MB/s, you want ~9 GB RAM just for cache.

Formula
Required RAM for cache ≈ (ingest throughput in MB/s) × 30 seconds

Raise vm.max_map_count for large clusters with many partitions or segments (e.g., set to 262144 or higher when required).

Formula
required_vm.max_map_count ≈ partitions_per_broker × segments_per_partition × 2

Increase file descriptor limits (ulimit -n) to at least 100k.
For networking, provision 10Gbps NICs for high-throughput clusters and tune socket buffers for cross–data center replication.

Availability, Replication, and Durability

Configure min.insync.replicas ≥ 2 when replication.factor = 3 to ensure durability even if one replica fails.
Require producers to use acks=all for critical topics to ensure writes are fully replicated before acknowledgment.
Enable rack awareness (broker.rack) so replicas are distributed across racks or availability zones for better fault tolerance.
Consider tiered storage (e.g., S3 or HDFS) for offloading cold data while keeping hot data local to brokers.

Security and Networking

Enable TLS encryption for both client–broker and inter-broker communication.
Use SASL authentication (SCRAM, mTLS, or GSSAPI depending on your environment).
Apply Kafka ACLs to enforce least-privilege access control.
Restrict broker ports to trusted networks and place brokers/controllers in private subnets.

Operations, Monitoring, and Alerting

Kafka’s monitoring flow begins with JMX exposing internal metrics, which are collected by a Prometheus exporter and visualized through Grafana dashboards for real-time tracking and alerting.

Key Metrics to Track
- Under-replicated or offline partitions
- Request latency across produce and fetch paths
- ISR size fluctuations and consumer lag
- Disk usage and I/O saturation
- GC pause duration and frequency
Critical Alerts
- Shrinking ISR or under-replicated partitions.
- Offline or missing replicas.
- Disk pressure or high utilization.
- Long GC pauses.
- Frequent rebalances.

13. Key Takeaways

Kafka is not just a queue : it’s a distributed event streaming platform for high-throughput, real-time data pipelines.
Core roles : Producers publish, Consumers subscribe, Topics organize, and Partitions enable horizontal scalability.
Immutable, ordered logs : guarantee replayable data streams and predictable processing.
Replication and ISR : leaders handle writes, followers stay synchronized to ensure fault tolerance.
KRaft replaces ZooKeeper : simplifying cluster metadata management and deployment complexity.
Performance is filesystem-driven : sequential disk I/O, OS page cache, and batching give Kafka exceptional throughput.
Exactly-once semantics (EOS) : achieved through idempotent + transactional producers combined with committed offsets.
Production readiness : comes from careful tuning: partitions, replication factor, monitoring, and security controls.

14. Conclusion

Kafka has become the backbone of modern data systems. Its distributed log architecture delivers scalability, fault tolerance, and speed—making it ideal for event-driven microservices, real-time analytics, and data pipelines.

By understanding core concepts (topics, partitions, logs, replication, controllers) and applying best practices in deployment and tuning, you can build robust, scalable, and future-proof systems powered by Kafka.

Appendix: Demo Project

To complement the concepts explored in this article, I’ve built a hands-on demo project that puts Kafka’s architecture and transactional patterns into practice.

GitHub Repository: Spring Boot Kafka Cluster

This project showcases a production-grade Kafka setup running in KRaft mode, integrated with Spring Boot and PostgreSQL. It includes:

A multi-node Kafka cluster with 3 controllers and 3 brokers
A RESTful producer service that publishes events to Kafka
Three consumer services demonstrating:
- Manual acknowledgment
- Kafka transactions
- Database transactions
A PostgreSQL-backed persistence layer
Docker Compose orchestration for easy startup
Scripts for testing, error simulation, and direct Kafka publishing

Whether you're exploring offset management, transactional guarantees, or deployment strategies, this demo gives you a practical playground to experiment with real-world Kafka patterns.

💡 Use it as a reference, a starting point, or a sandbox to deepen your Kafka mastery.

Kafka Made Simple: A Hands-On Quickstart with Docker and Spring Boot

Arata — Sat, 20 Sep 2025 07:56:42 +0000

Apache Kafka is a distributed, durable, real-time event streaming platform. It goes beyond a message queue by providing scalability, persistence, and stream processing capabilities.

In this guide, we’ll quickly spin up Kafka with Docker, explore it with CLI tools, and integrate it into a Spring Boot application.

1. What is Kafka?

Apache Kafka is a distributed, durable, real-time event streaming platform.

It was originally developed at LinkedIn and is now part of the Apache Software Foundation.

Kafka is designed for high-throughput, low-latency data pipelines, streaming analytics, and event-driven applications.

What is an Event?

An event is simply a record of something that happened in the system.

Each event usually includes:

Key → identifier (e.g., user ID, order ID).
Value → the payload (e.g., “order created with total = $50”).
Timestamp → when the event occurred.

Example event:

{
  "key": "order-123",
  "value": { "customer": "Alice", "total": 50 },
  "timestamp": "2025-09-19T10:15:00Z"
}

What is an Event Streaming Platform?

An event streaming platform is a system designed to handle continuous flows of data — or events — in real time.

Instead of working in batches (processing data after the fact), it allows applications to react as events happen.

2. What Kafka Can Do

Kafka is more than a message queue—it's a real-time event backbone for modern systems.

Messaging Like a Message Queue

Kafka decouples producers and consumers, enabling asynchronous communication between services.
Example:
A banking system publishes transaction events to Kafka. Fraud detection, ledger updates, and notification services consume these events independently.

Event Streaming

Kafka streams data in real time, allowing systems to react instantly.
Example:
An insurance platform streams claim events to trigger automated validation, underwriting checks, and customer updates in real time.

Data Integration

Kafka Connect bridges Kafka with databases, cloud storage, and analytics platforms.
Example:
A semiconductor company streams sensor data from manufacturing equipment into a data lake for predictive maintenance and yield optimization.

Log Aggregation

Kafka centralizes logs from multiple services for monitoring and analysis.
Example:
An industrial automation system sends logs from PLCs and controllers to Kafka, where they’re consumed by a monitoring dashboard for anomaly detection.

Replayable History

Kafka retains events for reprocessing or backfilling.
Example:
An insurance company replays past policy events to train a model that predicts claim risk or customer churn. This avoids relying on static snapshots and gives the model a dynamic, time-aware view of behavior.

Scalable Microservices Communication

Kafka handles high-throughput messaging across distributed services.
Example:
A financial institution uses Kafka to coordinate customer onboarding, KYC checks, and account provisioning across multiple microservices.

3. Architecture

Apache Kafka’s architecture is built for high throughput, fault tolerance, and horizontal scalability. At its core, Kafka relies on a log-based storage model and a distributed broker cluster.

Core Components

Producer → Publishes records (events/messages) to topics. Can be idempotent or transactional.
Topic → Logical category/feed for messages. Divided into partitions for parallelism.
Partition → Ordered, immutable commit log. Records have sequential offsets.
Broker → A Kafka server that stores partitions. Clusters have multiple brokers.
Consumer → Subscribes to topics and processes messages. Part of a consumer group for scaling.
Controller → Special broker role that manages metadata, leader election, and partition assignment.
Replication → Each partition has one leader and multiple followers in the ISR (in-sync replicas).

Data Flow

Producers send records to brokers.
Records are appended to the leader partition log.
Followers replicate the leader’s log for durability.
Consumers fetch records from leaders, tracking their offsets.

Architecture Diagram

               +-----------------+
               |    Producers    |
               +-----------------+
                   |    |    |
                   v    v    v
            +------------------------+
            |     Kafka Cluster      |
            |  +---------+           |
            |  | Broker 1|  <--------------- Partition 0 Leader
            |  +---------+           |
            |  | Broker 2|  <--------------- Partition 0 Follower
            |  +---------+           |
            |  | Broker 3|  <--------------- Partition 1 Leader
            |  +---------+           |
            +------------------------+
                   |    |    |
                   v    v    v
              +-------------------+
              |  Consumer Group   |
              |-------------------|
              | Consumer A → P0   |
              | Consumer B → P1   |
              +-------------------+

4. QuickStart with Docker

This configuration sets up a single-node Kafka broker using the KRaft. It’s ideal for development, testing scenarios

name: kafka
services:
  kafka:
    image: apache/kafka:4.1.0
    container_name: kafka
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: BROKER://:9092,CONTROLLER://:9093
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: BROKER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: BROKER:PLAINTEXT,CONTROLLER:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: BROKER://localhost:9092
      KAFKA_CLUSTER_ID: "kafka-1"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/data
    volumes:
      - kafka_data:/var/lib/kafka/data
    ports:
      - "9092:9092"
volumes:
  kafka_data:

How to Run

Start the Kafka container using:

docker compose up

Kafka will be available at localhost:9092 for producers and consumers, and internally at localhost:9093 for controller communication.

5. Kafka CLI

Before running Kafka commands, log into the Kafka container:

docker container exec -it localhost bash

Create Topic

Create a topic named quickstart with one partition and a replication factor of 1:

/opt/kafka/bin/kafka-topics.sh --create \
  --bootstrap-server localhost:9092 \
  --replication-factor 1 \
  --partitions 1 \
  --topic quickstart

List Topic

Check all existing topics:

/opt/kafka/bin/kafka-topics.sh --list \
  --bootstrap-server localhost:9092

Consume Message

Read messages from the order topic starting from the beginning:

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic quickstart \
  --from-beginning

Send Message

You can send messages to the quickstart topic using either direct input or a file.

Option A: Send a single message

echo 'This is Event 1' | \
/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 \
  --topic quickstart

Option B: Send multiple messages from a file

echo 'This is Event 2' > messages.txt
echo 'This is Event 3' >> messages.txt
cat messages.txt | \
/opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 \
  --topic quickstart

5. Spring Boot Integration

This configuration enables seamless integration between a Spring Boot application and an Apache Kafka broker. It defines both producer and consumer settings for message serialization, deserialization, and connection behavior.

pom.xml

<!-- spring-web -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <version>3.4.9</version>
</dependency>
<!-- kafka -->
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
    <version>3.3.9</version>
</dependency>
<!-- Lombok(optional) -->
<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.30</version>
    <optional>true</optional>
</dependency>

applicaiton.yml

spring:
  kafka:
    bootstrap-servers: localhost:9092
    template:
      default-topic: orders
    consumer:
      group-id: quickstart-group
      auto-offset-reset: latest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "dev.aratax.messaging.kafka.model"
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer

Topic Setup

@Bean
public NewTopic defaultTopic() {
    return new NewTopic("orders", 1, (short) 1);
}

Event Model

public class OrderEvent {
    private String id;
    private Status status;
    private BigDecimal totalAmount;
    private Instant createdAt = Instant.now();
    private String createdBy;

    public enum Status {
        IN_PROGRESS,
        COMPLETED,
        CANCELLED
    }
}

Producer Example

@RestController
@RequestMapping("/api")
@RequiredArgsConstructor
public class OrderEventController {

    private final KafkaTemplate<String, OrderEvent> kafkaTemplate;

    @PostMapping("/orders")
    public String create(@RequestBody OrderEvent event) {
        event.setId(UUID.randomUUID().toString());
        event.setCreatedAt(Instant.now());
        kafkaTemplate.sendDefault(event.getId(), event);
        return "Order sent to Kafka";
    }
}

Consumer Example

@Component
public class OrderEventsListener {

    @KafkaListener(topics = "orders")
    public void handle(OrderEvent event) {
        System.out.println("Received order: " + event);
    }
}

6. Demo Project

I built a demo project using Spring Boot and Kafka to demonstrate basic producer/consumer functionality.
Check it out on GitHub: springboot-kafka-quickstart

7. Key Takeaways

Kafka is more than a message queue—it's a scalable, durable event streaming platform.
Events are central to Kafka’s architecture, enabling real-time data flow across systems.
Docker makes setup easy, allowing you to spin up Kafka locally for development and testing.
Kafka CLI tools help you explore topics, produce messages, and consume events interactively.
Spring Boot integration simplifies Kafka usage with built-in support for producers and consumers.
Real-world use cases span industries like banking, insurance, semiconductor, and automation.

8. Conclusion

Apache Kafka empowers developers to build reactive, event-driven systems with ease. Whether you're streaming financial transactions, processing insurance claims, or monitoring factory equipment, Kafka provides the backbone for scalable, real-time communication.

With Docker and Spring Boot, you can get started in minutes—no complex setup required. This quickstart gives you everything you need to explore Kafka hands-on and begin building production-grade event pipelines.

Ready to go deeper? Try explore its design/implementation, stream processing, or Kafka Connect integrations next.

Redis Sentinel Made Simple: Hands-On High Availability

Arata — Sun, 24 Aug 2025 11:56:02 +0000

High availability is no longer a luxury — it’s a survival kit for modern applications. Databases crash, servers die, containers get killed (sometimes by accident, sometimes by design). In the world of Redis, Sentinel is the quiet guardian that keeps your cache cluster alive when chaos happens.

In this article, I’ll walk you through Redis Sentinel step by step, with a runnable Docker demo and a Spring Boot integration example. By the end, you’ll see failover happening live — and how your application can recover without manual intervention.

1. Introduction

Why does Redis Sentinel matter?

Picture this: you’ve got Redis set up with one master and a couple of replicas. Everything’s smooth… until the master suddenly crashes. Now what? Who decides which replica should take over? Who makes sure your clients know where to connect?
👉 That’s exactly the job Sentinel handles for you.

Monitors your Redis instances.
Notifies you when something goes wrong.
Automatically promotes a replica to master.
Redirects clients to the new master.

Sentinel is the difference between a cache outage and a smooth failover.

2. What is Redis Sentinel?

At its core, Redis Sentinel is a distributed system that provides:

Monitoring – constantly checking whether your master and replicas are alive.
Notification – alerting operators (or systems) when something goes wrong.
Automatic Failover – promoting a replica when the master is unavailable.
Client Redirection – letting apps connect to the new master automatically.

3 . Sentinel Architecture

A Sentinel deployment usually includes multiple Sentinel nodes plus your Redis master and replicas. Sentinels work together, reaching quorum before deciding a master is truly dead.

Key concepts:

SDOWN (Subjectively Down): One Sentinel thinks the master is down.
ODOWN (Objectively Down): Enough Sentinels agree the master is down.
Replica Priority: Determines which replica should be promoted first.

Deployment Diagram

+-------------------+       +-------------------+
|   Sentinel #1     |       |   Sentinel #2     |
+-------------------+       +-------------------+
           \                     /
            \                   /
             \   Quorum Vote   /
              \               /
            +-------------------+
            |   Sentinel #3     |
            +-------------------+
                   |
                   v
            +-------------------+
            | Redis Master      |
            +-------------------+
              /          \
             v            v
   +----------------+   +----------------+
   | Redis Replica1 |   | Redis Replica2 |
   +----------------+   +----------------+

4. Setting Up Redis Sentinel

We use Docker Compose with one master, two replicas, and three Sentinels.

Redis Sentinel Config

sentinel announce-ip "127.0.0.1"
sentinel announce-port 26379
# sentinel with version above 6.2 can resolve host names, but this is not enabled by default.
sentinel resolve-hostnames yes
# Monitor master named "mymaster" at 127.0.0.1(or domain name):6379 with quorum of 2
sentinel monitor mymaster 127.0.0.1 6379 2
# Master is considered down after 5 seconds of no response
sentinel down-after-milliseconds mymaster 5000
# Failover timeout 18 seconds
sentinel failover-timeout mymaster 18000

##Below line 'Generated by CONFIG REWRITE 'controlled by Redis Sentinel(Config file should be writable)
# Generated by CONFIG REWRITE

Ways to Run Sentinel:

redis-sentinel /etc/redis/sentinel.conf
# or
redis-server /etc/redis/sentinel.conf --sentinel

Redis CLI Useful commands:

#Start Sentinel's monitoring.
SENTINEL MONITOR <master name> 
#Stop Sentinel's monitoring.
SENTINEL REMOVE <master name>
#Set Sentinel's monitoring configuration. 
SENTINEL SET <master name> <option> <value>
#(>= 5.0) Show a list of replicas for this master, and their state.
SENTINEL REPLICAS <master name> 
#Show a list of sentinel instances for this master, and their state.
SENTINEL SENTINELS <master name>
#Force a failover as if the master was not reachable, and without asking for agreement to other Sentinels 
#(however a new version of the configuration will be published so that the other Sentinels will update their configurations.
#That's called 'Configuration propagation'
SENTINEL FAILOVER <master name>
#Display information by Role.
INFO

Docker Compse :

  redis-sentinel-1:
    image: bitnami/redis-sentinel:8.0.3
    container_name: redis-sentinel-1
    ports:
      # Sentinel, Docker, NAT, and possible issues. Set port-mapping 1:1
      - "26379:26379"
    environment:
      - ALLOW_EMPTY_PASSWORD=yes   
    volumes:
      # Use with caution regarding permissions.
      - redis-sentinel-1-data:/bitnami/redis-sentinel
      - ./redis-sentinel-1:/usr/local/etc/redis-sentinel
    # Sentinel, Docker, NAT, and possible issues. Use host for maximum compatibility.
    network_mode: host
    depends_on:
      - redis-master
      - redis-replica-1
      - redis-replica-2
    restart: unless-stopped
    command: ["redis-sentinel", "/usr/local/etc/redis-sentinel/sentinel.conf"]

5. Redis Docker Demo

Clone the demo project:

git clone https://github.com/arata-x/redis-ha.git

Docker Setup/Run

cd redis-ha/docker/redis/sentinel
docker-compose up

Simulate master crash:

docker kill redis-master

The Sentinels will detect the failure and promote a replica to do the Failover.

6. Spring Boot Integration

Spring Boot supports Sentinel natively via spring-boot-starter-data-redis. Here’s how to configure it.

pom.xml

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-redis-reactive</artifactId>
</dependency>

application.yml

spring:
  data:
    redis:
      sentinel:
        master: localhost
        nodes:
          - redis-sentinel-1:26379
          - redis-sentinel-2:26379
          - redis-sentinel-3:26379

Spring Boot Config for Pub/Sub messages（Optional）

  @Bean(destroyMethod = "shutdown")
  public RedisClient sentinelClient() {
    return RedisClient.create("redis://127.0.0.1:26379");
  }

  @Bean(destroyMethod = "close")
  public StatefulRedisPubSubConnection<String, String> sentinelPubSub(RedisClient client) {
    var conn = client.connectPubSub();
    conn.addListener(new RedisPubSubAdapter<>() {
      @Override public void message(String channel, String message) {
        log.info("Sentinel event [{}] {}", channel, message);
      }
    });

    // subscribe to key Sentinel events (or use psubscribe("*") to get all)
    conn.sync().subscribe(
        "+switch-master",        // master changed
        "+sdown", "-sdown",      // subjective down / cleared
        "+odown", "-odown",      // objective down / cleared (masters only)
        "+try-failover",
        "+failover-state-*"
    );
    return conn;
  }

This way, clients automatically reconnect after failover. And log Sentinel events.

7. Testing Failover & Logs

Failover Timeline

t0: Master alive
t1: Master killed  -> SDOWN
t2: Quorum reached -> ODOWN
t3: Leader elected -> VOTE
t4: Master elected -> PROMOTE
t5: New master active -> CLIENTS REDIRECT
t6: Replica detcted -> SLAVE
t7: Old master back -> SLAVE

Docker logs:

redis-sentinel-1  | 1:X 24 Aug 2025 01:29:56.652 * Sentinel ID is 45f2090cc345fd2a0a9afad89d45d3c212816390
redis-sentinel-3  | 1:X 24 Aug 2025 01:29:56.670 * Sentinel ID is 72098a7942ff006106511dbb0db3044b00fa5473
redis-sentinel-2  | 1:X 24 Aug 2025 01:29:56.690 * Sentinel ID is b87c2be6edf6192e03783f1ed1647af7fa2b51f6
# Simulate the master down via command 'docker container kill redis-master' and the Failover will start.
redis-sentinel-1  | 1:X 24 Aug 2025 01:30:32.047 # +sdown master mymaster redis-master 6379
redis-sentinel-2  | 1:X 24 Aug 2025 01:30:32.067 # +sdown master mymaster redis-master 6379
redis-sentinel-3  | 1:X 24 Aug 2025 01:30:32.107 # +sdown master mymaster redis-master 6379
redis-sentinel-2  | 1:X 24 Aug 2025 01:30:32.144 # +odown master mymaster redis-master 6379 #quorum 2/2
redis-sentinel-2  | 1:X 24 Aug 2025 01:30:32.144 # +try-failover master mymaster redis-master 6379
redis-sentinel-2  | 1:X 24 Aug 2025 01:30:32.151 # +vote-for-leader b87c2be6edf6192e03783f1ed1647af7fa2b51f6 1
redis-sentinel-3  | 1:X 24 Aug 2025 01:30:32.166 # +vote-for-leader b87c2be6edf6192e03783f1ed1647af7fa2b51f6 1
redis-sentinel-1  | 1:X 24 Aug 2025 01:30:32.167 # +vote-for-leader b87c2be6edf6192e03783f1ed1647af7fa2b51f6 1
redis-sentinel-2  | 1:X 24 Aug 2025 01:30:33.215 # +promoted-slave slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster redis-master 6379
redis-sentinel-2  | 1:X 24 Aug 2025 01:30:32.244 # +elected-leader master mymaster redis-master 6379
redis-sentinel-2  | 1:X 24 Aug 2025 01:30:32.244 # +failover-state-select-slave master mymaster redis-master 6379
redis-sentinel-2  | 1:X 24 Aug 2025 01:30:32.299 # +selected-slave slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster redis-master 6379
redis-sentinel-2  | 1:X 24 Aug 2025 01:30:32.299 * +failover-state-send-slaveof-noone slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster redis-master 6379
redis-sentinel-3  | 1:X 24 Aug 2025 01:30:33.263 # +switch-master mymaster redis-master 6379 127.0.0.1 6381
redis-sentinel-3  | 1:X 24 Aug 2025 01:30:33.264 * +slave slave redis-master:6379 redis-master 6379 @ mymaster 127.0.0.1 6381 
# Restore master via command 'docker container start redis-master' and master will be the replica.
redis-sentinel-2  | 1:X 24 Aug 2025 01:30:34.096 # -sdown slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6380
redis-master      | 1:S 24 Aug 2025 01:30:34.236 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
redis-master      | 1:S 24 Aug 2025 01:30:34.236 * Connecting to MASTER 127.0.0.1:6380
redis-sentinel-1  | 1:X 24 Aug 2025 01:30:34.236 * +convert-to-slave slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6380
redis-replica-1   | 1:M 24 Aug 2025 01:30:34.447 * Synchronization with replica 127.0.0.1:6379 succeeded
redis-master      | 1:S 24 Aug 2025 01:30:34.447 * MASTER <-> REPLICA sync: Successfully streamed replication buffer into the db (0 bytes in total)

Redis Event List

+slave -- A new replica was detected and attached.
+sdown -- The specified instance is now in Subjectively Down state.
+odown -- The specified instance is now in Objectively Down state.
+try-failover -- New failover in progress, waiting to be elected by the majority.
+elected-leader -- Won the election for the specified epoch, can do the failover.
+failover-state-select-slave -- New failover state is select-slave: we are trying to find a suitable replica for promotion.

Spring Boot log by Redis pub/sub:

2025-08-24T01:34:46.946+08:00  INFO 44256 --- [redis-reactive-demo] [ioEventLoop-7-1] d.a.redis.config.RedisConfigSentinel     : Sentinel event [+sdown] master mymaster 127.0.0.1 6379
2025-08-24T01:34:48.055+08:00  INFO 44256 --- [redis-reactive-demo] [ioEventLoop-7-1] d.a.redis.config.RedisConfigSentinel     : Sentinel event [+odown] master mymaster 127.0.0.1 6379 #quorum 3/2
2025-08-24T01:34:48.176+08:00  INFO 44256 --- [redis-reactive-demo] [ioEventLoop-7-1] d.a.redis.config.RedisConfigSentinel     : Sentinel event [+switch-master] mymaster 127.0.0.1 6379 127.0.0.1 6381

8. Best Practices

Run at least 3 Sentinels.
Distribute Sentinels across nodes for resilience.
Tune failover-timeout and down-after-milliseconds.

10. Final Thoughts

🚦 Think of Redis Sentinel as your system’s insurance policy.
Most of the time, you’ll never notice it quietly standing guard in the background. But the moment your master node takes a dive, Sentinel steps in to keep traffic flowing — and you’ll be very glad it was there all along.

👉 Use Sentinel when you want simple, lightweight high availability.
It doesn’t complicate your setup and gets the job done for most HA needs.

⚡ But if your workload demands both horizontal scaling (sharding) and HA, that’s where Redis Cluster shines. Sentinel won’t replace Cluster — they solve different problems.

🔗Demo project: Redis Sentinel

Redis Replication Made Simple: With Spring Boot Integration

Arata — Sat, 09 Aug 2025 15:54:45 +0000

Imagine it’s 3 AM. Your Redis server—yes, the one holding all your app’s session data—just crashed. Your team’s phones are buzzing. Users are locked out, and panic is rising.

What if I told you this nightmare could be avoided with a simple feature built right into Redis? Enter Redis replication—your built-in safeguard for data availability, read scaling, and peace of mind.

🔑 What Is Redis Replication?

At its core, Redis replication enables a single Redis instance (the primary) to automatically copy its data to one or more replicas.

The primary handles all write operations.
Replicas stay in sync and serve read requests, reducing the load on the primary.
If the primary fails, replicas can quickly take over.

This fundamental setup lays the groundwork for high availability and scaling in Redis environments.

⚙️ How Does It Work?

Redis replication works in three key stages:

Initial Sync: A replica requests a full snapshot (RDB) from the primary, loads it, and applies any updates.
Command Streaming: Once synced, the replica continuously receives write commands from the primary to stay current.
Partial Resync (PSYNC2): If a replica temporarily disconnects, it resumes from where it left off using Redis’s backlog buffer—avoiding a full resync.

This process is asynchronous, which means replicas may lag slightly but offer high throughput.

🖥 Setting Up Replication (Primary + Two Replicas)

Here’s how to launch a simple master-replica setup locally:

# Start Primary
redis-server --port 6379

# Start Replica 1
redis-server --port 6380 --replicaof 127.0.0.1 6379

# Start Replica 2
redis-server --port 6381 --replicaof 127.0.0.1 6379

At this point, reads can be routed to replicas while writes continue to flow to the primary.

🆕 Chained Replication (Replica of a Replica)

Beyond basic replication, Redis supports chained replication, where a replica can act as a source for another replica.

Why Use It?

Reduce primary load: Only one replica pulls directly from the primary.
Regional optimization: Place replicas closer to users while syncing through a nearer node.
Better bandwidth usage: Ideal for distributed or high-latency networks.

Example:

# Primary
redis-server --port 6379

# Replica 1 (syncs from primary)
redis-server --port 6380 --replicaof 127.0.0.1 6379

# Replica 2 (syncs from Replica 1)
redis-server --port 6381 --replicaof 127.0.0.1 6380

✒Deployment Diagram

This diagram shows a primary node with two direct replicas and one chained replica.

          +--------+
          | Server |
          +---+----+
              |
             WRITE
              v
          +--------+
          | Master |
          +---+----+
          /        \
     SYNC/          \SYNC
        v            v
+-------+--+    +----+------+
| Replica  |    |  Replica  |
+----+-----+    +-----+-----+
     |
 CHAINED SYNC
     v
+----+-----+
| Replica  |
+----------+

⚡ Diskless Replication

To further speed up initial synchronization, enable diskless replication, which streams snapshots directly to replicas:

redis.conf (master)

repl-diskless-sync yes
repl-diskless-sync-delay 5

redis.conf (replica)

replicaof redis-master 6379
replica-read-only yes
repl-diskless-load on-empty-db

This avoids writing intermediate files to disk and is ideal for large datasets or high-performance environments.

🔧 Spring Boot with Redis Replicas

Let’s integrate this into a Spring Boot project for practical use.

Dependency:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

Configuration:

@Bean
public LettuceConnectionFactory redisConnectionFactory() {
    RedisStaticMasterReplicaConfiguration masterReplicaConfig =
            new RedisStaticMasterReplicaConfiguration("127.0.0.1", 6379);
    masterReplicaConfig.addNode("127.0.0.1", 6380);
    masterReplicaConfig.addNode("127.0.0.1", 6381);
    masterReplicaConfig.setPassword(RedisPassword.of("myRedisPass"));

    LettuceClientConfiguration clientConfig = LettuceClientConfiguration.builder()
            .readFrom(ReadFrom.ANY_REPLICA)
            .build();

    return new LettuceConnectionFactory(masterReplicaConfig, clientConfig);
}

This configuration connects Spring Boot to a primary and its replicas, preferring replica reads automatically.

🛠 Verifying Reads Are Hitting Replicas

To confirm that reads hit replicas rather than the primary:

1️⃣ Monitor Replica Activity

redis-cli -p 6380 MONITOR

Execute a read query and see it logged on the replica.

2️⃣ In Spring Boot

@Autowired
private RedisTemplate<String, Object> redisTemplate;

@Bean
public CommandLineRunner testRedis() {
    return args -> {
        ValueOperations<String, Object> ops = redisTemplate.opsForValue();
        ops.set("user:1", "Alice"); // Write -> Master
        String value = ops.get("user:1"); // Read -> Replica
        System.out.println("Read value (replica preferred): " + value);
    };
}

⚠️ Limitations of Replicas Alone

However, while replication improves resilience, it doesn’t guarantee full HA on its own.

No automatic failover: Promotion must be done manually without Sentinel or Cluster.
Asynchronous replication: Recent writes might be lost if the primary fails before syncing.
Single control point: The primary remains the bottleneck for writes.

These gaps highlight why replication is essential but insufficient for full HA in production environments.

✅ Final Thoughts

Redis replication is a simple yet powerful way to protect against single points of failure, scale reads, and prepare for failover. Its nature—one primary continuously mirrored by one or more replicas—ensures that your data is redundant, accessible, and performance-optimized.

Why use replicas?

Keep a live backup ready for emergencies.
Reduce read load on the primary.
Improve latency with geographically placed replicas.

Key takeaway: Replication is your first step toward high availability. Pair it with Sentinel for automatic failover or Cluster for sharding to achieve a production-grade, fault-tolerant Redis deployment.

🛠 Demo Project for Readers

I have created a demo project that showcases practical usage of Redis replica.

Project Includes:

Docker: Pre-configured Redis master & replicas using docker-compose.
Spring Boot: Example backend service demonstrating Redis read/write splitting.

🔗 Access the Project

You can clone or explore the project from my repository :

git clone https://github.com/arata-x/redis-ha.git
cd redis-ha