Forem: Varun Patil

A Developer’s Guide to Apache Kafka: From Basics to Architecture in One Read

Varun Patil — Sat, 22 Nov 2025 17:26:04 +0000

In today’s world, applications are no longer simple systems with a single database and a few users. Modern platforms like Uber, Netflix, Zomato, Amazon, Instagram, and even banking apps generate millions of events every second—a ride request, a payment update, a login attempt, a notification, a cart update, a video play, and so on.

Handling this constant flow of data in real time is no longer a luxury—it’s a necessity.
And traditional systems struggle badly with this.

They are:

slow
tightly coupled
difficult to scale
easily break under heavy load.

To solve these modern data challenges, companies use Apache Kafka, a distributed event streaming platform designed to handle massive volumes of real-time data with high speed, fault tolerance, and scalability.

This blog will walk you through Kafka in the simplest possible way—from beginner concepts to intermediate architecture—using relatable examples, clear explanations, and real-life use cases.
Whether you’ve never touched Kafka or want to strengthen your basics before learning about microservices, this guide will provide you with a solid foundation.

Why Do We Need Apache Kafka? (The Problem Before the Solution)

Before understanding what Kafka is, it’s important to understand why it was created.
Modern systems generate enormous amounts of data every second, yet traditional communication patterns were never designed to handle this scale or speed.

Let’s break this down in simple terms.

2.1 Traditional Service Communication Was Broken

Most old systems communicated through direct API calls.

Service A → Service B → Service C

This approach creates several problems:

Tight Coupling:
If Service B goes down, Service A also fails.
Complex Dependencies:
You add one new service, and every old service must be updated.
High Latency:
Each service waits for the previous one to respond.
Difficult to Scale:
At high traffic, these direct calls easily collapse.

2.2 Real-Time Data Processing Was Hard

Traditional systems used batch processing—data is collected first, then processed after a few minutes or hours.

But today, apps require instant updates:

Uber must show the driver’s live location
Zomato must show live order status
Netflix must track what you’re watching right now
Banks must detect fraud in milliseconds
Batch processing is too slow for this world.

2.3 Databases Alone Could Not Handle Event Streams

Developers tried using databases as messaging systems:

Write event in DB
Another service reads it
Marks it as “processed”
This fails because:

Databases cannot handle millions of writes per second

Polling the DB constantly is expensive
No proper real-time streaming
Hard to replay events
No distributed scalability
Databases were never meant for continuous event flow.

2.4 Scaling Microservices Was a Nightmare

When companies switched to microservices, a new problem came:

Every service had to communicate with 5–10 other services.

Example:

Order Service → Payment Service → Delivery Partner Service → Notification Service → Live Tracking Service → Analytics Service

If any one service goes down, everything breaks.

Adding even a single new service (e.g., fraud detection) means:

Updating 5–10 other services
Changing code everywhere
More API calls = More delays

Microservices became too dependent on each other.

3. What Is Apache Kafka? (Simple Definition + Analogy)

Apache Kafka is a distributed event streaming platform designed to handle huge amounts of real-time data efficiently and reliably.

If that sounds complex, here’s the simplest explanation:

Kafka is like a high-speed delivery system that collects, stores, and distributes data (events) between different systems in real time.

Kafka acts as a middle layer between producers (systems that generate data) and consumers (systems that use the data).

No direct communication.
No dependency.
No delays.

3.1 A Simple Analogy (Easy to Remember)

Imagine a YouTube Channel.

Topic = The channel
Partition = Playlists inside the channel
Producers = People uploading videos
Consumers = Subscribers watching videos
Offset = The position of each video in the playlist
Broker = YouTube server

YouTube doesn't send videos directly to you.
You pull them whenever you want.

Kafka works the same way.

3.2 Real Definition (Technical But Clear)

Apache Kafka is a distributed publish-subscribe messaging system designed for high-throughput, fault-tolerant, real-time event streaming.

Let’s decode that:

Distributed → runs on multiple servers (brokers)
Publish-subscribe → producers publish, consumers subscribe
High-throughput → handles millions of events per second
Fault-tolerant → even if servers fail, data is safe
Real-time → consumers get events instantly
Event streaming → continuous flow of data

4. Kafka Architecture: Understanding the Structure of Kafka

To understand how Kafka actually works, you need to know its internal structure. Kafka is built from a few simple but powerful components. Once you understand these blocks, the whole system becomes easy.

4.1 High-Level Kafka Structure

Here is the simplest breakdown of Kafka’s structure:

Producers → Topics → Partitions → Brokers → Consumers → Consumer Groups

4.2 Core Components of Kafka Architecture

1️⃣ Topics
A topic is a category or channel where data is stored.

Example topics:

orders
payments
user-logins

Kafka topics are append-only logs—events are added at the end.

2️⃣ Partitions
Each topic is split into multiple partitions.

Why?

To increase speed
To process data in parallel
To scale horizontally

Each partition stores messages in order, like this:

**Partition 0:** [msg1, msg2, msg3...]
**Partition 1:** [msg4, msg5, msg6...]
**Partition 2:** [msg7, msg8...]

More partitions = higher throughput (millions of messages/second).

3️⃣ Brokers
A broker is a single Kafka server.

Kafka clusters contain multiple brokers:

Broker 1
Broker 2
Broker 3

Each broker:

Stores data
Manages partitions
Serves consumer requests
Ensures high availability

If one broker fails, Kafka still works because of replication.

4️⃣ Replication
Kafka replicates partitions across brokers.

Example:

Partition 0 → Leader on Broker 1
Replica copies on Broker 2 and Broker 3

If Broker 1 fails, Broker 2 becomes leader.

This makes Kafka fault-tolerant.

5️⃣ Producers
Producers send (publish) messages to topics.

They can:

choose which partition to write to
send millions of messages per second
handle failures using retries and acks

6️⃣ Consumers
Consumers read messages from topics.

They read data in sequence, based on offsets.

Consumers do NOT delete messages. Kafka keeps them until retention time is over.

7️⃣ Consumer Groups
A consumer group is a set of consumers reading from the same topic.

Kafka ensures:

one partition → only one consumer of the group
perfect load balancing
parallel processing

Example:

Topic: orders (3 partitions)
Consumer Group:
  Consumer 1 → Partition 0
  Consumer 2 → Partition 1
  Consumer 3 → Partition 2

8️⃣ Zookeeper / KRaft (Metadata Manager)
Kafka uses:

Zookeeper (older versions)
KRaft (new versions — built-in Kafka controller)

It manages:

Broker information
Leader elections
Cluster metadata

Modern Kafka uses KRaft only, making setup simpler.

4.3 Kafka Architecture Diagram

Conclusion: The Foundation You Need Before Diving Deeper

Apache Kafka has become the backbone of modern real-time systems—from ride-hailing platforms and food delivery apps to banking, e-commerce, IoT, and streaming services. Understanding how Kafka works at a conceptual level—its topics, partitions, brokers, replication, producers, consumers, and consumer groups—gives you the foundation needed to navigate event-driven architectures confidently.

By now, you should have a clear picture of:
Why traditional systems struggled
What problem Kafka solves
Kafka’s internal structure and architecture
How data flows through Kafka

Why Kafka is the preferred choice for scalable microservices

This knowledge sets the stage for the practical side of Kafka, where the real magic begins.

Understanding and Verifying JWT Authentication in Spring Boot

Varun Patil — Sat, 04 Oct 2025 20:11:17 +0000

What Is JWT (JSON Web Token)?

JWT (JSON Web Token) is a compact, URL-safe way to represent secure information between two parties — typically between a client (like your React frontend) and a server (like your Spring Boot backend).

Structure of JWT

1. Header – contains metadata about the token
Example:

{
  "alg": "HS256",
  "typ": "JWT"
}

alg: the algorithm used to sign the token (like HMAC SHA256)
typ: the type of token (always "JWT")

2. Payload– contains the actual data (called claims)
Example:

{
  "sub": "varunpatil",
  "role": "USER",
  "iat": 1696504500,
  "exp": 1696508100
}

sub: subject (usually the username or user ID)
iat: issued at (timestamp)
exp: expiration time

You can also add custom fields like email, role, etc.

3. Signature – ensures the token hasn’t been tampered with
Created by combining the header, payload, and a secret key:

HMACSHA256(base64UrlEncode(header) + "." + base64UrlEncode(payload), secret)

So a final JWT looks something like this 👇

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.
eyJzdWIiOiJ2YXJ1bnBhdGlsIiwicm9sZSI6IlVTRVIifQ.
X7a2qDg6m8i0wRJCrjXw9sM1D_Va8m2YH7JbZp8Qb6Y

⚙️ How JWT Works (Simple Flow)

The user logs in with credentials (email & password).
The server verifies credentials and generates a JWT.
The client stores the token (e.g., in localStorage or a cookie).
For every request, the client sends the token in the Authorization header:
Authorization: Bearer <your_jwt_token>
The server verifies the token. If valid, it processes the request.
If the token is expired or invalid → the request is denied.

🔍 JWT Verification — How the Server Confirms the Token Is Legit

When a user sends a JWT (usually in the Authorization header), the server doesn’t just trust it — it recomputes and verifies it.

1️⃣ Token Received

Example token:

xxxxx.yyyyy.zzzzz

Where:

xxxxx → Base64URL(header)
yyyyy → Base64URL(payload)
zzzzz → Base64URL(signature)

2️⃣ Server Splits the Token
The server separates these 3 parts.
Then it decodes the header and payload (Base64URL → JSON):

Header: { "alg": "HS256", "typ": "JWT" }
Payload: { "sub": "varunpatil", "exp": 1696508100 }
Signature: "zzzzz"

3️⃣ Server Recreates the Signature
The server takes the same header and payload (the first two parts) and runs them through the same signing algorithm using the secret key stored securely on the backend.

newSignature = HMACSHA256(
  base64UrlEncode(header) + "." + base64UrlEncode(payload),
  secretKey
)

This produces a new hash — essentially a digital fingerprint of the token’s content.

4️⃣ Signature Comparison
Now the server compares the newly created signature with the one received in the token:

if (newSignature == receivedSignature)
    → ✅ Token is authentic
else
    → ❌ Token has been tampered with

If someone even changes a single character in the payload (like user role or ID), the new hash will be completely different, and verification will fail instantly.

Key Features of JWT

1.Stateless (No Server-Side Session)

JWT is self-contained: it carries all the user information inside the token (claims).
The server doesn’t need to store session data — just validate the signature.
Makes scaling APIs easier because you don’t rely on in-memory sessions.

2.Compact and URL-Safe

Tokens are small and can be sent in:
- HTTP headers (Auth: Bearer )
- URL query parameters
- Cookies
Base64Url encoding ensures tokens are safe to transmit over the web.

3. Cross-Platform

JWT is language-agnostic — works with Java, Node.js, Python, Go, etc.
Can be used in mobile apps, web apps, or microservices.
Perfect for distributed systems where multiple services need to verify the same token.

4. Integrity Verified

The signature ensures the token hasn’t been tampered with.
If the payload is changed, verification will fail.
Protects sensitive operations like authentication, authorization, and API access.

Disadvantages of JWT

Payload Is Not Encrypted
Token Revocation Is Hard
Complexity in Key Management
Vulnerable If Secrets Are Compromised

Many people mistakenly believe that JWT encrypts data. In fact, JWT does not encrypt the data; it only encodes it and uses a signature to verify that the token was issued by a trusted source and hasn’t been tampered with.

Encoding vs Encryption

1. Encoding

Purpose: To transform data into a different format so it can be safely transmitted or stored.
Key Point: Encoding is not meant to be secret — anyone can decode it.
Example in JWT:
Header and payload are Base64Url encoded.
Anyone can decode and read the payload; the security comes from the signature, not the encoding.
Analogy: Like converting English text to Morse code — it’s readable if you know the rules.

2. Encryption

Purpose: To make data confidential so only authorized parties can read it.
Key Point: Encryption requires a secret key (or public/private key pair). Without the key, the data is unreadable.
Example: AES, RSA, or JWE (JSON Web Encryption).
Analogy: Like locking your letter in a safe — only someone with the key can open it.

Thank You....

What is a Bean in Spring Boot?

Varun Patil — Sat, 13 Sep 2025 19:33:32 +0000

If you are learning Spring Boot, one term you’ll hear again and again is “Bean”. But what exactly is a Bean, and why is it so important? Let’s break it down in simple terms for developers.

1. What is a Bean?
In Spring, a Bean is simply an object managed by the Spring IoC (Inversion of Control) container.

Instead of manually creating objects using new in your code, Spring creates and manages the objects for you. This allows Spring to:

Automatically inject dependencies into classes.
Manage the lifecycle of objects (initialization and destruction).
Promote loose coupling and reusability.

In short, a bean is just a Spring-managed object.

2. Beans Are Not Just Models
Many developers assume beans are only for model classes (like Car, User, etc.), but that’s not true. Beans can be:

Models / Entities
Services
Repositories / DAOs
Controllers
Configuration classes

Basically, any class you want Spring to manage can be a bean.

3. How to Create Beans in Spring Boot
a) Using Annotations (Recommended in Spring Boot)
Spring Boot automatically scans your packages for components and creates beans for you.

@Component
public class Car {
    public void drive() {
        System.out.println("Driving a Mercedes");
    }
}

@Service
public class CarService {
    private final Car car;

    @Autowired
    public CarService(Car car) {
        this.car = car;
    }

    public void startJourney() {
        car.drive();
    }
}

b) Using Configuration Class with @bean
You can also define beans in a central configuration class:

@Configuration
public class AppConfig {

    @Bean
    public Car car() {
        return new Car();
    }

    @Bean
    public CarService carService() {
        return new CarService(car());
    }
}

4. Why Beans Matter

Inversion of Control (IoC): Spring creates objects and injects dependencies, not your code.
Dependency Injection (DI): Makes your code loosely coupled, reusable, and testable.
Lifecycle Management: Spring can manage bean lifecycle (@PostConstruct, @PreDestroy, or init-method / destroy-method).
Reusability:A single bean can be used across multiple services, controllers, and other beans. **
Bean Lifecycle (Brief Overview)**
Spring instantiates the bean
Injects dependencies
Calls initialization methods
Bean is ready to use
On shutdown, calls destroy methods

6. Conclusion

A bean is simply a Spring-managed object.
Beans allow Spring to handle object creation, wiring, and lifecycle, letting you focus on building business logic.
Modern Spring Boot uses annotations + component scanning, so you rarely need XML anymore.