Forem: suyash200

Decoding Kafka Part 2

suyash200 — Sat, 06 Dec 2025 16:05:45 +0000

Hands-On & Under the Hood

In Part 1, we established that Kafka is the high-speed highway for data, handling real-time streams with high throughput. We covered the basic anatomy: Brokers, Topics, and Partitions.

Now, it’s time to get our hands dirty. In this part, we will spin up a local Kafka cluster, write code to produce and consume events, and—crucially—dive deeper into how Consumers actually track their progress using Offsets.

1. The Setup: Kafka via Docker Compose

Setting up Kafka manually can be complex (ZooKeeper, multiple brokers, etc.). To keep things clean, we’ll use Docker Compose. This allows us to spin up a Broker and a UI tool in a single command.

We will use the apache/kafka image, provectus/kafka-ui for monitoring and node.js runtime and kafkajs driver.

Download the docker-compose file on your system

docker-compose.yaml
Run the docker compose command

docker compose up -d

Thats it we've a running kafka broker, ready to churn out events 🙌

2. Managing Topics (The Admin API)

Before we send data, we need a destination. While you can let Kafka auto-create topics, defining them via the Admin API gives you control over two critical factors:

Partitions: How much parallelism do you need?
Replication Factor: How many copies of the data do you want for fault tolerance?


import { Kafka } from "kafkajs";

const kafka = new Kafka({
  clientId: "my-app",
  brokers: ["localhost:9094"],
});

const createTopicIfNotExists = async () => {
  const admin = kafka.admin();
  const topicName = "Test-topic";

  try {
    await admin.connect();

    // 1. Get the list of existing topics
    const topics = await admin.listTopics();
    console.log("Existing topics:", topics);

    // 2. Check if the specific topic exists
    if (!topics.includes(topicName)) {
      console.log(`Topic "${topicName}" not found. Creating...`);

      // 3. Create the topic
      await admin.createTopics({
        topics: [
          {
            topic: topicName,
            numPartitions: 1,     // Adjust based on your needs
            replicationFactor: 1, // Adjust based on your broker count
          },
        ],
      });
      console.log(`Topic "${topicName}" created successfully.`);
    } else {
      console.log(`Topic "${topicName}" already exists.`);
    }

  } catch (error) {
    console.error("Error in admin operation:", error);
  } finally {
    // 4. Always disconnect
    await admin.disconnect();
  }
};

createTopicIfNotExists();

3. The Producer: Keys vs. No Keys

Now, let's write some events. In Kafka, how you send the message determines where it lands.

Scenario A: Sending without a Key

If you send a message with key=null, the producer creates a "Round Robin" effect. It distributes messages evenly across all available partitions. This is great for load balancing but implies no guarantee of order relative to other messages.

import { Kafka } from "kafkajs";

const kafka = new Kafka({
  clientId: "my-app",
  brokers: ["localhost:9094"],
});

async function produceWithoutKey() {
  await producer.connect();
  await producer.send({
    topic: "Test-topic",
    messages: [
      {
        value: JSON.stringify({
          message: "Hello KafkaJS user!",
        }),
      },
    ],
  });

  await producer.disconnect();
}

produceWithoutKey(); // this will produce messages without a key

Scenario B: Sending with a Key

If you provide a Key (e.g., user_id or transaction_id), Kafka guarantees that all messages with the same key go to the same partition.

import { Kafka } from "kafkajs";

const kafka = new Kafka({
  clientId: "my-app",
  brokers: ["localhost:9094"],
});

const producer = kafka.producer();
async function produceWithKey() {
  await producer.connect();
  await producer.send({
    topic: "Test-topic",
    messages: [
      {
        key: Math.random().toString(36).substring(2, 15),
        value: JSON.stringify({
          message: "Hello KafkaJS user!",
        }),
      },
    ],
  });

  await producer.disconnect();
}
produceWithKey(); // this will produce messages with a key

Why does this matter? If you are processing payment updates for User A, you need "Payment Initiated" to arrive before "Payment Completed." Sending both with key=User A ensures they land in the same partition and are read in order.

TIP 💡: Check out the script multiple-events.zsh —it generates events repeatedly to simulate high-volume data, giving you a feel for the scale Kafka is designed to handle.

4. The Consumer

Consuming messages is where the logic gets interesting. This isn't just about reading data; it's about tracking state. GroupId is an important parameter for consumer groups it helps sharing events in parallel, rebalancing manage consumption and much more. Typically, consumers run in a separate microservice so they can process events independently from the producer or other services.

import { Kafka } from "kafkajs";

const kafka = new Kafka({
  clientId: "my-app",
  brokers: ["localhost:9094"],
});

const consumer = kafka.consumer({ groupId: "Test-group" });

await consumer.connect();
await consumer.subscribe({ topic: "Test-topic" });



await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    // processing as per need
    console.log({
      partition,
      offset: message.offset,
      value: JSON.parse(message.value.toString()),
      message: message,
      topic,
    });
  },
});

Understanding Offsets

As a consumer reads from a partition, it needs to keep track of its place. We call this the Offset.

TIP 💡: Consider the Offset as a bookmark while reading a book. It tells you exactly where you stopped reading so you can pick up from there next time (or after a crash).

There are three specific types of offsets you should know:

Log-End Offset: The offset of the very last message written to the partition (the newest data).
High-Watermark Offset: The point up to which all consumers are safe to read.
Committed Offset: The last offset the consumer successfully processed and reported done.

Consumer Lag

One of the most important metrics to watch is Consumer Lag. This is essentially the distance between the writer (Producer) and the reader (Consumer).

Consumer Lag = LogEndOffset - CommittedOffset

If your lag is high, your consumer is falling behind the producer 🚨

The Consumer Group Logic (Scaling)

When you start a consumer, you usually assign it to a Group. Kafka automatically balances the partitions among the consumers in that group.

Here is the relationship between Consumers n(C) and Partitions n(P):

n(C) < n(P):
- Result: Some consumers will read from multiple partitions.
- Status: Heavy load on individual consumers.
n(C) = n(P):
- Result: The ideal state. Each consumer handles exactly one partition.
- Status: Balanced.
n(C) > n(P):
- Result: Since a partition cannot be split, the extra consumers will have no work.
- Status: Idle. (Useful only as failover backups).

5. Monitoring with Kafka UI

Finally, we can visualize everything we just built. By opening localhost:9012 (or your configured port), we can see the Kafka UI.

What to look for:

Topics: Verify your topic exists with the correct partition count.

Messages : Here we can see each message with its partition assigned and offset, how partitioned are assigned is out of scope for now, we'll pick it up in future.

Consumers: Check your Consumer Lag. If you see the lag growing, it means your consumer script can't keep up with the producer!

Thats it for today, we'll see in-depth application by using it for some projects in the next part of the ongoing series.

See ya 👋, keep learning !

Follow me

X (Twitter): https://x.com/SuyashLade
LinkedIn: https://www.linkedin.com/in/suyash-lade/

Appendix

Decoding Kafka: A Streaming Powerhouse?

suyash200 — Sun, 24 Nov 2024 09:23:52 +0000

What is Kafka exactly?

Imagine a high-speed highway for data. That's essentially what Kafka is! It's a powerful tool for handling real-time data streams in large-scale systems. Think of it as a system that captures continuous data flow from various sources like applications, user interfaces, and servers, and stores it for further analysis and processing.

When to use Kafka?

Kafka is build for large scale systems that requires a high throughput. It's ideal for situations where real-time data transfer is crucial, such as managing payment transactions, Internet of Things (IoT) data flow, and monitoring systems. It's a popular choice in data platforms, event-driven architectures, and micro-services environments.

Example: Lets take an payment service where after successful payment we need to generate invoice send email, add database entry all at the same time. Along with subscription event where we have to capture analytics and enable users Kafka can help in streamlining this entire setup at scale and in real-time

Kafka Working

Kafka works on Kafka-protocol which is based on TCP protocol. It uses distributed systems server and client to process data.

Kafka-protocol

Kafka employs a binary protocol over TCP for efficient communication between clients and servers. This binary protocol is designed to minimize overhead and maximize performance. Unlike traditional protocols that require a handshake for every connection, Kafka's protocol establishes a long-lasting connection, reducing the overhead of repeated handshakes.

Kafka-server

Kafka-server is cluster of one or more brokers that handles distributed processing of events across various data-centers.

Brokers are individual Kafka servers responsible for orchestrating the event streaming.
A Kafka cluster is highly scalability and fault-tolerant, if any of its servers fails, the other servers will take over their work to ensure continuous operations without any data loss.

Kafka-client

Kafka clients are applications that interact with Kafka clusters to produce and consume events. Kafka-has many drivers available across various languages for Node.js, Python, C, C++, Java.

Producers generate and send events to specific Kafka topics. They can operate asynchronously for high throughput or synchronously for guaranteed delivery.
Consumers subscribe to topics of interest and actively poll the brokers for new events. They can be part of consumer groups, allowing multiple consumers to collaborate and share the load of processing events from a single topic.

Kafka-Terminologies

Event

We all are talking about event for quite some time, it simply is an action that has happened in the application. Events belong to a topic, it also has data(message) with it to help in further processing.Unlike traditional messaging systems, events are not deleted after consumption. We can define how long do we store an event in the Kafka broker or cluster.

Topic

Topic are the identification for an event. Similar to files in folder system topic helps organizing events to specific topic. Topics in Kafka are always multi-producer and multi-subscriber.

Partition

Imagine a topic as a book with multiple chapters (partitions). Each chapter (partition) can hold a sequence of events. When you produce an event, you essentially decide which chapter (partition) it belongs to based on its key.

For each partition, one broker is designated as the leader, while others are followers. The leader handles reads and writes for the partition, while followers replicate data from the leader. If the leader fails, one of the followers is promoted to become the new leader.
Kafka brokers maintain metadata about the cluster's state, including the topics, partitions, and their respective leaders. Clients fetch this metadata to determine which broker to send requests to.

Example of topic partition,entire event is assigned to a specific partition based on a partitioning key docs.

Kafka-APIs

All these API's are provided by client side packages to interact with Kafka broker/cluster servers

The Admin API to manage and inspect topics, brokers, and other Kafka objects.
The Producer API to publish (write) a stream of events to one or more Kafka topics.
The Consumer API to subscribe to (read) one or more topics and to process the stream of events produced to them.
The Kafka Streams API to implement stream processing applications and micro-services. Helpful in doing aggregation and pre-processing some data like joints, windowing.
The Kafka Connect API to build and run reusable data import/export connectors that consume (read) or produce (write) streams of events from and to external systems and applications so they can integrate with Kafka

More info on installing Kafka can be found here
Hope this give you basic idea of Kafka its working and its need. Will keep updating this series learning Kafka.

Setting Up Your Next Project

suyash200 — Sun, 25 Jun 2023 16:30:29 +0000

Looking for a guide to start your project setup? You found a right place. This blog will give you an basic idea for setting up your project and this series will help you as a beginner. I am using next but this can apply for any framework as well as vanilla.js

Prerequisite

-node.js
-npm
-HTML
-CSS

Lets start ...

1. Creating next project

npx create-next-app app-name

For beginners we can use ESlint in the options .

Project structure

lets understand project structure of the current project and update it to be more suitable structure.
How it looks now :

Following steps would be followed:

1. Removing the api folder from pages

2. Create component folder

3. Create an api folder in rood directory

4. Create an interceptor.js file and Endpoints folder

Now it looks like this

2. Working on CSS

lets clear the entire global CSS ...
Now starting fresh

CSS reset

  *{
    margin:0;
    padding:0;
    box-sizing: border-box;
   }

This reset the CSS padding and removes unnecessary padding, margin that is automatically added to your HTML components.

Root Selector

:root{
   --white:#fff,
   --black:#000,

}

The :root selector matches the document's root element. We can
declare our variable here , basically we can describe font
color that are repetitively used in the web app.

Body


body {
  background: var(--cream);
  font-family: "Playfair Display", "sans-serif";

}

The css declarations in this part are applied through out the the web app ,we can provide with background color for out website
font-size, font-family etc can be described here.

Understanding Basics of Node.js

suyash200 — Mon, 03 Apr 2023 13:02:45 +0000

Introduction

Node.js is a javascript runtime environment built on V8 engine. It is open-source, single threaded and is used to make server-side applications.

Lets understand by visualization how node.js works

Modules in Node

Core feature of node is modules. Modules are simple or complex
task that are pre-build in them. Each module in Node.js has its own context, so it cannot interfere with other modules or pollute global scope. It is based on common Javascript modules standard
Modules are of three types:

Core: In built modules that are pre installed with node.js
Local: Local modules are modules created locally in your Node.js application.
Third-Party: External packages that are installed via package managers like npm or yarn.

Node package Manager(NPM)

"There is a npm package for that."
This quote sums up importance of npm. It allows third party packages to be installed to your project. Anyone can upload their
packages to website and allow others to use.

Example

Now that we take a look at the basics of node.js lets write a program that covers http module in node used for backend development.

var http = require('http');

http.createServer(function (req, res) {
  res.write('Hello World!');  
  res.end();  
}).listen(8080);

In above code at line 1 http module is imported as it is a Core module in node.js.
createserver is used to create a server
res,req represents request and response of the server
.listen() represents on which port the requests are going to be heard.
The output of the following server would be on port 8000:

This has been basics of node.js. More documentation can always be found on node docs.Thank you for reading