Forem: Gokulnath P

Kafka Essentials: Building Scalable and Reliable Event Streaming

Gokulnath P — Mon, 13 Oct 2025 15:21:46 +0000

In our previous blog, we covered the basic building blocks like producers, consumers, and topics that help send messages from one service to another. To build a truly scalable app, we need to go beyond these basics. Let’s look at some other Kafka concepts that help us make communication production-ready.

Partitions

Imagine a topic in Kafka like a big folder where all messages about the same thing go. Now picture 1000 orders coming in at once — all those order messages go into this folder. The problem is, they get handled one by one, like people standing in a single line. That can be slow when there are lots of orders.

Partitions fix this by splitting the big folder into smaller pieces called partitions. Think of partitions like separate checkout lines at a grocery store — multiple lines help more people check out quickly.

Kafka makes sure messages stay in order within each partition, because sometimes order really matters. But messages in different partitions don’t have to be in order.

So partitions help Kafka work on many messages at the same time, speeding things up and making them smoother.

Message Key

Keys play an important role in deciding which partition a message goes to. Kafka uses the key to figure this out by running it through a process called hashing (think of it like a recipe that always gives the same result for the same key). This makes sure all messages with the same key end up in the same partition.

If a message doesn’t have a key, Kafka spreads those messages evenly across different partitions, like taking turns, to keep things balanced.

Using keys lets us group related messages together in one partition. For example, all events related to a single user will always go to the same partition, so they stay in order.

Consumer Group

Partitions let us split events into multiple streams so consumers can read them faster. But what if you have two partitions and two consumers? How do we make sure each consumer reads different partitions without both reading the same messages?

That’s where consumer groups come in. You can create multiple consumers that share the same group ID. Kafka then assigns partitions to each consumer in the group, making sure each partition is read by only one consumer. This way, consumers work in parallel without duplicate messages.

So, consumer groups help spread the work and make event processing faster and more efficient.

Offset

How do consumers know which message to read next? And if they stop for a while, how do other consumers in the group know where to continue?

That’s where offsets come in. Every message in a partition gets a unique number called an offset — it’s like a simple count starting at 0. Kafka keeps track of offsets separately for each partition to make sure messages stay in the right order.

Consumers “commit” their current offset to Kafka to remember which messages they’ve read. This way, if they stop or another consumer takes over, they know exactly where to pick up without missing or repeating messages.

Replication

We know a Kafka cluster has multiple brokers so if one broker goes down, others can keep things running. But how do those other brokers get the events they need to continue without missing anything?

That’s where replication comes in. Kafka makes copies of each partition and stores them on different brokers in the cluster. This means if one broker fails, the data is still safe and available from the other brokers.

When you create a topic, you decide the replication factor — basically how many copies of each partition you want. The more copies, the safer your data is.

A few other things to remember:

The replication factor can’t be bigger than how many brokers you have, because each copy lives on its own broker.
Every partition has a “leader” broker responsible for handling all reads and writes.
If the leader broker crashes, Kafka automatically switches leadership to another broker that has a copy—making sure your data stream never misses a beat.

Together, these concepts form the core of Kafka’s ability to handle large-scale event streaming, making sure your app can process messages quickly, reliably, and without losing data.

Ready to see these ideas in action? Keep an eye out for our next blog where we roll up our sleeves and get hands-on with Kafka!

Kafka Fundamentals: A quick look at the basic building blocks

Gokulnath P — Mon, 13 Oct 2025 15:07:44 +0000

So what’s Kafka, anyway? Kafka is a super-fast, reliable tool for moving messages (events) between apps and services. Instead of sending data directly (like a phone call), services share what’s happening through Kafka — almost like posting messages on a community board. Others can check those messages whenever they want.

This setup lets lots of microservices talk with each other in a flexible, reliable way — making your app easier to build and scale without all the tight connections that can cause problems.

Let’s start by breaking down each building block of Kafka to see how they help everything communicate behind the scenes.

Messages / Events

Messages, sometimes called events, are like little notes that one service sends to another. They carry data, along with some extra info like when the message was created. They also have something called keys — we’ll explain how keys help group messages later. The important thing is that once a message is sent, it can’t be changed — just like sending a letter in the mail. This helps keep a clear and reliable record of what happened.

Topic

Think of a topic like a folder that groups together similar events. Producers send their messages to a topic, and consumers subscribe to the topics they care about to get those events.

Broker

A broker is a server that manages the event streaming behind the scenes. It’s responsible for receiving messages, storing them safely, and then sending them to other services when they ask for them.

Cluster

A Kafka cluster is created by running multiple brokers together. The idea is that if one broker stops working, the others can keep everything running without any problems, so the event streaming never stops.

Producer

A producer is an app or client that sends messages. It connects to the Kafka cluster and sends its messages to a specific topic.

Consumer

A consumer is an app or client that reads messages. It subscribes to one or more topics and pulls messages from the Kafka cluster to process or use them.

Together, these building blocks form the heart of Kafka’s messaging system, letting different parts of your application communicate smoothly and reliably.

In future blogs, we’ll dive deeper into Kafka topics like partitions, consumer groups, offsets, and replication to get the full picture of how Kafka works in production. This way, you’ll build a strong foundation and be ready to use Kafka effectively in your own projects.

Kafka for beginners: An event streaming platform

Gokulnath P — Mon, 13 Oct 2025 14:13:30 +0000

Have you ever wondered how big apps handle millions of messages every second without slowing down or crashing? As digital platforms keep changing, so does our code. To build apps that can grow smoothly, we break down a big app into smaller, independent parts called microservices — each doing its own job.

One big question when building apps with lots of these microservices is: how do they talk to each other?

The usual way, using REST APIs, can make these parts too tightly connected. That means if you change one, it might accidentally break others — not good for growing or fixing your app.

This is where event streaming comes in. Think of it as services sending out messages about what they’ve done, and other services listening if they’re interested. This keeps everything more flexible and easier to manage.

By the end of this blog series, you’ll understand how Kafka, a popular event streaming tool, helps these chatting services work smoothly together.

React: Effective UI State Management Practices

Gokulnath P — Mon, 06 Oct 2025 15:01:38 +0000

As web development evolves, it is important to choose the right practices that help us create a scalable, readable, maintainable, and testable codebase. In this blog series, we will look at different practices that help us create such a codebase.

This blog series is divided into three topics:

Application Design
State Management
Testing Practices

State Management

Bridging the gap between business goals and React implementation requires thoughtful state design. In today's blog, we will be exploring the following practices under State Management:

Understanding the state
Efficiently managing the form state
You Might Not Need a useEffect

Understanding the state

To build performance-heavy applications, it is important to understand the scope of state and what should go into it. State should be placed as close as possible to the components that require the data, to avoid unnecessary re-rendering. Identify what doesn't qualify as state

Does it remain unchanged over time? If yes, it isn’t state.
Is it passed in from a parent via props? If yes, it isn’t state.
Can you compute it based on existing state or props in your component? If yes, it definitely isn’t state.

Reference: Find the minimal but complete representation of UI state

Efficiently Managing Form State

Maintaining the state of forms is as important as managing application-level state. Forms have unique needs such as handling frequent changes, applying validation, and processing submissions. Since form state changes frequently, place it close to the form component to:

Improve application performance by avoiding unnecessary re-rendering of other components.
Test the behaviour of the form, not its internal implementation.

You Might Not Need a useEffect

useEffect is an escape hatch in React, allowing interaction with external systems such as non-React widgets, network calls, or the browser DOM. Avoid using useEffect just to update a component’s state from prop or state changes.

You don’t need Effects to transform data for rendering.
You don’t need Effects to handle user events.

Reference: You Might Not Need an Effect

That’s all for today. In the next blog, we’ll explore best practices for testing React applications.

React: Effective UI Application Design Practices

Gokulnath P — Wed, 22 May 2024 03:25:34 +0000

This blog series is divided into three topics:

Application Design
State Management
Testing Practices

Application Design

Understanding how to convert business requirements into a React codebase with minimal effort is crucial. In today's blog, we will be exploring the following practices under Application Design:

Extracting Design Components
Building Application Logic as Custom Hooks
Embracing Component Architecture

Extracting Design Components

UI applications are nothing but a composition of design components such as buttons, tables, and accordions. These design components can be as granular as a button, icon, or form row, or as large as a table, accordion, or form. It is important to decouple these design components from business logic so that:

We can reuse them across all places without redesigning them every time.
Design changes can be applied in a single place and reflect everywhere.
We can test the behaviour of these design components in an isolated environment since they are decoupled from business logic.

Building Application Logic as Custom Hooks

React Hooks are not just another feature; they represent a paradigm shift in the React development journey. By embracing React Hooks, developers can focus more on building the application logic rather than grappling with React’s complexities. This shift orchestrates:

Efficient code organisation, which helps improve the reusability and maintainability of behaviours.
Easier testing.
Improved overall application performance by avoiding unnecessary re-renders.

Reference: Reusing Logic with Custom Hooks

Embracing Component Architecture

Component architecture is a fundamental concept in React development. Breaking our application into smaller components instead of having one bulk component is as important as extracting design components and behaviours. As the complexity of a component increases, it is a good idea to split it into separate components to make your code:

More readable and maintainable.
Easier to test.
Loosely coupled with other behaviours and components.

Reference: Break the UI into a component hierarchy

That is all for today. In our next blog, we will look at some of the best practices for managing and designing state in React.

Boost Your Productivity with a Node JS Code Generator

Gokulnath P — Mon, 06 Mar 2023 09:37:55 +0000

A code generator is a tool that automates the process of creating code. It can save you a lot of time and effort, especially if you need to create similar code repeatedly. Rather than writing the same code over and over again, you can use a code generator to create templates that you can reuse for different projects.

Using a code generator can also help you to maintain consistency across your projects. You can define coding standards and conventions in your code generator, and then use it to generate code that adheres to these standards. This can help to ensure that your code is more reliable, easier to maintain, and less prone to errors.

Another benefit of using a code generator is that it can reduce the amount of manual work involved in the coding process. By automating repetitive tasks, you can free up your time to focus on more complex and challenging tasks that require your attention.

Overall, a code generator can be a powerful tool for developers who want to save time, improve code quality, and reduce errors in their projects. With a code generator, you can create code faster, more consistently, and with less effort.

Prerequisites:

Before we get started, you'll need to have NPM installed on your computer. If you don't have NPM installed, you can download it from the official website.

In this post, we will cover the following topics:

How to create a NPM package
How to create Command-Line Interface (CLI) application in NPM
How to create a generator using the "fs-extra", "enquirer", and "replace-in-file" modules
Using the generator to create a custom web application

Each section will provide a step-by-step guide to help you create your own code generator and customise it to your needs.

1. Creating a NPM Package: A Step-by-Step Guide

Creating an NPM package is a simple and straightforward process that allows you to share your code with others or use it in other projects. In this example, we'll walk you through how to create an NPM package that exports a simple greeting function.

Step 1: Create the Package Directory

First, we need to create a new directory for our package. In our example, we'll call it "custom-package". Inside this directory, we'll create two files: index.js and package.json.

mkdir custom-package
cd custom-package

Step 2: Create the index.js File

In the index.js file, we'll export a function that will output a greeting to the console.

const greet = () => console.log('Welcome to the exciting world of creating your own NPM packages!');

module.exports = { greet };

Step 3: Create the package.json File

In the package.json file, we'll include metadata about our package such as its name, version, and main entry point.

{
  "name": "custom-package",
  "version": "1.0.0",
  "main": "index.js"
}

Step 4: Link the Package Locally

Before we can use our package in another project, we need to link it locally using the "npm link" command.

npm link

Step 5: Create the Consumer Directory

Next, we need to create a new directory for our consumer project. In our example, we'll call it "consumer". Inside this directory, we'll create two files: index.js and package.json.

cd ..
mkdir consumer
cd consumer

Step 6: Install the Package

Now that we have our package linked locally, we can install it in our consumer project using the "npm link" command with the name of our package.

npm link custom-package

Step 7: Use the Package

In the index.js file of our consumer project, we'll require the "custom-package" package and call its greet function.

const { greet } = require("custom-package");

greet();

Step 8: Run the Consumer Project

Finally, we can run the consumer project using the "npm start" command, which will execute the index.js file and output the greeting from our "custom-package" package.

{
  "name": "consumer",
  "version": "1.0.0",
  "scripts": {
    "start": "node index.js"
  }
}

And that's it! We've successfully created an NPM package and used it in a consumer project. This is just the beginning of what you can do with NPM packages, so feel free to experiment and explore all the possibilities.

2. Creating a Command-Line Interface (CLI) Application in NPM: A Step-by-Step Guide

Welcome to the world of creating command-line interface (CLI) applications in NPM! In this section, we will walk through the steps to create a simple CLI application that prints a welcome message to the console.

Step 1: Create the Package Directory

First, we need to create a new directory for our package. In our example, we'll call it "npm-cli". Inside this directory, we'll create two files: index.js and package.json.

mkdir npm-cli
cd npm-cli

Step 2: Create the index.js File

In the index.js file, we will print a welcome message to the console when the CLI is run..

#!/usr/bin/env node

console.log("Welcome to the world of creating command-line interface (CLI) applications in NPM!");

The first line (#!/usr/bin/env node) tells your terminal that this file should be executed using Node.js.

Step 3: Create the package.json File

In the package.json file, we'll include metadata about our package such as its name, version and bin (path and command for executables).

{
  "name": "npm-cli",
  "version": "1.0.0",
  "bin": {
    "npm-cli": "index.js"
  }
}

This tells NPM that your package has a bin executable called npm-cli, which should be linked to the index.js file you created in the previous step.

Step 4: Link the Package Locally

Before we can use our package, we need to link it locally using the "npm link" command.

npm link

This will create a symbolic link from your package directory to your global NPM directory.

Step 5: Use the Package

Finally, we can run "npm-cli" command in terminal, which will execute the index.js file and output the welcome message.

npm-cli

That's it! You've created a simple NPM CLI. Now you can modify the index.js file to add more functionality to your CLI.

3. Creating a Custom Generator with Node js: A Step-by-Step Guide

In this section, we will walk through the steps to create a simple code generator that will bootstrap a project from existing template.

Step 1: Create a New Directory

Create a new directory named "create-custom-webapp" and navigate into it

mkdir create-custom-webapp
cd create-custom-webapp

Step 2: Create a package.json File

This file will contain information about our custom generator, including its name, version, description, and dependencies. Here's what the "package.json" file should look like:

{
  "name": "create-custom-webapp",
  "version": "1.0.0",
  "description": "A tool to generate custom webapp",
  "bin": {
    "create-custom-webapp": "cli.js",
    "ccw": "cli.js"
  },
  "dependencies": {
    "commander": "10.0.0",
    "enquirer": "2.3.6",
    "fs-extra": "11.1.0",
    "replace-in-file": "6.3.5"
  }
}

Note that we have included the "bin" field in our "package.json" file, which tells NPM that our custom generator will have two command line interfaces ("create-custom-webapp" and "ccw") that will be executed by running the "cli.js" file.

Step 3: Create the Command-Line Interface (CLI)

Create a new file named cli.js in the root of the project and add the following code:

#!/usr/bin/env node

const { Command } = require('commander');
const packageJson = require('./package.json');
const { prompt } = require('enquirer');
const { bootstrap } = require('./src');

const readUserInput = () => {
  return prompt([
    {
      type: 'input',
      name: 'appName',
      message: 'What is your application name?',
      required: true,
    },
  ]);
};

const program = new Command();

program
  .name(packageJson.name)
  .version(packageJson.version)
  .description(packageJson.description)
  .action(() => readUserInput().then(bootstrap));

program.parse();

This code sets up the "commander" package to handle the command-line interface for your generator. The readUserInput function uses the "enquirer" package to prompt the user for input, and the bootstrap function (which we will define in the next step) is called to generate the project with the user's input.

Step 4: Define the `bootstrap` Function

Create a new directory named src and a file named index.js inside the src directory. Add the following code to index.js

const fs = require('fs-extra');
const path = require('path');
const { execSync } = require('child_process');
const replaceInFile = require('replace-in-file');

/**
 * A function to bootstrap a project from template.
 * @param {Object} config - The config object.
 * @param {string} config.appName - The name of the application.
 */
const bootstrap = (config) => {
  const templatePath = path.join(__dirname, '..', 'templates');
  const destinationPath = path.join(process.cwd(), config.appName);

  console.log('Bootstrapping simple React web app');
  fs.copySync(templatePath, destinationPath);

  console.log('Customizing a app with information provided by user');
  replaceInFile.sync({
    files: path.join(destinationPath, '**', '*'),
    from: ['CGD_APP_NAME'],
    to: [config.appName],
  });

  console.log('Installing required dependencies to start the application');
  execSync('npm i', { cwd: destinationPath });

  console.log(`Inorder to run an application, navigate to ${config.appName} folder and run 'npm start'`);
};

module.exports = { bootstrap };

The bootstrap function takes a configuration object as input that includes the appName property, which is the name of the application to be generated. The function first sets up paths for the source and destination folders using the Node JS "path" module. Then, it copies the contents of the templates directory (which contains a basic React web app) to a new directory with the name specified by the user.

After copying the template files, the function uses the "replace-in-file" package to replace all instances of the placeholder string "CGD_APP_NAME" with the actual appName provided by the user. This ensures that the name of the application is correctly reflected in all files.

Finally, the function uses the "child_process" module to execute the command npm install in the new directory, which installs all necessary dependencies for the generated application. Once the dependencies are installed, the function logs a message to the console to inform the user how to start the application.

That's it! With this bootstrap function, our custom generator can now generate a basic React web app with the user-specified name and all required dependencies installed.

Step 5: Create the Template Files

Create a new directory names templates and copy the file from here into the templates folder.

This folder contains the files that will be used to generate the web application. The folder structure and files inside the templates folder are based on the type of application we want to generate. In this case, we're generating a simple web application using React and Webpack.

The folder structure is as follows:

templates/
  |- public/
  |   |- index.html
  |- src/
  |   |- index.js
  |- package.json
  |- webpack.config.js

Let's take a closer look at each of these files and folders:

public/: This folder contains the static assets that will be served by the application. In our case, it contains an index.html file.
src/: This folder contains the source code of the application. It contains an index.js file, which is the entry point for our application.
package.json: This file contains the metadata and dependencies for the application. It specifies the dependencies required to run our application, including React and Webpack.
webpack.config.js: This file contains the configuration for Webpack. It specifies the entry point for our application, the output directory for the compiled assets, and the loaders and plugins required to compile the application code.

When the user generates a new web application using our custom generator, these files will be copied to the new project directory with any necessary modifications based on the user's input.

Step 6: Link the Package Locally

In the create-custom-webapp directory, run the following command to link the package to the global module directory

npm link

Step 7: Use the Package

Navigate to the directory where you want to create your new project and run the following command:

create-custom-webapp

This will start the custom generator and prompt you to enter a name for your application. Once the details are entered it will generate a project and will do install dependencies inside it.

To start your application, navigate to the new directory and run the following command:

npm start

Congratulations, you have successfully created a custom generator to create a new web application from a template!

Summary

In conclusion, code generators, such as the one we built in this tutorial, can save you time and effort by automating repetitive tasks and providing a starting point for your projects. By creating an npm package with a CLI, we were able to create a custom generator that asks for user input and generates a customized application with the necessary dependencies. By leveraging the "fs-extra", "commander", "enquirer", and "replace-in-file" modules, we were able to make the generator easy to use and customizable for different projects.

We hope this tutorial has been helpful in showing you how to create your own custom generator. With these tools, you can easily create new applications and get started on your next project with minimal effort. Happy coding!

You can find the code snippets used in this blog here.

Forem: Gokulnath P

Kafka Essentials: Building Scalable and Reliable Event Streaming

Partitions

Message Key

Consumer Group

Offset

Replication

Kafka Fundamentals: A quick look at the basic building blocks

Messages / Events

Topic

Broker

Cluster

Producer

Consumer

Kafka for beginners: An event streaming platform

React: Effective UI State Management Practices

State Management

Understanding the state

Efficiently Managing Form State

You Might Not Need a useEffect

React: Effective UI Application Design Practices

Application Design

Extracting Design Components

Building Application Logic as Custom Hooks

Embracing Component Architecture

Boost Your Productivity with a Node JS Code Generator

Prerequisites:

Contents

1. Creating a NPM Package: A Step-by-Step Guide

Step 1: Create the Package Directory

Step 2: Create the index.js File

Step 3: Create the package.json File

Step 4: Link the Package Locally

Step 5: Create the Consumer Directory

Step 6: Install the Package

Step 7: Use the Package

Step 8: Run the Consumer Project

2. Creating a Command-Line Interface (CLI) Application in NPM: A Step-by-Step Guide

Step 1: Create the Package Directory

Step 2: Create the index.js File

Step 3: Create the package.json File

Step 4: Link the Package Locally

Step 5: Use the Package

3. Creating a Custom Generator with Node js: A Step-by-Step Guide

Step 1: Create a New Directory

Step 2: Create a package.json File

Step 3: Create the Command-Line Interface (CLI)

Step 4: Define the bootstrap Function

Step 5: Create the Template Files

Step 6: Link the Package Locally

Step 7: Use the Package

Summary

Step 4: Define the `bootstrap` Function