Forem: Amool Kuldiya

Understanding the Thundering Herd Problem: Taming the Stampede in Distributed Systems

Amool Kuldiya — Tue, 24 Feb 2026 15:21:46 +0000

Imagine a popular store opening its doors at 9 AM sharp. Hundreds of customers lined up outside rush in simultaneously, overwhelming the cashiers and causing chaos. This is exactly what happens in distributed systems—the Thundering Herd Problem—when too many requests hit a shared resource at once.

NORMAL OPERATION (Cache Hit)                  THUNDERING HERD (Cache Miss Stampede)
    Fast path                                                 Failure path
┌─────────────┐                                             ┌─────────────┐
│   Clients   │                                             │   Clients   │
│ 10k users   │                                             │ 10k users   │
└──────┬──────┘                                             └──────┬──────┘
       │                                                           │
       ▼                                                           ▼
 ┌─────▼─────┐   Cache Hit   ┌──────────────┐                      ▼
 │ App Server│◄──────────────│ Redis Cache  │               ┌──────▼──────┐
 │   Node 1  │               │ key=product1 │               │ 10k Cache   │
 └─────┬─────┘               │ TTL=60s      │               │   MISSES    │
       │                     └──────┬───────┘               └──────┬──────┘
       │             Cache Miss     │                              │
       ▼                            ▼                              ▼
 ┌─────▼─────┐                     ┌──────────────┐         ┌──────────────┐
 │ App Server│                     │   Database   │         │   Database   │
 │   Node 2  │◄─── 1 Query ────────│ 1 Query Only │         │ 10k Queries! │
 └─────┬─────┘                     │ Returns Data │         │ CPU=1000%    │
       │                           └──────┬───────┘         └──────────────┘
       │                                  │                   💥 OVERLOAD
       └──────────┬───────────────────────┘
                  │
             ┌────▼────┐
             │Cache Set│ ← Serves all 10k clients
             └─────────┘

What is the Thundering Herd Problem?

The Thundering Herd Problem occurs when numerous clients or processes simultaneously compete for the same shared resource, like a database or cache, creating a sudden traffic spike that overwhelms the system.

Unlike gradual load increases, this is a synchronized burst—think cache keys expiring at the exact same timestamp across millions of requests.

Where It Commonly Occurs

This issue plagues several system components:

Caching systems: Popular cache entries expire together, triggering mass backend fetches.
Databases: Multiple app servers hammer the DB after a cache miss.
Load balancers: Requests flood a single healthy node during failures elsewhere.
Lock acquisition: Processes race for mutexes on critical sections.

In a typical app architecture, clients query an app server, which checks Redis cache first. Cache hit? Serve instantly. Miss? Fetch from DB and repopulate.

Real-World Example: Cache Expiry Spike

Consider Netflix releasing a hot new show—millions request episode data cached with 60-second TTL. At expiry (say 10,000 req/sec), all clients miss simultaneously:

Normal: Cache serves 10k req/sec at 1ms latency
Expiry moment: 10k DB queries at 100ms each → 5-10x overload

Database connections exhaust, latency jumps to seconds, cascading failures hit the entire app.

Similar spikes occur during IPL match ticket sales in India or Black Friday e-commerce rushes.

Normal Spike vs Thundering Herd

Aspect	Normal Traffic Spike	Thundering Herd
Cause	Organic growth (marketing, events)	Synchronized event (TTL expiry, cron jobs)
Pattern	Gradual ramp-up	Instant burst
Impact	Autoscaling handles	Overwhelms even scaled capacity
Duration	Minutes-hours	Seconds (but devastating)

Key difference: Herd is predictable but synchronized, amplifying tiny windows of vulnerability into outages.

Why Dangerous in Distributed Systems

Clients → App → DB overload → Timeouts → Retries → More DB load → 💥

Amplification: 1 cache miss → N DB queries (N=concurrent clients).
Tail latency: Slowest DB query blocks everyone.
Cascading failure: Overloaded DB slows apps → more timeouts → retry storms.
Autoscaling lag: Spikes are too brief for new instances to spin up.

In multi-region setups, one region's stampede ripples globally.

System Impacts Breakdown

CPU Overload

Sudden thread explosions thrash scheduler; context switches skyrocket.

Database Strain

Connection pools exhaust; query queues balloon → timeouts cascade.

Cache Ineffectiveness

Becomes useless during stampede—worse than no cache!

Latency Explosion

P99 jumps 100x; users abandon sessions.

Prevention Techniques

1. Stale-While-Revalidating

Only one request refreshes the cache; others serve stale data and reuse the result.

2. MUTEX

Use a distributed lock (e.g., Redis SETNX) so only one request hits DB.

3. Jitter on TTL

TTL = base + random(0, maxJitter) to avoid synchronized expiry.

4. Probability Early Computation

Refresh hot keys early based on access frequency / near-expiry.

5. Rate Limiting

Limit requests per key/user to prevent backend overload.

6: Cache Warming

Preload hot keys before traffic spikes or deployments.

Real outage example: Facebook's 2010 cache stampede took hours to resolve Link.

Final Thoughts

The Thundering Herd turns "working at scale" into outages without proper safeguards. Master these patterns—staggered TTLs + coalescing + backoff.

Next time your cache expires, remember: one cow is fine, the herd is deadly.

Building a Centralized, Type-Safe Modal System in Next.js

Amool Kuldiya — Wed, 07 Jan 2026 04:34:23 +0000

Modals are one of the most common UI patterns in web applications.

They’re used for:

Confirmations
Forms
Previews
Warnings
Details panels

In small applications, managing modals is simple.

But as a Next.js app grows, modal handling often becomes messy and hard to maintain.

In this article, we’ll build a centralized, type-safe modal system in Next.js using:

React Context
TypeScript generics
A modal registry pattern
shadcn/ui Dialog (UI layer only)

This approach scales well for large production apps and keeps modal logic clean, predictable, and reusable.

The Problem

In small applications, handling modals locally works fine.
But as a Next.js app grows, modal management quickly becomes messy.

Imagine a page that needs multiple modals:

A confirmation modal
A details modal
A form modal

A typical implementation looks like this:

const [isConfirmOpen, setConfirmOpen] = useState(false);
const [isDetailsOpen, setDetailsOpen] = useState(false);
const [isFormOpen, setFormOpen] = useState(false);

return (
  <>
    <ConfirmModal
      open={isConfirmOpen}
      onClose={() => setConfirmOpen(false)}
      onConfirm={handleConfirm}
    />

    <DetailsModal
      open={isDetailsOpen}
      onClose={() => setDetailsOpen(false)}
      data={selectedItem}
    />

    <FormModal
      open={isFormOpen}
      onClose={() => setFormOpen(false)}
      onSubmit={handleSubmit}
    />
  </>
);

Now imagine this same set of modals is needed on five different pages.

Each page ends up:

Duplicating modal state
Passing slightly different props
Re-implementing the same open/close logic

Over time, this leads to:

❌ Repeated and fragile state logic
❌ Inconsistent modal behavior across pages
❌ Harder refactors when modal props change
❌ Poor developer experience as the app scales

This approach works initially, but it does not scale well in medium to large applications.

To solve this, we can move modal state and rendering into a single, centralized system.

The Solution: A Centralized Modal System

Instead of managing modal state locally, we can design a global modal architecture.

Core ideas

One global Modal Context
A modal registry that maps modal names to components
Type-safe modal props enforced by TypeScript
Open any modal from anywhere in the app
UI libraries stay at the edges

Step 1: Define Generic Shared Types

Create app/(modals)/types.ts.

This file defines the contract for your entire modal system.

import type { ReactNode } from "react";

// Base props passed to all modals
export interface BaseModalProps {
  open: boolean;
  onOpenChange: (open: boolean) => void;
  close: () => void;
}

// Example shared data type
export interface Entity {
  id: string;
  title: string;
  description?: string;
}

// Modal-specific props
interface EntityModalProps {
  entity: Entity;
}

// Map modal names to their required props
export interface ModalPropsMap {
  confirm: EntityModalProps & {
    onConfirm?: () => void;
    onCancel?: () => void;
  };

  details: EntityModalProps;

  form: {
    initialValues?: Record<string, unknown>;
    onSubmit: (values: Record<string, unknown>) => void;
  };
}

export type ModalType = keyof ModalPropsMap;

export type ModalState<K extends ModalType = ModalType> = {
  type: K;
  props: ModalPropsMap[K];
  open: boolean;
};

export type ModalComponent<K extends ModalType> =
  React.ComponentType<ModalPropsMap[K] & BaseModalProps>;

Why this design works

Every modal has strictly typed props
You cannot open a modal without required data
TypeScript autocomplete works everywhere
Refactors are safe and predictable

Step 2: Create Generic Modal Components

Example 1: Details Modal

"use client";
import { Dialog, DialogContent, DialogHeader, DialogTitle } from "@/components/ui/dialog";
import type { ModalPropsMap, BaseModalProps } from "./types";

type Props = ModalPropsMap["details"] & BaseModalProps;

export default function DetailsModal({
  open,
  onOpenChange,
  close,
  entity,
}: Props) {
  return (
    <Dialog open={open} onOpenChange={onOpenChange}>
      <DialogContent>
        <DialogHeader>
          <DialogTitle>{entity.title}</DialogTitle>
        </DialogHeader>

        <p>{entity.description}</p>

        <button onClick={close}>Close</button>
      </DialogContent>
    </Dialog>
  );
}

Example 2: Confirmation Modal

type Props = ModalPropsMap["confirm"] & BaseModalProps;

export function ConfirmModal({
  open,
  onOpenChange,
  close,
  entity,
  onConfirm,
  onCancel,
}: Props) {
  return (
    <Dialog open={open} onOpenChange={onOpenChange}>
      <DialogContent>
        <p>Are you sure you want to proceed with "{entity.title}"?</p>

        <div className="flex gap-2 mt-4">
          <button onClick={() => { onConfirm?.(); close(); }}>
            Confirm
          </button>
          <button onClick={() => { onCancel?.(); close(); }}>
            Cancel
          </button>
        </div>
      </DialogContent>
    </Dialog>
  );
}

Step 3: Modal Registry

All modals are registered in one place.

import ConfirmModal from "./ConfirmModal";
import DetailsModal from "./DetailsModal";
import FormModal from "./FormModal";
import type { ModalComponent, ModalType } from "./types";

export const MODALS: Record<ModalType, ModalComponent<any>> = {
  confirm: ConfirmModal,
  details: DetailsModal,
  form: FormModal,
};

Why a registry?

Centralized control
No accidental imports across pages
Easy to add or remove modals
Clear overview of available modals

Step 4: Modal Context

"use client";

import { createContext, useContext, useState } from "react";
import { MODALS } from "../(modals)/registry";
import type { ModalPropsMap, ModalType, ModalState } from "../(modals)/types";

interface ModalContextValue {
  openModal: <K extends ModalType>(
    type: K,
    props: ModalPropsMap[K]
  ) => void;
  closeModal: () => void;
}

const ModalContext = createContext<ModalContextValue | null>(null);

export function ModalProvider({ children }: { children: React.ReactNode }) {
  const [modal, setModal] = useState<ModalState | null>(null);

  const openModal = <K extends ModalType>(
    type: K,
    props: ModalPropsMap[K]
  ) => {
    setModal({ type, props, open: true });
  };

  const closeModal = () => {
    setModal(prev => (prev ? { ...prev, open: false } : null));
  };

  const ModalComponent = modal ? MODALS[modal.type] : null;

  return (
    <ModalContext.Provider value={{ openModal, closeModal }}>
      {children}

      {ModalComponent && (
        <ModalComponent
          {...modal.props}
          open={modal.open}
          onOpenChange={(open) => {
            if (!open) closeModal();
          }}
          close={closeModal}
        />
      )}
    </ModalContext.Provider>
  );
}

export function useModal() {
  const ctx = useContext(ModalContext);
  if (!ctx) throw new Error("useModal must be used inside ModalProvider");
  return ctx;
}

Step 5: Where to Use `ModalProvider`

After creating the ModalProvider, it needs to wrap your application so that any page or component can open modals.

In Next.js (App Router), the best place to do this is your root layout.

Example: Wrapping the App

// app/layout.tsx
import { ModalProvider } from "@/context/ModalContext";

export default function RootLayout({
  children,
}: {
  children: React.ReactNode;
}) {
  return (
    <html lang="en">
      <body>
        <ModalProvider>
          {children}
        </ModalProvider>
      </body>
    </html>
  );
}

Now the modal system is available globally across the app.

Step 6: Using the Modal Anywhere

"use client";
import { useModal } from "../(context)/ModalContext";

export default function ExamplePage() {
  const { openModal } = useModal();

  const entity = {
    id: "1",
    title: "Example Item",
    description: "This is a generic entity used for demonstration.",
  };

  return (
    <div className="flex gap-2">
      <button
        onClick={() =>
          openModal("details", { entity })
        }
      >
        Open Details
      </button>

      <button
        onClick={() =>
          openModal("confirm", {
            entity,
            onConfirm: () => console.log("Confirmed"),
          })
        }
      >
        Open Confirm
      </button>
    </div>
  );
}

TypeScript guarantees:

❌ You can’t open a modal with wrong props
❌ You can’t open an unregistered modal
✅ Autocomplete works everywhere

Bonus: Generic Promise-Based Modals

This architecture can be easily extended to support promise-based modals:

const confirmed = await openConfirmModal({ entity });

This is especially useful for:

Delete confirmations
Destructive actions
Async or step-based workflows

Promise-based modals fit naturally into the same centralized system and help keep async logic clean and readable.

Benefits of This Approach

✅ No duplicated modal state
✅ Fully type-safe modal usage
✅ Works across the entire application
✅ Scales cleanly as the app grows
✅ UI implementation stays flexible

Possible Enhancements

Promise-based modals – Let modals return values (true/false, form data) so they can be used with async/await instead of callbacks.
Modal stacking – Support opening multiple modals on top of each other for complex or nested workflows.
URL-driven modals – Sync modal state with the URL to enable deep links, browser navigation, and refresh persistence.
Animations & transitions – Add custom enter/exit animations using libraries like Framer Motion while keeping logic centralized.
Auto-unmount & cleanup – Delay unmounting to finish animations and ensure forms, timers, or requests are properly cleaned up.
Access control & guards – Prevent certain modals from opening based on permissions, feature flags, or application state.

Conclusion

A centralized modal system built with React Context + TypeScript generics provides:

Predictable modal behavior
Strong compile-time guarantees
Cleaner page components
Better developer experience

This pattern is ideal for medium to large Next.js applications and is worth setting up early as your app grows in complexity.

#NextJS #TypeScript #React #Frontend #Architecture #DeveloperExperience #WebDevelopment

Expressando (Part 2): A real-time sign language detection system

Amool Kuldiya — Fri, 21 Jan 2022 20:49:36 +0000

This is a follow-up article from our previous post.

Expressando (Part 1): A real-time sign language detection system

Pritam for GNU/Linux Users' Group, NIT Durgapur ・ Jan 22 '22

#machinelearning #opencv #tensorflow #python

In the previous post, we discussed the features like What is a 'Virtual Environment'?, Configuring Input through Webcam using OpenCV, Checking for Convexity Defects in the Camera Input, What is a 'Convex Hull'? How to display the Convex Hull ?, What are 'Convexity Defects'?. Today, you will be learning about Collecting data through OpenCV, Demonstration of Data-Collection and TensorFlow, Convolutional Neural Networks (CNN), and many more things. You will also learn about how to live prediction works and all the things.

Collecting data through OpenCV and labeling them

Create a file named collect-data.py inside the directory of TDoC-2021. As the name suggests, collecting data collects the images taken by webcam using the OpenCV library. creating one or more dataset(s) depending on the input, storing them in the directories, and pursuing them according to classes.

First, activate your virtual environment (This is very important as OpenCV is installed inside the virtual environment and not globally). Then import OpenCV and OS (included in python library) as cv2 and os.


import cv2

import os

We will be checking for the presence of the data directory, which will be used to store the train and test data. If the data folder is already created then, it will not create the folders, and proceed to store the data in the form of datasets, each representing a class.


if not os.path.exists("data"): #True

  os.makedirs("data")

  os.makedirs("data/train") 

  os.makedirs("data/test")

  os.makedirs("data/train/0") 

  os.makedirs("data/train/1")

  os.makedirs("data/train/2")

  os.makedirs("data/train/3")

  os.makedirs("data/train/4")

  os.makedirs("data/train/5")

  os.makedirs("data/test/0")

  os.makedirs("data/test/1")

  os.makedirs("data/test/2")

  os.makedirs("data/test/3")

  os.makedirs("data/test/4")

  os.makedirs("data/test/5")

The if not statement checks if the directory data exists or not, whereas the os.path.exists method returns Boolean value & depending on which next lines are executed. Next if os.path.exists returns False then os.makedirs method will create the necessary directories.

Now we will create some variables for the path of the folders where we will save our dataset of images. After that, we need to collect the required images via webcam. For that, we will use the already learned how to use the cv2.VideoCapture() method to initialize.

To make the process of collecting the data more interactive and visually appealing, we will display some statistics in the live view screen of the webcam. So to know how many images we have collected inside the folder, we can simply use these key bindings and record them as count objects and keep the counts of images as a dictionary for every number. we can display it in the live view of the webcam. It can be done very easily, made possible by OpenCV. This can be achieved by the following lines of code:


cv2.putText(frame, "MODE : "+mode, (30, 50), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,255), 1)

cv2.putText(frame, "IMAGE COUNT", (10, 100), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,255), 1)

cv2.putText(frame, "ZERO : "+str(count['zero']), (10, 120), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

cv2.putText(frame, "ONE : "+str(count['one']), (10, 140), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

cv2.putText(frame, "TWO : "+str(count['two']), (10, 160), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

cv2.putText(frame, "THREE : "+str(count['three']), (10, 180), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

cv2.putText(frame, "FOUR : "+str(count['four']), (10, 200), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

cv2.putText(frame, "FIVE : "+str(count['five']), (10, 220), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1

Next, we will learning about the Region of Interest (R.O.I).A region of interest (ROI) is an area of an image defined for further analysis or processing. Here, the region of interest will basically contain the region of the hand, used for portraying the gesture. We will be defining the region of interest using the following code:


x1 = int(0.5*frame.shape[1])

y1 = 10

x2 = frame.shape[1]-10

x1 = int(0.5*frame.shape[1])

y1 = 10

x2 = frame.shape[1]-10

y2 = int(0.5*frame.shape[1])

cv2.rectangle(frame, (x1-1, y1-1), (x2+1, y2+1), (255,0,0) ,3)

roi = frame[y1:y2, x1:x2] 

roi = cv2.resize(roi, (200, 200)) 

cv2.putText(frame, "R.O.I", (440, 350), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,225,0), 3)

cv2.imshow("Frame", frame)



roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)

_, roi = cv2.threshold(roi, 120, 255, cv2.THRESH_BINARY)

cv2.imshow("ROI", roi)  y2 = int(0.5*frame.shape[1])

cv2.rectangle(frame, (x1-1, y1-1), (x2+1, y2+1), (255,0,0) ,3)

roi = frame[y1:y2, x1:x2] 

roi = cv2.resize(roi, (200, 200)) 

cv2.putText(frame, "R.O.I", (440, 350), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,225,0), 3)

cv2.imshow("Frame", frame)



roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)

_, roi = cv2.threshold(roi, 120, 255, cv2.THRESH_BINARY)

cv2.imshow("ROI", roi)

Here, frame.shape[1] returns the shape of the frame or the camera input. Next, we will be defining the variables x1, y1, x2, and y2, which will serve as the diagonals of the rectangle (x1,y1) and (x2,y2). Next, we draw a rectangle surrounding the ROI using the rectangle() function, so that we can record our gesture inside the rectangle. After extraction, we resize and enlarge the region of interest using the resize() function.

we convert the ROI into its grayscale form, using the module cv2.COLOR_BGR2GRAY. After converting the ROI into grayscale, we will apply a simple threshold using the function cv2.threshold(). We will be using the cv2.THRESH_BINARY module to apply the threshold.

Next, we will be recording the images in the dataset according to the key bindings. We first initialize the interrupt variable, which returns 10 frames per second here. Then we define all the key bindings. Whenever, we press 0, data for the 'zero' key will get recorded. Similarly, it applies to the other datasets as well.

Now your collect-data.py should look like the following.


import cv2

import os 



if not os.path.exists("data"): #True

  os.makedirs("data")

  os.makedirs("data/train") 

  os.makedirs("data/test")

  os.makedirs("data/train/0") 

  os.makedirs("data/train/1")

  os.makedirs("data/train/2")

  os.makedirs("data/train/3")

  os.makedirs("data/train/4")

  os.makedirs("data/train/5")

  os.makedirs("data/test/0")

  os.makedirs("data/test/1")

  os.makedirs("data/test/2")

  os.makedirs("data/test/3")

  os.makedirs("data/test/4")

  os.makedirs("data/test/5")





mode = 'train' 

directory = 'data/'+mode+'/' #data/train/



cap=cv2.VideoCapture(0)



while True:

  _, frame = cap.read()

  frame = cv2.flip(frame, 1)



  cv2.putText(frame, "Expressando - TDOC 2021", (175, 450), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,255), 3)



  count = {'zero': len(os.listdir(directory+"/0")), 

       'one': len(os.listdir(directory+"/1")),

       'two': len(os.listdir(directory+"/2")),

       'three': len(os.listdir(directory+"/3")),

       'four': len(os.listdir(directory+"/4")),

       'five': len(os.listdir(directory+"/5"))} 



  cv2.putText(frame, "MODE : "+mode, (30, 50), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,255), 1)

  cv2.putText(frame, "IMAGE COUNT", (10, 100), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,255), 1)

  cv2.putText(frame, "ZERO : "+str(count['zero']), (10, 120), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

  cv2.putText(frame, "ONE : "+str(count['one']), (10, 140), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

  cv2.putText(frame, "TWO : "+str(count['two']), (10, 160), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

  cv2.putText(frame, "THREE : "+str(count['three']), (10, 180), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

  cv2.putText(frame, "FOUR : "+str(count['four']), (10, 200), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)

  cv2.putText(frame, "FIVE : "+str(count['five']), (10, 220), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (255,255,255), 1)





  x1 = int(0.5*frame.shape[1])

  y1 = 10

  x2 = frame.shape[1]-10

  y2 = int(0.5*frame.shape[1])

  cv2.rectangle(frame, (x1-1, y1-1), (x2+1, y2+1), (255,0,0) ,3)

  roi = frame[y1:y2, x1:x2] 

  roi = cv2.resize(roi, (200, 200)) 

  cv2.putText(frame, "R.O.I", (440, 350), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,225,0), 3)

  cv2.imshow("Frame", frame)



  roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)

  _, roi = cv2.threshold(roi, 120, 255, cv2.THRESH_BINARY)

  cv2.imshow("ROI", roi)



  interrupt = cv2.waitKey(10) 

  if interrupt & 0xFF == 27:

    break

  if interrupt & 0xFF == ord('0'):

    cv2.imwrite(directory+'0/'+str(count['zero'])+'.jpg', roi)

  if interrupt & 0xFF == ord('1'):

    cv2.imwrite(directory+'1/'+str(count['one'])+'.jpg', roi)

  if interrupt & 0xFF == ord('2'):

    cv2.imwrite(directory+'2/'+str(count['two'])+'.jpg', roi)

  if interrupt & 0xFF == ord('3'):

    cv2.imwrite(directory+'3/'+str(count['three'])+'.jpg', roi)

  if interrupt & 0xFF == ord('4'):

    cv2.imwrite(directory+'4/'+str(count['four'])+'.jpg', roi)

  if interrupt & 0xFF == ord('5'):

    cv2.imwrite(directory+'5/'+str(count['five'])+'.jpg', roi)



cap.release()

cv2.destroyAllWindows()

Demonstration of Data-Collection and Introduction to TensorFlow

Now, we are going to have a demonstration on how to collect data with increased precision. This is a vital step, as it governs the future processes which are required in making the model.

What is a 'Dataset'?

A Dataset is a collection of data that is treated as a single unit by a computer. This means that a dataset contains a lot of separate pieces of data/images but can be used to train an algorithm with the goal of finding predictable patterns inside the whole dataset.

Datasets are of three types:

Training Dataset - This dataset which contains the necessary data with which the model is to be trained. This dataset contains the maximum amount of data, and it is primarily used by the machine for comparison.
Validation Dataset - This dataset checks for the validation of the input in accordance with the training dataset. It is a subset of the training dataset. It eliminates unnecessary inputs and increases the speed of operation.
Test Dataset - This dataset is used for testing the models and determining the accuracy and losses which are incurred while training the machine. This is very much important, as it serves as a test set for the training data, and the user can verify the results.

Now let us go through a quick example of how to collect data:

Run the collect-data.py file using the command:


python collect-data.py

The window will open, taking a reference input from the webcam. It will also display a thresholded image of the area under the rectangle (Region of Interest) in another window. This image will be recorded under the respective directories of the dataset.

Let us first begin with the "zero" gesture, which basically involves a closed fist. Bring the gesture close inside the Region of Interest, as much that it is completely enclosed within the rectangle, and covers most of the area of the region.
Make sure, you have a clear background and do not let unexpected disturbances appear inside the rectangle/ROI. Try to collect clear data as much as possible. Make sure all your fingers are visible.
Then, start collecting the data by pressing '0' of the Numpad. This will capture the images, which will be the images stored in the dataset under the train directory in the 0 folder.
Collect approximately 100-120 images to train the model. Make sure to collect data, while changing your orientation over the rectangle, so that the machine can recognize the gesture whenever you show the gesture as input inside the rectangle.

Similarly for other datasets.

Now, as you have collected the data, now let us see what TensorFlow has in store for us.

TensorFlow

Implementing machine learning is a complex stream mainly the concerns related to the creation of real-time models. Hence the end-to-end open-source framework - TensorFlow is used for collecting datasets, training systems through models and providing the results based on it.

TensorFlow is an open-source framework created by Google Creative Labs. It executes machine learning and helps to build neural networks easily (along with Keras) in Python as well as in Javascript. It builds an environment of networks to experiment with the algorithms of machine learning and visualizes it using flow graphs. The graphs represent the progression of all nodes where nodes are the operations in the model.

Why TensorFlow

TensorFlow provides pre-built functions and advanced operations API to ease the task of building different neural network models. It provides the required infrastructure and hardware which makes them one of the leading libraries used extensively by researchers and students in the deep learning domain.

To begin with Tensorflow, you can go through the following tutorials:

Convolutional Neural Networks (CNN)

Artificial Intelligence has been witnessing a monumental growth in bridging the gap between the capabilities of humans and machines. Researchers and enthusiasts alike, work on numerous aspects of the field to make amazing things happen. One of many such areas is the domain of Computer Vision.

The agenda for this field is to enable machines to view the world as humans do, perceive it in a similar manner, and even use the knowledge for a multitude of tasks such as Image & Video recognition, Image Analysis & Classification, Media Recreation, Recommendation Systems, Natural Language Processing, etc. The advancements in Computer Vision with Deep Learning have been constructed and perfected with time, primarily over one particular algorithm — Convolutional Neural Network.

To know more about CNN go through the following link: https://deepai.org/machine-learning-glossary-and-terms/convolutional-neural-network

Now let us go through the code:

Create a file named as 'train_model.py' inside the directory of 'TDoC-2021'. As the name suggests, it will be training the models using TensorFlow and Keras with the Convolutional Neural Network being trained on it.

At first, we will import the model and layers, which we will be using from Keras. Here, Keras uses TensorFlow as it's backend, i.e, the functions in Keras also makes uses of the functions in TensorFlow for all of the results.

Here, we will be making use of the Sequential model. This model is used when you have one input, and one output at a time. This model basically stacks the layers and serves as a container for the layers in connection.

To know more about the Sequential model, go here:

The process of building a Convolutional Neural Network always involves four major steps.

Convolution
Pooling
Flattening
Full Connection

Now, we will create an object of the sequential class below:


from keras.models import Sequential

from keras.layers import Convolution2D, MaxPooling2D, Flatten, Dense



classifier = Sequential()

Next, we will define the first convolutional layer. This will modulate the image input tensor and result in convoluted matrices. The matrices produced will be merged and amalgamated into a single matrix. This basically consists of mathematical operands, each of which serves as a node. The operands are decided on the basis of image size, color, and characteristics. After the convolution, we will be Max-Pooling the matrix, so that the size of the matrix is reduced, and it will help in the efficient determination. It will be achieved by the following code:


classifier.add(Convolution2D(32, (3, 3), input_shape=(64, 64, 1), activation='relu'))

classifier.add(MaxPooling2D(pool_size=(2, 2)))

Here, the add() function helps to add successive layers in the Convolution Neural Network object.

We will be using two types of activation function:

relu: It stands for Rectified Linear Unit. It helps in the independent activation of neurons/nodes in the layers.
softmax: It helps to activate the neurons if there are multiple branching present, and it requires detecting simultaneous figures.

To know more about activation functions, go here: https://www.v7labs.com/blog/neural-networks-activation-functions

Similarly, we will be adding a second Convolutional Layer to it, in order to increase the efficiency of detection. However, we cannot add the infinite number of layers, as it would decrease the speed and also, can result in wrong detections.

Next, we will be flattening the layers, and connecting them to produce our entire Neural Network. Flattening is a very important step to understand. What we are basically doing here is taking the 2-D array, i.e pooled image pixels, and converting them to a one-dimensional single vector. We’ve used flatten function to perform flattening, we no need to add any special parameters, Keras will understand that the “classifier” object is already holding pooled image pixels and they need to be flattened.


classifier.add(Flatten())

Now, we need to create a fully connected layer, and to this layer, we are going to connect the set of nodes we got after the flattening step, these nodes will act as an input layer to these fully-connected layers. As this layer will be present between the input layer and output layer, we can refer to it as a hidden layer.

Dense() is the function to add a fully connected layer. Units is where we define the number of nodes that should be present in this hidden layer, these units value will be always between the number of input nodes and the output nodes but the art of choosing the most optimal number of nodes can be achieved only through experimental tries. Though it’s a common practice to use a power of 2 and the activation function will be a rectifier function.


classifier.add(Dense(units=128, activation='relu'))

classifier.add(Dense(units=6, activation='softmax'))

As, the model has been prepared, you need to compile and compose it.


classifier.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

To know more about Adam Optimisation, go here: https://www.geeksforgeeks.org/intuition-of-adam-optimizer/

Next, we will import our datasets: train and test, and then will pass them through the model for compilation, and ultimately compose the model on the basis of the images fed into them.


from keras.preprocessing.image import ImageDataGenerator



train_datagen = ImageDataGenerator(

    rescale=1./255,

    shear_range=0.2,

    zoom_range=0.2,

    horizontal_flip=True)



test_datagen = ImageDataGenerator(rescale=1./255)



training_set = train_datagen.flow_from_directory('data/train',

                         target_size=(64, 64),

                         batch_size=5,

                         color_mode='grayscale',

                         class_mode='categorical')



test_set = test_datagen.flow_from_directory('data/test',

                      target_size=(64, 64),

                      batch_size=5,

                      color_mode='grayscale',

                      class_mode='categorical')

Here, we will be using the ImageDataGenerator. It is used to manipulate the image input, and convert them into a machine-readable format using the CNN building. We will be first defining the Data Generators for the test and train data individually, and naming them as test_datagen and train_datagen. We will be using the rescale parameter to rescale the images, zoom_range and shear_range to adjust the image identification, so that when the image can be still be recognized under zoomed conditions up to the scale of 0.2. At last, horizontal_flip helps to check for a correct orientation for the image.

After the data generators are created, we need to generate the training_set and test_set for making our model, and testing it. We define the directory, color-space (grayscale), and the class mode(which can be binary or categorical.) We also define the input size of the image as 64 x 64. The batch size refers to the size of the batches, which is used here as 5. Batch is a continuous length of data, that is passed as once to the model for training.

After the datasets are prepared to feed into the model, we fit them into the model layers, for analysis and metrics calculation.


classifier.fit_generator(

    training_set,

    epochs=10,

    validation_data=test_set)

Here, we define the sets, that have been generated in the previous steps. We will be testing the training_set against the test_set, and determine the accuracy and losses. We will pass them in 10 epochs. Epoch is the number of passes of the entire training dataset the machine learning algorithm has completed. It is similar to test cases, which the training set needs to pass. These check for valid input and test the accuracy and losses.

After the model has been prepared and tested, it is necessary to save the model in the form of hierarchial data-model. We will also save the model in the form of JSON format (JavaScript Object Notation). We are saving it in the JSON format so that the user can know about the model composition and characteristics.


model_json = classifier.to_json()

with open("model-bw.json", "w") as json_file:

  json_file.write(model_json)

classifier.save_weights('model-bw.h5')

Now your train_model.py should look like the following.


from keras.models import Sequential

from keras.layers import Convolution2D, MaxPooling2D, Flatten, Dense



classifier = Sequential()



classifier.add(Convolution2D(32, (3, 3), input_shape=(64, 64, 1), activation='relu'))

classifier.add(MaxPooling2D(pool_size=(2, 2)))



classifier.add(Convolution2D(32, (3, 3), activation='relu'))

classifier.add(MaxPooling2D(pool_size=(2, 2)))



classifier.add(Flatten())



classifier.add(Dense(units=128, activation='relu'))

classifier.add(Dense(units=6, activation='softmax'))



classifier.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])





from keras.preprocessing.image import ImageDataGenerator



train_datagen = ImageDataGenerator(

    rescale=1./255,

    shear_range=0.2,

    zoom_range=0.2,

    horizontal_flip=True)



test_datagen = ImageDataGenerator(rescale=1./255)



training_set = train_datagen.flow_from_directory('data/train',

                         target_size=(64, 64),

                         batch_size=5,

                         color_mode='grayscale',

                         class_mode='categorical')



test_set = test_datagen.flow_from_directory('data/test',

                      target_size=(64, 64),

                      batch_size=5,

                      color_mode='grayscale',

                      class_mode='categorical') 



classifier.fit_generator(

    training_set,

    epochs=10,

    validation_data=test_set)



model_json = classifier.to_json()

with open("model-bw.json", "w") as json_file:

  json_file.write(model_json)

classifier.save_weights('model-bw.h5')

Run the code in your Powershell/terminal using:


python train_model.py

Live Prediction of Customised Sign Language

Create a file named as 'prediction.py' inside the directory of 'TDoC-2021'. As the name suggests, it will be predicting the customized sign with which we have trained the model, when we give a similar gesture to what we have trained it. First activate your virtual environment. Also, make sure you have the model-bw.h5 and model-bw.json Prepared before running the code.

Next, we will be importing cv2, operator(an inherent module present in the standard Python library), and model_from_json, which is to be imported to read the model content in the JSON file.


from keras.models import model_from_json

import operator

import cv2

Here, we will be declaring the object json_file, which will parse the model-bw.json present in the same directory as the prediction.py file. Next, we will be parsing the contents from the json_file, which has the model-bw.json already passed in it.

Then, we use the read() and close() function to store the contents in the file in the object model_json.

Next we will load the data model and JSON model simultaneously using the load_weights() and model_from_json() function respectively. When the models are successfully loaded, it will print "Loaded model from disk" in the terminal/Powershell.

Next, we have initialized a Video-Capture object called "cap". Then, we have declared a dictionary called categories, having integer numbers as their key index. This will be helpful in rendering the string value of their respective numbers, which will be shown in the prediction.


json_file = open("model-bw.json", "r")

model_json = json_file.read()

json_file.close()

loaded_model = model_from_json(model_json)

loaded_model.load_weights("model-bw.h5")

print("Loaded model from disk")

cap = cv2.VideoCapture(0) 



categories = {0: 'ZERO', 1: 'ONE', 2: 'TWO', 3: 'THREE', 4: 'FOUR', 5: 'FIVE'}

Then, we will initialize a while-loop which will run infinitely, as long as the video is being captured, or a break statement has been encountered. Then we read the frames of the video using cap.read() function, and invert the reading frame using the flip() function.


while True:

  _, frame = cap.read()

  frame = cv2.flip(frame, 1)



  x1 = int(0.5*frame.shape[1])

  y1 = 10

  x2 = frame.shape[1]-10

  y2 = int(0.5*frame.shape[1])



  cv2.putText(frame, "Expressando - TDOC 2021", (175, 450), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,0), 3)

  cv2.rectangle(frame, (x1-1, y1-1), (x2+1, y2+1), (255,255,255) ,3)

  roi = frame[y1:y2, x1:x2]



  roi = cv2.resize(roi, (64, 64)) 

  roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)

  cv2.putText(frame, "R.O.I", (440, 350), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,225,0), 3)



  _, test_image = cv2.threshold(roi, 120, 255, cv2.THRESH_BINARY)

  cv2.imshow("ROI", test_image)

Next, we will be defining the Region of Interest, where we will be showing our hand gestures. We have previously discussed how to define the region of interest in our previous documentation. Then, we will be using the rectangle() function to enclose the region of interest, within the rectangle for the user to know where the sign will be detected.


  result = loaded_model.predict(test_image.reshape(1, 64, 64, 1))

  prediction = {'ZERO': result[0][0], 

         'ONE': result[0][1], 

         'TWO': result[0][2],

         'THREE': result[0][3],

         'FOUR': result[0][4],

         'FIVE': result[0][5]} 

  prediction = sorted(prediction.items(), key=operator.itemgetter(1), reverse=True)

  cv2.putText(frame, "PREDICTION:", (30, 90), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2)

  cv2.putText(frame, prediction[0][0], (80, 130), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2)   

  cv2.imshow("Frame", frame)

We will use the predict() function, which enables us to predict the labels of the data values on the basis of the trained model. It accepts only a single argument which is usually the data to be tested. It returns the labels of the data passed as an argument based upon the learned or trained data obtained from the model. Thus, the predict() function works on top of the trained model and makes use of the learned label to map and predict the labels for the data to be tested. We will pass the test_image, reshaped to the input size which we will be passing into the Convolutional Neural Network, which is 64 x 64 x 1.

Then, we will declare a dictionary called prediction, which has the integer name in the string as the key, which in turn returns the result[0][key]. Here result[0] refers to the first element in the list, which is the categories dictionary. Then, we parse the results depending on the key value in result[0][key].

Next, we are sorting the prediction.items(), which are the individual probabilities that the input gesture you are showing in the Region of Interest (R.O.I) resembles the images/dataset with which you have trained the models. The operator.itemgetter(1) helps to sort with more than one column. Also, reverse=True refers to sorting in the descending order such that prediction[0][0] contains the string integer with the maximum probability.

We will then show the string using the putText() function on the frame, and then show the frame under the name Frame using the imshow() function.

Next, we initialize the interrupt variable, which returns 10 frames per second here. 0xFF collects and recognizes the keyboard response and matches it to the Escape Key. Then, after the detection is over, we release our webcam and deallocate all the memory rendered to the image array vectors. This has been previously explained in the past documentation.

Now your prediction.py should look like the following.


from keras.models import model_from_json

import operator

import cv2



json_file = open("model-bw.json", "r")

model_json = json_file.read()

json_file.close()

loaded_model = model_from_json(model_json)

loaded_model.load_weights("model-bw.h5")

print("Loaded model from disk")



cap = cv2.VideoCapture(0) 



categories = {0: 'ZERO', 1: 'ONE', 2: 'TWO', 3: 'THREE', 4: 'FOUR', 5: 'FIVE'}



while True:

  _, frame = cap.read()

  frame = cv2.flip(frame, 1)



  x1 = int(0.5*frame.shape[1])

  y1 = 10

  x2 = frame.shape[1]-10

  y2 = int(0.5*frame.shape[1])



  cv2.putText(frame, "Expressando - TDOC 2021", (175, 450), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (225,255,0), 3)

  cv2.rectangle(frame, (x1-1, y1-1), (x2+1, y2+1), (255,255,255) ,3)

  roi = frame[y1:y2, x1:x2]



  roi = cv2.resize(roi, (64, 64)) 

  roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)

  cv2.putText(frame, "R.O.I", (440, 350), cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, (0,225,0), 3)



  _, test_image = cv2.threshold(roi, 120, 255, cv2.THRESH_BINARY)

  cv2.imshow("ROI", test_image)



  result = loaded_model.predict(test_image.reshape(1, 64, 64, 1))

  prediction = {'ZERO': result[0][0], 

         'ONE': result[0][1], 

         'TWO': result[0][2],

         'THREE': result[0][3],

         'FOUR': result[0][4],

         'FIVE': result[0][5]} 

  prediction = sorted(prediction.items(), key=operator.itemgetter(1), reverse=True) 

  cv2.putText(frame, "PREDICTION:", (30, 90), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2)

  cv2.putText(frame, prediction[0][0], (80, 130), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2)   

  cv2.imshow("Frame", frame)



  interrupt = cv2.waitKey(10)

  if interrupt & 0xFF == 27:

    break





cap.release()

cv2.destroyAllWindows()