Forem: ronak navadia

Level Up Your NestJS App with BullMQ Queues, DLQs & Bull Board

ronak navadia — Thu, 28 Aug 2025 17:03:18 +0000

In this guide, we’ll walk through how to integrate BullMQ with NestJS using the WorkerHost processor pattern, as recommended in the official NestJS documentation. Along the way, we’ll enhance our job processing system with Bull Board for real-time monitoring and set up a Dead Letter Queue (DLQ) to catch and manage poison jobs. To tie everything together, we'll build a simple authentication system with a signup endpoint that enqueues a welcome email job—demonstrating how background processing can streamline user workflows.

📂 Folder Structure

src/
│── app.module.ts
│── main.ts
│
├── config/
│   └── bull.config.ts
│
├── modules/
│   ├── auth/
│   │   ├── auth.controller.ts
│   │   ├── auth.service.ts
│   │   └── auth.module.ts
│   │
│   └── queue/
│       ├── queue.constants.ts
│       ├── queue.module.ts
│       ├── queue.service.ts         # producers
│       └── processors/
│           ├── appointment.processor.ts  # WorkerHost
│           └── dlq.processor.ts          # WorkerHost (DLQ)
│
└── .env

📦 Install Dependencies

npm install bullmq ioredis class-validator class-transformer
npm install @bull-board/api @bull-board/express

⚙️ Environment Configs (.env)

PORT=3000

# Redis
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0
REDIS_USE_TLS=false

# Queue defaults
QUEUE_DEFAULT_ATTEMPTS=4

🧭 Queue Constants (src/modules/queue/queue.constants.ts)

export enum QueueNames {
  APPOINTMENT = 'appointment',
  APPOINTMENT_DLQ = 'appointment-dlq', // Dead Letter Queue
}

export const JobNames = {
  SCHEDULE_APPOINTMENT: 'schedule-appointment',
  APPOINTMENT_DIGEST: 'appointment-digest',
  DEAD_LETTER: 'dead-letter',
};

🔧 Bull Config (src/config/bull.config.ts)

// src/config/bull.config.ts
import { ConfigModule, ConfigService } from '@nestjs/config';

export const bullQueueConfig = {
  imports: [ConfigModule],
  inject: [ConfigService],
  useFactory: async (configService: ConfigService) => {
    const useTls =
      configService.get<string>('REDIS_USE_TLS') === 'true' ||
      (configService.get<boolean>('REDIS_USE_TLS') as any) === true;

    return {
      connection: {
        host: configService.get<string>('REDIS_HOST'),
        port: Number(configService.get<number>('REDIS_PORT')),
        password: configService.get<string>('REDIS_PASSWORD') || undefined,
        db: Number(configService.get<number>('REDIS_DB') || 0),
        ...(useTls && { tls: {} }),
      },
      defaultJobOptions: {
        attempts: Number(configService.get<number>('QUEUE_DEFAULT_ATTEMPTS') || 4),
        backoff: { type: 'exponential', delay: 3000 },
        removeOnComplete: { age: 3600, count: 1000 },
        removeOnFail: { age: 24 * 3600, count: 1000 },
      },
    };
  },
};

🏗 App Module (src/app.module.ts)

import { Module } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import { BullModule } from '@nestjs/bullmq';
import { bullQueueConfig } from './config/bull.config';
import { QueueModule } from './modules/queue/queue.module';
import { AuthModule } from './modules/auth/auth.module';

@Module({
  imports: [
    ConfigModule.forRoot({ isGlobal: true }),
    BullModule.forRootAsync(bullQueueConfig), // <- as requested
    QueueModule,
    AuthModule,
  ],
})
export class AppModule {}

📦 Queue Module (src/modules/queue/queue.module.ts)

Registers both the main queue and the DLQ, and wires processors.

import { Module } from '@nestjs/common';
import { BullModule } from '@nestjs/bullmq';
import { QueueNames } from './queue.constants';
import { QueueService } from './queue.service';
import { AppointmentProcessor } from './processors/appointment.processor';
import { DlqProcessor } from './processors/dlq.processor';

@Module({
  imports: [
    BullModule.registerQueue({
      name: QueueNames.APPOINTMENT,
      // per-queue defaults (can override root defaults)
      defaultJobOptions: {
        attempts: 4,
        backoff: { type: 'exponential', delay: 3000 },
        removeOnComplete: { age: 1800, count: 500 },
        removeOnFail: { age: 24 * 3600, count: 1000 },
      },
    }),
    BullModule.registerQueue({
      name: QueueNames.APPOINTMENT_DLQ,
      defaultJobOptions: {
        removeOnComplete: true,
      },
    }),
  ],
  providers: [QueueService, AppointmentProcessor, DlqProcessor],
  exports: [QueueService],
})
export class QueueModule {}

📨 Producer Service (src/modules/queue/queue.service.ts)

Adds jobs, delayed jobs, and a repeatable digest. (We keep producer isolated here and call it from AuthService.)

import { Injectable } from '@nestjs/common';
import { InjectQueue } from '@nestjs/bullmq';
import { Queue, JobsOptions } from 'bullmq';
import { JobNames, QueueNames } from './queue.constants';

@Injectable()
export class QueueService {
  constructor(
    @InjectQueue(QueueNames.APPOINTMENT) private readonly appointmentQueue: Queue,
  ) {}

  async addAppointmentJob(data: { userId: string; email: string; name: string }, opts?: JobsOptions) {
    // Idempotency via jobId (prevents duplicates)
    const jobId = opts?.jobId ?? `welcome:${data.email}`;
    return this.appointmentQueue.add(JobNames.SCHEDULE_APPOINTMENT, data, {
      jobId,
      attempts: 4, // ensure attempts is set on the job so DLQ logic can read it
      backoff: { type: 'exponential', delay: 3000 },
      ...opts,
    });
  }

  async addDelayedAppointment(data: { userId: string; email: string; name: string }, delayMs: number) {
    return this.addAppointmentJob(data, { delay: delayMs });
  }

  async addRepeatableDigest() {
    return this.appointmentQueue.add(JobNames.APPOINTMENT_DIGEST, {}, { repeat: { cron: '0 9 * * *' } });
  }
}

⚙️ Appointment Processor (WorkerHost) with DLQ routing

Implements work in process(job).
Emits worker events (waiting, active, stalled, paused, resumed, drained).
On final failure (attempts exhausted), it adds a job to DLQ with a small payload.

// src/modules/queue/processors/appointment.processor.ts
import { Injectable, Logger } from '@nestjs/common';
import { Processor, WorkerHost, OnWorkerEvent, InjectQueue } from '@nestjs/bullmq';
import { Job, Queue } from 'bullmq';
import { JobNames, QueueNames } from '../queue.constants';

@Processor(QueueNames.APPOINTMENT)
@Injectable()
export class AppointmentProcessor extends WorkerHost {
  private readonly logger = new Logger(AppointmentProcessor.name);

  constructor(
    @InjectQueue(QueueNames.APPOINTMENT_DLQ) private readonly dlq: Queue,
  ) {
    super();
  }

  // Main work
  async process(job: Job<{ userId: string; email: string; name: string }>): Promise<any> {
    this.logger.log(`Processing job ${job.id} for ${job.data.email}`);
    await job.updateProgress(10);

    // Simulate sending a welcome email (replace with real provider call)
    await simulateSendWelcomeEmail(job.data.email, job.data.name);

    await job.updateProgress(100);
    this.logger.log(`Completed job ${job.id}`);
    return { ok: true };
  }

  // Worker events (observability)
  @OnWorkerEvent('waiting')
  onWaiting(jobId: string) {
    this.logger.log(`Event: waiting jobId=${jobId}`);
  }

  @OnWorkerEvent('active')
  onActive(job: Job) {
    this.logger.log(`Event: active jobId=${job?.id}`);
  }

  @OnWorkerEvent('stalled')
  onStalled(job: Job) {
    this.logger.warn(`Event: stalled jobId=${job?.id}`);
  }

  @OnWorkerEvent('paused')
  onPaused() {
    this.logger.warn('Event: worker paused');
  }

  @OnWorkerEvent('resumed')
  onResumed() {
    this.logger.log('Event: worker resumed');
  }

  @OnWorkerEvent('drained')
  onDrained() {
    this.logger.log('Event: worker drained');
  }

  @OnWorkerEvent('failed')
  async onFailed(job: Job | undefined, err: Error) {
    const id = job?.id ?? 'unknown';
    const attemptsMade = job?.attemptsMade ?? 0;
    const maxAttempts = job?.opts?.attempts ?? 1; // we set attempts on each job in the producer

    this.logger.error(`Event: failed jobId=${id} attemptsMade=${attemptsMade}/${maxAttempts} err=${err?.message}`);

    // **DLQ routing**: only when attempts exhausted (poison job)
    if (job && attemptsMade >= maxAttempts) {
      await this.routeToDLQ(job, err);
    }
  }

  private async routeToDLQ(job: Job, err: Error) {
    try {
      await this.dlq.add(JobNames.DEAD_LETTER, {
        originalJobId: job.id,
        name: job.name,
        queue: QueueNames.APPOINTMENT,
        data: job.data,                // keep small—prefer references/IDs in real apps
        reason: err?.message ?? 'unknown',
        attemptsMade: job.attemptsMade,
        maxAttempts: job.opts.attempts ?? null,
        failedAt: new Date().toISOString(),
      }, { removeOnComplete: true });

      this.logger.warn(`Job ${job.id} moved to DLQ (${QueueNames.APPOINTMENT_DLQ})`);
    } catch (dlqErr) {
      this.logger.error(`Failed to push job ${job.id} to DLQ: ${dlqErr?.message}`);
    }
  }
}

// Simulated email sender (10% transient failure to demo retries/backoff)
async function simulateSendWelcomeEmail(email: string, name: string) {
  await new Promise((r) => setTimeout(r, 200));
  if (!email) throw new Error('Invalid email');
  if (Math.random() < 0.1) throw new Error('Transient mail provider error');
  return true;
}

🧰 DLQ Processor (WorkerHost)

Handles jobs placed in DLQ (e.g., persist to DB, alert ops).

// src/modules/queue/processors/dlq.processor.ts
import { Injectable, Logger } from '@nestjs/common';
import { Processor, WorkerHost } from '@nestjs/bullmq';
import { Job } from 'bullmq';
import { JobNames, QueueNames } from '../queue.constants';

@Processor(QueueNames.APPOINTMENT_DLQ)
@Injectable()
export class DlqProcessor extends WorkerHost {
  private readonly logger = new Logger(DlqProcessor.name);

  async process(job: Job<{
    originalJobId: string;
    name: string;
    queue: string;
    data: any;
    reason: string;
    attemptsMade: number;
    maxAttempts: number | null;
    failedAt: string;
  }>) {
    this.logger.warn(
      `DLQ: jobId=${job.id} original=${job.data.originalJobId} reason="${job.data.reason}"`,
    );

    // TODO:
    // - Persist to DB for triage
    // - Create an incident/alert
    // - Optionally implement auto-retry logic back to main queue after inspection

    return { handled: true };
  }
}

👤 Auth Module — Signup API (Producer outside queue module)

src/modules/auth/auth.controller.ts

import { Body, Controller, Post } from '@nestjs/common';
import { IsEmail, IsString, MinLength } from 'class-validator';
import { AuthService } from './auth.service';

class SignupDto {
  @IsEmail()
  email: string;

  @IsString()
  @MinLength(3)
  name: string;

  @IsString()
  @MinLength(6)
  password: string;
}

@Controller('auth')
export class AuthController {
  constructor(private readonly authService: AuthService) {}

  @Post('signup')
  async signup(@Body() dto: SignupDto) {
    const res = await this.authService.signup(dto);
    return { ok: true, queuedJobId: res.jobId };
  }
}

src/modules/auth/auth.service.ts

import { Injectable } from '@nestjs/common';
import { QueueService } from '../queue/queue.service';

@Injectable()
export class AuthService {
  constructor(private readonly queueService: QueueService) {}

  async signup(dto: { email: string; name: string; password: string }) {
    // 1) Create user in DB (omitted)
    const userId = 'generated-user-id';

    // 2) Enqueue welcome/appointment job (idempotent by jobId)
    const job = await this.queueService.addAppointmentJob({
      userId,
      email: dto.email,
      name: dto.name,
    });

    return { jobId: job.id };
  }
}

src/modules/auth/auth.module.ts

import { Module } from '@nestjs/common';
import { AuthController } from './auth.controller';
import { AuthService } from './auth.service';
import { QueueModule } from '../queue/queue.module';

@Module({
  imports: [QueueModule],
  controllers: [AuthController],
  providers: [AuthService],
})
export class AuthModule {}

🌐 Bull Board at /bull-board (src/main.ts)

Mounts Bull Board using the queue instances from Nest DI.

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { ValidationPipe } from '@nestjs/common';
import { createBullBoard } from '@bull-board/api';
import { BullMQAdapter } from '@bull-board/api/bullMQAdapter';
import { ExpressAdapter } from '@bull-board/express';
import { getQueueToken } from '@nestjs/bullmq';
import { QueueNames } from './modules/queue/queue.constants';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  app.useGlobalPipes(new ValidationPipe({ whitelist: true, transform: true }));

  const serverAdapter = new ExpressAdapter();
  serverAdapter.setBasePath('/bull-board');

  // Get queues from Nest DI
  const appointmentQueue = app.get(getQueueToken(QueueNames.APPOINTMENT));
  const dlqQueue = app.get(getQueueToken(QueueNames.APPOINTMENT_DLQ));

  createBullBoard({
    queues: [new BullMQAdapter(appointmentQueue), new BullMQAdapter(dlqQueue)],
    serverAdapter,
  });

  app.use('/bull-board', serverAdapter.getRouter());

  const port = process.env.PORT || 3000;
  await app.listen(port);
  console.log(`App:        http://localhost:${port}`);
  console.log(`Bull Board: http://localhost:${port}/bull-board`);
}
bootstrap();

📘 Topics & Best Practices

🔁 Retries & Backoff

Set attempts and backoff on each job (or via defaultJobOptions).
Throw in process(job) to trigger retry.
Exponential backoff is safer for flaky downstream providers.

⏱ Delayed & Repeatable Jobs

Delay: queue.add(name, data, { delay: ms }).
Repeat: queue.add(name, data, { repeat: { cron: '0 9 * * *' }}).
Remove a repeatable with queue.removeRepeatable(...) when no longer needed.

⚠️ Error Handling (Throw vs Fail, DLQ, Poison Jobs)

Throw inside process() → failed attempt recorded, retried until attempts exhausted.
On final failure, route to DLQ (see AppointmentProcessor.onFailed()), store minimal payload (IDs/refs) and reason.
Poison jobs should end up in DLQ; review and requeue manually (or via an ops flow).

🔐 Idempotency & Dedup

Use jobId like welcome:${email} to dedupe enqueues.
Make processing idempotent (e.g., DB flag “email_sent” checked before sending).

🚦 Rate Limiting & Throughput Tuning

Prefer horizontal scaling (multiple worker processes/instances).
Keep concurrency reasonable to avoid saturating downstream services.
For provider rate limits, add throttling middleware or use Worker-level limiter (if you run custom workers).

📡 Worker/Queue Events

Handled via @OnWorkerEvent: waiting, active, stalled, paused, resumed, progress, completed, failed, drained.
Use these for metrics, logs, alerts, and DLQ routing.

🧹 Cleanup Policies

Use removeOnComplete / removeOnFail to avoid Redis bloat.
Periodically clean old jobs (Bull Board or code).

🛑 Graceful Shutdown

Nest + @nestjs/bullmq gracefully tears down workers.
Avoid SIGKILL; let in-flight jobs finish.
If you hold raw Queue instances elsewhere, close them in onModuleDestroy.

🔭 Observability

Bull Board for inspection.
Log worker events.
Export metrics (counts, durations, failure ratio) to Prometheus/Grafana.

🔎 Accessing Bull Board

Start the app: npm run start:dev
Visit: http://localhost:3000/bull-board
Inspect queues, retry/remove failed jobs, watch job progress.

Bull: The Robust Job and Queue System.

ronak navadia — Mon, 25 Aug 2025 17:16:10 +0000

Now that we understand what queues are and why they’re important, let’s talk about Bull — one of the most widely used job queue libraries in Node.js.

If your app needs to send emails, process payments, generate reports, or handle background tasks, Bull makes your life a lot easier.

What is Bull?

Bull is a job and message queue built for Node.js. It uses Redis under the hood, which makes it:

Fast → thanks to in-memory operations.
Durable → jobs won’t just vanish if something crashes.
Scalable → you can run multiple workers and share the load.

Out of the box, Bull gives you advanced features like retries, delayed jobs, repeat jobs, and monitoring.

Think of it as your app’s task manager — organizing, scheduling, and making sure things get done, even if your app crashes.

How Bull Works (Architecture)

Bull revolves around a few simple building blocks:

Queue → Stores jobs in Redis and manages them.
Job → A unit of work (like “send an email” or “generate a PDF”).
Worker (Processor) → A function that picks up jobs and executes them.
Events → Notifications when a job completes, fails, or stalls.

Redis guarantees atomic operations. That means jobs won’t get lost or processed twice — super important in production.

Job Lifecycle

Every job in Bull goes through a series of states:

Created → You add the job (producer creates it).
Waiting → Job sits in the queue until a worker is ready.
Active → A worker picks it up and starts processing.
Completed or Failed → Job finishes successfully or errors out.
Stalled → If a worker crashes midway, Bull puts the job back into the queue.

👉 This lifecycle ensures that no jobs “silently disappear.”

Core Features (Why Developers Love Bull)

Bull comes packed with features that make it production-ready:

Retries & Backoff → Failed jobs retry automatically. You can add backoff strategies like exponential delay to avoid hammering external APIs.
Delayed Jobs → Schedule jobs in the future (like sending a reminder 24 hours later).
Repeatable Jobs → Run jobs on a cron schedule (like daily reports).
Priorities → Important jobs (like fraud alerts) can skip the line.
Rate Limiting → Control how many jobs run per second to avoid hitting API rate limits.
Concurrency → Let one worker handle multiple jobs at once.
Progress Updates → Track job progress (like 30% done on video encoding).
Auto Cleanup → Prevent Redis from filling up by removing completed jobs.

💡 Pro Tip: Combine retries + dead-letter queues to make your system both resilient and debuggable.

Operational Superpowers

Bull takes care of a lot of tricky details for you:

Detects stalled jobs automatically and requeues them.
Uses Lua scripts to guarantee atomic operations in Redis.
Ensures jobs survive app restarts or worker crashes.

That’s why many devs trust Bull for critical workloads.

Configurations Made Easy

You can fine-tune Bull at both the queue and job level:

Job options → attempts, backoff, priority, delay.
Queue options → FIFO vs. LIFO processing, rate limits, global defaults.

This flexibility means you can have one queue for “emails” that retries 5 times, and another for “payments” that retries 10 times with exponential backoff.

Events & Hooks

Bull emits useful events so you can plug in your own logic:

completed → Job finished successfully.
failed → Job failed (after retries).
progress → Worker reported progress.
stalled → Worker died midway.
drained → No jobs left in the queue.

👉 These hooks are perfect for logging, monitoring, or triggering other workflows.

Monitoring Made Simple

Observability is key. Bull integrates with tools like Bull Board or Arena, giving you a neat dashboard where you can:

View pending, active, completed, and failed jobs.
Retry or remove failed jobs with one click.
Monitor worker speed and error rates.

For bigger setups, you can also send metrics to Prometheus + Grafana for full-blown monitoring.

Scaling Bull

Scaling is straightforward:

Run multiple workers consuming the same queue.
Split jobs across multiple queues (emails, payments, media processing).
Auto-scale workers in the cloud depending on queue length.

This makes Bull suitable for both small apps and enterprise-scale systems.

Best Practices for Job Data

Keep job payloads lightweight. Don’t put raw files in Redis — store them in S3 or a DB and pass just the reference (like file path or ID).
Add versioning to job data in case your schema changes later.
Validate job data before enqueueing to prevent poison jobs.

Security & Reliability Tips

Make jobs idempotent → running the same job twice shouldn’t cause duplication (e.g., double charges).
Don’t put raw sensitive data (like passwords or PII) into Redis. Use references or encrypted tokens.
Clean up completed jobs regularly to keep Redis memory healthy.

Bull vs Alternatives

BullMQ → The next-gen version of Bull, better TypeScript support, more features.
Agenda → MongoDB-based, good for cron-like scheduling.
Bree → Node.js-native job scheduler, doesn’t need Redis.
SQS / Kafka → Heavy-duty, distributed solutions for very large systems.

👉 For most Node.js apps, Bull hits the sweet spot between power and simplicity.

Common Pitfalls (Watch Out!)

Redis memory bloat if you don’t clean up old jobs.
Infinite retries if misconfigured.
Long-running jobs may block the event loop — use child processes or workers.

Deployment Tips

For production setups:

Use Redis persistence (AOF or RDB snapshots).
Set up Redis with replicas or cluster mode for high availability.
Keep workers in the same region/close to Redis to minimize latency.

✅ With Bull, you don’t just get a queue — you get a full-fledged background job system that’s reliable, scalable, and battle-tested. Whether you’re running a startup project or a large distributed system, Bull can handle it.

Understanding Queueing in Node.js

ronak navadia — Mon, 25 Aug 2025 17:00:06 +0000

Introduction to Queues in Node.js

When you build applications in Node.js, you’ll often deal with tasks that are either time-consuming or don’t need to run immediately. For example, sending confirmation emails, generating reports, or processing uploaded images. If you try to perform such tasks directly inside your API response cycle, your server might become slow and unresponsive.

This is where queueing comes into play. Let’s dive deeper.

Why Queues?

A queue helps in managing workloads efficiently. Instead of handling everything instantly, tasks can be lined up for processing later. This ensures that:

The application doesn’t get overwhelmed during heavy traffic.
Time-consuming jobs don’t block user-facing requests.
The system can recover gracefully from failures.

Think of it like a restaurant: the waiter (producer) takes multiple orders and places them in a queue for the chef (consumer/worker) to prepare.

What is a Queue?

A queue is a First In, First Out (FIFO) data structure. In software, it acts as a middle layer between the producer (who creates jobs) and the consumer (who processes them).

For example, if five users upload files at the same time, all requests can be added to the queue. Workers then pick them one by one (or in parallel) to process without overloading the system.

What is a Job?

A job is a unit of work inside the queue. It contains:

Payload – the actual data required to complete the task.
Metadata – such as job ID, number of retries, scheduled time, or priority.

Example jobs:

An email job may contain the recipient, subject, and body.
A video processing job may contain the file path and encoding settings.

Jobs help break down complex workloads into manageable pieces.

What is a Job Producer?

A producer is the part of the system that creates and submits jobs to the queue. This is usually your application server or API layer.

Example: When a user signs up, your API endpoint can immediately return a response while also creating a “send welcome email” job and placing it into the queue. The actual email sending will happen later by a worker.

This ensures responsiveness and prevents delays in user interactions.

What is a Worker (Job Consumer)?

A worker (or consumer) is a dedicated process that continuously listens to the queue and executes jobs as they arrive.

Multiple workers can run in parallel to handle higher loads.
Workers can retry failed jobs or handle errors gracefully.
They’re often separated from the main server to avoid blocking real-time traffic.

Example: An “email worker” continuously checks the queue for email jobs and sends them using an SMTP service.

Why Use Queues?

Queues are useful for several reasons:

Reliability: If a worker crashes, the job remains safe in the queue and will be retried.
Scalability: You can add more workers as your workload increases.
Performance: Heavy tasks don’t slow down your main application.
Flexibility: Producers and consumers don’t need to know about each other’s implementation.

Without queues, you risk slower responses, timeouts, and lost jobs during failures.

Common Use Cases

Queues are everywhere in modern applications:

Sending bulk notifications (emails, SMS, push messages).
Image/video processing in the background.
Handling third-party API requests that may be slow.
Managing scheduled tasks like billing, reports, or reminders.

Basically, anything that can be done later should go into a queue.

Core Queue Features (In Depth)

Modern queue systems provide powerful features that go beyond simple FIFO processing:

1. Delayed Jobs

Execute after a specific time.
Example: “Send password reset reminder email 30 minutes later.”

2. Scheduled/Recurring Jobs

Run jobs at fixed intervals using cron expressions.
Example: “Generate sales reports every midnight.”

3. Priority Jobs

High-priority jobs jump ahead of others in the queue.
Example: “Fraud alerts” take precedence over “weekly newsletters.”

4. Concurrency

Multiple jobs can be processed simultaneously by the same worker.
Example: A worker handling 10 image-processing jobs in parallel.

5. Rate Limiting & Throttling

Restrict how many jobs run in a given time frame.
Example: Call a third-party API only 100 times per minute to avoid rate-limit errors.

6. Job Progress & Events

Workers can update job status (e.g., 30% done).
Example: Tracking progress of a video upload and transcoding.

These features make queues versatile and capable of handling real-world workloads gracefully.

Reliability Patterns (Making Queues Robust)

Queues must be resilient to failures, otherwise they create more problems than they solve. Some common patterns are:

1. Acknowledgements (ACKs)

A job is marked complete only when a worker explicitly acknowledges success.
Prevents job loss if a worker crashes mid-task.

2. Retries with Backoff

Failed jobs are retried automatically with increasing delays.
Example: Retry after 5s, then 15s, then 1 minute, etc.

3. Dead-letter Queues (DLQ)

Jobs that fail after multiple retries are moved to a separate queue.
Example: “Invalid email addresses” jobs end up in DLQ for manual review.

4. Idempotency

Jobs are designed so running them multiple times doesn’t cause duplicate effects.
Example: A billing system checks if a payment is already processed before charging again.

5. Job Deduplication

Prevents enqueueing the same job multiple times.
Example: Avoid sending multiple welcome emails if a user triggers signup twice.

These patterns ensure that even in the face of crashes, network errors, or bad data, your system stays consistent.

Observability (Monitoring Queues)

A queue without monitoring is like flying blind. You should always track:

Job States – pending, active, completed, failed.
Worker Metrics – how fast workers process jobs, average latency.
Failure Rates – recurring errors often signal bugs or bad data.
Throughput – number of jobs processed per second/minute.

Tools like Bull Board, Arena, or custom dashboards give real-time visibility into job lifecycles. Alerts and logs can notify you of unusual delays or spikes in failures.

Gotchas (Challenges with Queues)

While queues are powerful, they come with challenges:

1. Poison Jobs

A job with invalid data keeps failing forever.
Fix: Use dead-letter queues and validation before enqueueing.

2. Duplicate Jobs

Same work gets queued multiple times accidentally.
Fix: Add deduplication keys or job uniqueness constraints.

3. Ordering vs Parallelism

Running jobs in parallel can cause them to complete out of order.
Fix: Use priority queues or partitioning for strict ordering needs.

4. Resource Starvation

A few heavy jobs might block lightweight ones.
Fix: Split queues by type (e.g., email queue vs. video processing queue).

When Not to Use Queues

Queues aren’t always necessary:

For very small apps, queues add unnecessary complexity.
For real-time tasks (like “user is typing” indicators in chat), queues introduce unwanted delays.

Use queues when they truly solve a scalability or reliability issue.

If you're interested in learning more about bulls and how they can be used in various contexts like task queues, job management, and real-time systems, check out our detailed post on Bull: The Robust Job and Queue System.

Setting Up ESLint and Prettier in a Node.js Project: A Complete Guide.

ronak navadia — Sun, 11 May 2025 17:30:10 +0000

Have you ever worked on a project where quotes were sometimes single, sometimes double? Where some files used semicolons and others didn't? Or worse, have you ever spent hours debugging because you used a variable that wasn't imported? If you answer yes to any of these questions, this guide is for you!

In this blog post, I'll walk you through setting up ESLint and Prettier in your Node.js project in simple, easy-to-follow steps. I'll explain what each tool does, why you need it, and how to configure it properly - all in plain language that anyone can understand.

What are ESLint and Prettier?

Before diving into the setup, let's understand what these tools do:

ESLint is like a code detective. It checks your code for potential errors and enforces coding style rules. For example, it can warn you if you're using a variable that hasn't been defined, or if you're not following the project's coding standards.

Prettier is like an automatic code formatter. It takes your code, regardless of how it's formatted, and makes it look consistent according to the rules you set. It handles things like indentation, spacing, and line breaks.

Together, these tools help make your code more consistent, easier to read, and less prone to bugs.

Why Do I Need These Tools?

You might be wondering, "Why bother with all this setup?" Here are a few real-world problems these tools solve:

Catching errors before they happen: ESLint can spot potential bugs before you even run your code.
Consistency across projects: No more debating whether to use tabs or spaces, or single or double quotes.
Saving time: Instead of manually formatting your code, Prettier does it for you instantly.
Improved collaboration: Everyone on the team follows the same coding standards automatically.

Step-by-Step Setup Guide

Let's break down the process into simple steps:

Step 1: Install the Required Packages

First, we need to install ESLint, Prettier, and some helper packages:

yarn add -D eslint prettier eslint-config-prettier eslint-plugin-import globals

What each package does:

eslint: The main linting tool that checks your code for problems
prettier: The code formatter that makes your code look consistent
eslint-config-prettier: Turns off ESLint rules that might conflict with Prettier
eslint-plugin-import: Helps ESLint check import/export statements (super helpful!)
globals: Provides global variables for different environments (like Node.js)

Step 2: Create Configuration Files

Next, we need to create configuration files for ESLint and Prettier.

**ESLint Configuration (eslint.config.js)

This file tells ESLint how to check your code:

import eslint from '@eslint/js';
import prettierConfig from 'eslint-config-prettier';
import importPlugin from 'eslint-plugin-import';
import globals from 'globals';

export default [
  eslint.configs.recommended,
  {
    files: ['**/*.js'],
    languageOptions: {
      ecmaVersion: 'latest',
      sourceType: 'module',
      globals: {
        ...globals.node,
        ...globals.es2021
      }
    },
    plugins: {
      import: importPlugin,
    },
    rules: {
      // Enforce import/export syntax
      'import/no-unresolved': 'error',
      'import/named': 'error',
      'import/default': 'error',
      'import/export': 'error',

      // Prevent usage of variables before they are defined
      'no-use-before-define': 'error',

      // Enforce consistent quote style (single quotes)
      'quotes': ['error', 'single', { 'avoidEscape': true }],

      // Require semicolons
      'semi': ['error', 'always'],

      // Disallow unused variables
      'no-unused-vars': ['error', { 'argsIgnorePattern': '^_' }],
    },
  },
  prettierConfig,
];

**Key points about this config:

It's set up for modern JavaScript using ES modules
It adds specific rules to catch missing imports
It enforces style rules like single quotes and semicolons
It includes Node.js globals, so you won't get errors about console or process

**Prettier Configuration (.prettierrc)
This file tells Prettier how to format your code:

{
  "singleQuote": true,
  "trailingComma": "es5",
  "semi": true,
  "tabWidth": 2,
  "printWidth": 100,
  "bracketSpacing": true,
  "arrowParens": "avoid"
}

This configuration:

Uses single quotes instead of double quotes
Adds semicolons at the end of statements
Sets tab width to 2 spaces
Limits line length to 100 characters
Adds spaces inside brackets
Avoids parentheses around single arrow function parameters

Step 3: Set Up Editor Integration

To make ESLint and Prettier work seamlessly with your code editor, we need to add some editor-specific settings.

Create a .vscode folder in your project root, then add a settings.json file inside it:

{
  "editor.formatOnSave": true,
  "editor.defaultFormatter": "esbenp.prettier-vscode",
  "editor.codeActionsOnSave": {
    "source.fixAll.eslint": true
  },
  "eslint.validate": ["javascript"],
  "prettier.singleQuote": true
}

This tells VS Code to:

Format your code automatically when you save
Use Prettier as the default formatter
Fix ESLint issues automatically when you save
Apply ESLint to JavaScript files
Use single quotes with Prettier

**For Cursor:
Since Cursor is built on VS Code, the same settings work there too! Just create the same .vscode folder and settings.json file.

Step 4: Install Editor Extensions

To make everything work smoothly, install these extensions in your editor:

**For VS Code or Cursor:

ESLint (by Microsoft)
Prettier - Code formatter (by Prettier)

You can install them from the Extensions marketplace in your editor.

Step 5: Add NPM/Yarn Scripts

Add these scripts to your package.json file:

{
  "scripts": {
    "lint": "eslint . --ext .js",
    "lint:fix": "eslint . --ext .js --fix",
    "format": "prettier --write ."
  }
}

These scripts let you:

Check your code for issues (yarn lint)
Fix linting issues automatically (yarn lint:fix)
Format your entire project (yarn format)

Testing Your Setup

To make sure everything is working correctly:

Create a test file with some issues:

// This should show an error (missing import)
const result = someFunction();

// This should be converted to single quotes
const message = "This should be converted to single quotes";

// This should show an unused variable warning
const unusedVar = 42;

Save the file and check if:

ESLint highlights the errors
The quotes are automatically converted to single quotes
The indentation and formatting are fixed

Conclusion

Setting up ESLint and Prettier might seem like extra work at first, but it pays off enormously in the long run. Your code will be more consistent, you'll catch errors earlier, and you'll spend less time on formatting and style issues.

The configuration we've set up in this guide is particularly useful for Node.js projects using ES modules, but you can easily adapt it for other types of projects by adjusting the rules and plugins.

Happy coding!

P.S. If you ever need to reference this setup again, bookmark this blog post or save the configuration files in a GitHub repository for easy access!

The Complete Beginner’s Guide to Git: Setup, SSH, Tokens, Remotes, and More.

ronak navadia — Sun, 11 May 2025 15:45:04 +0000

If you’re learning development, sooner or later you'll hear everyone saying “use Git!”
But no one really stops to explain it like a normal human being.

Let's fix that.

This post is your no-jargon, no-overcomplicated guide to Git.
We’ll start with what Git is, why you need it, and how to set it up properly — even covering the small details like SSH vs HTTPS vs Tokens.

What is Git?(Magic Time Machine for Your Code)

Imagine you're writing a story and want to save versions as you go—so you can look back or undo changes. Git does that for coding projects.

🔧 Definition:
Git is a version control system. It tracks changes, helps you work with others, and lets you go back to earlier versions of your files.

🧠 Why Use Git?

Track changes over time
Collaborate without conflicts
Experiment safely in branches
Backup your work using platforms like GitHub

GitHub

📘 Example:
On Monday: <h1>Hello World</h1>
On Tuesday: <h1>Welcome to My Website</h1>
With Git, you can easily switch between these versions and see what changed.

🖥️ How It Works:

Start with git init
Save changes using git commit
Upload to GitHub using git push

What is GitHub, GitLab, Bitbucket? (Code Banks)

Git is only local by default — like saving your essay on your personal laptop.

But what if:

You want a backup in case your laptop crashes?
You want to share it with friends across the world?

You need an online bank for your code like GitHub, GitLab, Bitbucket

Think of GitHub / GitLab / Bitbucket as Google Drive for Code.

GitHub – The most widely used, great for open source and teams. Features include issues, pull requests, and GitHub Actions for automation.
GitLab – Offers Git hosting plus built-in DevOps tools like CI/CD. Great for teams wanting an all-in-one solution.
Bitbucket – Made by Atlassian (creators of Jira), it’s ideal for teams already using Jira or Trello. Includes Bitbucket Pipelines for CI/CD.

How to Set Up Git on Your Local Machine (Step-by-Step)

🖥️ Step 1: Install Git

Windows: Download from git-scm.com and run the installer (default settings are fine). Open Git Bash after install.
macOS: Run git in Terminal—macOS will prompt you to install Xcode CLI tools.
Linux:

sudo apt update  
sudo apt install git

🧪 Check Git Version

git --version

⚙️ Step 2: Set Your Identity
Tell Git who you are:

git config --global user.name "Your Name"  
git config --global user.email "you@example.com"

Check settings with:

git config --list

📁 Step 3: Initialize Git
In your project folder:

cd your/project/path  
git init

📝 Step 4: Make Your First Commit
Create a file (e.g., index.html), then run:

git add index.html  
git commit -m "Initial commit"

✅ Your project is now tracked by Git!

🔍 Optional: Local Git Config vs Global Git Config

--global: Applies settings to all projects on your computer.
(No --global): Applies settings only to the current project (inside that .git folder).

git config user.email "another@example.com"

Only applies to the current project.

Understanding What Remote and Origin in Git is.

🌐 What’s a Remote?
When you use Git on your computer, everything is local. But to share or back up your project online (like on GitHub), you need to connect it to a remote.

🛰️ Remote = Online version of your project
When you run:

git push

You’re saying: “Send my changes to the remote.”

When you run:

git pull

You’re saying: “Get the latest changes from the remote.”

🔖 What is origin?
When you first connect your local project to a remote, Git gives it a default name:
origin = nickname for your main remote (you can rename it, but "origin" is the standard).

🧪 Example: Connect to GitHub
If your GitHub repo is:
https://github.com/yourname/myproject.git

Connect it with:

git remote add origin https://github.com/yourname/myproject.git

Now push your code:

git push -u origin main

🔍 View Remotes

git remote -v

🔄 Multiple Remotes? No problem.
You can add more:

git remote add github https://github.com/you/repo.git  
git remote add gitlab https://gitlab.com/you/repo.git

Push to each one:

git push github main  
git push gitlab main

Understanding the Difference Between HTTPS and SSH URLs in Git

When connecting your local Git project to GitHub (or similar), you'll see two types of URLs:

HTTPS: https://github.com/yourname/repo.git
SSH: git@github.com:yourname/repo.git

Both work, but they connect differently.

🔐 HTTPS

Uses your username + token to authenticate
Easy to set up—no special tools or keys
But you might have to enter your token often (unless cached)
GitHub doesn’t allow plain passwords anymore

🛡️ SSH

Uses a public/private key pair (no password after setup)
Takes a bit more time to set up
More secure and seamless for daily use

What is SSH? And How to Set It Up for Git (Step-by-Step)

🔐 What is SSH (Simple Version)
SSH (Secure Shell) is a safe way for your computer to connect to services like GitHub without needing to type your password every time.

Instead of logging in manually, you use:

Public Key → Added to GitHub/GitLab/etc.
Private Key → Stays on your computer and proves your identity automatically

It's like using a digital key to unlock the door without knocking.

🛠️ Set Up SSH for Git (Quick Steps)
✅ 1. Check for existing keys:

Windows (Git Bash):

ls -al ~/.ssh

Windows (PowerShell):

Get-ChildItem -Path $env:USERPROFILE\.ssh

Windows (CMD):

dir %USERPROFILE%\.ssh

macOS / Linux:

ls -al ~/.ssh

🔐 2. Generate a new key:

ssh-keygen -t ed25519 -C "your_email@example.com"

Press Enter through prompts. It creates two files:

Private key: ~/.ssh/id_ed25519 (keep safe)
Public key: ~/.ssh/id_ed25519.pub (share)

🚀 3. Add your key to the SSH agent:
The SSH agent is a background program that securely holds your SSH key in memory, so you don’t have to re-enter your passphrase every time you use Git. It makes authentication seamless and more secure during your session.

eval "$(ssh-agent -s)"  
ssh-add ~/.ssh/id_ed25519

🌐 4. Add your public key to your Git host:
Show the key:

cat ~/.ssh/id_ed25519.pub

Copy the output and paste it into:

GitHub → Settings → SSH Keys
GitLab → Preferences → SSH Keys
Bitbucket → Personal Settings → SSH Keys

🔍 5. Test your connection:

ssh -T git@github.com

You should see:
“Hi username! You've successfully authenticated...”

Different Ways to Connect Git with Online Services (GitHub, GitLab, Bitbucket)

🔹 Connecting Git Using Username and Password (Only on GitLab and Bitbucket)

🚫 GitHub no longer allows username + password for Git over HTTPS.
✅ This still works on GitLab and Bitbucket (for now), but is considered less secure

How it works:
You clone the repo using the HTTPS URL:

git clone https://gitlab.com/username/project.git

When you push:

git push

Git will prompt you to enter:

Your username
Your password

💡 You can cache these credentials using Git Credential Manager (on Windows/macOS) or helper scripts.

Why avoid it?

Easily forgettable.
Passwords can be compromised.
Many platforms are moving away from this for security reasons.

🔹 b. Connecting Git Using SSH (Recommended Method)
**How it works:
You clone the repo using the SSH URL:

git clone git@gitlab.com:username/project.git

Your SSH key is automatically used to authenticate (no username or password prompts).

✅ Secure and fast once set up.
🔁 Works well for frequent Git users or developers managing many projects.

🔹 c. Connecting Git Using Personal Access Token (PAT)

🔐 GitHub requires a Personal Access Token (PAT) for HTTPS connections.
GitLab and Bitbucket support PATs as an alternative to passwords.

Step-by-Step:

Generate a token from your Git platform:

GitHub: https://github.com/settings/tokens
GitLab: User → Preferences → Access Tokens
Bitbucket: Personal Settings → Access Management

Use HTTPS to clone the repo:

git clone https://github.com/yourusername/project.git

When prompted:

Username: your GitHub/GitLab username
Password: your access token

💡 You can use a credential helper to cache the token so you don’t retype it every time:

git config --global credential.helper cache

Handling Multiple Git Accounts on the Same Device

If you work with more than one GitHub, GitLab, or Bitbucket account (e.g., one personal and one work), things can get tricky.

Here’s how to manage multiple accounts without mixing them up.

✅ Method 1: Use SSH with Separate Keys for Each Account
Generate a second SSH key:

ssh-keygen -t ed25519 -C "you@work-email.com" -f ~/.ssh/id_ed25519_work

Add it to the SSH agent:

ssh-add ~/.ssh/id_ed25519_work

Add the public key to your second Git account (e.g., work GitHub/GitLab).

Edit ~/.ssh/config to tell Git which key to use for each domain:

# Personal GitHub
Host github.com-personal
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_ed25519

# Work GitHub
Host github.com-work
    HostName github.com
    User git
    IdentityFile ~/.ssh/id_ed25519_work

Clone using custom hostname:

git clone git@github.com-work:workuser/project.git

In your local repo, set the correct remote:

git remote set-url origin git@github.com-work:workuser/project.git

✅ Method 2: Use HTTPS + Git Credential Manager (Separate Git Identities)
You can also manage multiple accounts by:

Setting per-project Git identities:

git config user.name "Work Name"
git config user.email "work@example.com"

Using Git Credential Manager (built-in on Windows/macOS) to store different credentials per URL.

🧾 Wrapping Up
Getting started with Git and remote repositories might feel a bit technical at first, but once you understand the core concepts—like what Git is, how remotes work, and the difference between HTTPS and SSH—it becomes second nature.

Whether you're a solo developer or working in a team, properly setting up Git and knowing how to securely connect to platforms like GitHub, GitLab, or Bitbucket will save you time and headaches down the road.

Now that you've got the essentials down, you're ready to collaborate, contribute, and code with confidence.

Happy committing! 🚀

Getting Started with Node.js: Understanding Node, npm, nvm, and npx (and How to Install Node.js)

ronak navadia — Sun, 20 Apr 2025 10:30:58 +0000

When you first hear about Node.js, it's easy to feel overwhelmed with all the extra terms like npm, nvm, and npx flying around.
Don't worry — by the end of this post, you'll not only know what each of them means, but you'll also know why they exist and how they fit together.

Let's break it down, in the simplest way possible.

What is Node (or Node.js)?

Imagine you have a sports car (JavaScript) — but it's stuck inside a racetrack (the browser).
You can only drive it inside that track.

Now, what if you wanted to drive it on highways, through cities, anywhere you like?

Node.js is like building a powerful engine that lets JavaScript drive outside the racetrack — on the entire internet! 🚗💨

In technical terms:

Node.js allows JavaScript to run outside a web browser.
It’s built on Google Chrome’s V8 engine, which makes it very fast.
With Node.js, you can create servers, tools, APIs, scripts, and even desktop apps — all using JavaScript.

In short: Node.js takes JavaScript from the browser to the real world.

What is npm?

Alright, now that you have your sports car (JavaScript with Node.js), you realize:

"I need a GPS."
"I want better tires."
"I need fancy headlights."

Are you going to build all of that from scratch?
Of course not!
You’ll go to a shop and buy those accessories.

npm is that shop 🛒 — but for code.

npm (Node Package Manager) is a marketplace where developers share reusable pieces of code (called "packages").
You can install these packages into your project in seconds instead of building everything yourself.

Example:
Want to send emails in your app? Instead of coding it manually, you can just install an email package:

npm install nodemailer

And it's ready to use!
Over 2 million packages are available on npm — everything you can imagine.

In short: npm is the world's largest supermarket for ready-made JavaScript tools.

What is nvm?

Imagine now that you have two cars 🚗:

One is for racing.
One is for everyday driving.

You can’t use racing tires on city roads — you need different setups depending on the situation.

Similarly, different projects may need different Node.js versions.
Some old projects work only on Node.js 14, but new ones need Node.js 20.

nvm (Node Version Manager) is like your car garage — it lets you easily switch between different Node engines whenever needed.

Using nvm, you can:

Install multiple Node versions on your system.
Switch between them instantly.
Test if your app works on different versions.

nvm install 16   # installs Node.js 16
nvm install 20   # installs Node.js 20
nvm use 16       # switches to Node.js 16
nvm use 20       # switches to Node.js 20

Without nvm, switching versions would be a real headache.

In short: nvm is your personal garage manager for different Node.js engines.

What is npx?

Now, back to our sports car analogy 🏎️ —
Sometimes, you just want to test drive a car without buying it.

npx is like a test drive for npm packages.

npx lets you run any package directly without permanently installing it.
It’s useful when you need a tool just once.

Example:
You want to create a new React app, but you don’t want to install the create-react-app tool permanently.

Just run:

npx create-react-app my-project

It downloads and runs create-react-app temporarily, and then cleans up.

In short: npx lets you borrow a tool for a quick ride without putting it in your garage.

Different Ways to Install Node.js

Now that you understand the players (Node, npm, nvm, npx), let's talk about how you can install Node.js on your machine.

Installing Node.js on Windows
Option 1: Direct Install (Simple but not flexible)

Visit https://nodejs.org/.
Download the LTS (Long Term Support) version.
Run the installer and click "Next" → "Next" → "Install."

It will install both Node.js and npm automatically.

You can check installation:

node -v
npm -v

✅ Good for beginners. ❌ Bad if you later need multiple Node.js versions.

Option 2: Install Node.js using nvm for Windows (Recommended)

Download nvm-setup.exe.
Install nvm (very easy setup).
Use it to install Node.js versions:

nvm install 18.16.0
nvm use 18.16.0

✅ Good for future-proofing (manage different versions). ✅ Easy switching between Node versions. ✅ Cleaner system.

Installing Node.js on Ubuntu (Linux)

Option 1: Install from Ubuntu’s Software Repo

sudo apt update
sudo apt install nodejs npm

⚠️ Warning:

Ubuntu’s repo may give you an outdated version.

Option 2: Install Latest Node.js from NodeSource

curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -
sudo apt install -y nodejs

✅ Gets you the latest stable version. ✅ Comes with npm.

Option 3: Install Using nvm (Best Way)

First, install nvm:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash

Then install Node:

nvm install 18
nvm use 18

✅ Best for flexibility. ✅ Lets you manage Node versions easily.

Quick Commands Checklist After Installation

After installing Node.js, verify everything:

node -v    # See Node.js version
npm -v     # See npm version
npx -v     # See npx version

If these commands work, you’re all set! 🎉

Wrapping Up

Node.js has completely changed how developers use JavaScript —
From front-end only, to building full applications, websites, servers, APIs, and even real-time apps (like chat apps)!

If you're starting out — mastering Node, npm, nvm, and npx will unlock endless possibilities in your development journey 🚀.