Forem: Mayank Singh

LocalStack: The Complete Guide to Running AWS Locally — From Zero to Production-Like Pipelines

Mayank Singh — Sun, 15 Mar 2026 13:32:45 +0000

Why I Wrote This

I spent months fighting with AWS bills, slow feedback loops, and "works on my machine but not on AWS" bugs. Then I discovered LocalStack — a tool that emulates AWS services on your laptop using Docker. It transformed our team's local development: no more shared dev accounts, no more waiting for CloudFormation stacks to deploy just to test a Lambda trigger.

This guide is everything I wish existed when I started. Whether you're setting up LocalStack for the first time or wiring up a multi-service pipeline with Kinesis, Lambda, SQS, and S3, it's all here.

What is LocalStack?
Quick Docker Primer
Installing LocalStack (Mac / Windows / Linux)
Your First LocalStack Container
Core Concepts: Endpoints, Credentials, and Regions
Setting Up Individual AWS Services
- S3 — Object Storage
- SQS — Message Queues
- Kinesis — Event Streaming
- Lambda — Serverless Functions
- IAM — Roles and Policies
- CloudWatch Logs — Monitoring
- SSM Parameter Store — Configuration
Building a Real Pipeline: Kinesis → Lambda → S3
Adding SQS for Decoupled Processing
Monitoring Everything with CloudWatch
AWS SDK Integration (Node.js / TypeScript)
Docker Compose for the Full Stack
Initialization Scripts — Automating Resource Creation
Advanced Patterns
Troubleshooting Common Issues
Free vs Pro — What You Actually Need
Conclusion

1. What is LocalStack?

LocalStack is a cloud service emulator that runs entirely on your local machine inside a Docker container. It provides the same API endpoints as AWS, so your code, CLI commands, and IaC tools (Terraform, CloudFormation, Serverless Framework) work against it without modification.

Why use it?

Zero AWS costs for development and testing
Instant feedback — no waiting for cloud deployments
Offline development — works without internet
Isolated environments — every developer gets their own "AWS"
CI/CD integration — run integration tests against real AWS APIs

What it supports (Community / Free tier):

S3, SQS, SNS, Lambda, Kinesis, DynamoDB, CloudWatch, IAM, SSM, STS, CloudFormation, API Gateway, EventBridge, and many more.

2. Quick Docker Primer

LocalStack runs inside a Docker container. If you're already comfortable with Docker, skip ahead. Otherwise, here's the 60-second version:

Docker packages applications into isolated containers that include everything they need to run. Think of it as a lightweight virtual machine.

# Install Docker Desktop (Mac/Windows) from https://docker.com
# Verify installation
docker --version
docker info

Key concepts you'll need:

Container: A running instance of an image (like a process)
Image: A blueprint for a container (like localstack/localstack:3.8)
Volume: Persistent storage that survives container restarts
Port mapping: 14566:4566 means "host port 14566 maps to container port 4566"
Docker Compose: A YAML file that defines multi-container setups

That's it. Everything else in this guide will explain Docker bits as we go.

3. Installing LocalStack (Mac / Windows / Linux)

Prerequisites

Docker Desktop installed and running (download here)
Verify: docker info should show no errors

macOS (Homebrew — Recommended)

brew install localstack/tap/localstack-cli

macOS / Linux (Python pip)

python3 -m pip install --upgrade localstack

macOS / Linux (Binary Download)

# macOS (Intel)
curl -Lo localstack-cli.tar.gz \
  https://github.com/localstack/localstack-cli/releases/download/v4.14.0/localstack-cli-4.14.0-darwin-amd64-onefile.tar.gz

# Linux (x86-64)
curl -Lo localstack-cli.tar.gz \
  https://github.com/localstack/localstack-cli/releases/download/v4.14.0/localstack-cli-4.14.0-linux-amd64-onefile.tar.gz

sudo tar xvzf localstack-cli.tar.gz -C /usr/local/bin

Windows

Download the binary from the GitHub releases page
Extract the .zip file
Add the extracted folder to your system PATH
Or use Python: python3 -m pip install --upgrade localstack

Install the AWS CLI (Required)

You also need the AWS CLI to interact with LocalStack:

# macOS
brew install awscli

# Or via pip (all platforms)
pip install awscli

# Verify
aws --version

Install `awslocal` (Optional but Recommended)

awslocal is a thin wrapper around aws that automatically sets the endpoint to LocalStack:

pip install awscli-local

# Instead of: aws --endpoint-url=http://localhost:4566 s3 ls
# You can just: awslocal s3 ls

Verify Installation

localstack --version
# Output: 4.14.0 (or similar)

4. Your First LocalStack Container

Quick Start

localstack start -d

The -d flag runs it in the background. Check status:

localstack status services

Or Use Docker Directly

docker run -d \
  --name localstack \
  -p 4566:4566 \
  -e SERVICES=s3,sqs,kinesis,lambda,iam,logs,ssm,events \
  -e AWS_DEFAULT_REGION=us-east-1 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  localstack/localstack:3.8

Health Check

curl http://localhost:4566/_localstack/health | python3 -m json.tool

Expected output:

{
  "services": {
    "s3": "running",
    "sqs": "running",
    "kinesis": "running",
    "lambda": "running",
    ...
  }
}

5. Core Concepts: Endpoints, Credentials, and Regions

The Endpoint

All AWS services are available through a single endpoint: http://localhost:4566

Unlike real AWS where S3 is at s3.amazonaws.com and SQS is at sqs.us-east-1.amazonaws.com, LocalStack serves everything from one port.

Credentials

LocalStack doesn't validate credentials, but the AWS CLI and SDKs still require them. Use dummy values:

export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_DEFAULT_REGION=us-east-1

Account ID

LocalStack uses 000000000000 as the default account ID. You'll see this in ARNs:

arn:aws:kinesis:us-east-1:000000000000:stream/my-stream
arn:aws:sqs:us-east-1:000000000000:my-queue

The `--endpoint-url` Flag

Every aws CLI command needs to point to LocalStack:

aws --endpoint-url=http://localhost:4566 s3 ls

Or use awslocal to skip it:

awslocal s3 ls

6. Setting Up Individual AWS Services

Let's create each service one by one. Later, we'll wire them together.

S3 — Object Storage

# Create a bucket
awslocal s3 mb s3://my-data-bucket

# Upload a file
echo '{"hello": "world"}' > test.json
awslocal s3 cp test.json s3://my-data-bucket/data/test.json

# List bucket contents
awslocal s3 ls s3://my-data-bucket/data/

# Download a file
awslocal s3 cp s3://my-data-bucket/data/test.json downloaded.json

# Add CORS (needed if a frontend will access S3 directly)
awslocal s3api put-bucket-cors --bucket my-data-bucket --cors-configuration '{
  "CORSRules": [{
    "AllowedHeaders": ["*"],
    "AllowedMethods": ["GET", "PUT", "POST", "DELETE", "HEAD"],
    "AllowedOrigins": ["*"],
    "ExposeHeaders": ["ETag"]
  }]
}'

Tip: Always set forcePathStyle: true in your S3 SDK clients when using LocalStack. Without it, the SDK tries virtual-hosted-style URLs (bucket.localhost:4566) which don't resolve.

SQS — Message Queues

# Standard queue
awslocal sqs create-queue --queue-name processing-queue

# FIFO queue (guaranteed ordering + deduplication)
awslocal sqs create-queue \
  --queue-name processing-queue.fifo \
  --attributes '{
    "FifoQueue": "true",
    "ContentBasedDeduplication": "true"
  }'

# Create a Dead Letter Queue (DLQ)
awslocal sqs create-queue --queue-name processing-dlq

# Get DLQ ARN
DLQ_ARN=$(awslocal sqs get-queue-attributes \
  --queue-url http://localhost:4566/000000000000/processing-dlq \
  --attribute-names QueueArn \
  --query 'Attributes.QueueArn' --output text)

# Set DLQ policy on main queue
awslocal sqs set-queue-attributes \
  --queue-url http://localhost:4566/000000000000/processing-queue \
  --attributes "{
    \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"${DLQ_ARN}\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\"
  }"

# Send a message
awslocal sqs send-message \
  --queue-url http://localhost:4566/000000000000/processing-queue \
  --message-body '{"orderId": "12345", "action": "process"}'

# Receive messages
awslocal sqs receive-message \
  --queue-url http://localhost:4566/000000000000/processing-queue

Kinesis — Event Streaming

# Create a stream
awslocal kinesis create-stream \
  --stream-name event-stream \
  --shard-count 1

# Verify
awslocal kinesis describe-stream --stream-name event-stream

# Put a record
awslocal kinesis put-record \
  --stream-name event-stream \
  --partition-key user-123 \
  --data '{"event": "page_view", "userId": "user-123", "page": "/products"}'

# Read records (get shard iterator first)
SHARD_ITERATOR=$(awslocal kinesis get-shard-iterator \
  --stream-name event-stream \
  --shard-id shardId-000000000000 \
  --shard-iterator-type TRIM_HORIZON \
  --query 'ShardIterator' --output text)

awslocal kinesis get-records --shard-iterator "$SHARD_ITERATOR"

Note: Kinesis data is base64-encoded. Decode it with: echo "<data>" | base64 -d

Lambda — Serverless Functions

First, create a simple handler:

// handler.js
exports.handler = async (event) => {
  console.log('Received event:', JSON.stringify(event, null, 2));

  const records = event.Records || [];
  const processed = records.map(record => {
    // Kinesis records have base64-encoded data
    if (record.kinesis) {
      const payload = Buffer.from(record.kinesis.data, 'base64').toString('utf-8');
      console.log('Kinesis payload:', payload);
      return JSON.parse(payload);
    }
    // SQS records have a body string
    if (record.body) {
      console.log('SQS message:', record.body);
      return JSON.parse(record.body);
    }
    return record;
  });

  return {
    statusCode: 200,
    body: JSON.stringify({ processed: processed.length }),
  };
};

Deploy it:

# Zip the handler
zip function.zip handler.js

# Create the function
awslocal lambda create-function \
  --function-name event-processor \
  --runtime nodejs20.x \
  --zip-file fileb://function.zip \
  --handler handler.handler \
  --role arn:aws:iam::000000000000:role/lambda-role \
  --timeout 30 \
  --memory-size 256

# Wait for it to be active
awslocal lambda wait function-active-v2 \
  --function-name event-processor

# Test invoke
awslocal lambda invoke \
  --function-name event-processor \
  --cli-binary-format raw-in-base64-out \
  --payload '{"test": true}' \
  output.json

cat output.json

IAM — Roles and Policies

Lambda needs an execution role:

# Create the role
awslocal iam create-role \
  --role-name lambda-execution-role \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "lambda.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

# Attach basic Lambda execution policy
awslocal iam attach-role-policy \
  --role-name lambda-execution-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

# Add Kinesis read access
awslocal iam put-role-policy \
  --role-name lambda-execution-role \
  --policy-name KinesisReadPolicy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Action": [
        "kinesis:GetRecords",
        "kinesis:GetShardIterator",
        "kinesis:DescribeStream",
        "kinesis:ListShards"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:000000000000:stream/*"
    }]
  }'

# Add S3 write access
awslocal iam put-role-policy \
  --role-name lambda-execution-role \
  --policy-name S3WritePolicy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject"],
      "Resource": "arn:aws:s3:::*/*"
    }]
  }'

Note: LocalStack Community edition doesn't enforce IAM policies. Creating roles and policies is still useful for parity with production and to avoid surprises when deploying to real AWS.

CloudWatch Logs — Monitoring

Lambda functions automatically create log groups in LocalStack:

# List log groups
awslocal logs describe-log-groups

# View log streams for a Lambda function
awslocal logs describe-log-streams \
  --log-group-name /aws/lambda/event-processor

# Fetch recent log events
awslocal logs get-log-events \
  --log-group-name /aws/lambda/event-processor \
  --log-stream-name '<stream-name-from-above>'

# Create a custom log group
awslocal logs create-log-group \
  --log-group-name /custom/my-application

# Tail logs (requires aws logs tail — AWS CLI v2)
aws --endpoint-url=http://localhost:4566 logs tail \
  /aws/lambda/event-processor --follow

SSM Parameter Store — Configuration

Store configuration that your services read at runtime:

# Store parameters
awslocal ssm put-parameter \
  --name "/myapp/local/database/url" \
  --value "postgresql://user:pass@db:5432/mydb" \
  --type String

awslocal ssm put-parameter \
  --name "/myapp/local/redis/url" \
  --value "redis://redis:6379" \
  --type String

awslocal ssm put-parameter \
  --name "/myapp/local/api-key" \
  --value "sk-local-test-key" \
  --type SecureString

# Read a parameter
awslocal ssm get-parameter \
  --name "/myapp/local/database/url" \
  --query 'Parameter.Value' --output text

# List parameters by path
awslocal ssm get-parameters-by-path \
  --path "/myapp/local/" \
  --recursive

7. Building a Real Pipeline: Kinesis → Lambda → S3

Now let's wire everything together into a realistic data pipeline.

Architecture

┌──────────────┐     ┌─────────────┐     ┌──────────────────┐     ┌──────────────┐
│  Application  │────▶│   Kinesis    │────▶│  Lambda Function │────▶│   S3 Bucket   │
│  (Producer)   │     │   Stream     │     │  (Processor)     │     │  (Storage)    │
└──────────────┘     └─────────────┘     └──────────────────┘     └──────────────┘
                                                  │
                                                  ▼
                                          ┌──────────────┐
                                          │  CloudWatch   │
                                          │  Logs         │
                                          └──────────────┘

Step 1: Create the Infrastructure

# 1. Create the Kinesis stream
awslocal kinesis create-stream \
  --stream-name event-pipeline \
  --shard-count 1

# 2. Create the S3 bucket for processed data
awslocal s3 mb s3://processed-events

# 3. Create the IAM role (see Section 6)
awslocal iam create-role \
  --role-name pipeline-lambda-role \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "lambda.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

Step 2: Write the Lambda Handler

This Lambda reads from Kinesis, processes the records, and writes results to S3:

// pipeline-handler.js
const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');

const s3Client = new S3Client({
  region: process.env.AWS_DEFAULT_REGION || 'us-east-1',
  endpoint: process.env.AWS_ENDPOINT || undefined,
  forcePathStyle: true, // Required for LocalStack
});

const BUCKET_NAME = process.env.S3_BUCKET || 'processed-events';

exports.handler = async (event) => {
  console.log(`Processing ${event.Records.length} Kinesis records`);

  const results = [];

  for (const record of event.Records) {
    // Decode Kinesis record (base64 encoded)
    const payload = Buffer.from(record.kinesis.data, 'base64').toString('utf-8');
    const data = JSON.parse(payload);

    console.log('Processing event:', data);

    // --- Your business logic here ---
    const enrichedData = {
      ...data,
      processedAt: new Date().toISOString(),
      partitionKey: record.kinesis.partitionKey,
      sequenceNumber: record.kinesis.sequenceNumber,
      shardId: record.eventSourceARN.split('/').pop(),
    };

    // Write each record to S3
    const key = `events/${data.eventType}/${Date.now()}-${record.kinesis.sequenceNumber}.json`;

    await s3Client.send(new PutObjectCommand({
      Bucket: BUCKET_NAME,
      Key: key,
      Body: JSON.stringify(enrichedData, null, 2),
      ContentType: 'application/json',
    }));

    console.log(`Saved to s3://${BUCKET_NAME}/${key}`);
    results.push(key);
  }

  console.log(`Successfully processed ${results.length} records`);
  return { statusCode: 200, processedKeys: results };
};

Step 3: Deploy and Connect

# Package the Lambda (with dependencies)
# For a real project, you'd use esbuild or webpack. For this demo:
npm init -y
npm install @aws-sdk/client-s3
zip -r function.zip handler.js node_modules/

# Create the Lambda
awslocal lambda create-function \
  --function-name kinesis-to-s3-processor \
  --runtime nodejs20.x \
  --zip-file fileb://function.zip \
  --handler pipeline-handler.handler \
  --role arn:aws:iam::000000000000:role/pipeline-lambda-role \
  --timeout 60 \
  --memory-size 256 \
  --environment "Variables={
    AWS_ENDPOINT=http://localhost:4566,
    S3_BUCKET=processed-events
  }"

# Wait for function to be active
awslocal lambda wait function-active-v2 \
  --function-name kinesis-to-s3-processor

# Create the Event Source Mapping (Kinesis → Lambda)
awslocal lambda create-event-source-mapping \
  --function-name kinesis-to-s3-processor \
  --event-source-arn arn:aws:kinesis:us-east-1:000000000000:stream/event-pipeline \
  --batch-size 10 \
  --starting-position LATEST \
  --maximum-retry-attempts 3

Step 4: Test the Pipeline

# Publish some events to Kinesis
for i in {1..5}; do
  awslocal kinesis put-record \
    --stream-name event-pipeline \
    --partition-key "user-$((RANDOM % 100))" \
    --cli-binary-format raw-in-base64-out \
    --data "{
      \"eventType\": \"page_view\",
      \"userId\": \"user-$((RANDOM % 100))\",
      \"page\": \"/products/$i\",
      \"timestamp\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"
    }"
  echo "Published event $i"
done

# Wait a few seconds for Lambda to process
sleep 5

# Check S3 for processed files
awslocal s3 ls s3://processed-events/events/ --recursive

# Download and inspect a processed file
awslocal s3 cp s3://processed-events/events/page_view/ ./output/ --recursive
cat output/*.json | python3 -m json.tool

Step 5: Check CloudWatch Logs

# List log groups (Lambda auto-creates one)
awslocal logs describe-log-groups

# Get log streams
awslocal logs describe-log-streams \
  --log-group-name /aws/lambda/kinesis-to-s3-processor

# Read the logs
LOG_STREAM=$(awslocal logs describe-log-streams \
  --log-group-name /aws/lambda/kinesis-to-s3-processor \
  --query 'logStreams[0].logStreamName' --output text)

awslocal logs get-log-events \
  --log-group-name /aws/lambda/kinesis-to-s3-processor \
  --log-stream-name "$LOG_STREAM"

8. Adding SQS for Decoupled Processing

Let's extend the pipeline. After saving to S3, the Lambda also pushes a message to SQS for a downstream notification service to pick up.

Extended Architecture

                                              ┌─────────────┐
                                         ┌───▶│  S3 Bucket   │
                                         │    │  (Storage)   │
┌──────────┐    ┌─────────┐    ┌────────┐│    └─────────────┘
│ Producer  │───▶│ Kinesis  │───▶│ Lambda ││
└──────────┘    └─────────┘    └────────┘│    ┌─────────────┐    ┌───────────────┐
                                         └───▶│  SQS Queue   │───▶│ Notification  │
                                              │  (Buffer)    │    │  Service      │
                                              └─────────────┘    └───────────────┘
                                                    │
                                              ┌─────────────┐
                                              │  SQS DLQ     │
                                              │  (Failures)  │
                                              └─────────────┘

Create the Queues

# Notification queue
awslocal sqs create-queue --queue-name notification-queue

# Dead letter queue for failed notifications
awslocal sqs create-queue --queue-name notification-dlq

# Link DLQ to main queue
DLQ_ARN=$(awslocal sqs get-queue-attributes \
  --queue-url http://localhost:4566/000000000000/notification-dlq \
  --attribute-names QueueArn \
  --query 'Attributes.QueueArn' --output text)

awslocal sqs set-queue-attributes \
  --queue-url http://localhost:4566/000000000000/notification-queue \
  --attributes "{
    \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"${DLQ_ARN}\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\"
  }"

Updated Lambda — Write to S3 + SQS

// extended-handler.js
const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');
const { SQSClient, SendMessageBatchCommand } = require('@aws-sdk/client-sqs');

const REGION = process.env.AWS_DEFAULT_REGION || 'us-east-1';
const ENDPOINT = process.env.AWS_ENDPOINT || undefined;

const s3 = new S3Client({
  region: REGION,
  endpoint: ENDPOINT,
  forcePathStyle: true,
});

const sqs = new SQSClient({
  region: REGION,
  endpoint: ENDPOINT,
});

const BUCKET = process.env.S3_BUCKET || 'processed-events';
const QUEUE_URL = process.env.SQS_QUEUE_URL
  || 'http://localhost:4566/000000000000/notification-queue';

exports.handler = async (event) => {
  console.log(`Processing ${event.Records.length} records`);

  const sqsEntries = [];

  for (const [index, record] of event.Records.entries()) {
    const payload = Buffer.from(record.kinesis.data, 'base64').toString('utf-8');
    const data = JSON.parse(payload);

    // 1. Save to S3
    const key = `events/${data.eventType}/${Date.now()}-${index}.json`;
    await s3.send(new PutObjectCommand({
      Bucket: BUCKET,
      Key: key,
      Body: JSON.stringify({ ...data, processedAt: new Date().toISOString() }),
      ContentType: 'application/json',
    }));

    // 2. Queue notification
    sqsEntries.push({
      Id: String(index),
      MessageBody: JSON.stringify({
        type: 'EVENT_PROCESSED',
        s3Key: key,
        eventType: data.eventType,
        userId: data.userId,
        timestamp: new Date().toISOString(),
      }),
    });
  }

  // Send SQS messages in batches of 10 (SQS limit)
  for (let i = 0; i < sqsEntries.length; i += 10) {
    const batch = sqsEntries.slice(i, i + 10);
    await sqs.send(new SendMessageBatchCommand({
      QueueUrl: QUEUE_URL,
      Entries: batch,
    }));
    console.log(`Queued ${batch.length} notification messages`);
  }

  return { statusCode: 200, processed: event.Records.length };
};

Create a Notification Consumer Lambda

// notification-handler.js
exports.handler = async (event) => {
  for (const record of event.Records) {
    const message = JSON.parse(record.body);
    console.log(`Notification: Event "${message.eventType}" processed for user ${message.userId}`);
    console.log(`  S3 location: s3://processed-events/${message.s3Key}`);
    // In production: send email, push notification, webhook, etc.
  }

  return { statusCode: 200, processed: event.Records.length };
};

Deploy and wire it:

# Deploy notification consumer
zip notification.zip notification-handler.js

awslocal lambda create-function \
  --function-name notification-consumer \
  --runtime nodejs20.x \
  --zip-file fileb://notification.zip \
  --handler notification-handler.handler \
  --role arn:aws:iam::000000000000:role/pipeline-lambda-role \
  --timeout 30

# Wire SQS → Lambda
awslocal lambda create-event-source-mapping \
  --function-name notification-consumer \
  --event-source-arn arn:aws:sqs:us-east-1:000000000000:notification-queue \
  --batch-size 10

9. Monitoring Everything with CloudWatch

Viewing Lambda Logs

Every Lambda function automatically logs to CloudWatch:

# List all log groups
awslocal logs describe-log-groups --query 'logGroups[].logGroupName'

# Output:
# [
#   "/aws/lambda/kinesis-to-s3-processor",
#   "/aws/lambda/notification-consumer"
# ]

Create a Simple Log Viewer Script

#!/bin/bash
# view-logs.sh — View logs for any Lambda function

FUNCTION_NAME=${1:-"kinesis-to-s3-processor"}
LOG_GROUP="/aws/lambda/$FUNCTION_NAME"
ENDPOINT="http://localhost:4566"

echo "=== Logs for $FUNCTION_NAME ==="

# Get latest log stream
STREAM=$(aws --endpoint-url=$ENDPOINT logs describe-log-streams \
  --log-group-name "$LOG_GROUP" \
  --order-by LastEventTime \
  --descending \
  --query 'logStreams[0].logStreamName' \
  --output text 2>/dev/null)

if [ "$STREAM" = "None" ] || [ -z "$STREAM" ]; then
  echo "No log streams found."
  exit 0
fi

# Fetch events
aws --endpoint-url=$ENDPOINT logs get-log-events \
  --log-group-name "$LOG_GROUP" \
  --log-stream-name "$STREAM" \
  --query 'events[].message' \
  --output text

Usage:

chmod +x view-logs.sh
./view-logs.sh kinesis-to-s3-processor
./view-logs.sh notification-consumer

Custom CloudWatch Metrics (Advanced)

# Push a custom metric
awslocal cloudwatch put-metric-data \
  --namespace "MyApp/Pipeline" \
  --metric-name "EventsProcessed" \
  --value 42 \
  --unit Count

# Query metrics
awslocal cloudwatch get-metric-statistics \
  --namespace "MyApp/Pipeline" \
  --metric-name "EventsProcessed" \
  --start-time "$(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --period 300 \
  --statistics Sum

10. AWS SDK Integration (Node.js / TypeScript)

In a real application, you interact with LocalStack through the AWS SDK — not the CLI. Here's how to set up clients that work against both LocalStack and production AWS.

Pattern: Environment-Aware Client Factory

// aws-clients.ts
import { S3Client, S3ClientConfig } from '@aws-sdk/client-s3';
import { SQSClient } from '@aws-sdk/client-sqs';
import { KinesisClient } from '@aws-sdk/client-kinesis';
import { LambdaClient } from '@aws-sdk/client-lambda';
import { CloudWatchLogsClient } from '@aws-sdk/client-cloudwatch-logs';

const REGION = process.env.AWS_DEFAULT_REGION || 'us-east-1';
const ENDPOINT = process.env.AWS_ENDPOINT; // Set only for LocalStack

// S3 — Note forcePathStyle is critical for LocalStack
export function createS3Client(): S3Client {
  const config: S3ClientConfig = { region: REGION };
  if (ENDPOINT) {
    config.endpoint = ENDPOINT;
    config.forcePathStyle = true; // ← Required for LocalStack!
  }
  return new S3Client(config);
}

// SQS
export function createSQSClient(): SQSClient {
  return new SQSClient({
    region: REGION,
    ...(ENDPOINT && { endpoint: ENDPOINT }),
  });
}

// Kinesis
export function createKinesisClient(): KinesisClient {
  return new KinesisClient({
    region: REGION,
    ...(ENDPOINT && { endpoint: ENDPOINT }),
    retryMode: 'adaptive', // Better throttle handling
  });
}

// Lambda
export function createLambdaClient(): LambdaClient {
  return new LambdaClient({
    region: REGION,
    ...(ENDPOINT && { endpoint: ENDPOINT }),
  });
}

// CloudWatch Logs
export function createCloudWatchLogsClient(): CloudWatchLogsClient {
  return new CloudWatchLogsClient({
    region: REGION,
    ...(ENDPOINT && { endpoint: ENDPOINT }),
  });
}

Pattern: Singleton Client (Performance — Lambda Best Practice)

// In Lambda, reuse clients across invocations (connection pooling)
let s3Client: S3Client | null = null;

export function getS3Client(): S3Client {
  if (!s3Client) {
    s3Client = createS3Client();
  }
  return s3Client;
}

Pattern: S3 ETag-Based Caching

Avoid unnecessary S3 downloads by checking if the object changed:

import { HeadObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';

class ConfigLoader {
  private etag: string | null = null;
  private cachedConfig: any = null;

  async loadConfig(bucket: string, key: string): Promise<any> {
    const s3 = getS3Client();

    // Check ETag (lightweight HEAD request)
    const head = await s3.send(new HeadObjectCommand({ Bucket: bucket, Key: key }));

    if (head.ETag === this.etag && this.cachedConfig) {
      console.log('Config unchanged, using cache');
      return this.cachedConfig;
    }

    // Config changed — download it
    const response = await s3.send(new GetObjectCommand({ Bucket: bucket, Key: key }));
    const body = await response.Body?.transformToString();
    this.cachedConfig = JSON.parse(body!);
    this.etag = head.ETag!;

    return this.cachedConfig;
  }
}

Pattern: SQS Batch Publisher with Chunking

import { SendMessageBatchCommand, SendMessageBatchRequestEntry } from '@aws-sdk/client-sqs';

async function publishBatch(
  messages: Array<{ id: string; body: object; groupId?: string }>,
  queueUrl: string
): Promise<void> {
  const sqs = createSQSClient();

  // SQS allows max 10 messages per batch
  const chunks = chunkArray(messages, 10);

  for (const chunk of chunks) {
    const entries: SendMessageBatchRequestEntry[] = chunk.map((msg) => ({
      Id: msg.id,
      MessageBody: JSON.stringify(msg.body),
      ...(msg.groupId && {
        MessageGroupId: msg.groupId,
        MessageDeduplicationId: `${msg.id}-${Date.now()}`,
      }),
    }));

    const result = await sqs.send(new SendMessageBatchCommand({
      QueueUrl: queueUrl,
      Entries: entries,
    }));

    if (result.Failed?.length) {
      console.error('Failed to send:', result.Failed);
    }
  }
}

function chunkArray<T>(arr: T[], size: number): T[][] {
  return Array.from({ length: Math.ceil(arr.length / size) }, (_, i) =>
    arr.slice(i * size, i * size + size)
  );
}

11. Docker Compose for the Full Stack

Here's a production-grade docker-compose.yml that sets up LocalStack alongside your application:

version: "3.8"

# Reusable AWS configuration
x-common-aws-config: &common-aws-config
  AWS_ACCESS_KEY_ID: test
  AWS_SECRET_ACCESS_KEY: test
  AWS_DEFAULT_REGION: us-east-1
  AWS_ENDPOINT: http://localstack:4566

services:
  # ──────────────────────────────────────────
  # LocalStack — AWS emulator
  # ──────────────────────────────────────────
  localstack:
    image: localstack/localstack:3.8
    container_name: localstack
    ports:
      - "4566:4566"         # Gateway (all services)
    environment:
      <<: *common-aws-config
      SERVICES: kinesis,s3,sqs,iam,lambda,logs,ssm,cloudwatch,events
      PERSISTENCE: "1"                           # Survive restarts
      LAMBDA_KEEPALIVE_MS: 60000                 # Reuse Lambda containers
    volumes:
      - localstack-data:/var/lib/localstack
      - /var/run/docker.sock:/var/run/docker.sock  # Required for Lambda
    healthcheck:
      test: >
        curl -sf http://localhost:4566/_localstack/health || exit 1
      interval: 3s
      timeout: 5s
      retries: 15
      start_period: 5s

  # ──────────────────────────────────────────
  # Init container — creates all AWS resources
  # ──────────────────────────────────────────
  localstack-init:
    image: amazon/aws-cli:latest
    container_name: localstack-init
    depends_on:
      localstack:
        condition: service_healthy
    environment:
      <<: *common-aws-config
      AWS_ENDPOINT_URL: http://localstack:4566
    entrypoint: ["/bin/bash", "/scripts/init.sh"]
    volumes:
      - ./scripts/init-localstack.sh:/scripts/init.sh:ro

  # ──────────────────────────────────────────
  # Your application
  # ──────────────────────────────────────────
  app:
    build: .
    depends_on:
      localstack-init:
        condition: service_completed_successfully
    environment:
      <<: *common-aws-config
      S3_BUCKET: processed-events
      KINESIS_STREAM: event-pipeline
      SQS_QUEUE_URL: http://localstack:4566/000000000000/notification-queue

volumes:
  localstack-data:

Key Notes

Setting	Purpose
`PERSISTENCE: "1"`	Resources survive container restarts
`LAMBDA_KEEPALIVE_MS: 60000`	Reuses Lambda containers (faster warm starts)
`docker.sock` volume	Required for Lambda execution (spawns sibling containers)
`x-common-aws-config`	YAML anchor avoids repeating credentials everywhere
`service_healthy`	Init script waits for LocalStack to be fully ready
`service_completed_successfully`	App waits for all resources to be created

12. Initialization Scripts — Automating Resource Creation

Create a script that runs once when LocalStack starts, setting up all your AWS resources:

#!/bin/bash
# scripts/init-localstack.sh
set -euo pipefail

ENDPOINT="http://localstack:4566"

echo "⏳ Waiting for LocalStack..."
until curl -sf "$ENDPOINT/_localstack/health" > /dev/null 2>&1; do
  sleep 2
done
echo "✅ LocalStack is ready"

# ── Kinesis Streams ──────────────────────────
echo "Creating Kinesis streams..."
aws --endpoint-url "$ENDPOINT" kinesis create-stream \
  --stream-name event-pipeline \
  --shard-count 1

aws --endpoint-url "$ENDPOINT" kinesis wait stream-exists \
  --stream-name event-pipeline

echo "✅ Kinesis: event-pipeline"

# ── S3 Buckets ───────────────────────────────
echo "Creating S3 buckets..."
for BUCKET in processed-events app-config deployment-artifacts; do
  aws --endpoint-url "$ENDPOINT" s3 mb "s3://$BUCKET" 2>/dev/null || true
  echo "  ✅ s3://$BUCKET"
done

# CORS for buckets accessed by frontend
aws --endpoint-url "$ENDPOINT" s3api put-bucket-cors \
  --bucket processed-events \
  --cors-configuration '{
    "CORSRules": [{
      "AllowedHeaders": ["*"],
      "AllowedMethods": ["GET", "PUT", "POST", "DELETE", "HEAD"],
      "AllowedOrigins": ["*"],
      "ExposeHeaders": ["ETag"]
    }]
  }'

# ── SQS Queues ───────────────────────────────
echo "Creating SQS queues..."

# Dead Letter Queue first
aws --endpoint-url "$ENDPOINT" sqs create-queue \
  --queue-name notification-dlq

# Main queue with DLQ
DLQ_ARN="arn:aws:sqs:us-east-1:000000000000:notification-dlq"
aws --endpoint-url "$ENDPOINT" sqs create-queue \
  --queue-name notification-queue \
  --attributes "{
    \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"${DLQ_ARN}\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\"
  }"

# FIFO queue for ordered processing
aws --endpoint-url "$ENDPOINT" sqs create-queue \
  --queue-name processing-queue.fifo \
  --attributes '{
    "FifoQueue": "true",
    "ContentBasedDeduplication": "true"
  }'

echo "✅ SQS queues created"

# ── IAM Roles ────────────────────────────────
echo "Creating IAM roles..."
aws --endpoint-url "$ENDPOINT" iam create-role \
  --role-name lambda-execution-role \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "lambda.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }' 2>/dev/null || true

aws --endpoint-url "$ENDPOINT" iam attach-role-policy \
  --role-name lambda-execution-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole \
  2>/dev/null || true

echo "✅ IAM roles created"

# ── SSM Parameters ───────────────────────────
echo "Storing SSM parameters..."
declare -A PARAMS=(
  ["/myapp/local/s3/bucket"]="processed-events"
  ["/myapp/local/kinesis/stream-arn"]="arn:aws:kinesis:us-east-1:000000000000:stream/event-pipeline"
  ["/myapp/local/sqs/notification-queue-url"]="http://localstack:4566/000000000000/notification-queue"
)

for KEY in "${!PARAMS[@]}"; do
  aws --endpoint-url "$ENDPOINT" ssm put-parameter \
    --name "$KEY" \
    --value "${PARAMS[$KEY]}" \
    --type String \
    --overwrite 2>/dev/null || true
  echo "  ✅ $KEY"
done

# ── Upload seed config to S3 ────────────────
echo '{"version":"1.0","features":{"darkMode":true}}' | \
  aws --endpoint-url "$ENDPOINT" s3 cp - s3://app-config/config.json

echo ""
echo "🎉 All LocalStack resources initialized!"
echo "   Kinesis: event-pipeline (1 shard)"
echo "   S3:      processed-events, app-config, deployment-artifacts"
echo "   SQS:     notification-queue (+ DLQ), processing-queue.fifo"
echo "   IAM:     lambda-execution-role"
echo "   SSM:     3 parameters under /myapp/local/"

Pro tip: Use 2>/dev/null || true on creation commands so the script is idempotent — re-running it won't fail if resources already exist.

13. Advanced Patterns

Lambda Aliases for Zero-Downtime Deployment

In production, you'd use aliases to switch between Lambda versions atomically:

# Publish a version
VERSION=$(awslocal lambda publish-version \
  --function-name event-processor \
  --query 'Version' --output text)

# Create/update alias
awslocal lambda create-alias \
  --function-name event-processor \
  --name live \
  --function-version "$VERSION"

# Point event source mapping to the alias
awslocal lambda create-event-source-mapping \
  --function-name "event-processor:live" \
  --event-source-arn arn:aws:kinesis:us-east-1:000000000000:stream/event-pipeline \
  --batch-size 10 \
  --starting-position LATEST

When deploying a new version, update the alias — the event source mapping automatically picks it up.

EventBridge Scheduled Rules

Trigger Lambda on a cron schedule:

# Create a rule that fires every minute
awslocal events put-rule \
  --name every-minute-rule \
  --schedule-expression "rate(1 minute)"

# Add Lambda as target
awslocal events put-targets \
  --rule every-minute-rule \
  --targets '[{
    "Id": "1",
    "Arn": "arn:aws:lambda:us-east-1:000000000000:function:event-processor"
  }]'

# Grant EventBridge permission to invoke Lambda
awslocal lambda add-permission \
  --function-name event-processor \
  --statement-id eventbridge-invoke \
  --action lambda:InvokeFunction \
  --principal events.amazonaws.com \
  --source-arn arn:aws:events:us-east-1:000000000000:rule/every-minute-rule

Lambda Failure Destinations (S3)

Capture failed Lambda invocations for debugging:

awslocal s3 mb s3://lambda-failures

awslocal lambda put-function-event-invoke-config \
  --function-name event-processor \
  --maximum-retry-attempts 2 \
  --destination-config '{
    "OnFailure": {
      "Destination": "arn:aws:s3:::lambda-failures"
    }
  }'

Docker Networking Gotcha

When Lambda containers try to reach LocalStack internally, they can't use localhost. The init script may need to rewrite URLs:

# LocalStack returns URLs like:
#   http://sqs.localhost.localstack.cloud:4566/000000000000/my-queue
#
# But Lambda containers need:
#   http://localstack:4566/000000000000/my-queue

QUEUE_URL_RAW=$(aws --endpoint-url "$ENDPOINT" sqs get-queue-url \
  --queue-name my-queue --query 'QueueUrl' --output text)

QUEUE_URL=$(echo "$QUEUE_URL_RAW" | \
  sed 's|http://sqs\.[^/]*localhost\.localstack\.cloud:[0-9]*/|http://localstack:4566/|')

Serverless Framework + LocalStack

If you use the Serverless Framework, the serverless-localstack plugin makes deployment seamless:

# serverless.yml
plugins:
  - serverless-esbuild
  - serverless-localstack

custom:
  localstack:
    stages:
      - local
    host: http://localstack
    edgePort: 4566
    autostart: false
    endpoints:
      S3: http://localstack:4566
      Lambda: http://localstack:4566
      Kinesis: http://localstack:4566
      SQS: http://localstack:4566
      IAM: http://localstack:4566
      CloudFormation: http://localstack:4566
      CloudWatchLogs: http://localstack:4566

Deploy to LocalStack:

npx serverless deploy --stage local

14. Troubleshooting Common Issues

"Could not connect to the endpoint URL"

# Check if LocalStack is running
docker ps | grep localstack

# Check health
curl http://localhost:4566/_localstack/health

# Check logs
docker logs localstack

S3 "bucket does not exist" with valid bucket name

You're probably missing forcePathStyle:

// WRONG — SDK uses virtual-hosted style: http://bucket.localhost:4566
const s3 = new S3Client({ endpoint: 'http://localhost:4566' });

// CORRECT — SDK uses path style: http://localhost:4566/bucket
const s3 = new S3Client({
  endpoint: 'http://localhost:4566',
  forcePathStyle: true,  // ← THIS
});

Lambda "Docker not available" error

Mount the Docker socket in your docker-compose.yml:

volumes:
  - /var/run/docker.sock:/var/run/docker.sock

Lambda can't reach other Docker services

Lambda containers run in separate Docker networks. If your Lambda needs to reach a database or API in another container, ensure they share a Docker network:

# docker-compose.yml
services:
  localstack:
    networks:
      - backend

  my-database:
    networks:
      - backend

networks:
  backend:
    driver: bridge

SQS URL format issues

LocalStack SQS URLs can vary. Always use the format:

http://localstack:4566/000000000000/queue-name  # Inside Docker
http://localhost:4566/000000000000/queue-name    # From host machine

Data doesn't persist after restart

Set PERSISTENCE=1 and use a named volume:

environment:
  PERSISTENCE: "1"
volumes:
  - localstack-data:/var/lib/localstack

Slow Kinesis performance

Enable the Scala engine:

environment:
  KINESIS_MOCK_PROVIDER_ENGINE: scala

15. Free vs Pro — What You Actually Need

Feature	Community (Free)	Pro
S3, SQS, SNS, Kinesis	Yes	Yes
Lambda, IAM, CloudWatch	Yes	Yes
DynamoDB, SSM, EventBridge	Yes	Yes
Persistence	Yes	Yes
IAM policy enforcement	No	Yes
Cognito, RDS, ECS, EKS	No	Yes
Cloud Pods (state snapshots)	No	Yes
CI analytics dashboard	No	Yes

My recommendation: Start with Community. It covers 90% of typical use cases. Upgrade to Pro only if you need a specific service (like RDS or Cognito) or IAM enforcement.

16. Conclusion

What We Built

┌──────────────┐     ┌─────────────┐     ┌──────────────────┐     ┌──────────────┐
│  Application  │────▶│   Kinesis    │────▶│  Lambda          │────▶│   S3 Bucket   │
│  (Events)     │     │   Stream     │     │  (Processor)     │     │  (Storage)    │
└──────────────┘     └─────────────┘     └──────────────────┘     └──────────────┘
                                                  │                       │
                                                  ▼                       ▼
                                          ┌──────────────┐     ┌──────────────────┐
                                          │   SQS Queue   │     │  Frontend / API   │
                                          │ + Dead Letter  │     │  (reads from S3) │
                                          └──────┬───────┘     └──────────────────┘
                                                  │
                                                  ▼
                                          ┌──────────────────┐
                                          │  Lambda           │
                                          │  (Notification)   │
                                          └──────────────────┘
                                                  │
                                          ┌───────▼──────────┐
                                          │   CloudWatch      │
                                          │   (All Logs)      │
                                          └──────────────────┘

We covered:

Installation on Mac, Windows, and Linux
Individual service setup — S3, SQS, Kinesis, Lambda, IAM, CloudWatch, SSM
A complete pipeline — events flow from Kinesis through Lambda into S3 and SQS
SDK integration — TypeScript patterns that work against both LocalStack and real AWS
Docker Compose — a production-grade setup with health checks, init scripts, and persistence
Advanced patterns — aliases, EventBridge cron, failure destinations, Serverless Framework
Troubleshooting — the issues you will actually hit

The key mindset shift: treat LocalStack as your local AWS account. Same APIs, same SDKs, same CLI — just no bills and instant feedback.

Have questions? Drop them in the comments — I'll do my best to answer. If this saved you time or money, a like helps others find it too.

All code examples in this article are available as standalone scripts you can copy-paste and run. No AWS account required.

Revise All Basics of MongoDB in 10 Minutes 🚀

Mayank Singh — Mon, 19 May 2025 17:05:33 +0000

Want to revise MongoDB quickly before an interview or task? Here's your ultimate 10-minute refresher!

MongoDB is a powerful, open-source, NoSQL document-oriented database, widely recognized for its scalability and flexibility. It’s written in C++ and uses BSON (a binary representation of JSON) for storing data.

In this article, we’ll explore all the essential MongoDB concepts, setup steps, and commands—perfect for a quick revision!

📚 Core Concepts

1. Database

A database in MongoDB is a physical container for collections. Each database gets its own files on the system. A single MongoDB server can have multiple databases.

2. Collection

Think of a collection as the equivalent of a table in relational databases. It's a group of MongoDB documents. Collections are schema-less, allowing documents within them to have different fields and structures.

3. Document

The fundamental unit of data in MongoDB—a document is a key-value pair structure, stored in BSON format. Example:

{
  "name": "John Doe",
  "age": 30,
  "email": "john@example.com"
}

🔤 MongoDB Data Types

String: Most common. UTF-8 valid.
Integer: 32-bit or 64-bit.
Boolean: true/false.
Double: Floating point numbers.
Arrays: Store lists.
Object: Embedded documents.
Date: Stored in UNIX time.
ObjectID: Unique document identifier.
Null, Binary, Code, Regex, Symbol, etc.

⚙️ Installing MongoDB (macOS via Homebrew)

brew tap mongodb/brew
brew update
brew install mongodb-community@8.0

Key MongoDB executables:

mongod: MongoDB server daemon.
mongosh: MongoDB shell (CLI client).
mongos: For sharded cluster queries.

Setting Up (Windows Example)

mkdir C:\data\db
mongod

In a new terminal:

mongosh

🧪 Essential MongoDB Shell Commands

🔎 Current Database

db

📃 List All Databases

show databases

➡️ Switch Database

use myDB

If myDB doesn’t exist, it gets created when data is inserted.

❌ Drop Current Database

db.dropDatabase()

📂 Collections and Documents

✅ Create a Collection

db.createCollection("testCollection")

OR insert directly to auto-create:

db.testCollection.insertOne({ name: "Alice", age: 25 })

📥 Insert Documents

// Single
db.testCollection.insertOne({ name: "Bob", age: 30 })

// Multiple
db.testCollection.insertMany([
  { name: "Carol", age: 22 },
  { name: "Dave", age: 29, city: "Delhi" }
])

🔍 Query Documents

db.testCollection.find()                   // All documents
db.testCollection.find().pretty()         // Prettified

db.testCollection.find({ name: "Carol" }) // Filter
db.testCollection.find({ age: { $lt: 25 } }) // Less than

More operators: $gt, $gte, $lte, $ne, $and, $or

db.testCollection.find({
  $and: [{ age: { $lt: 25 } }, { city: "Delhi" }]
})

🔄 Update & Delete

✏️ Update

db.testCollection.update({ age: 22 }, { $set: { age: 24 } })

// Remove a field
db.testCollection.update({ name: "Carol" }, { $unset: { age: "" } })

❌ Delete

db.testCollection.remove({ name: "Carol" }) // Delete specific
db.testCollection.remove({})                // Delete all

🗑️ Drop Collection

db.testCollection.drop()

📌 Sorting and Limiting

db.testCollection.find().limit(2)
db.testCollection.find().sort({ name: 1 }) // Ascending

⚡ Indexing

Without indexes, MongoDB scans all documents—inefficient!

✅ Create Index

db.testCollection.createIndex({ name: 1 }) // Ascending

❌ Drop Index

db.testCollection.dropIndex({ name: 1 })

🔎 Get All Indexes

db.testCollection.getIndexes()

🔥 Drop All Indexes

db.testCollection.dropIndexes()

🧠 MongoDB vs RDBMS

Feature	MongoDB (NoSQL)	RDBMS (SQL)
Data Model	Document-oriented (BSON)	Relational (tables & rows)
Schema	Dynamic, schema-less	Fixed schema
Scalability	Horizontally scalable via sharding	Mostly vertical scaling
Joins	Limited support (via `$lookup`)	Native and complex join support
Transactions	Multi-document ACID (since v4.0)	Full ACID-compliant
Data Retrieval	JSON-like query language	SQL
Performance	Faster for unstructured or semi-structured data	Better for structured data
Use Case	Big data, real-time analytics, fast iteration	Banking, legacy systems, ERP
Storage	BSON (Binary JSON)	Relational format (tables)
Examples	MongoDB, Couchbase, Cassandra	MySQL, PostgreSQL, Oracle

🏗️ How MongoDB Handles Millions of Incoming API Requests

MongoDB is designed for high concurrency and horizontal scalability. Here's how it deals with massive loads:

🚀 Efficient Indexing

MongoDB supports single field, compound, text, geospatial, and hashed indexes.

It automatically uses indexes to quickly locate documents.
You can create indexes on frequently queried fields to reduce read latency.
Background indexing allows index creation without blocking operations.

db.collection.createIndex({ field: 1 })

You can monitor index usage via:

db.collection.getIndexes()
db.collection.explain("executionStats").find({ field: "value" })

⚖️ Sharding (Horizontal Scaling)

Sharding divides large data sets across multiple machines.
MongoDB uses a shard key to distribute documents.
A query router (mongos) sends requests to appropriate shards.
Each shard is a replica set ensuring redundancy and high availability.

Sharding helps:

Balance read/write load
Store more data than one server can handle
Achieve linear horizontal scaling

sh.enableSharding("myDatabase")
sh.shardCollection("myDatabase.myCollection", { userId: 1 })

🔁 Joins via Aggregation Framework
MongoDB didn’t support joins traditionally but now allows joins using $lookup in the aggregation pipeline.

db.orders.aggregate([
  {
    $lookup: {
      from: "users",
      localField: "userId",
      foreignField: "_id",
      as: "userDetails"
    }
  }
])

While not as performant or flexible as SQL joins, this is effective for many use cases where data is somewhat normalized.

🧠 Connection Management

MongoDB uses non-blocking I/O and asynchronous processing.
Connection pooling is supported by most drivers.
Caching and efficient memory management via WiredTiger engine.

✅ Why Use MongoDB?

Schema-less and flexible structure
High performance at scale
Easier horizontal scaling
Fast development lifecycle
Ideal for modern applications using microservices

🔚 Conclusion

MongoDB’s flexibility and dynamic nature make it a great fit for modern, scalable applications. With the commands and concepts above, you’re now ready to dive into MongoDB or revise it whenever needed in just 10 minutes!

Have any MongoDB experience, tricks, or tips? Share in the comments! 👇

Author: Mayank Singh
Portfolio: https://msn2106.github.io/
LinkedIn: @msn2106

# Is Node.js Really Single-Threaded? Here’s What Most Developers Get Wrong About Node’s Backend Power

Mayank Singh — Sun, 18 May 2025 19:23:57 +0000

When I recently hit a production issue where one of our Azure Functions unexpectedly fired hundreds of API calls within seconds, it pushed me down a rabbit hole.

🔍 What caused it?

Node.js was the culprit — or rather, my misunderstanding of how Node.js handles caching and concurrency.

Due to a cold start, the in-memory cache reset, and with Azure's high parallelism settings, each parallel thread triggered its own set of calls. Boom — API surge.

This got me asking...

How does Node.js actually work behind the scenes?

Is Node.js really "single-threaded"?

How does it compare to Java and other backend technologies?

Let’s break it down.

🔧 Node.js Architecture — A Quick Recap

At its core, Node.js is built around:

🧵 A single-threaded event loop
⏳ A non-blocking I/O model
🔧 An internal thread pool (libuv) for some tasks
🧠 Optional worker threads for CPU-heavy work

This model shines in high I/O scenarios: chat apps, API gateways, real-time apps, etc.

🚦 How Request Handling Works

I/O-bound requests (e.g., DB reads, file reads) → delegated to non-blocking handlers
CPU-heavy work (e.g., image processing) → needs special treatment or it blocks the event loop
Event Loop picks up completed work and executes callbacks

⚡ What Makes Node.js Different?

✅ Strengths:

Feature	Node.js
Event Loop	Yes
Single Threaded	By default
Asynchronous by Design	Yes
CPU-bound Work	Needs manual scaling
Native Multithreading	Now possible via Worker Threads
Scales via Cluster or Workers	Yes
Best Use	APIs, Microservices, Real-time apps

🔄 Comparing with Java (and Others)

Feature	Node.js	Java
Thread Model	Single (default) + workers	Multi-threaded
Default Concurrency	Event Loop	Thread Pools
Memory Overhead	Low	Higher
Scaling	Manual (cluster/fork/worker)	Automatic via thread pools
Heavy CPU Work	Needs worker threads	Native threading
Ideal Use	Real-time, async I/O	Complex business logic, CPU-heavy apps

💡 Java inherently manages threads and thread pools.

Node requires explicit configuration to utilize more CPU cores (via cluster, worker threads, etc.)

🛠️ Worker Threads in Node.js — Multithreading in Action

const { Worker } = require('worker_threads');

function runWorker(path, data) {
  return new Promise((resolve, reject) => {
    const worker = new Worker(path, { workerData: data });
    worker.on('message', resolve);
    worker.on('error', reject);
    worker.on('exit', (code) => {
      if (code !== 0) reject(new Error(`Worker stopped with code ${code}`));
    });
  });
}

// Usage:
runWorker('./heavyTask.js', { payload: 123 }).then(console.log);

Using this, you can spawn real threads up to the number of CPU cores available on the system.

🤯 Real-World Problem: Azure + Node.js + Cold Start = Disaster

In my recent experience:

Node.js Azure Function used in-memory cache
Cold start triggered cache reset
High degree of parallelism (16 threads!) meant every invocation missed the cache
Result: hundreds of external API calls = system overload

This led me to study Node’s internals more deeply and compare with Java, which would’ve used:

JVM-level thread pooling
Shared memory space
Potentially avoided the burst entirely

📊 Thread Utilization and Scaling

A system with 8 cores, 16 logical processors can theoretically handle 16 threads in parallel
But OS and system apps use some threads → leave 2–4 for system
Use max 12–14 threads (Java or Node workers)

In Node.js:

const os = require('os');
const usableThreads = os.cpus().length - 2; // e.g., 14 threads

📌 When Should You Choose Node.js?

Choose Node.js if:

You’re building real-time apps (chat, WebSocket)
You need API gateways, proxies, or high I/O throughput
You can split heavy work into services or offload to queues

Avoid Node.js if:

Your app involves heavy CPU-bound synchronous logic
You need robust multithreading natively
You prefer strongly typed OOP-heavy codebases

✅ Final Thoughts

Node.js isn’t just single-threaded — it’s smartly single-threaded, with options to scale via worker threads or clusters.
But those choices are manual, and you must understand the architecture to avoid pitfalls like I did on Azure.
If you’re choosing a backend tech stack, factor in:
Type of load: I/O-heavy vs CPU-heavy
Scaling model: thread-based or event-driven
Operational complexity

.
.
.
NodeJs, Backend, Backend Architecture, Event Loop, Multi-threading, Worker Threads, Java, Java Vs NodeJs, Performance, Azure, Cold Start, High Parallelism, CPU Bound Tasks

Never Worry About Project Dependencies: A Comprehensive Guide to package.json and package-lock.json

Mayank Singh — Sat, 01 Jul 2023 15:14:37 +0000

Introduction:

Managing dependencies is a crucial aspect of any software project. By correctly declaring and tracking dependencies, developers can ensure consistent and reliable application builds. In the JavaScript ecosystem, package.json and package-lock.json files play a vital role in managing project dependencies. This article will delve into the intricacies of these files, exploring their attributes and various ways to declare dependencies.

Understanding package.json:

package.json is a file that contains metadata about your Node.js project. This includes the project's name, version, author, and dependencies. The dependencies section of package.json is a list of all the packages that your project needs. Each dependency is listed with its name, version, and optionally, a range of versions that are acceptable.

{
  "name": "package.json-file",
  "version": "1.0.0",
  "description": "creating a new package.json file",
  "private": false,
  "main": "index.js",
  "scripts": {
    "start": "node index",
    "dev": "nodemon index",
    "test": "jest"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/Easybuoy/package.json-mastery.git"
  },
  "keywords": [
    "node",
    "javascript",
    "npm",
    "yarn"
  ],
  "author": "Ezekiel Ekunola",
  "license": "ISC",
  "bugs": {
    "url": "https://github.com/Easybuoy/package.json-mastery/issues"
  },
  "homepage": "https://github.com/Easybuoy/package.json-mastery#readme",
  "engines": {
    "npm": "6.10.0",
    "node": "10.14.1"
  },
  "dependencies": {
    "bcryptjs": "~2.4.3",
    "cors": "^2.8.5",
    "dotenv": "^6.1.0",
    "express": "^4.16.4"
  },
  "devDependencies": {
    "eslint": "^4.19.1",
    "mocha": "~6.2.0",
    "nodemon": "^1.19.1"
  },
  "nyc": {
    "exclude": [
      "server/app.js",
      "server/config/",
      "server/build"
    ]
  }
}

"name":
The "name" attribute specifies the project's name. It should be unique and typically follows the format: "@author/project-name".
"version":
The "version" attribute represents the project's current version. It follows the Semantic Versioning (SemVer) format (e.g., "1.0.0").
"description":
The "description" attribute provides a brief description of the project.
"author" and "contributors":
The "author" attribute denotes the project's primary author, while "contributors" lists additional contributors.
"license":
The "license" attribute specifies the license under which the project is distributed (e.g., "MIT," "Apache-2.0").

Dependency Declaration:

In package.json, dependencies are declared under the "dependencies" attribute. There are different ways to specify dependencies:

{
  "dependencies": {
    "foo": "1.0.0 - 2.9999.9999",
    "bar": ">=1.0.2 <2.1.2",
    "baz": ">1.0.2 <=2.3.4",
    "boo": "2.0.1",
    "qux": "<1.0.0 || >=2.3.1 <2.4.5 || >=2.5.2 <3.0.0",
    "asd": "http://asdf.com/asdf.tar.gz",
    "til": "~1.2",
    "elf": "~1.2.3",
    "two": "2.x",
    "thr": "3.3.x",
    "lat": "latest",
    "dyl": "file:../dyl"
  }
}

2.1. Version Ranges:

Tilde (~) Range: This symbol indicates that any version of the package that is greater than or equal to the specified version, but less than the next major version, is acceptable. For example, ~4.17.1 means that any version of your package that is greater than or equal to 4.17.1 and less than 5.0.0 is acceptable.

Caret (^) Range: This symbol indicates that any version of the package that is greater than or equal to the specified version is acceptable. For example, ^4.17.1 means that any version of your package that is greater than or equal to 4.17.1 is acceptable.

2.2. Exact Version:
Specifying the exact version ensures that only that specific version is installed. Example: "lodash": "4.17.21"

2.3. Version Ranges with Operators:
You can use operators to define version ranges:
Greater than (>): Example: "axios": ">0.21.1"
Greater than or equal to (>=): Example: "jest": ">=26.0.0"

Working with package-lock.json:

package-lock.json is a file that is generated by npm when you install your dependencies. It contains the exact versions of all the packages that are installed in your project. This means that even if you install your project on a different machine or use a different version of npm, you can be sure that the same versions of the dependencies will be installed.

Best Practices:

To ensure smooth dependency management, consider the following best practices:

5.1. Regularly update dependencies:
Keeping dependencies up to date ensures security patches and feature enhancements. Use tools like "npm outdated" or "npm-check-updates" to identify outdated packages.

5.2. Lock dependencies:
Commit both package.json and package-lock.json to your version control system to ensure reproducible builds across different environments.

5.3. Use semantic versioning:
Follow SemVer guidelines when declaring dependencies to ensure compatibility and avoid breaking changes.

Conclusion:

Understanding the attributes of package.json and leveraging package-lock.json is essential for managing project dependencies effectively. By adopting best practices and utilizing the various dependency declaration methods, developers can ensure reliable and hassle-free dependency management in their projects.

Remember, a well-managed dependency ecosystem contributes to the stability and maintainability of your applications, enabling you to focus on delivering valuable features to your users.

References :-

Dependencies Versions details Semver:- https://github.com/npm/node-semver#versions
Package.json details from NPM:- https://docs.npmjs.com/cli/v8/configuring-npm/package-json
Package-lock.json details from NPM:- https://docs.npmjs.com/cli/v8/configuring-npm/package-lock-json
More Better Explanation with Diagram:- https://www.atatus.com/blog/package-json-vs-package-lock-json/

Lol found the 4th one in last. Felt mine as 🥔
Most images are taken from Atatus rest Google