Forem: Jayesh Shinde

How We Fixed Intermittent ECS Image-Not-Found Errors in AWS CDK

Jayesh Shinde — Fri, 13 Mar 2026 09:00:11 +0000

At one point, our ECS deployments started failing in a way that felt random.

Sometimes a deployment would work perfectly. Sometimes the service would try to roll forward and fail because the container image it expected was no longer available. Nothing was wrong with the application code. The problem was in the deployment asset flow.

We were using AWS CDK to deploy container-based workloads, and like many teams, we were relying on CDK’s default bootstrap ECR repository for Docker image assets. That was convenient at first, but it became a problem once repository retention rules were tightened for cost control.

In environments with frequent deployments, older intermediate images were being cleaned up faster than our deployment flow could safely tolerate. The result was intermittent ECS deploy failures caused by missing images.

The Root Cause

AWS CDK Docker assets are published during the asset publishing phase, which happens before CloudFormation starts deploying stacks.

That means two things:

CDK is not just defining infrastructure, it is also managing where deployable image assets are stored.
If the default asset repository has aggressive cleanup policies, your deployments can become fragile.

This is especially painful in non-production environments where deployment frequency is high and image churn is constant.

The Strategy We Took

We wanted a solution that was simple, low-risk, and did not require redesigning the whole build pipeline.

So instead of pushing ECS image assets to the shared default CDK ECR repository, we moved to a dedicated ECR repository per environment/application area.

At a high level, the fix looked like this:

create a dedicated ECR repository ahead of time
configure the CDK synthesizer to publish image assets there
keep lifecycle control on that repository
deploy the ECR stack first, then the app stacks

This gave us isolation from the shared bootstrap repository while keeping the rest of the CDK deployment model mostly unchanged.

Sample: Dedicated ECR Stack

Here is a simplified example of creating a dedicated ECR repository with a lifecycle policy:

import { Stack, RemovalPolicy } from 'aws-cdk-lib';
import { Repository } from 'aws-cdk-lib/aws-ecr';
import type { Construct } from 'constructs';

type AppEnvProps = {
  stage: string;
};

export class ContainerAssetRepoStack extends Stack {
  constructor(scope: Construct, id: string, props: AppEnvProps) {
    super(scope, id);

    new Repository(this, 'AppContainerRepo', {
      repositoryName: `myapp-assets-${props.stage.toLowerCase()}`,
      removalPolicy: RemovalPolicy.RETAIN,
      lifecycleRules: [
        {
          rulePriority: 1,
          description: 'Keep only the latest 100 images',
          maxImageCount: 100,
        },
      ],
    });
  }
}

A few important details here:

RETAIN protects the repository if the stack is deleted later.
lifecycle rules still clean up old images over time.
the repository name is normalized to lowercase, which is important for ECR.

Sample: Point CDK to the Dedicated Repo

Once the repository exists, the application stack can tell CDK to publish image assets there using DefaultStackSynthesizer.

import { App, DefaultStackSynthesizer, Stack } from 'aws-cdk-lib';
import type { Construct } from 'constructs';

type ServiceStackProps = {
  stage: string;
};

class ServiceStack extends Stack {
  constructor(scope: Construct, id: string, props: ServiceStackProps) {
    super(scope, id, {
      synthesizer: new DefaultStackSynthesizer({
        imageAssetsRepositoryName: `myapp-assets-${props.stage.toLowerCase()}`,
      }),
    });

    // ECS service, task definition, container asset usage, etc.
  }
}

const app = new App();

new ServiceStack(app, 'ServiceStackDev', {
  stage: 'dev',
});

This keeps the existing CDK asset publishing model, but moves the destination away from the shared default bootstrap repository.

One Important Gotcha

A stack dependency is not enough if the same deployment run tries to create the ECR repository and publish assets into it.

Why?

Because asset publishing happens before CloudFormation stack deployment.

So if the repository does not already exist, the asset publish step can fail before your “repo stack” is even deployed.

The safest pattern is:

deploy the ECR repository stack first
run the normal application deployment after that

That sequencing matters.

Another Important Gotcha: IAM Permissions

Changing the repository target is not enough by itself.

The identity or role that CDK uses to publish Docker assets must also have permission to push to the new ECR repository.

That usually means allowing actions such as:

ecr:PutImage
ecr:InitiateLayerUpload
ecr:UploadLayerPart
ecr:CompleteLayerUpload
ecr:BatchCheckLayerAvailability
ecr:BatchGetImage
ecr:GetDownloadUrlForLayer
ecr:GetAuthorizationToken

If you forget this part, the deployment simply moves from “image missing” problems to “access denied” problems.

Why This Worked Well for Us

We liked this approach because it was a practical middle ground.

It did not require:

rebuilding our CI/CD image strategy from scratch
changing every ECS service definition
introducing a more complex app-owned image publishing flow immediately

But it did give us:

predictable image retention
environment-specific isolation
fewer surprises during ECS deployments
better control over cost and cleanup behavior

When to Use This Pattern

This approach makes sense if:

you already use CDK-managed Docker/container assets
the default bootstrap ECR repository is shared across too many deployments
retention rules on that shared repository are causing instability
you want a fast, low-disruption improvement

If you want a more explicit long-term model, the next step is usually:

build image in CI
push image to a named ECR repository yourself
reference the image directly in ECS by repo and tag

That gives maximum control, but it also requires more changes.

Final Thought

CDK defaults are great for getting started, but they are not always ideal once platform constraints like retention, cost control, and deployment frequency start to matter.

In our case, moving Docker assets to dedicated ECR repositories was a small change with a big operational impact. It made deployments more predictable without forcing a major rework of the pipeline.

The Silent Connection Killer: MySQL2 and AWS Lambda's Freeze/Thaw Problem

Jayesh Shinde — Thu, 05 Feb 2026 09:03:13 +0000

The Mystery Error

You're running a Node.js Lambda with MySQL2, everything works great in testing, but production logs show intermittent failures:

Error: Connection lost: The server closed the connection.code: "PROTOCOL_CONNECTION_LOST"
fatal: true

No pattern. No warning. Just random failures that make you question your life choices.

Understanding the Real Problem : How Lambda Actually Works

Lambda doesn't spin up a fresh container for every request. AWS keeps containers "warm" for reuse:

Request 1 → Lambda runs → Response
                ↓
           [FREEZE] ← Container paused (not terminated)
                ↓     
         (5-15 min pass)
                ↓
           [THAW] ← Container resumed
                ↓
Request 2 → Lambda runs → 💥 PROTOCOL_CONNECTION_LOST

During freeze, your Lambda is literally paused. The JavaScript event loop stops. Timers stop. Everything stops.
But here's the catch: the outside world doesn't stop.

What Happens to Your Database Connection

Lambda creates a MySQL connection pool
Connections sit idle in the pool
Lambda freezes (container paused)
Real world time passes (5-30 minutes)
Network timeouts occur, NAT gateways clear state, RDS Proxy cleans up
The TCP socket dies, but your pool doesn't know
Lambda thaws, tries to use the dead connection
💥 PROTOCOL_CONNECTION_LOST

Why idleTimeout Doesn't Help

You might think: "I'll set idleTimeout: 60000 to clean up idle connections!"
Here's why it doesn't work:

Timer starts (60s countdown)
    ↓
Lambda FREEZES at 1s elapsed
    ↓
████████████████████████████████
█  15 minutes pass in REAL WORLD  █
█  Timer is PAUSED at 1s          █
████████████████████████████████
    ↓
Lambda THAWS - timer resumes at 1s
    ↓
Connection still in pool (timer thinks 2s passed)
    ↓
Connection is DEAD but pool doesn't know

The timer doesn't run during freeze. Your 60-second timeout is useless against a 15-minute freeze.

The Solution: Detect and Retry
Since we can't prevent stale connections, we detect them and retry transparently.

Step 1: Enable TCP Keep-Alive

const pool = mysql.createPool({
  ...config,
  enableKeepAlive: true,
  keepAliveInitialDelay: 10000,
});

This helps get clear error codes (ECONNRESET, PROTOCOL_CONNECTION_LOST) instead of hanging indefinitely.

Step 2: Implement Retry Logic

async executeQuery(sql, params) {
  const maxRetries = 1;
  let lastError = null;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    let connection = null;

    try {
      connection = await this.getConnectionFromPool();
      const result = await this.executeQueryWithConnection(connection, sql, params);
      connection.release();
      return result;
    } catch (error) {
      // Non-recoverable error - throw immediately
      if (!this.isConnectionLostError(error)) {
        if (connection) connection.release();
        throw error;
      }

      // Connection lost - destroy stale connection
      if (connection) connection.destroy();

      // Retry if attempts left
      lastError = error;
      if (attempt < maxRetries) {
        console.warn("Connection lost, retrying...", { attempt: attempt + 1 });
        continue;
      }
    }
  }

  throw lastError;
}

isConnectionLostError(error) {
  const recoverableCodes = [
    "PROTOCOL_CONNECTION_LOST",  // Server closed connection
    "ECONNRESET",                // TCP reset
    "EPIPE",                     // Broken pipe
    "ETIMEDOUT",                 // Connection timeout
    "ECONNREFUSED",              // Connection refused
  ];
  return recoverableCodes.includes(error?.code);
}

Key Points:

connection.destroy() - Removes stale connection from pool (don't reuse it!)
connection.release() - Returns healthy connection to pool
One retry is usually enough - The second attempt gets a fresh connection

What About Pool Settings?

Do I Need to Tune connectionLimit, maxIdle, etc.?
Short answer: Not really.

Setting	Helps with freeze/thaw?	Why?
`idleTimeout`	No	Timer paused during freeze
`maxIdle`	Marginally	Fewer connections = fewer stale ones, but adds reconnection overhead
`connectionLimit`	No	Doesn't affect stale connections
Retry logic	Yes	Handles stale connections at runtime

If you're using RDS Proxy, it handles connection pooling at the infrastructure level. Keep your Lambda pool settings simple and let the retry logic do the heavy lifting.

Using RDS Proxy?

RDS Proxy's "Idle client connection timeout" (default: 30 minutes) is separate from MySQL's wait_timeout. The proxy manages Lambda→Proxy connections independently.
But even with RDS Proxy, connections can die due to:

NAT gateway timeouts (typically 5-15 minutes for idle TCP)
Network state table cleanup
Proxy internal connection recycling

The retry logic is still your safety net.

The Final Architecture

┌─────────────────────────────────────────────────────┐
│                    Your Lambda                       │
├─────────────────────────────────────────────────────┤
│                                                      │
│   executeQuery()                                     │
│       ↓                                             │
│   for (attempt = 0; attempt <= 1; attempt++)        │
│       ↓                                             │
│   getConnection() → Try query                       │
│       ↓                                             │
│   Success? → return result                          │
│       ↓                                             │
│   Connection lost? → destroy() → retry              │
│       ↓                                             │
│   Other error? → release() → throw                  │
│                                                      │
└─────────────────────────────────────────────────────┘
           ↓
    [RDS Proxy] (optional)
           ↓
    [MySQL/Aurora]

Summary

Problem	Solution
Lambda freezes, connections go stale	Retry logic detects and recovers
Pool doesn't know connections are dead	`enableKeepAlive` for faster detection
`idleTimeout` doesn't work during freeze	Accept it, rely on retry instead
Random `PROTOCOL_CONNECTION_LOST`	Transparent retry = users don't notice

The key insight: You can't prevent stale connections in a serverless environment. But you can detect them instantly and retry transparently.

BUT....

The Problem With Just Retrying Once

The previous article suggested detecting a stale connection error and retrying once. That works — but only if a single connection went stale. After a longer Lambda freeze (10–15+ minutes), the entire pool goes stale. Here's the scenario:

Pool has 5 connections → Lambda freezes → all 5 TCP sockets die

Request comes in after thaw:
→ gets conn #1 from _freeConnections → FAILS (stale)
→ retry: gets conn #2 from _freeConnections → FAILS (also stale!)

One retry is not enough because _freeConnections is a queue — the retry just picks the next dead connection in line.

How mysql2's Pool Actually Works Internally

Looking at mysql2's pool.js source, getConnection() does this:

// Simplified from mysql2 internals
getConnection(cb) {
  if (this._freeConnections.length > 0) {
    const connection = this._freeConnections.shift(); // FIFO — no health check
    return cb(null, connection);
  }
  // if allConnections.length < connectionLimit → create a NEW connection
  // otherwise → queue the request in _connectionQueue
}

There is zero validation when pulling from _freeConnections. The pool hands you whatever is sitting there, stale or not.

The inverse is also useful to know — when _freeConnections is empty AND _allConnections.length < connectionLimit, mysql2 will automatically create a brand new TCP connection. This is the behavior we want to exploit.

The Fix: Drain All Free Connections on a Stale Error

Instead of retrying once and hoping the next connection is healthy, destroy every connection in _freeConnections the moment you detect a stale error. The pool's own logic then forces a fresh connection on retry.

const STALE_ERRORS = new Set([
  'PROTOCOL_CONNECTION_LOST',
  'ECONNRESET',
  'EPIPE',
  'ETIMEDOUT',
  'ECONNREFUSED',
]);

function drainFreeConnections(pool) {
  // Destroy all idle connections in one sweep
  for (const conn of pool._freeConnections) {
    conn.destroy(); // sets conn._pool = null, removes from _allConnections
  }
  pool._freeConnections.length = 0; // clear the array in-place
  // pool._allConnections.length is now reduced → next getConnection()
  // sees: freeConnections empty + allConnections < connectionLimit
  // → creates a fresh TCP connection automatically
}

async function executeQuery(pool, sql, params) {
  let connection;
  try {
    connection = await pool.getConnection();
    const [rows] = await connection.query(sql, params);
    connection.release();
    return rows;
  } catch (err) {
    if (connection) connection.destroy(); // kill the one that triggered the error

    if (STALE_ERRORS.has(err.code)) {
      // Nuke all remaining free connections — they're all suspect
      drainFreeConnections(pool);

      // Retry — pool is now forced to open a fresh connection
      const freshConn = await pool.getConnection();
      try {
        const [rows] = await freshConn.query(sql, params);
        freshConn.release();
        return rows;
      } catch (retryErr) {
        freshConn.destroy();
        throw retryErr;
      }
    }

    throw err;
  }
}

Why `pool._freeConnections.length = 0` instead of a `while` loop with `shift()`?

pool._freeConnections is an array used as a FIFO queue — mysql2 uses shift() when getting connections and push() when releasing them. Since we're destroying all of them, iterating with for...of then zeroing the length is simpler and safer than mutating the array mid-iteration with shift().

What `conn.destroy()` does under the hood

When you call destroy(), mysql2 does:

// Simplified from connection._removeFromPool()
this._allConnections.splice(this._allConnections.indexOf(connection), 1);
connection._pool = null;
// TCP socket is closed

So after drainFreeConnections:

_freeConnections → empty ✅
_allConnections.length → drops back toward 0 ✅
Next getConnection() → pool sees room under connectionLimit → creates fresh TCP connection ✅

Comparison: Single Retry vs. Drain All

Approach	After 1 stale conn	After full pool stale
Single retry (article)	✅ Works	❌ Retry hits another stale conn
Drain all + retry (this approach)	✅ Works	✅ Pool forced to create fresh conn

A Note on `_freeConnections` Being a Private API

_freeConnections is not exported in mysql2's TypeScript typings (it's missing from Pool.d.ts). In TypeScript, you'll need a cast:

(pool as any)._freeConnections

It has been stable and present since the beginning of the library. But since it's not officially part of the public API, it's worth keeping an eye on across major version upgrades.

Why Prisma Doesn't Have This Problem

If you're using Prisma, you may have noticed it doesn't suffer from the freeze/thaw stale connection issue as badly. There's a concrete reason for this — it's not magic, it's the Rust query engine.

Prisma uses a connection pool built on the mobc library inside its Rust engine. Before handing a connection to your query, it performs a time-gated pre-ping:

Connection pulled from pool
         ↓
Has more than 15 seconds passed since this connection was last used?
    YES → run SELECT 1
              ↓
         SELECT 1 succeeds? → proceed with query
         SELECT 1 fails?    → discard, open fresh connection
    NO  → skip ping, proceed directly (optimization for rapid queries)

This is essentially the same pattern as SQLAlchemy's pool_pre_ping=True, with a 15-second grace window to avoid pinging on every rapid-fire query.

After a Lambda freeze of any meaningful duration (seconds to minutes), the timer has expired, so Prisma will ping before your query even runs — and silently replace any dead connection. Your application code never sees the error.

How it stacks up

	mysql2 (raw pool)	Prisma (Rust engine)
Pre-ping on checkout	❌ None	✅ If >15s idle
Handles full pool going stale	❌ Needs manual drain logic	✅ Each connection validated individually
Error surfaces to app code	✅ Yes — you must handle it	❌ Transparent — retried internally
Overhead	None (no extra queries)	One `SELECT 1` per connection after idle period

Even Prisma Isn't Bulletproof

Worth noting: Prisma's pre-ping protects against stale connections, but the 15-second threshold means a freeze shorter than 15 seconds could still theoretically slip through. And connection-level issues outside the pool (e.g. NAT gateway state tables, RDS Proxy recycling) can still cause failures that the pre-ping doesn't catch. Retry logic at the application layer remains a good safety net regardless of ORM.

Summary

The key insight is to work with mysql2's internal pool logic rather than against it:

On a stale connection error, don't just retry — drain _freeConnections first
mysql2 will automatically open fresh connections to fill the gap (it's built into getConnection())
Your retry then gets a genuinely new TCP connection instead of another dead one from the queue

If you want this behavior without managing it yourself, Prisma's Rust engine gives you a time-gated pre-ping out of the box — which is the more principled long-term solution for serverless MySQL workloads.

Building a Clean Event Pipeline in Spring: From Simple Events to Async Listeners to the Outbox Pattern

Jayesh Shinde — Sat, 03 Jan 2026 07:43:05 +0000

Event‑driven architecture sounds simple on paper: “emit an event when something happens.”

But once you start implementing it inside a real Spring Boot service, you quickly discover the hidden trade‑offs.

In this post, we’ll walk through a real‑world progression:

emitting domain events inside a service
handling them with @EventListener
realizing enrichment logic slows down the request
making listeners async
adding a production‑grade executor
and finally touching the gold standard: the Outbox Pattern

Let’s dive in.

1. The initial requirement: emit an event inside the service

Imagine a simple use case: when a user is created, we want to emit an event so other parts of the system can react.

A clean way to do this in Spring is to wrap ApplicationEventPublisher:

@Component
@AllArgsConstructor
public class UserEventPublisher {
    private final ApplicationEventPublisher applicationEventPublisher;

    public <T> void publish(T event) {
        applicationEventPublisher.publishEvent(event);
    }
}

Now inside your service:

userEventPublisher.publish(new UserCreatedEvent(userId, email));

2. Handling the event with `@EventListener`

A simple listener:

@Component
public class UserAuditListener {

    @EventListener
    public void handleUserCreateEvent(UserCreatedEvent event) {
        System.out.println("User created: " + event);
    }
}

This works beautifully… until you need to do more than just print.

3. The problem: enrichment logic slows down the request

Let’s say before publishing to Kafka, you want to:

fetch additional data from DB
call another service
enrich the event payload

Since @EventListener is synchronous by default, all this work blocks the original request thread.

Your API response time suddenly spikes.

Not good.

4. Making listeners async with `@Async` and `@EnableAsync`

Spring makes this easy:

@EnableAsync
@SpringBootApplication
public class App { }

And in the listener:

@Async
@EventListener
public void handleUserCreateEvent(UserCreatedEvent event) {
    // runs in a background thread
}

Now the main request returns immediately while the listener does its work asynchronously.

But there’s a catch…

5. The default executor is not production‑grade

If you don’t configure anything, Spring uses SimpleAsyncTaskExecutor:

creates a new thread per task
no pooling
no backpressure
no monitoring

This is fine for demos, not for real systems.

6. Adding a custom executor

A better approach is to define your own thread pool:

@Configuration
@EnableAsync
public class AsyncConfig {

    @Bean(name = "taskExecutor")
    public Executor taskExecutor() {
        return Executors.newCachedThreadPool();
    }
}

Now all @Async methods use this executor.

You can replace it with a tuned ThreadPoolTaskExecutor for even more control.

7. The gold standard: the Outbox Pattern

Async listeners solve the latency problem, but they don’t solve the reliability problem.

What if:

the DB transaction commits
but the async listener fails before sending to Kafka?
or the service crashes?

You lose the event.

This is why mature systems use the Outbox Pattern.

How the Outbox Pattern works (high‑level)

Write the event into an “outbox” table inside the same DB transaction
- If the user is created, the outbox record is also created
- Atomic, consistent, no partial failures
A background process reads the outbox table

This can be:
- a scheduled Spring job
- a Kafka Connect Debezium connector
- a lightweight polling thread
The background process publishes the event to Kafka
After successful publish, the outbox record is marked as processed

Why this is the gold standard

no lost events
no double‑publishing
no dependency on async listeners
fully decoupled from request latency
battle‑tested at Uber, Netflix, Stripe, Shopify

8. Summary

Here’s the journey we walked through:

Start with simple Spring events
Add @EventListener to react to them
Realize enrichment logic slows down the request
Add @Async + @EnableAsync to make listeners non‑blocking
Add a custom executor for production‑grade async processing
Finally, adopt the Outbox Pattern for guaranteed delivery and reliability

This progression mirrors how real systems evolve as they scale.

If you’re building event‑driven microservices, the outbox pattern is the foundation you eventually want to reach.

How a Cache Invalidation Bug Nearly Took Down Our System - And What We Changed After

Jayesh Shinde — Fri, 05 Dec 2025 01:57:56 +0000

A few weeks ago, we had one of those production incidents that quietly start in the background and explode right when the traffic peaks.
This one involved Aurora MySQL, a Lambda with a 30-second timeout, and a poorly-designed cache invalidation strategy that ended up flooding our database.

Here’s the story, what went wrong, and the changes we made so it never happens again.

🎬 The Setup

The night before the incident, we upgraded our Aurora MySQL engine version.
Everything looked good. No alarms. No red flags.

The next morning around 8 AM, our daily job kicked in — the one responsible for:

deleting the stale “master data” cache
refetching fresh master data from the DB
storing it back in cache

This master dataset is used Application to work correctly, so if the cache isn’t warm, the DB gets hammered.

💥 The Explosion

Right after the engine upgrade, a specific query in the Lambda suddenly started taking 30+ seconds.
But our Lambda had a 30-second timeout.

So what happened?

The cacheInvalidate → cacheRebuild flow failed.
The cache remained empty.
Every user request resulted in a cache miss.
All those requests hit the DB directly.
Aurora CPU spiked to 99%.
Application responses stalled across the board.

Classic cache stampede.

We eventually triggered a failover, and luckily the same query ran in ~28.7 seconds on the new writer, just under the Lambda timeout. That bought us a few minutes to stabilize.

Later that night, we found the real culprit:
➡️ The query needed a new index, and the upgrade changed its execution plan.

We created the index via a hotfix, and the DB stabilized.

But the deeper problem was our cache invalidation approach.

🧹 Our Original Cache Invalidation: Delete First, Hope Later

Our initial flow was:

Delete the existing cache key
Fetch fresh data from DB
Save it back to cache

If step 2 fails, everything collapses.

It’s simple… until it isn’t.
In our case, the Lambda failed to fetch fresh data, so the cache stayed empty.

🔧 What We Changed (and Recommend)

1. Never delete the cache before you have fresh data

We inverted the flow:

Fetch → Validate → Update cache
Only delete if we already have fresh data ready

This eliminates the “empty cache” window.

2. Use “stale rollover” instead of blunt deletion

If the refresh job fails, we now:

rename the key
- "Master-Data" → "Master-Data-Stale"
keep the old value available
add an internal notification so the team can investigate

This ensures that even if the DB is slow or down, the system still has something to serve.

It’s not ideal, but it prevents a meltdown.

3. API layer now returns stale data when fresh data is unavailable

The API logic became:

Try to read "Master-Data"
If not found:
- Attempt to rebuild (only if allowed)
- If rebuild fails → return stale data

This avoids cascading failures.

4. Add a Redis distributed lock to prevent cache stampede

Without this, even if stale data existed, multiple API nodes or Lambdas could all try to rebuild simultaneously — hammering the DB again.

With a Redis lock:

Only one request gets the lock and rebuilds
Others:
- Do not hit DB
- Simply return stale data
- Wait for the winner to repopulate the cache

This one change alone eliminates 90% of stampede risk.

Node.js — Acquire Distributed Lock (Redis)

Below is a simple Redis-based lock using SET NX PX (no external library).
You can replace redis client with ioredis or node-redis based on your stack.

// redis.js
const { createClient } = require("redis");

const redis = createClient({
  url: process.env.REDIS_URL
});
redis.connect();

module.exports = redis;

Acquiring and Releasing the Lock

// lock.js
const redis = require("./redis");
const { randomUUID } = require("crypto");

const LOCK_KEY = "lock:master-data-refresh";
const LOCK_TTL = 10000; // 10 seconds

async function acquireLock() {
  const lockId = randomUUID();

  const result = await redis.set(LOCK_KEY, lockId, {
    NX: true,
    PX: LOCK_TTL
  });

  if (result === "OK") {
    return lockId; // lock acquired
  }

  return null; // lock not acquired
}

async function releaseLock(lockId) {
  const current = await redis.get(LOCK_KEY);

  if (current === lockId) {
    await redis.del(LOCK_KEY);
  }
}

module.exports = { acquireLock, releaseLock };

Usage

const { acquireLock, releaseLock } = require("./lock");

async function refreshMasterData() {
  const lockId = await acquireLock();

  if (!lockId) {
    console.log("Another request is refreshing. Returning stale data.");
    return getStaleData();
  }

  try {
    const newData = await fetchFromDB();
    await saveToCache(newData);
    return newData;
  } finally {
    await releaseLock(lockId);
  }
}

5. Add observability around refresh times

We now record:

query execution time
cache refresh duration
lock acquisition metrics
alerts when a refresh exceeds a threshold

The goal is to catch slowdowns before timeout happens.

📝 Key Takeaways

Engine upgrades can change execution plans, sometimes dramatically.
Always benchmark critical queries after major DB changes.
Cache invalidation strategies must assume that refresh can fail.
Serving stale-but-valid data is often better than serving errors.
Distributed locks are essential in preventing cache stampede.

🚀 Final Thoughts

The incident was stressful, but the learnings were worth it.
Caching problems rarely show up during normal traffic — they appear right when your system is busiest.

If you have a similar “delete-then-refresh” pattern somewhere in your application… you may want to review it before it reviews you.

🧩 From 15 Minutes to Infinite: Scaling STT Jobs with AWS Batch

Jayesh Shinde — Sun, 09 Nov 2025 05:25:17 +0000

💡 The Problem

We recently ran into a production issue — our Speech-to-Text (STT) service stopped working for a few hours.
The feature was fixed quickly, but the transcripts for that downtime were missing.

Luckily, in Amazon Connect, all call recordings are stored in S3.
So the audio was there, but no transcripts.

We needed to reprocess all those missed files — fast.

🧠 First Attempt: Lambda (and its Limitations)

We quickly built a Lambda function to process unprocessed files from S3.

It worked fine — until it didn’t.
AWS Lambda has a 15-minute execution limit, and processing large audio files can easily exceed that.

We could have switched to EC2, but that felt like using a hammer for a small screw — no auto-scaling, no graceful shutdown, no built-in retry or job management.

We needed something that behaved like a job, not a script.

🚀 Enter AWS Batch + Fargate

That’s when AWS Batch came to the rescue.
It’s perfect for this kind of workload — long-running, batch-style, event-driven jobs.

Here’s the setup we used:

Created a Compute Environment

Backed by AWS Fargate → no EC2 management.
Scales automatically depending on job load.

Defined a Job Queue

All reprocessing jobs will be submitted here.
The queue ensures controlled concurrency and retries.

Built a Job Definition

Packaged our STT processing logic as a Docker image.
Uploaded it to Amazon ECR.
Defined required vCPU and memory for each job.

Triggered via Lambda

A small Lambda fetches a list of unprocessed S3 files.
For each batch (say 50 files), it submits a Batch Job.

⚙️ The Flow in Action

Lambda → Checks for unprocessed audio files in S3.
Lambda → AWS Batch: Submits a job to process them.
AWS Batch (Fargate) spins up compute, runs the job.
Job → Downloads audio → runs STT → uploads transcript → updates metadata.
Fargate shuts down automatically when the job finishes.

No idle servers, no manual cleanup, no stress.

🧩 Why This Design Rocks

✅ Serverless all the way — Lambda + Fargate + S3
✅ Auto-scaling compute — no EC2 to babysit
✅ Long-running safe zone — runs beyond Lambda’s 15-min cap
✅ Reusable — we can reprocess any backlog anytime
✅ Cost-efficient — pay only for what’s used

🪄 Bonus Tip

You can even schedule a “missed transcript” job to run daily or weekly,
checking for any files without transcripts and triggering a Batch job automatically.

🧩 Understanding AWS Batch Scaling

In AWS Batch, the number of tasks (containers) that run in parallel depends on three things working together:

Compute Environment capacity
→ e.g., your environment has a maximum of 10 vCPUs.
Job Definition requirements
→ e.g., each job needs 1 vCPU.
How many jobs are in the queue (and their array size, if used).

🔹 Case 1: You Submit Multiple Independent Jobs

If you submit 10 jobs, each with 1 vCPU, and your environment allows 10 vCPUs,
then AWS Batch can run all 10 in parallel (subject to available Fargate capacity).

Example:

# pseudo example
for i in {1..10}; do
  aws batch submit-job \
    --job-name process-audio-$i \
    --job-queue my-queue \
    --job-definition my-job-def
done

Each job = 1 vCPU → up to 10 can run simultaneously.

AWS Batch’s Job Scheduler will automatically pack as many as possible based on available compute.

🔹 Case 2: You Use an Array Job

Instead of manually looping, you can submit an array job.

Example:

aws batch submit-job \
  --job-name process-audios \
  --job-queue my-queue \
  --job-definition my-job-def \
  --array-properties size=10

This creates 10 child jobs under a single parent, each running independently (great for S3 list chunking).

Same result — 10 parallel containers, each with 1 vCPU.

🔹 Case 3: You Submit a Single Job that Needs More vCPUs

If you set in your job definition:

"vcpus": 4

and your environment has 10 total vCPUs →
then Batch will reserve 4 vCPUs for that job, leaving room for other smaller jobs.

So the compute environment doesn’t spawn “10 copies automatically” —
it just enforces a maximum pool of total CPU that concurrent jobs can consume.

⚙️ TL;DR — How to Scale

Goal	What to Do
Run multiple tasks concurrently	Submit multiple jobs or an array job
Each job’s CPU need	Defined in Job Definition (e.g., 1 vCPU)
Max parallel limit	Based on compute environment capacity
Control at runtime	You can pass `--array-properties size=N` dynamically
Scaling behavior	Batch automatically scales Fargate/EC2 capacity up/down

🏁 Closing Thoughts

This experience reminded me —

“When your script starts feeling like a job, give it job-like powers.”

AWS Batch (especially with Fargate) is often underrated,
but it’s a powerful tool when you need on-demand, containerized, long-running compute
without managing any servers.

Reusing HTTP and SDK clients in AWS Lambda to avoid “too many open files” (FD) errors

Jayesh Shinde — Wed, 15 Oct 2025 05:04:25 +0000

TL;DR

We hit sporadic network errors in a high-throughput Lambda that made HTTP calls (Axios) and AWS SDK calls.
Root cause: creating new HTTP clients/agents per invocation ballooned the number of open sockets (file descriptors).
Fix: initialize clients and their https.Agent once at module scope with keep-alive and reuse them across warm invocations. For AWS SDK v2, also set AWS_NODEJS_CONNECTION_REUSE_ENABLED=1.

The scenario

We had a Lambda that was invoked asynchronously to process a large dataset (thousands of events). Inside the handler, we created an Axios client and AWS SDK client(s) for each invocation. Under sustained concurrency, we started seeing intermittent network failures.

Symptoms we saw

These popped up in CloudWatch logs while the Lambda was busy:

“too many open files” errors:
- Error: EMFILE: too many open files, open
- NodeError: getaddrinfo ENFILE
Connection instability:
- AxiosError: socket hang up
- Error: read ECONNRESET
- Error: connect ECONNRESET
Occasional timeouts and throttling-like behavior despite healthy downstream services

These were worse during bursts when many async invocations overlapped.

What’s really happening (FDs and sockets in Lambda)

Every TCP connection (HTTP/HTTPS) consumes a file descriptor (FD).
Lambda execution environments have a relatively low per-process FD limit (commonly around 1024).
If you create a new HTTP client (and thus a new https.Agent) per invocation, each agent can open many sockets. Under high concurrency, you exhaust FDs, leading to the errors above.
Lambda reuses the same execution environment for multiple “warm” invocations. Objects created at module scope are kept alive and reused, which is exactly what we want for clients and connection pools.

Why Node’s `https.Agent` matters

The agent controls connection pooling and keep-alive.
Creating a new agent per invocation increases the number of socket pools and the total sockets in use.
Reusing a single agent keeps the number of open sockets bounded and allows connection reuse across requests, reducing FD pressure and latency.

The anti-pattern (what we had)

Creating new clients and agents inside the handler:

import axios from 'axios';
import https from 'https';

// Anti-pattern: runs on every invocation
export const handler = async () => {
  const ax = axios.create({
    httpsAgent: new https.Agent(), // new agent each time
  });

  const resp = await ax.get('https://api.example.com/data');
  return resp.data;
};

Same issue with AWS SDK if you new a client per invocation, especially if you also create its own agent.

The fix (module-level reuse with keep-alive)

Move client and agent creation to module scope so they’re created once per warm environment and then reused.

Axios

import axios from 'axios';
import https from 'https';

const httpsAgent = new https.Agent({
  keepAlive: true,
  maxSockets: 60,         // tune based on expected concurrency per environment
  maxFreeSockets: 10,
  timeout: 30_000,        // socket idle timeout
  freeSocketTimeout: 30_000,
});

const ax = axios.create({
  headers: { 'Content-Type': 'application/json' },
  httpsAgent,
});

export const handler = async () => {
  const resp = await ax.get('https://api.example.com/data');
  return resp.data;
};

AWS SDK v3

import https from 'https';
import { NodeHttpHandler } from '@aws-sdk/node-http-handler';
import { S3Client } from '@aws-sdk/client-s3';

const httpsAgent = new https.Agent({
  keepAlive: true,
  maxSockets: 60,
  maxFreeSockets: 10,
  timeout: 30_000,
  freeSocketTimeout: 30_000,
});

const s3 = new S3Client({
  region: process.env.AWS_REGION,
  requestHandler: new NodeHttpHandler({
    httpsAgent,
    connectionTimeout: 3_000,
    socketTimeout: 30_000,
  }),
});

export const handler = async () => {
  const out = await s3.listBuckets({});
  return out;
};

AWS SDK v2

Reuse clients, and enable connection reuse via env var.

import https from 'https';
import AWS from 'aws-sdk';

// Also set in Lambda env: AWS_NODEJS_CONNECTION_REUSE_ENABLED=1
AWS.config.update({
  region: process.env.AWS_REGION,
  httpOptions: { agent: new https.Agent({ keepAlive: true, maxSockets: 60 }) },
});

const s3 = new AWS.S3();

export const handler = async () => {
  const out = await s3.listBuckets().promise();
  return out;
};

Results after the change

FD-related errors (EMFILE, ENFILE, socket hang ups) disappeared under the same workload.
Lower p95 latency due to connection reuse.
Fewer outbound connection spikes visible on NAT Gateway/ENI metrics (for VPC Lambdas).
More predictable behavior during bursts.

Bonus mitigations

Concurrency control: use SQS with a sane maxConcurrency/batchSize, reserved concurrency, or step-wise throttling to prevent bursts from scaling FD usage across many environments at once.
Timeouts and retries: set realistic timeouts; add backoff with jitter to avoid synchronized retries.
context.callbackWaitsForEmptyEventLoop = false: can help the handler return even if the agent keeps idle sockets open (don’t overuse).
Consider undici for HTTP in Node 18+; it provides efficient HTTP/1.1 keep-alive by default.

Quick checklist

Initialize HTTP clients and SDK clients at module scope.
Use a shared https.Agent with keepAlive: true; set maxSockets, maxFreeSockets, and timeouts.
For AWS SDK v2, set AWS_NODEJS_CONNECTION_REUSE_ENABLED=1.
Avoid creating clients/agents inside loops or inside the handler.
Monitor and tune under realistic concurrency.

Closing thoughts

FD exhaustion is easy to miss until traffic scales. In serverless, the simplest lever is to reuse resources across warm invocations. One shared agent + one shared client per execution environment eliminates a whole class of flaky, intermittent network issues.

🧠 How We Upgraded Our WordPress Search with OpenSearch Neural + Cohere for Multilingual Semantic Search

Jayesh Shinde — Sat, 11 Oct 2025 04:32:51 +0000

At our company, we use WordPress as a knowledge base (KB) for internal articles.

We indexed those articles in OpenSearch, but the default keyword search felt… old-school.

Searching “Netflix subscription” missed “How to manage ネットフリックス plans”.
Searching “AWS cost optimization” returned random hits because keywords didn’t align.

So we upgraded our search to semantic search using the OpenSearch ML plugin + Cohere embeddings served via Amazon Bedrock — a great combination for multilingual understanding and secure enterprise integration.

⚙️ The Problem

Our initial setup used simple keyword mapping:

{
  "properties": {
    "title": { "type": "text" },
    "content": { "type": "text" }
  }
}

Even after tweaking analyzers, users searching in Japanese (katakana) or English weren’t getting expected matches.

For example, “netflix” and “ネットフリックス” should be the same — but OpenSearch treated them as completely different tokens.

🚀 The Plan

We wanted to add semantic search on top of our existing index.
That means converting both documents and queries into vectors using the same embedding model, and comparing them using cosine similarity.

Our plan:

Create an ML connector for the Cohere embedding API
Create a model for document embeddings (input_type = search_document)
Create another model for query embeddings (input_type = search_query)
Update our ingestion pipeline to generate vectors during indexing
Use script_score (or kNN query) to retrieve the best semantic matches

🔌 Step 1: Create a Connector (to Cohere)

First, we create an ML connector in OpenSearch to call Cohere’s API.

POST _plugins/_ml/connectors/_create
{
  "name": "bedrock-cohere-doc-connector",
  "description": "Amazon Bedrock connector for Cohere embeddings (document)",
  "protocol": "aws_sigv4",
  "parameters": {
    "region": "us-east-1",
    "service_name": "bedrock",
    "model_id": "cohere.embed-multilingual-v3",
    "input_type": "search_document"
  }
}

Here:

protocol: aws_sigv4 makes OpenSearch sign Bedrock API calls with IAM credentials.

model_id refers to Cohere’s multilingual embedding model hosted on Bedrock.

We use input_type=search_document for document-level embeddings.

🧩 Step 2: Register the Model

Now, we create a model using the connector:

POST _plugins/_ml/models/_register
{
  "name": "bedrock-cohere-doc-embed",
  "function_name": "embedding",
  "connector_id": "bedrock-cohere-doc-connector"
}

POST _plugins/_ml/models/bedrock-cohere-doc-embed/_deploy

This model will be used in our ingestion pipeline.

Then deploy it:

POST _plugins/_ml/models/cohere-doc-embed-model/_deploy

OpenSearch now knows how to call Cohere to embed documents.

💾 Step 3: Create the Index with a Vector Field

Next, we create our knowledge base index that includes a vector field for embeddings.

PUT kb-articles
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "title": { "type": "text" },
      "content": { "type": "text" },
      "embedding": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "name": "hnsw",
          "space_type": "cosinesimil",
          "engine": "nmslib",
          "parameters": {
            "ef_construction": 512,
            "m": 16
          }
        }
      }
    }
  }
}

The vector field embedding will hold our document-level embeddings from Cohere (1024 dimensions).

🧠 Step 4: Add an Ingest Pipeline

We’ll generate document embeddings automatically during ingestion.

PUT _ingest/pipeline/kb-embed-pipeline
{
  "processors": [
    {
      "ml_inference": {
        "model_id": "bedrock-cohere-doc-embed",
        "input_map": { "title": "text" },
        "output_map": { "embedding": "embedding" }
      }
    }
  ]
}

Now, when we index a document:

POST kb-articles/_doc?pipeline=kb-embed-pipeline
{
  "title": "Netflix subscription help",
  "content": "Steps to manage your Netflix account and billing."
}

OpenSearch automatically calls Amazon Bedrock, retrieves Cohere’s embedding, and stores it in the embedding vector field.

💬 Step 5: Handle User Queries with Another Connector

Initially, I used the same connector (with input_type=search_document) for both documents and queries.
That caused a mismatch — “ネットフリックス” (Katakana) and “Netflix” were still not matching.

The fix was to create another connector and model specifically for query embeddings.

POST _plugins/_ml/connectors/_create
{
  "name": "bedrock-cohere-query-connector",
  "description": "Amazon Bedrock connector for Cohere embeddings (query)",
  "protocol": "aws_sigv4",
  "parameters": {
    "region": "us-east-1",
    "service_name": "bedrock",
    "model_id": "cohere.embed-multilingual-v3",
    "input_type": "search_query"
  }
}

and then register and deploy another model:

POST _plugins/_ml/models/_register
{
  "name": "bedrock-cohere-query-embed",
  "function_name": "embedding",
  "connector_id": "bedrock-cohere-query-connector"
}

POST _plugins/_ml/models/bedrock-cohere-query-embed/_deploy

This ensures both documents and queries are embedded in compatible vector spaces.

Now, when we run a search, we first generate a query embedding via the ML model’s /predict endpoint:

POST _plugins/_ml/models/bedrock-cohere-query-embed/_predict
{
  "text": "ネットフリックス subscription"
}

🔍 Step 6: Semantic Search Query

Finally, we plug the embedding into a script_score query to rank results by cosine similarity:

POST kb-articles/_search
{
  "size": 5,
  "query": {
    "script_score": {
      "query": { "match_all": {} },
      "script": {
        "source": "cosineSimilarity(params.query_vector, 'embedding') + 1.0",
        "params": {
          "query_vector": [/* embedding array from _predict */]
        }
      }
    }
  },
  "sort": [
    { "_score": { "order": "desc" } }
  ]
}

Now “Netflix” and “ネットフリックス” both match beautifully.
🎯 Cohere’s multilingual embeddings + OpenSearch vector search did the trick.

🧩 Why We Chose Cohere

Strong multilingual understanding — perfect for English + Japanese content
Easy integration — just plug in an API key via OpenSearch ML connector
Consistent embedding dimensions — works well with knn_vector
Fast inference — good for production-scale pipelines

🪛 Troubleshooting Tips

Problem	Cause	Fix
Katakana & English not matching	Used `search_document` for query embeddings	Create a separate connector with `input_type=search_query`
“Dimension mismatch” errors	Wrong embedding model or field dimension	Make sure both model & field use same `dimension`
Inference timeout	Cohere API rate limits	Batch or cache embeddings during ingestion
Weird scores	Missing `space_type: cosinesimil`	Use cosine similarity in mapping

✅ Summary

We started with basic keyword search and ended up with multilingual semantic search using:

OpenSearch ML plugin
Cohere embedding models
Cosine similarity on knn_vector
Separate connectors for documents vs. queries

✨ Outcome

After integrating Cohere via Bedrock and separating the input types:

✅ We can now search using phrases, not just keywords

🌏 Cross-lingual search works — “Netflix” ≈ “ネットフリックス”

💬 Semantic matching improved drastically (e.g., “streaming issue” finds “再生エラー”)

📈 Search relevance and recall are noticeably better, even for non-English content

The combination of OpenSearch ML plugin + Cohere embeddings via Bedrock turned our keyword search into a truly semantic multilingual search engine — all running within the AWS ecosystem.

💡 If you’re building multilingual or brand-sensitive search, don’t skip the input_type difference — it can make or break your semantic matching.

🛠️ Fixing Lost SecurityContext and Correlation IDs in Async Calls with Spring Boot

Jayesh Shinde — Sun, 05 Oct 2025 11:57:58 +0000

When we started parallelizing API calls in our Spring Boot service using CompletableFuture and a custom ExecutorService, everything looked great… until we checked the logs.

Our JWT SecurityContext wasn’t available in the async threads.
Our MDC correlation IDs (used for distributed tracing/log correlation) were missing too.

That meant downstream services didn’t know who was calling, and our logs lost the ability to tie requests together. Not good.

🚨 The Problem

Spring Security stores authentication in a ThreadLocal (SecurityContextHolder).

SLF4J’s MDC (Mapped Diagnostic Context) also uses ThreadLocal to store correlation IDs.

When you hop threads (e.g., via CompletableFuture.supplyAsync), those ThreadLocal values don’t magically follow along. So in worker threads:

SecurityContextHolder.getContext() → empty
MDC.get("correlationId") → null

✅ The Solution: Wrap the Executor

We solved this by wrapping our ExecutorService in a lightweight decorator that captures the MDC + SecurityContext from the submitting thread and restores them inside the worker thread.

Here’s the implementation:

public class ContextPropagatingExecutorService extends AbstractExecutorService {

    private final ExecutorService delegate;

    public ContextPropagatingExecutorService(ExecutorService delegate) {
        this.delegate = delegate;
    }

    private Runnable wrap(Runnable task) {
        final Map<String, String> mdc = MDC.getCopyOfContextMap();
        final SecurityContext securityContext = SecurityContextHolder.getContext();

        return () -> {
            if (mdc != null) MDC.setContextMap(mdc);
            if (securityContext != null) SecurityContextHolder.setContext(securityContext);
            try {
                task.run();
            } finally {
                MDC.clear();
                SecurityContextHolder.clearContext();
            }
        };
    }

    @Override
    public void execute(Runnable command) {
        delegate.execute(wrap(command));
    }

    // delegate lifecycle methods...
}

⚙️ Wiring It Up

In our Spring Boot config:

@Configuration
public class AsyncConfig {

    @Bean(name = "executorService")
    public ExecutorService executorService() {
        ExecutorService base = Executors.newCachedThreadPool();
        return new ContextPropagatingExecutorService(base);
    }
}

Now, whenever we do:

CompletableFuture<AccountDTO> fromAccount =
    CompletableFuture.supplyAsync(() -> accountClient.getAccountsById(id), executorService);

…the async thread has the same SecurityContext and MDC as the request thread.

📊 Before vs After

Aspect	Before	After
`SecurityContextHolder.getContext()`	Empty in async thread	Correctly populated
`MDC.get("correlationId")`	`null`	Same correlation ID as request thread
Logs	Missing trace IDs	Full traceability across async calls
Downstream services	No JWT propagated	JWT available for Feign/RestTemplate

🔑 Takeaways

ThreadLocals don’t cross thread boundaries — you need to propagate them manually.
Wrapping your ExecutorService is a clean, reusable fix.
This pattern works not just for MDC + SecurityContext, but for any contextual data you need across async boundaries.

🚀 Closing Thoughts

If you’re building microservices with Spring Boot and using async execution (CompletableFuture, @Async, Kafka listeners, etc.), don’t forget about context propagation. Without it, your logs and security checks will silently break.

Wrapping your executor is a small change that pays off big in observability and security consistency.

Why My CDK Deploys Started Failing After Org Added Strict SCP Rules

Jayesh Shinde — Sat, 27 Sep 2025 07:56:29 +0000

Recently, I ran into a head‑scratcher while deploying a CDK stack. Everything used to work fine, but once my organization introduced strict SCP rules based on tags, my cdk deploy started failing with:

AccessDenied: action cloudformation:CreateChangeSet is not authorized

At first glance, it didn’t make sense. I was already tagging everything in my CDK code. I even had a loop that pulled tags from props.Tags and attached them with cdk.Tags.of(this).add(...). These tags used to flow down nicely to all resources — including the CloudFormation stack itself.

So why did it suddenly stop working? 🤔

What Changed? SCPs and Request‑Time Enforcement

The key is where SCP rules get evaluated.

An SCP can restrict not only what resources exist, but also what API calls are allowed. In my case, the org had a policy like this:(dummy example)

{
  "Effect": "Deny",
  "Action": "cloudformation:CreateChangeSet",
  "Resource": "*",
  "Condition": {
    "StringNotEquals": {
      "aws:RequestTag/Org": "ABC"
    }
  }
}

This means: if the CreateChangeSet request doesn’t already include the required tag, the call is blocked right away.

CDK Tagging: Two Different Worlds

This is where CDK behavior matters:

StackProps.tags
When you pass tags into the super(scope, id, props) constructor, CDK includes those tags in the CreateChangeSet API call. These show up as RequestTags. That’s what SCPs check.
cdk.Tags.of(resource).add(...)
This method attaches tags to resources inside the CloudFormation template. They are applied after the stack is already created.

So my old approach of looping through props.Tags and calling cdk.Tags.of(this).add(...) worked fine in the past, but now fails because the SCP never lets the stack get created in the first place. The required tags simply aren’t present yet at request time.

Fix: Pass Tags via StackProps

The solution was simple once I understood the difference. Previously my props had Tags property which I replaced with tags which props:cdk.StackProps expects and uses to initialize CDK’s in-memory construct tree (the Stack object inside your app).
When you run cdk deploy, the CLI uses the CloudFormation SDK to call:

CreateChangeSet (or UpdateChangeSet) → this is the first API call to AWS.

This is where stack-level tags (props.tags) are injected into the request payload

export class MyStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

      for (const [k, v] of Object.entries(props.tags)) {
        cdk.Tags.of(this).add(k, v);
      }

  }
}

new MyStack(app, "MyTaggedStack", {
  tags: {
    Org: "ABC",
    Owner: "TeamA"
  }
});

Now:

StackProps.tags → go straight into the CreateChangeSet request (SCP passes ✅).
cdk.Tags.of(this).add(...) → still ensures all resources get the same tags after creation.

Takeaway

If your organization enforces strict SCP rules on cloudformation:CreateChangeSet, you can’t rely on cdk.Tags.of(...) alone. Those tags arrive too late. You need to use StackProps.tags so the tags are present in the request itself.

It’s a subtle but important difference — and once I understood it, the “AccessDenied” error finally made sense.

Lessons Learned: CompletableFuture Security and Kafka Topic Grouping in Microservices

Jayesh Shinde — Tue, 23 Sep 2025 08:27:53 +0000

Issue 1: CompletableFuture + Missing Security Context = 401 Unauthorized

When we started using CompletableFuture.supplyAsync() in our Spring Boot microservice, some calls to other services (via Feign client) started failing with 401 Unauthorized.

The reason:

CompletableFuture runs in a separate thread.
By default, the Spring Security context (and JWT token) is not propagated to that new thread.
As a result, our Feign interceptor couldn’t attach the Authorization header, and the downstream service rejected the request.

Fix: We wrapped our executor with DelegatingSecurityContextExecutorService so that the SecurityContext (including the JWT token) travels along with async tasks. After this change, the token was correctly added and the 401s disappeared.

Issue 2: Kafka Consumers and Group IDs

We also ran into a Kafka consumer issue. Initially, we used the same consumer group ID for multiple topics. This caused unexpected behavior:

Consumers in the same group share messages.
With a single group ID, messages from different topics were getting “competed” for by the same group, leading to missed or unbalanced consumption.

Fix: We gave each topic its own unique group ID. This way, each topic is consumed independently, and no messages are skipped.

Takeaway:

For async code in Spring Security, always make sure the security context is propagated.
For Kafka, think carefully about group IDs — same group = load balancing, different groups = independent consumption.

Debugging Helm Template Errors: Lessons Learned

Jayesh Shinde — Sun, 21 Sep 2025 11:53:06 +0000

Today while working with Helm to package my Kubernetes deployments, I hit a couple of tricky but interesting issues. Writing them down so that it may help others facing the same problem.

1. Keys with `-` in values.yaml

In my values.yaml, I had a key like this:

postgresql-pvc:
  storage: 1Gi

And in my template (postgresql.yaml), I tried to reference it like this:

storage: {{ .Values.postgresql.postgresql-pvc.storage }}

This failed with an error:

[ERROR] templates/: parse error: bad character U+002D '-'

Why?
Helm templates treat the - in postgresql-pvc as a subtraction operator.

Fix:
Wrap the key in quotes and use index notation:

storage: {{ index .Values.postgresql "postgresql-pvc" "storage" }}

This tells Helm clearly that it’s a key lookup, not math.

Better to avoid using "-" in the key simply use like below in values.yaml file

postgresqlPvc:
  storage: 1Gi

and then in helm template

storage: {{ index .Values.postgresql.postgresqlPvc.storage" }}

2. Numbers in `values.yaml` need quotes

The second issue I faced was with environment variables in a Deployment. My values.yaml had:

postgresql:
  servicePort: 5432

And in the Deployment template:

env:
  - name: POSTGRES_PORT
    value: {{ .Values.postgresql.servicePort }}

This failed with:

cannot unmarshal number into Go struct field EnvVar.spec.template.spec.containers.env.value of type string

Why?
Kubernetes expects environment variable values to always be strings, but Helm rendered it as a number.

Fix:
Wrap the value in quotes:

env:
  - name: POSTGRES_PORT
    value: "{{ .Values.postgresql.servicePort }}"

Or Use quote function

env:
  - name: POSTGRES_PORT
    value: {{ .Values.postgresql.servicePort | quote }}

Now Helm correctly renders it as a string.

Checklist for Debug

Env vars (value: in env) → always string

Labels and annotations → always string

Secrets/ConfigMaps → always string

Ports, replicas, resource requests → can stay as numbers

Check the Kubernetes API spec for the field type (string vs number)

Takeaway

Helm is powerful, but also picky:

Keys with - need quotes and index lookup.
Numbers used as environment variables should be wrapped in quotes.

Small things, but they save a lot of head-scratching once you know them.

Fixing Kafka KRaft Cluster ID Mismatch on Kubernetes

Jayesh Shinde — Sat, 20 Sep 2025 05:22:59 +0000

Running Kafka on Kubernetes is usually smooth—but we hit a tricky problem when using KRaft mode.

For context, KRaft (Kafka Raft Metadata mode) is Kafka’s new way of managing cluster metadata without ZooKeeper. Each cluster has a unique Cluster ID stored in its metadata. All nodes need to agree on this ID to join the cluster.

The Problem

We noticed that every time a Kafka pod restarted, it would generate a new Cluster ID. Since our pods used persistent storage, the metadata on disk still had the old Cluster ID. This caused Kafka to fail on startup with a clear error:

Cluster ID mismatch: Expected <old-id>, Found <new-id>

In short, the pod thought it was a new cluster, but the storage said otherwise.

How We Fixed It

For testing, we solved it by manually specifying the Cluster ID in the Kubernetes deployment. This ensured that every pod picked the same ID as the metadata on the persistent volume. After that, the pods started without errors and rejoined the cluster seamlessly. Below is the sample Kubernetes env variable to set it

            - name: KAFKA_KRAFT_CLUSTER_ID
              value: "kraft-local-cluster"

Lessons Learned

In KRaft mode, the Cluster ID must persist when using Kubernetes and stateful pods.
Reusing old volumes can cause mismatches if the Cluster ID changes.
For production, automate cluster ID propagation or initialize nodes with a pre-set ID to avoid manual fixes.

KRaft mode is promising, but small details like the Cluster ID can trip you up. Once you know what to look for, fixing it is straightforward—and now our Kafka cluster is more stable than ever.

Forem: Jayesh Shinde

How We Fixed Intermittent ECS Image-Not-Found Errors in AWS CDK

The Root Cause

The Strategy We Took

Sample: Dedicated ECR Stack

Sample: Point CDK to the Dedicated Repo

One Important Gotcha

Another Important Gotcha: IAM Permissions

Why This Worked Well for Us

When to Use This Pattern

Final Thought

The Silent Connection Killer: MySQL2 and AWS Lambda's Freeze/Thaw Problem

The Mystery Error

Understanding the Real Problem : How Lambda Actually Works

What Happens to Your Database Connection

Why idleTimeout Doesn't Help

Step 1: Enable TCP Keep-Alive

Step 2: Implement Retry Logic

Key Points:

What About Pool Settings?

Using RDS Proxy?

The Final Architecture

Summary

BUT....

The Problem With Just Retrying Once

How mysql2's Pool Actually Works Internally

The Fix: Drain All Free Connections on a Stale Error

Why pool._freeConnections.length = 0 instead of a while loop with shift()?

What conn.destroy() does under the hood

Comparison: Single Retry vs. Drain All

A Note on _freeConnections Being a Private API

Why Prisma Doesn't Have This Problem

How it stacks up

Even Prisma Isn't Bulletproof

Summary

Building a Clean Event Pipeline in Spring: From Simple Events to Async Listeners to the Outbox Pattern

1. The initial requirement: emit an event inside the service

2. Handling the event with @EventListener

3. The problem: enrichment logic slows down the request

4. Making listeners async with @Async and @EnableAsync

5. The default executor is not production‑grade

6. Adding a custom executor

7. The gold standard: the Outbox Pattern

How the Outbox Pattern works (high‑level)

Why this is the gold standard

8. Summary

How a Cache Invalidation Bug Nearly Took Down Our System - And What We Changed After

🎬 The Setup

💥 The Explosion

🧹 Our Original Cache Invalidation: Delete First, Hope Later

🔧 What We Changed (and Recommend)

1. Never delete the cache before you have fresh data

2. Use “stale rollover” instead of blunt deletion

3. API layer now returns stale data when fresh data is unavailable

4. Add a Redis distributed lock to prevent cache stampede

Node.js — Acquire Distributed Lock (Redis)

Acquiring and Releasing the Lock

Usage

5. Add observability around refresh times

📝 Key Takeaways

🚀 Final Thoughts

🧩 From 15 Minutes to Infinite: Scaling STT Jobs with AWS Batch

💡 The Problem

🧠 First Attempt: Lambda (and its Limitations)

🚀 Enter AWS Batch + Fargate

⚙️ The Flow in Action

🧩 Why This Design Rocks

🪄 Bonus Tip

🧩 Understanding AWS Batch Scaling

🔹 Case 1: You Submit Multiple Independent Jobs

🔹 Case 2: You Use an Array Job

🔹 Case 3: You Submit a Single Job that Needs More vCPUs

⚙️ TL;DR — How to Scale

🏁 Closing Thoughts

Reusing HTTP and SDK clients in AWS Lambda to avoid “too many open files” (FD) errors

TL;DR

The scenario

Symptoms we saw

What’s really happening (FDs and sockets in Lambda)

Why Node’s https.Agent matters

Why `pool._freeConnections.length = 0` instead of a `while` loop with `shift()`?

What `conn.destroy()` does under the hood

A Note on `_freeConnections` Being a Private API

2. Handling the event with `@EventListener`

4. Making listeners async with `@Async` and `@EnableAsync`

Why Node’s `https.Agent` matters

1. Keys with `-` in values.yaml

2. Numbers in `values.yaml` need quotes