Forem: Adam - The Developer

Your Caching Strategy Is Not a Strategy

Adam - The Developer — Mon, 06 Apr 2026 04:46:45 +0000

You have a slow endpoint. Someone suggests Redis. You add Redis. The endpoint gets faster. You ship it. Six months later you are debugging a production incident where users see stale data, your cache hit rate is 12%, and you have no idea what is actually in Redis anymore.

That is not a caching strategy. That is a prayer with an expiry time.

This post is not here to roast you. It is here to give you the patterns, the code, and the mental model to do this right.

Caching Is a Contract
The Three Classic Patterns (And When to Use Each)
Build a Cache Client Worth Using
Cache Key Design: More Important Than It Looks
Cache Invalidation: The Part Everyone Skips
- TTL vs. Event Driven Invalidation
The Thundering Herd and How to Solve It
- Solution 1: Jitter
- Solution 2: Stale While Revalidate
- Solution 3: Distributed Lock on Cache Miss
Observability: Know What Is Actually Happening
The Decision Checklist Before You Add a Cache
Putting It All Together

Caching Is a Contract

Before you touch Redis, understand what you are agreeing to.

Caching is a contract. You are telling your system: "I accept that this data may be slightly wrong for a period of time, in exchange for speed." Most teams sign that contract without reading it.

Before you add any cache entry, answer these four questions:

What is the acceptable staleness window for this data?
Who invalidates this entry and when?
What happens when the cache is cold?
What happens when the cache is wrong?

If you cannot answer all four, you do not have a caching strategy. You have optimism.

The rest of this post is about answering each of those questions with real code.

The Three Classic Patterns (And When to Use Each)

There are three fundamental ways to integrate a cache with your database. Most teams only know one and use it everywhere.

Pattern	How It Works	When to Use It
Cache Aside	App checks cache, misses go to DB, app writes to cache	Default. Works well for most read heavy workloads.
Write Through	Every write goes to DB and cache together, atomically	Read heavy data that changes infrequently. Keeps cache always warm.
Write Behind	Write to cache immediately, flush to DB asynchronously	Very high write throughput: analytics, metrics, event ingestion, rate limiting counters.

Most engineers default to cache aside everywhere, which is fine until it is not. Write behind in particular is underused. When you are recording analytics events or incrementing rate limit counters, you do not need each write to round trip to a database. Write to Redis, flush to Postgres in batches. Your database handles a fraction of the load.

The important thing is that you choose consciously. Each pattern has tradeoffs. Write behind carries real risk of data loss if Redis fails before the flush. That is acceptable for a view counter and unacceptable for a financial transaction. Know which one you are dealing with.

Build a Cache Client Worth Using

Before diving into patterns, establish a typed, reusable cache client. This becomes the foundation for everything below.

import { createClient, RedisClientType } from "redis";

export interface CacheOptions {
  ttl: number;      // base TTL in seconds
  jitter?: number;  // max random seconds to add (prevents stampedes)
  stale?: number;   // extra seconds to serve stale while revalidating
}

export interface CacheEntry<T> {
  value: T;
  cachedAt: number;
  expiresAt: number;
}

export class CacheClient {
  private client: RedisClientType;

  constructor(redisUrl: string) {
    this.client = createClient({ url: redisUrl }) as RedisClientType;
  }

  async connect(): Promise<void> {
    await this.client.connect();
  }

  protected getUnderlyingClient(): RedisClientType {
    return this.client;
  }

  private effectiveTTL(options: CacheOptions): number {
    const jitter = options.jitter
      ? Math.floor(Math.random() * options.jitter)
      : 0;
    return options.ttl + jitter;
  }

  async get<T>(key: string): Promise<CacheEntry<T> | null> {
    const raw = await this.client.get(key);
    if (!raw) return null;
    return JSON.parse(raw) as CacheEntry<T>;
  }

  async set<T>(key: string, value: T, options: CacheOptions): Promise<void> {
    const ttl = this.effectiveTTL(options);
    const staleTTL = ttl + (options.stale ?? 0);

    const entry: CacheEntry<T> = {
      value,
      cachedAt: Date.now(),
      expiresAt: Date.now() + ttl * 1000,
    };

    await this.client.setEx(key, staleTTL, JSON.stringify(entry));
  }

  async del(key: string): Promise<void> {
    await this.client.del(key);
  }

  async delByPattern(pattern: string): Promise<void> {
    const keys = await this.client.keys(pattern);
    if (keys.length > 0) {
      await this.client.del(keys);
    }
  }
}

The CacheEntry wrapper stores cachedAt and expiresAt alongside the value. This unlocks stale while revalidate later without a second Redis call.

Cache Key Design: More Important Than It Looks

This is one of the most overlooked parts of a caching system. Your key schema is an architectural decision, not a naming convention.

A good cache key answers five questions, in order:

app : v2 : user : 123 : profile
 ^     ^     ^     ^      ^
 |     |     |     |      shape or query variant
 |     |     |     entity ID
 |     |     entity type
 |     cache version
 app namespace

More examples:

app:v2:user:123:permissions
app:v2:feed:user:123:page:2:limit:20
app:v2:product:456:inventory:warehouse:uk

Bad keys look like this:

user:123
profile_123_v2
user_data_new_123
temp_user_123

No namespace means you cannot isolate patterns for deletion. No version means changing the shape of a cached object requires flushing all of Redis. No structure means you cannot delete "everything for user 123" with a single pattern.

The key schema also tells you something about your architecture. If your keys look inconsistent, your caching layer grew organically without a plan. Standardize the schema early and enforce it through a key builder:

const CACHE_VERSION = process.env.CACHE_VERSION ?? "v1";

export const keys = {
  userProfile:     (userId: string) =>
    `app:${CACHE_VERSION}:user:${userId}:profile`,
  userPermissions: (userId: string) =>
    `app:${CACHE_VERSION}:user:${userId}:permissions`,
  userFeed:        (userId: string, page: number, limit: number) =>
    `app:${CACHE_VERSION}:feed:user:${userId}:page:${page}:limit:${limit}`,
  userAll:         (userId: string) =>
    `app:${CACHE_VERSION}:user:${userId}:*`,
};

Now when you need to invalidate all data for a user, it is one call: delByPattern(keys.userAll(userId)).

To handle a breaking shape change, bump CACHE_VERSION in your deploy config. Old keys expire naturally. New requests populate the new shape. No coordinated flush against production Redis.

Cache Invalidation: The Part Everyone Skips

Most cache bugs are not "we cached the wrong thing." They are "we forgot to uncache it when the underlying data changed."

Here is the pattern that causes incidents:

// The read path is carefully thought through
async function getUserProfile(userId: string) {
  const key = keys.userProfile(userId);
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  const profile = await db.findUser(userId);
  await redis.setEx(key, 3600, JSON.stringify(profile));
  return profile;
}

// The write path does not think about the cache at all
async function updateUserProfile(userId: string, data: Partial<User>) {
  await db.updateUser(userId, data);
  // cache is now wrong. silently. for the next hour.
}

The fix is to own both paths in the same service:

export class UserService {
  constructor(
    private db: Database,
    private cache: CacheClient
  ) {}

  async getProfile(userId: string): Promise<UserProfile> {
    const key = keys.userProfile(userId);
    const entry = await this.cache.get<UserProfile>(key);
    if (entry) return entry.value;

    const profile = await this.db.findUser(userId);
    await this.cache.set(key, profile, { ttl: 3600, jitter: 300 });
    return profile;
  }

  async updateProfile(userId: string, data: Partial<UserProfile>): Promise<UserProfile> {
    const updated = await this.db.updateUser(userId, data);

    // Write through: update cache immediately with fresh data
    await this.cache.set(keys.userProfile(userId), updated, {
      ttl: 3600,
      jitter: 300,
    });

    return updated;
  }

  async suspendAccount(userId: string): Promise<void> {
    await this.db.suspendUser(userId);

    // Permissions must be immediately consistent.
    // Never serve stale permission data. Delete, do not wait for TTL.
    await this.cache.del(keys.userPermissions(userId));
  }
}

Notice suspendAccount does not use write through. It deletes the permissions key outright. Serving a stale permission is a security issue, not a UX issue. The next read will hit the database and get the correct answer.

TTL vs. Event Driven Invalidation

These are two different tools for two different problems.

TTL invalidation is for data where being slightly stale is acceptable. Exchange rates, public blog posts, product catalog pages. Set a TTL and let entries expire naturally.

Event driven invalidation is for data where correctness matters. User permissions, account status, pricing. Delete or update the cache entry at the moment of the change, not on a timer.

Most systems use TTL for everything because it requires less upfront thinking. Then they get burned when a permissions update does not take effect for an hour. The fix is not to lower the TTL. Lowering the TTL is how you get a thundering herd.

The Thundering Herd and How to Solve It

A typical Redis hit takes 1 to 3ms. A typical database query takes 50 to 300ms. That means one cache miss can cost as much as 100 cache hits. At scale, getting this wrong does not just make things slow. It brings services down.

Imagine a hot cache entry expires at 3:00:00 AM. You have 500 concurrent users. At 3:00:01, all 500 get a cache miss simultaneously and fire a database query. Your database, handling 10 queries per second comfortably, suddenly receives 500 in the same second and collapses.

You just traded slightly stale data for a full outage.

Without protection:

500 requests ──► 500 DB queries ──► DB overwhelmed ──► Outage

With jitter + lock:

500 requests ──► 1 DB query ──► Cache filled ──► 499 served from cache

Three solutions, applied in layers.

Solution 1: Jitter

The simplest fix. Add randomness to expiry so entries do not all expire at the same moment. Already built into the CacheClient above via the jitter option.

// Without jitter: 500 entries for a popular endpoint all expire at 03:00:00
await cache.set(key, value, { ttl: 3600 });

// With jitter: entries expire anywhere between 3600 and 3900 seconds
// Stampede risk drops dramatically for zero added complexity
await cache.set(key, value, { ttl: 3600, jitter: 300 });

Add jitter everywhere. It costs nothing.

Solution 2: Stale While Revalidate

Serve the stale entry immediately. Trigger a background refresh. The next request gets fresh data. This is how HTTP caching has worked for decades.

Request ──► Cache hit? ──► Yes, and fresh? ──► Return immediately
                │
                │ Yes, but stale?
                ├──► Return immediately (user does not wait)
                │    └──► Trigger background refresh
                │
                │ No (full miss)
                └──► Fetch from DB ──► Populate cache ──► Return

type FetchFn<T> = () => Promise<T>;

export async function getWithStaleRevalidate<T>(
  cache: CacheClient,
  key: string,
  fetchFn: FetchFn<T>,
  options: CacheOptions & { stale: number }
): Promise<T> {
  const entry = await cache.get<T>(key);

  if (entry) {
    const isStale = Date.now() > entry.expiresAt;

    if (isStale) {
      // Serve the stale value immediately, refresh in the background
      void refreshInBackground(cache, key, fetchFn, options);
    }

    return entry.value;
  }

  // Full miss: fetch synchronously
  const value = await fetchFn();
  await cache.set(key, value, options);
  return value;
}

async function refreshInBackground<T>(
  cache: CacheClient,
  key: string,
  fetchFn: FetchFn<T>,
  options: CacheOptions
): Promise<void> {
  try {
    const value = await fetchFn();
    await cache.set(key, value, options);
  } catch (err) {
    console.error(`Background cache refresh failed for ${key}:`, err);
  }
}

Usage at call sites is one line:

const profile = await getWithStaleRevalidate(
  cache,
  keys.userProfile(userId),
  () => db.findUser(userId),
  { ttl: 300, stale: 600, jitter: 60 }
);

Users never wait on a cache refresh. The background task handles it. The next user gets the fresh value.

Solution 3: Distributed Lock on Cache Miss

For hot single keys, when a miss happens only one process should fetch and repopulate. Others wait briefly. This prevents 500 processes all querying the database for the same row at the same time.

import Redlock from "redlock";

export class LockedCacheClient extends CacheClient {
  private redlock: Redlock;

  constructor(redisUrl: string) {
    super(redisUrl);
    this.redlock = new Redlock([this.getUnderlyingClient()]);
  }

  async getOrFetch<T>(
    key: string,
    fetchFn: FetchFn<T>,
    options: CacheOptions
  ): Promise<T> {
    const entry = await this.get<T>(key);
    if (entry) return entry.value;

    const lockKey = `lock:${key}`;
    let lock;

    try {
      lock = await this.redlock.acquire([lockKey], 5000);

      // Re-check after acquiring the lock.
      // Another process may have already populated the cache while we waited.
      const recheck = await this.get<T>(key);
      if (recheck) return recheck.value;

      const value = await fetchFn();
      await this.set(key, value, options);
      return value;
    } catch {
      // Lock contention: fall back to a direct DB fetch rather than failing the request
      const value = await fetchFn();
      await this.set(key, value, options);
      return value;
    } finally {
      await lock?.release();
    }
  }
}

The re-check after acquiring the lock is critical. Without it, the second process acquires the lock and queries the database anyway even though the first process just populated the cache. This is the double checked locking pattern applied to distributed systems.

Observability: Know What Is Actually Happening

You cannot improve what you cannot see. Wrap your cache client with metrics once and label call sites with a pattern name.

import { Counter, Histogram, Registry } from "prom-client";

export class ObservableCacheClient extends CacheClient {
  private hits: Counter;
  private misses: Counter;
  private hitLatency: Histogram;
  private missLatency: Histogram;

  constructor(redisUrl: string, registry: Registry) {
    super(redisUrl);

    this.hits = new Counter({
      name: "cache_hits_total",
      help: "Total cache hits",
      labelNames: ["key_pattern"],
      registers: [registry],
    });

    this.misses = new Counter({
      name: "cache_misses_total",
      help: "Total cache misses",
      labelNames: ["key_pattern"],
      registers: [registry],
    });

    this.hitLatency = new Histogram({
      name: "cache_hit_duration_seconds",
      help: "Latency of cache hits",
      labelNames: ["key_pattern"],
      registers: [registry],
    });

    this.missLatency = new Histogram({
      name: "cache_miss_duration_seconds",
      help: "Latency of cache misses including the upstream fetch",
      labelNames: ["key_pattern"],
      registers: [registry],
    });
  }

  async getOrFetchObserved<T>(
    key: string,
    keyPattern: string,
    fetchFn: FetchFn<T>,
    options: CacheOptions
  ): Promise<T> {
    const start = performance.now();
    const entry = await this.get<T>(key);
    const elapsed = () => (performance.now() - start) / 1000;

    if (entry) {
      this.hits.inc({ key_pattern: keyPattern });
      this.hitLatency.observe({ key_pattern: keyPattern }, elapsed());
      return entry.value;
    }

    this.misses.inc({ key_pattern: keyPattern });
    const value = await fetchFn();
    await this.set(key, value, options);
    this.missLatency.observe({ key_pattern: keyPattern }, elapsed());
    return value;
  }
}

The three metrics that matter most:

Hit rate per key pattern, not aggregate. If your overall hit rate is 80% but a critical key pattern sits at 20%, the aggregate number is hiding a real problem. Always break this down by pattern.

Eviction rate. If Redis is evicting keys because you are out of memory, you are thrashing, not caching. A Redis hit at 1ms becomes meaningless if the key you need was evicted 30 seconds ago. Increase memory, shorten TTLs, or stop caching data that expires before it is ever read again.

Miss latency. At scale, a cache miss costs 50 to 300ms of database time. If your service depends on sub-10ms responses, one cold endpoint can blow your entire p99. Miss latency tells you exactly how bad the degradation is when your cache fails.

The Decision Checklist Before You Add a Cache

Not every performance problem needs a cache. Run through this before reaching for Redis.

Is the query actually slow or just called too often? N+1 patterns make caching look like the answer when a JOIN is. Profile the query execution plan before adding a cache layer on top of a structural problem.

Can the data be denormalized instead? If you always cache the same join result, consider materializing it in the schema. A cache is sometimes a workaround for a schema designed for write convenience rather than read performance.

Is this actually read heavy? If data is written more often than it is read, cache invalidation overhead exceeds the savings. You are paying the write cost twice: once to the database, once to update or invalidate the cache entry.

What is the cost of serving stale data? Product listings: probably acceptable. Account balance: no. Permissions and access control: never. Make this decision explicitly. Saying "we will just set a short TTL" is not an answer. It is a way of avoiding the question.

What is your cold start story? If your service falls over every time it restarts because the cache is cold, write through caching or a warming script is the fix, not hoping traffic is light during the deploy window.

Putting It All Together

Here is a service that uses everything from this post: typed versioned keys, write through on mutations, stale while revalidate on reads, jitter throughout, immediate delete for permissions, and full observability.

export class ProductionUserService {
  constructor(
    private db: Database,
    private cache: ObservableCacheClient
  ) {}

  async getProfile(userId: string): Promise<UserProfile> {
    return getWithStaleRevalidate(
      this.cache,
      keys.userProfile(userId),
      () => this.db.findUser(userId),
      { ttl: 300, stale: 600, jitter: 60 }
    );
  }

  async updateProfile(userId: string, data: Partial<UserProfile>): Promise<UserProfile> {
    const updated = await this.db.updateUser(userId, data);

    // Write through: fresh data goes straight to cache on every mutation
    await this.cache.set(
      keys.userProfile(userId),
      updated,
      { ttl: 300, stale: 600, jitter: 60 }
    );

    return updated;
  }

  async updatePermissions(userId: string, permissions: Permission[]): Promise<void> {
    await this.db.updatePermissions(userId, permissions);

    // Access control data is deleted immediately, never served stale
    await this.cache.del(keys.userPermissions(userId));
  }
}

Reads use stale while revalidate so users never block on a cache refresh. Profile writes use write through so the cache stays warm after every mutation. Permission changes delete immediately because serving a stale permission is a security issue, not a UX issue.

Closing

The teams that handle caching well treat it as a first class architectural concern. They document cache contracts: what is cached, for how long, what triggers invalidation, and what the acceptable staleness window is. They test cold start and stampede scenarios explicitly. They monitor cache health with the same seriousness as database health.

The teams that handle caching poorly add Redis when things get slow and ship it.

Both approaches work right up until they do not. The difference is that the first team knows exactly what breaks and why when it does. The second team opens their laptop at past midnight and starts reading documentation for the first time or throw in " pls help me, redis no work, idk what is redis no more " into ChatGPT.

The patterns in this post are not theoretical. They are the things you wish were already in the codebase when the incident starts. Put them in before it does.

Episode 2: I Was a Junior Developer and I Must Be Stopped

Adam - The Developer — Thu, 02 Apr 2026 03:36:54 +0000

Welcome back.

Last episode, we reviewed a Laravel function that used three loops to reconstruct an array it already had, fired two database queries per item instead of one, and named a variable $r_arr with the confidence of a man who has never once been held accountable for anything.

It was a good one, some of us liked it.

So today, we are reviewing my hotel invoice generator. It is a PHP class called ExcelController. It imports guest data from an uploaded Excel file and exports a personalised invoice for each guest, packed into a zip file for download.

It was never merged.

The room numbers are completely random.

My senior opened the PR. He read through it. He left seven comments. He never raised his voice. He never sent a follow-up Slack. He just reviewed it with the quiet, deliberate calm of a man who has already accepted his fate and is simply choosing, each day, to keep showing up anyway.

Here it is:

class ExcelController
{
    public function import(Request $request, ImportedFile $filedata, User $user, Excelconfig $config)
    {
        try {
            $file = $request->file('file');
            if (!$file) {
                return redirect()->back()->with('error', 'Please choose a file to import');
            }

            $spreadsheet = IOFactory::load($file);
            $numSheets = $spreadsheet->getSheetCount();
            if ($numSheets > 1) {
                return redirect()->back()->with('error', 'Excel file has more than one sheet.');
            }

            Excel::import(new DataImport, $file);
            $this->export($filedata, $user, $config);
            return redirect()->back()->with('success', 'File imported successfully.');
        } catch (\Exception $e) {
            DB::rollBack();
            return redirect()->back()->with('error', 'File import failed: ' . $e->getMessage());
        }
    }

    public function export($filedata, $user, $config)
    {
        $getfile_name = $user->where('id', 1)->pluck('default_file')[0];

        $file_path = base_path("/storage/app/main_file/$getfile_name");
        if (file_exists($file_path)) {
            $file = $file_path;
        }
        $file = base_path("/storage/app/main_file/main_file.xls");

        $num_copies = count($filedata::all()) - 1;
        $spreadsheet = IOFactory::load($file);

        $guest_names = $filedata::all()->pluck('guest_name');
        $invoice_number = $filedata::all()->pluck('reservation_number');
        $payment_date = $filedata::all()->pluck('payment_date');
        $room = "Room" . " " . rand(1, 19);
        $nights_stayed = $filedata::all()->pluck('number_nights_stayed');
        $price = $filedata::all()->pluck('total_revenue');

        $zip = new ZipArchive;
        $zip_name = now()->format('Ymd_His') . '.zip';
        if ($zip->open(storage_path('app/' . $zip_name), ZipArchive::CREATE) === TRUE) {
            for ($i = 1; $i <= $num_copies; $i++) {
                $zip_file_name = 'Guest' . $i . '.xlsx';
                $worksheet = $spreadsheet->getActiveSheet();

                $message_index = $i - 1;

                $_guestNameRow = $config->where('id', 1)->pluck('guest_name')[0] ?? 'A21';
                $_invoiceNumRow = $config->where('id', 1)->pluck('invoice_number')[0] ?? 'B15';
                $_dateRow = $config->where('id', 1)->pluck('payment_date')[0] ?? 'B17';
                $_roomRow = $config->where('id', 1)->pluck('room')[0] ?? 'B21';
                $_quantity = $config->where('id', 1)->pluck('nights_stayed')[0] ?? 'C21';
                $_unitPrice = $config->where('id', 1)->pluck('unit_price')[0] ?? 'D21';
                $_amountPrice = $config->where('id', 1)->pluck('amount_price')[0] ?? 'E21';
                $_amountTotal = $config->where('id', 1)->pluck('amount_price_row_total')[0] ?? 'D34';
                $_unitTotal = $config->where('id', 1)->pluck('unit_price_total')[0] ?? 'E34';

                $spreadsheet->getActiveSheet()->setCellValue($_guestNameRow, $guest_names[$message_index] ?? null);
                $spreadsheet->getActiveSheet()->setCellValue($_invoiceNumRow, $invoice_number[$message_index] ?? null);
                $spreadsheet->getActiveSheet()->setCellValue($_dateRow, $payment_date[$message_index]);
                $spreadsheet->getActiveSheet()->setCellValue($_roomRow, $room);
                $spreadsheet->getActiveSheet()->setCellValue($_quantity, $nights_stayed[$message_index] ?? null);
                $spreadsheet->getActiveSheet()->setCellValue($_unitPrice, ((float)$price[$message_index] / (int)$nights_stayed[$message_index]) ?? null);
                $spreadsheet->getActiveSheet()->setCellValue($_amountPrice, $price[$message_index] ?? null);
                $spreadsheet->getActiveSheet()->setCellValue($_unitTotal, ((float)$price[$message_index] / (int)$nights_stayed[$message_index]) ?? null);
                $spreadsheet->getActiveSheet()->setCellValue($_amountTotal, $price[$message_index] ?? null);

                $writer = new Xlsx($spreadsheet);
                ob_start();
                $writer->save('php://output');
                $file_contents = ob_get_clean();
                $zip->addFromString($zip_file_name, $file_contents);
            }
            $zip->close();

            header("Content-Type: application/zip");
            header("Content-Disposition: attachment; filename=$zip_name");
            header("Content-Length: " . filesize(storage_path('app/' . $zip_name)));
            readfile(storage_path('app/' . $zip_name));
            exit;
        }
    }
}

Take it in. Sit with it. Let it wash over you like a wave you saw coming from a mile away and decided not to move for.

Now. Let's go through it together. I recommend water. I recommend a snack. I recommend telling someone you love them before we start, just in case.

The `import()` Method, Which Does Not Import

public function import(Request $request, ImportedFile $filedata, User $user, Excelconfig $config)

Four injected dependencies. FOUR. Request. ImportedFile. User. Excelconfig. A full starting lineup. An ensemble cast. A heist crew assembled with purpose and intention.

And what does import() do with them? It validates the file exists, hands it to Excel::import(), and then — immediately, in the next breath, before the import has even had time to feel good about itself — calls $this->export() and passes all four dependencies over like a baton.

import() does not import. import() is a method that watches another method work and then takes credit at standup. import() is the guy who responds "yeah, we shipped that" while making eye contact with the person who actually shipped it, daring them to say something.

Three of the four injected dependencies are not used in import() at all. They exist solely to be forwarded. They showed up for a job they were not given, were immediately redirected, and had to pretend that was the plan all along.

My senior's very first comment on this PR was: "why does import() call export()?"

Not aggressive. Not horrified. Just a quiet, five-word question with a question mark that somehow conveyed the weight of every PR he had ever reviewed before this one.

I did not have a good answer. I replied anyway. The reply was not good.

The Rollback That Has Nothing to Roll Back

} catch (\Exception $e) {
    DB::rollBack();
    return redirect()->back()->with('error', 'File import failed: ' . $e->getMessage());
}

There is no DB::beginTransaction() anywhere in this class.

I want you to understand the full scope of what has happened here. DB::rollBack() is an instruction to undo a database transaction. A transaction that was never opened. There is nothing to undo. There has never been anything to undo. The database received this rollback instruction, checked its records, found no open transaction, shrugged in whatever way databases are capable of shrugging, and moved on.

This is not a bug. It's a philosophy. It's the worldview of someone who believes that calling rollBack() near a database operation is close enough. The vibes are correct. The implementation is a void.

It's like calling refund() on a purchase you never made. It's like issuing a formal apology for something you didn't do. It's like cancelling a reservation at a restaurant you never booked, and then waiting at home for confirmation that you don't have a table.

My senior's comment here was two words: "no transaction."

Two words. No question mark. No exclamation. Two words, written by a man conserving his energy for what he was about to find next.

User ID 1. Just User ID 1. Only Ever User ID 1.

$getfile_name = $user->where('id', 1)->pluck('default_file')[0];

This system has one user. His name is User ID 1. He is the client, the admin, the guest, and apparently the sole known resident of the hotel this invoice generator was built for. He has no colleagues. He has no hierarchy above or below him. He simply is, hardcoded and eternal, where('id', 1), forever.

Multi-tenancy is a concept this code has never heard of. Role-based access control is a myth, like dragons, or good variable names. There is only User ID 1. He is in the controller. He is in the database. He is in my code. He is everywhere. He always will be.

To be clear: this was a single-client internal tool. There technically was only one user. But still. To look at a codebase, decide that ->where('id', 1) is fine to hardcode into a controller, and commit it — that's not just a shortcut. That's a statement of values.

The `if` Block That Tried So Hard and Was Immediately Betrayed

$file_path = base_path("/storage/app/main_file/$getfile_name");
if (file_exists($file_path)) {
    $file = $file_path;
}
$file = base_path("/storage/app/main_file/main_file.xls");

We check whether a dynamic file exists. If it does, we assign it to $file. This is good. This is correct. This is the code doing exactly what code is supposed to do.

And then — on the very next line, with no condition, no ceremony, no acknowledgment whatsoever of what just happened — we overwrite $file with a hardcoded path to main_file.xls.

The if block is a ghost. It built $file with love and intention, watched it get immediately overwritten, and now haunts the codebase unable to affect anything, just flickering the lights occasionally to let us know it was here once and it mattered and we chose not to listen.

Every single guest received the same template. The dynamic file path never made it. The if block knew. It stood there and watched and couldn't stop it.

I think about this if block sometimes. I think about its optimism. I think about how it ran, found the file, did its job perfectly, and then got erased before it could even tell anyone.

We're not so different, the if block and I.

Six Queries. Same Table. In a Row. Like a Person Who Has Completely Lost the Plot.

$num_copies = count($filedata::all()) - 1;  // query 1
$guest_names = $filedata::all()->pluck('guest_name');       // query 2
$invoice_number = $filedata::all()->pluck('reservation_number'); // query 3
$payment_date = $filedata::all()->pluck('payment_date');    // query 4
$nights_stayed = $filedata::all()->pluck('number_nights_stayed'); // query 5
$price = $filedata::all()->pluck('total_revenue');          // query 6

Six. Six times. $filedata::all(). The same query. SELECT * FROM imported_files. Six full round trips to the database before the loop has even started. Six times I loaded the entire table into memory, grabbed one column off it, and threw the rest away, and then immediately did it again.

The correct code is one query followed by five collection operations:

$filedata = ImportedFile::all(); // once. just once. please.
$num_copies = $filedata->count() - 1;
$guest_names = $filedata->pluck('guest_name');

That's it. That's the whole fix. Load it once. It stays in memory. Use it five times. The database does not need to hear from you six times about the same table in the same second. The database is not a service counter. You don't need to take a new number each time.

If the database could file a complaint it would have. It did not, because databases are more professional than this code deserved.

The Room Number Is Random. I Need You To Understand That The Room Number Is Random.

$room = "Room" . " " . rand(1, 19);

I am going to say it plainly and I need you to receive it plainly:

The room number is randomly generated.

This is a hotel invoice generator. Its entire reason for existing is to produce legally adjacent documents that guests may use for expense reimbursement, business travel records, or billing disputes. A guest stays in Room 7. They receive an invoice. The invoice says Room 11. They ask for a corrected invoice. We regenerate it. It says Room 4. We regenerate it again. Room 17.

There is no floor. There is no ceiling on the chaos. There is only rand(1, 19), spinning eternally, indifferent to consequences, assigning rooms with the energy of a hotel that has completely given up on the concept of room assignments as a coherent system.

The data is there, in the uploaded file. The actual room number exists. It is a column. It was imported. It is sitting in the database. And instead of retrieving it, past me looked at the data, decided rand(1, 19) captured the spirit of the thing, and moved on.

My senior's comment on this line was: "the room number is random."

Not a question. Not a fix suggestion. Just a statement. An observation delivered by someone who had clocked what they were looking at, accepted it, and decided that documentation was the only appropriate response.

I replied: "yes I'll fix it."

He replied: "ok."

That was the entire conversation. We both agreed, silently, that there was nowhere productive to go from there.

Nine Queries Per Guest. Inside The Loop. Per Guest. Nine. Per Guest. Inside The Loop.

for ($i = 1; $i <= $num_copies; $i++) {
    $_guestNameRow = $config->where('id', 1)->pluck('guest_name')[0] ?? 'A21';
    $_invoiceNumRow = $config->where('id', 1)->pluck('invoice_number')[0] ?? 'B15';
    $_dateRow = $config->where('id', 1)->pluck('payment_date')[0] ?? 'B17';
    // six more of these

Nine. $config->where('id', 1). Nine times. Per iteration. Inside the loop.

This is the same row. It is always the same row. It was the same row on the first iteration. It will be the same row on the hundredth iteration. The config table does not update between guests. The laws of physics have not changed. The database did not go anywhere. The row is still there, completely unchanged, waiting patiently to be fetched for the 847th time.

100 guests: 906 total queries. And that's on top of the 6 we fired before the loop even started, which we haven't forgiven yet.

The fix is to fetch the config once. Before the loop. Assign it to a variable. Use that variable. The database would send a fruit basket. The database would cry. Not from sadness. From relief.

My senior left his longest comment here. Four sentences. Query optimization. N+1 problems. Eager loading. The works. It was generous. It was thorough. It was the kind of comment you leave when you genuinely believe the person on the other side can be better.

I read it. I understood approximately 60% of it. I said "thanks, will fix."

The PR was closed the next day. Not merged. Closed.

`$message_index`, a Variable That Means `$i - 1` and Wants You to Know It

$message_index = $i - 1;

The loop starts at 1. Arrays start at 0. We need $i - 1. This is fine. This is correct. I have no complaints about the arithmetic.

What I have complaints about is the name.

$message_index. In a hotel invoice generator. There are no messages. There has never been a message. No message is sent, received, composed, parsed, or gestured at anywhere in this codebase. But the variable needed a name, and the name it received was $message_index, with the full confidence of someone who is naming things based on vibes and hoping nobody asks follow-up questions.

It's not wrong. It functions. But it's like naming a file document1_FINAL_v3_USE_THIS_ONE_actually_final.xls. Technically a name. Technically communicating something. Not the something it thinks it's communicating.

The Ghost of PHP 4, Given the Keys to a Laravel Application

header("Content-Type: application/zip");
header("Content-Disposition: attachment; filename=$zip_name");
header("Content-Length: " . filesize(storage_path('app/' . $zip_name)));
readfile(storage_path('app/' . $zip_name));
exit;

We are inside a Laravel controller. Laravel has a response system. It has response()->download(). It is one line. It is documented. It handles headers automatically. It does not require you to manually compute Content-Length. It does not require you to call readfile() like you're writing a blog post about PHP in 2005. And it absolutely does not require you to call exit to kill the entire PHP process when you're done.

But we didn't do that. Instead we reached back through time, found PHP 4, handed it a zip file, said "you know what to do," watched it do a little dance with raw header() calls, and then exited out of existence.

exit. Not return. exit. We terminated the process. We didn't return a response. We didn't let Laravel wrap up gracefully. We just... left. Walked out mid-conversation. The framework stood there, holding the door open, waiting for a response object that was never coming.

The Final Tally

things that went wrong:
- import() that exists to not import:                                     ✅
- DB::rollBack() haunting a transaction that was never born:              ✅
- if block built $file perfectly then watched it die immediately:         ✅
- SELECT * fired 6 times on the same table, consecutively, on purpose:    ✅
- 9 config queries per guest inside the loop, 100 guests = 906 total:     ✅
- room numbers: rand(1, 19). not the actual room. a random room. always:  ✅
- $message_index: means $i-1, works nowhere, explains nothing:            ✅
- exit. just exit. in a Laravel controller. in the year of our lord:      ✅

PR comments left by senior:                                               7
PR comments addressed:                                                    0
PR status:                                                           closed

The Verdict

This code never shipped. The PR was closed. My senior reviewed every line of it, the phantom rollback, the six identical queries, the rand(1, 19) that could have ruined an expense report, all of it — and he left real comments with real explanations, like someone who looked at this disaster and somehow still saw a person who could learn.

He didn't laugh in the thread. He didn't forward it to anyone. He didn't bring it up in the next sprint. We agreed, without ever agreeing, to never speak of it, the way you agree to never speak of certain things that happen at work parties, certain things that happen on difficult deployments, certain things you open in a PR at 11am on a Tuesday and close by lunch.

But here is what I think about, now that I am far enough away from it to think:

He reviewed it anyway. That's the part that stays with me. He could have closed it in thirty seconds. He didn't. He went through it. He explained the N+1 problem to someone who didn't fully understand it yet. He noted the rollback. He pointed at the room number, stated plainly that it was random, and waited.

That's not a small thing. That is, in fact, the whole thing.

If you're a senior reading this: I see you. I know what it costs. Every comment you write on a PR from someone still figuring it out is a small act of faith that they will eventually figure it out. That faith is not nothing.

If you're a junior reading this: somewhere there is a folder. Not a literal one, maybe. But your senior has a folder. Full of PRs that keep them up. You are in that folder. The random room numbers are in that folder.

Do better. They're rooting for you. They reviewed the whole thing. They always do.

Episode 1: I Was a Junior Developer and I Must Be Stopped

Adam - The Developer — Tue, 31 Mar 2026 04:15:12 +0000

We all start somewhere.

Some developers begin their careers writing clean, well-structured code, carefully following best practices, naming variables properly, and writing tests like responsible adults.

Others start by shipping a few bugs here and there, learning, improving, and slowly developing their own style.

And then there's me.

Today, we are reviewing a function I wrote years ago that is somehow still running in production, untouched, unbothered, and — against all odds — still working. No one has dared to refactor it. No one has tried to rewrite it. It has achieved something most code never will:

It became too scary to change.

This function is called multipleUpdate.
It lives in a Laravel application.
It updates multiple items.
I think.

Here it is, in all its glory:

public function multipleUpdate(
    Scenario $scenario,
    Request $request
) {
    $r_arr = intval($request->right_arr);
    $id_array = $request->item_number ?? $r_arr;
    $name_ja_array = $request->name_ja ?? null;
    $name_en_array = $request->name_en ?? null;
    $type_array = $request->type ?? null;

    $added = []; // final res
    array_push($added, $r_arr);
    $difference = [];

    for ($i = 0; $i < count($id_array) - 1; $i++) {
        array_push($difference, $id_array[$i + 1] - $id_array[$i]);
    }

    for ($i = 0; $i < count($difference); $i++) {
        array_push($added, $r_arr += $difference[$i]);
    }

    for ($i = 0; $i < count($added); $i++) {
        if ($scenario->items()->where('item_number', $added[$i])->exists()) {
            $scenario->items()->where('item_number', $added[$i])->update([
                'name_ja' => ($name_ja_array[$i] ?? $request->default_name_ja) ?? null,
                'name_en' => ($name_en_array[$i] ?? $request->default_name_en) ?? null,
                'kind' => ($type_array[$i] ?? $request->default_type) ?? null,
            ]);
        } else {
            $scenario->items()->create([
                'scenario_id' => $scenario->id ?? null,
                'item_number' => $added[$i] ?? null,
                'name_ja' => $name_ja_array[$i] ?? null,
                'name_en' => $name_en_array[$i] ?? null,
                'kind' => $type_array[$i] ?? null,
            ]);
        }
    }
}

Take a moment. Let it wash over you. I'll be here.

Now let's go through it together, line by line, like a crime scene investigator who is also crying.

The Mysterious `$r_arr` Variable

// oh cool, a variable named $r_arr. what does r stand for? right? raw? random?
// who the hell knows. the author was apparently too busy to leave a single hint.
$r_arr = intval($request->right_arr);

$r_arr. Short for right_arr, which comes from the request field right_arr.

What is "right"? Right as in correct? Right as in direction? Right as in the author just started typing and committed before their brain caught up?

We will never know. There are no comments. There is no documentation. There is only $r_arr, staring back at you, completely unashamed.

Sir, This Is an Int

// sir this is an int. you are about to count() and loop over a single goddamn int.
// i genuinely hope you tested this. i genuinely fear that you did.
$id_array = $request->item_number ?? $r_arr;

The variable is called $id_array. It implies an array. It promises an array. It has the word array right there in the name.

The fallback is $r_arr, which we just established is an int.

So if $request->item_number is missing, $id_array is a single integer, and we are about to call count() on it, loop over it, and index into it like it's an array.

PHP, famously lenient, will try its best. PHP will not succeed.

The `?? null` Epidemic

// breaking news: local developer discovers that $request->name_ja ?? null
// and $request->name_ja are the same fucking thing. more at 11.
$name_ja_array = $request->name_ja ?? null;
$name_en_array = $request->name_en ?? null; // still doing it. unbelievable.
$type_array = $request->type ?? null;       // i am losing my mind.

$x ?? null means: "if $x is null, use... null."

This is the coding equivalent of a safety net made of holes. It is null, wearing a seatbelt, driving into a wall.

The author wrote this three times in a row. On three separate lines. Without stopping to wonder if something had gone wrong in their life.

The Comment That Explains Nothing

$added = []; // final res

"final res."

Final result? Final response? Final resignation letter?

You cannot write // final res and then disappear like a deadbeat dad. Come back. Explain yourself. We deserve that much.

Also: array_push($added, $r_arr) to append a single value. The author could have written $added[] = $r_arr. It's shorter. It's faster. It doesn't make me feel things.

But they didn't. They used array_push. For one value. And then kept going.

The Round Trip to Nowhere

Now we reach the heart of it. The crown jewel. The part that made me go outside and stand in a field for ten minutes.

// oh no. oh hell no. i see where this is going and i am grabbing you by the shoulders.
// you're going to calculate the gaps between IDs, aren't you.
// you absolute disaster of a human being.
for ($i = 0; $i < count($id_array) - 1; $i++) {
    array_push($difference, $id_array[$i + 1] - $id_array[$i]);
}

// and NOW you're adding them back together. you took [10, 12, 15],
// blew it up into [2, 3], and then painstakingly rebuilt [10, 12, 15].
// you did a round trip to absolutely nowhere and burned CPU cycles for the privilege.
// i am genuinely asking: are you okay? do you need water?
// did someone hurt you?
for ($i = 0; $i < count($difference); $i++) {
    array_push($added, $r_arr += $difference[$i]);
}

Let me walk you through what's happening here.

We have an array of IDs: [10, 12, 15].

Loop one: compute the differences between consecutive IDs. [2, 3].

Loop two: add those differences back, cumulatively, to reconstruct... [10, 12, 15].

We took the array. We destroyed it. We rebuilt it. We are exactly where we started.

This is a mathematical round trip that accomplishes zero. The only thing it changes is $r_arr, which started life as a request input, briefly became a loop accumulator, and is now something else entirely.

$r_arr has had more identity changes in one function than most people have in a lifetime. Leave $r_arr alone. $r_arr has been through enough.

The Database Loop of Sadness

We are now in loop three. The database loop. God help us.

for ($i = 0; $i < count($added); $i++) {
    if ($scenario->items()->where('item_number', $added[$i])->exists()) {
        $scenario->items()->where('item_number', $added[$i])->update([...]);
    } else {
        $scenario->items()->create([...]);
    }
}

Two database queries per item. One to check if it exists. One to update or create it.

No transaction wrapper. No bulk operation. Just a prayer and a dream that nothing fails halfway through and leaves the database looking like it was updated by a raccoon who found a keyboard.

And then there's this:

$scenario->items()->create([
    'scenario_id' => $scenario->id ?? null,
    'item_number' => $added[$i] ?? null,
    ...
]);

$scenario->id ?? null. We dependency-injected $scenario into the function signature. Laravel loaded it, validated it, handed it to us on a silver platter. If $scenario->id is null at this point, ?? null is not saving us. Nothing is saving us. Say your goodbyes.

$added[$i] ?? null. We are inside the loop that iterates over $added. $added[$i] is guaranteed to exist. The ?? null is not a safety net. It is a tiny umbrella in a drink that is already on fire.

The Method That Was Right There the Whole Time

Here is the part that gets me.

Laravel has had updateOrCreate() since version 5.3.

$scenario->items()->updateOrCreate(
    ['item_number' => $added[$i]],
    [
        'name_ja' => $name_ja_array[$i] ?? $request->default_name_ja,
        'name_en' => $name_en_array[$i] ?? $request->default_name_en,
        'kind'    => $type_array[$i] ?? $request->default_type,
    ]
);

One method. One query. Handles both cases. No manual existence check. No separate create/update branch.

It was sitting right there the whole time, like a golden retriever waiting by the door.

And I looked it in the face, said "no thanks," and wrote a three-loop differential equation instead.

I'm not angry at myself.

I'm disappointed.

(I am absolutely furious.)

The Final Tally

crimes committed:
- 3 loops to rebuild the array we already had:                       ✅
- up to 2N db queries with zero transaction safety:                  ✅
- $r_arr living three different lives in one function:               ✅
- ?? null deployed purely as emotional support:                      ✅
- updateOrCreate() left completely untouched like a jilted lover:    ✅
- "final res" as a comment explaining nothing:                       ✅

And yet.

And yet.

It works. It has always worked. It is running in production right now, at this very moment, as you read this. No one has touched it. No one will. It works and that is the most infuriating part of all of this.

The real multipleUpdate was the trust issues we developed along the way.

If you have code like this living rent-free in your production environment — code that works, code that you are afraid to touch, code that you wrote and no longer recognize — I want you to know: you are not alone.

We all have a multipleUpdate. Some of us just haven't found ours yet.

Episode 2 coming whenever I'm emotionally ready.

Resume-Driven Development Is Quietly Killing Your Product

Adam - The Developer — Wed, 25 Mar 2026 06:51:34 +0000

There is a conversation that happens in engineering teams all over the world, constantly, and it almost always goes the same way.

Someone proposes a new project. Before the requirements are even clear, before anyone has talked to a user, before the scope has been defined, someone opens their mouth and says: "What if we used this for the Rust rewrite?" Or: "This could be a great chance to try out that new edge runtime." Or: "I've been wanting to get some GraphQL experience."

And just like that, the technical decision has been made. Not by the problem. By someone's LinkedIn headline in 18 months.

This is Resume-Driven Development. It is everywhere. Almost nobody talks about it honestly. Partly because it's awkward to name, and partly because most of the people who should be naming it are also doing it.

What It Actually Looks Like

RDD rarely announces itself. It doesn't walk into the room wearing a sandwich board. It hides inside reasonable-sounding language, nodding along to the business requirements before quietly steering the conversation toward whatever technology someone bookmarked last weekend.

"We need something more scalable." Scalable to what? You have 200 users. Postgres hasn't broken a sweat. What you actually need is three more customers, not a distributed event-driven microservices architecture that requires four new AWS services, a dedicated DevOps hire, and someone brave enough to own the on-call rotation.

"The developer experience with this new tool is so much better." For whom? For the one engineer who spent two weekends learning it and is now the team's only expert? That is not better DX. That is a bus factor of one wearing a conference lanyard.

"We should use what the industry is moving toward." The industry is moving toward approximately forty things simultaneously and half of them will be deprecated before you finish the migration. Bun, Deno, whatever JavaScript framework launched last Tuesday, that new database that promises to replace all other databases while also making coffee. The industry is always moving. Products still need to ship.

The giveaway is always the same: the technology choice happens before the problem is fully understood. The cart is so far in front of the horse it has its own Kubernetes cluster.

The Honest Reason This Happens

Let's just say it plainly. The job market for developers rewards people who have used the new thing more than it rewards people who deeply understand boring, proven things. A resume with Kubernetes, GraphQL, and a trendy new runtime gets more recruiter messages than a resume that says "kept a Rails monolith running smoothly for five years and scaled it to a million users."

The incentives are completely misaligned with good engineering.

So developers, who are rational people, optimize for what the market rewards. They look for projects where they can justify using the new thing. They advocate for the new thing in architecture discussions. They get the experience on paper. Then they move on, often right around the time the true cost of that decision starts showing up in sprint retrospectives.

The product is not their primary concern in that moment. Their career is. That is human. That is understandable. It is also quietly catastrophic for the companies on the receiving end who wonder why everything takes so long and costs so much.

RDD Is Not the Actual Problem

Here is where most essays like this go wrong: they treat RDD as the villain. It is not.

Wanting to grow your skills is healthy. Wanting to work with modern tools is healthy. Advocating for better technology on your team is healthy. If developers never pushed to try new things, we would all still be writing SOAP XML by hand and arguing about tabs versus spaces in Vim. Some of us would enjoy that. Most of us would not.

The real problem is not that someone wanted to use a new tool. The real problem is that nobody stopped to have an honest conversation about what the business actually needed before the new tool got greenlit.

RDD becomes toxic at exactly the moment when the technology choice gets made in a vacuum. When nobody asked "what are we trying to accomplish in the next six months?" When nobody checked whether the existing stack had actually failed or was merely unfashionable. When the pitch was "this is better" without anyone defining better for whom, by what measure, over what time horizon.

A team that wants to use Rust and has a genuine performance problem that Rust would solve? Great. Go forth and rewrite. A team that wants to use Rust because a senior engineer just finished a side project in it and the current service is written in Go and works perfectly fine? That is RDD doing damage in a fleece vest and acting like it's doing the company a favor.

The difference is a conversation. A specific, slightly uncomfortable conversation where someone asks "what does the business need right now" and everyone has to sit with the answer even if it is boring.

What It Costs When That Conversation Doesn't Happen

The costs of undiscussed RDD are almost never measured because they happen slowly and diffusely and get attributed to other causes.

A team picks a new framework because it is exciting and nobody pushed back with requirements. Six months later, onboarding takes twice as long because the framework has steep learning curves and opinionated patterns that new hires have to unlearn their previous experience to adopt. That cost does not get traced back to the framework choice. It gets called "hiring is hard."

A team builds on a cutting-edge but immature library because it looked great at the time and the business case was hand-wavy. Two years later, the maintainer abandons it. Now the team owns it. That cost shows up as "tech debt" in a planning meeting, orphaned from the original decision that created it.

A team introduces a microservices architecture to a product that did not need one because someone wanted the experience. Local development becomes a nightmare involving Docker Compose files that nobody fully understands. Debugging a request means tracing it across six services. Deployments are complicated. None of this gets credited to the architectural decision. It gets called "complexity" as though complexity is weather that just rolls in and there is nothing to be done about it.

The engineers who made those choices are often long gone by the time the costs fully materialize. They have the experience on their resumes. The company has the mess. And a new round of engineers eager to rewrite it all in something more modern.

The Boring Stack Works

There is a version of this essay that collapses into a "just use Rails" post and I want to resist that because it misses the point. The point is not that new technology is bad. The point is that the decision should follow from the problem, and following from the problem requires first understanding the problem, which requires talking to the people who have the problem.

Boring, mature, well-understood technology has compounding advantages that are consistently undervalued because they are invisible. You do not see the bugs that did not happen because the ORM is stable. You do not see the hours not spent reading documentation for a library that has fifteen years of Stack Overflow answers covering every possible mistake. You do not see the hires who got productive in two weeks instead of two months because they already knew the stack.

Boring technology has been debugged by thousands of teams in thousands of situations. The edge cases are documented. The failure modes are known. When something breaks at 3am, someone has already written the blog post about it and the fix is four lines.

You are paying the price of novelty whether you account for it or not. The question is whether you made that tradeoff deliberately or just kind of drifted into it because the senior engineer seemed really enthusiastic in that one architecture meeting.

When New Technology Is Actually the Right Call

There are real situations where reaching for something new is the correct engineering decision. When the problem genuinely cannot be solved well with mature tools. When performance requirements are extreme and you have actual benchmarks showing existing tools fail. When the new technology has enough production adoption that its failure modes are understood rather than theoretical. When the team has the bandwidth to absorb the learning cost without stalling product delivery.

The difference in all these cases is that the technology choice follows from a requirement. You started with a real constraint. You evaluated options against it. You chose the best fit.

That is engineering. It is also, not coincidentally, a much easier decision to defend when things get hard and someone above your pay grade starts asking questions.

What is not engineering is starting with "I want to use X" and working backward to a justification. What is especially not engineering is doing that without ever asking the product manager, the founder, or the customer what they actually need from the next six months of development.

The Conversation That Makes the Difference

The question that almost never gets asked before a major stack decision is embarrassingly simple: what does this business need to be true in twelve months, and how does this technology choice help or hinder that?

Not "what's interesting." Not "what's scalable in theory." Not "what would be impressive in a conference talk." What does the business need.

That question is boring to ask. It often produces boring answers. A monolith. A managed database. Server-rendered HTML. A framework your entire team already knows. Things that do not make for interesting blog posts but do make for products that ship on time and do not require a tribal knowledge handoff every time someone quits.

The engineering leader's job, in part, is to make sure this conversation happens before the exciting choice gets made. That means creating space for someone to say "we want to try this, and here is how it maps to what we actually need" and requiring that the second half of that sentence be filled in with something real, not vibes.

If someone can genuinely make the case that the new tool serves the business, then by all means use the new tool. Put it on the roadmap. Put it on your resume. You earned it honestly. The goal was never to ban ambition. The goal was to make sure ambition and business reality are at least on speaking terms before you start the sprint.

The new framework will not save your product. The exciting database will not fix your architecture. The Rust rewrite will not make your users happier.

But a team that openly discusses what they want to learn, weighs it honestly against what the business needs, and makes a deliberate choice? That team can pick almost anything and make it work.

The technology was never really the point. The conversation was.

No Redis. No Kafka. Just Postgres. I Built Chat.

Adam - The Developer — Wed, 18 Mar 2026 08:07:27 +0000

Everyone reaches for Redis and Kafka the moment they need real-time. What if you didn't have to?

I'm not anti-Redis. I'm not anti-Kafka. I'm anti-"add another distributed system because we're nervous." I've watched teams ship a broker cluster before they even had a throughput problem, then spend the next year learning the operational and failure semantics of that cluster while their database sat there, underused, doing a fraction of what it was capable of.

So I built a chat application that does the hard parts: multi-node real-time delivery, per-room ordering, presence, typing indicators, using Postgres as both the source of truth and the real-time message bus.

No brokers. No Redis. No RabbitMQ. No separate pub/sub service. Just Postgres, NestJS, and raw WebSockets.

TL;DR - Repo url here: https://github.com/adamreaksmey/chat-application-with-postgres

The Bet: "Just Postgres" Is Enough
The Core Mechanism: Triggers + LISTEN/NOTIFY as the Message Bus
Schema Decision: Per-Room Sequences for Gapless Ordering
Schema Decision: NOTIFY on Insert, With the Full Payload
Node Mechanics: A Dedicated LISTEN Client + In-Process Event Routing
Reference-Counted Room Subscriptions
Exponential Backoff Reconnect for the LISTEN Connection
Horizontal Scaling: Stateless Nodes + Nginx least_conn + Postgres Fanout
The Unsexy Part: Postgres Settings That Mattered
The Detour I Didn't Keep: Sharding LISTEN/NOTIFY
The Performance Turning Point: Batched Fanout + One Batch Send per Room per Tick
Proof: Load Test Results
Tradeoffs (The Part You Should Actually Read)
The Takeaway

1. The Bet: "Just Postgres" Is Enough

The architecture decision was intentionally provocative:

Persist everything in Postgres: users, rooms, room_members, messages, presence, typing.
Emit real-time events from Postgres using triggers + pg_notify.
Consume events in Node using a dedicated LISTEN connection.
Fan out to clients over raw WebSockets (JSON-framed events).
Scale horizontally by running multiple identical, stateless app nodes behind Nginx.

The thesis: in most apps, the database is already your most reliable coordination point. If you're writing a row anyway, you can publish the corresponding event from the same transaction boundary. No second system. No "eventual consistency between your DB and your broker." One write, one source of truth.

Why not Redis pub/sub?

Redis pub/sub is fast, but it introduces questions you now have to answer:

What happens to messages when a subscriber is offline? Redis doesn't care.
How do you replay history for a reconnecting client? Redis doesn't store it.
Who owns ordering? You do, now, in application code.

Postgres already answers all of these. Transactional writes give you ordering. The messages table gives you history. LISTEN/NOTIFY gives you the real-time transport. The only thing you give up is the comfort of reaching for a familiar tool.

2. The Core Mechanism: Triggers + LISTEN/NOTIFY as the Message Bus

At the center is a single idea:

When a message row is inserted, Postgres fires a trigger. The trigger calls pg_notify. Every app node that is LISTENing on that channel receives the payload and pushes it to its connected WebSocket clients.

That sounds almost too simple. The complexity is in doing it correctly:

Per-room ordering that holds under concurrent inserts.
Efficient subscribe/unsubscribe as clients join and leave rooms.
Automatic reconnect when the LISTEN connection drops.
Backpressure so one slow client doesn't stall delivery to everyone else.

Let's walk through each piece.

3. Schema Decision: Per-Room Sequences for Gapless Ordering

Chat is not just "messages exist." Chat requires: clients can ask for history since the last message they saw and expect a cursor that is gapless, stable, and per-room.

BIGSERIAL primary keys don't give you that. Postgres sequences are non-transactional by design: rolled-back transactions burn sequence values, leaving gaps. You can't use id > last_seen as a reliable "catch me up" cursor.

So instead, I track a per-room monotonic counter in a room_sequences table:

CREATE TABLE IF NOT EXISTS room_sequences (
  room_id   UUID PRIMARY KEY REFERENCES rooms(id) ON DELETE CASCADE,
  next_seq  BIGINT NOT NULL DEFAULT 1
);

A BEFORE INSERT trigger atomically claims the next value and stamps it onto the new message row:

CREATE OR REPLACE FUNCTION assign_room_sequence()
RETURNS trigger AS $$
BEGIN
  UPDATE room_sequences
  SET next_seq = next_seq + 1
  WHERE room_id = NEW.room_id
  RETURNING next_seq - 1 INTO NEW.seq;

  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER set_message_seq
  BEFORE INSERT ON messages
  FOR EACH ROW EXECUTE FUNCTION assign_room_sequence();

The UPDATE ... RETURNING is the serialization point. Postgres takes a row lock on the room_sequences row for that room, increments it atomically, and returns the previous value. Concurrent inserts into the same room queue up on that row lock. No application-level locking needed, no gaps, no duplicates.

Clients store last_seen_seq. On reconnect they send it, and the server replays exactly what they missed:

SELECT * FROM messages
WHERE room_id = $1 AND seq > $2
ORDER BY seq ASC LIMIT 100;

4. Schema Decision: NOTIFY on Insert, With the Full Payload

Once seq is assigned and the row is durable, a second trigger fires the notification:

CREATE OR REPLACE FUNCTION notify_new_message()
RETURNS trigger AS $$
BEGIN
  PERFORM pg_notify(
    'room:' || NEW.room_id::text,
    json_build_object(
      'id',         NEW.id,
      'seq',        NEW.seq,
      'room_id',    NEW.room_id,
      'user_id',    NEW.user_id,
      'username',   NEW.username,
      'content',    NEW.content,
      'created_at', NEW.created_at
    )::text
  );
  RETURN NEW;
END;
$$ LANGUAGE plpgsql;

The payload is a complete message event, not just an ID. This is a deliberate choice: the fanout path never touches the database again. No "fetch the message after receiving the notification" round-trip. One insert, one notify, one broadcast. Read amplification stays flat regardless of how many nodes are running.

5. Node Mechanics: A Dedicated LISTEN Client + In-Process Event Routing

The Node app maintains two separate Postgres interfaces:

A connection pool for all queries and transactions.
A single dedicated pg.Client that never runs queries. It only LISTENs and receives notifications.

Keeping these separate matters. A pooled connection can be checked in and out by different requests. A LISTEN client needs to stay alive and stateful for the lifetime of the process. Mixing the two leads to subtle bugs where your LISTEN connection gets recycled by the pool at the worst possible moment.

When a notification arrives, it's decoded and re-emitted as an in-process EventEmitter event:

client.on('notification', (msg: { channel: string; payload?: string }) => {
  const { channel, payload } = msg;
  if (!payload) return;

  if (channel.startsWith('room:')) {
    const roomId = channel.slice('room:'.length);
    const data = JSON.parse(payload);
    emitter.emit('room_message', roomId, data);
  }
});

This keeps the service boundary clean: PostgresService owns the DB layer, ChatWsService owns the socket layer, and they communicate through typed in-process events rather than direct coupling.

6. Reference-Counted Room Subscriptions

You don't want to LISTEN to every room channel globally. A busy system might have thousands of active rooms, and you only care about the ones where this node has connected clients.

The solution is reference counting. Each node tracks how many local sockets are in each room. When the count goes from 0 to 1, issue LISTEN. When it drops back to 0, issue UNLISTEN.

async subscribeToRoomChannel(roomId: string): Promise<void> {
  const count = this.roomSubscriptionCounts.get(roomId) ?? 0;
  this.roomSubscriptionCounts.set(roomId, count + 1);

  if (count === 0) {
    await this.listenClient.query(`LISTEN "room:${roomId}"`);
  }
}

async unsubscribeFromRoomChannel(roomId: string): Promise<void> {
  const count = this.roomSubscriptionCounts.get(roomId) ?? 0;
  if (count === 0) return;

  const next = count - 1;
  if (next > 0) {
    this.roomSubscriptionCounts.set(roomId, next);
    return;
  }

  this.roomSubscriptionCounts.delete(roomId);
  await this.listenClient.query(`UNLISTEN "room:${roomId}"`);
}

One important detail: if LISTEN or UNLISTEN throws, roll back the ref count before rethrowing. Otherwise your in-memory count diverges from the actual Postgres subscription state, and the system silently delivers or drops messages incorrectly.

7. Exponential Backoff Reconnect for the LISTEN Connection

pg.Client does not auto-reconnect. If the dedicated LISTEN connection drops (network blip, Postgres restart, container reschedule), your entire real-time fanout silently dies. The app keeps running, inserts keep succeeding, but no notifications ever arrive. Nothing logs an error. Clients just stop receiving messages.

This is the kind of failure that looks like a different problem entirely until you know to look for it.

The fix is a reconnect loop with exponential backoff:

On error, schedule a reconnect attempt after a delay.
Double the delay on each failure, up to a cap (30 seconds works well).
On successful reconnect, re-issue LISTEN for every channel that was active before the drop.
Reset the backoff counter on success.

That last step, re-subscribing to all previously active channels, is why the roomSubscriptionCounts map exists beyond just reference counting. It's also the recovery manifest.

8. Horizontal Scaling: Stateless Nodes + Nginx least_conn + Postgres Fanout

Here is where the architecture earns its keep.

Each app node is completely stateless except for its in-memory socket registry and its LISTEN subscriptions. Add another node, and it connects to the same Postgres, starts LISTENing on channels as clients join rooms, and begins receiving the same NOTIFY events as every other node.

Cross-node fanout is free. You don't configure it. You don't pay for a broker to coordinate it. Postgres notifies everyone who is listening, and each node pushes to its own clients.

upstream app {
  least_conn;
  server app-1:3000;
  server app-2:3000;
  server app-3:3000;
  server app-4:3000;
  server app-5:3000;
  keepalive 1024;
}

least_conn routes new connections to whichever node has the fewest active ones, distributing WebSocket connections naturally as load grows.

With raw WebSockets and stateless nodes, you don't need sticky sessions at all. That's one less thing to configure.

9. The Unsexy Part: Postgres Settings That Mattered

Running Postgres with default container settings and then complaining it can't handle real-time load is like putting regular fuel in a race car and wondering why it's slow. The defaults are conservative. They're designed for safety, not throughput.

Under sustained insert load, Postgres periodically flushes dirty pages to disk in a checkpoint. With default settings, these checkpoints can cause multi-second write stalls. During a load test, that shows up as sudden latency spikes that look completely unrelated to your application code.

The fix:

checkpoint_completion_target = 0.9   # spread I/O across 90% of the checkpoint interval
checkpoint_timeout = 15min           # checkpoint less frequently
max_wal_size = 2GB                   # give WAL more room before forcing a checkpoint
wal_level = logical                  # needed for logical replication and event-oriented setups
max_wal_senders = 20                 # headroom for replication sender processes under load

Before applying these, p95 delivery latency occasionally spiked into the seconds mid-test. After: consistent sub-800ms p95 across consecutive runs. The application code didn't change. Only the database configuration did.

Treat Postgres like a primary system component, not a black box, and it behaves like one.

10. The Detour I Didn't Keep: Sharding LISTEN/NOTIFY

At one point I tried sharding room channels by hashing room IDs into a fixed set of channels and routing in the application layer. The idea was to reduce the number of active LISTEN channels per node.

It worked, in the sense that messages moved. But it immediately introduced everything I was trying to avoid: hot channels with uneven load distribution, routing logic in the app that Postgres used to handle transparently, and debugging sessions spent asking "why is this room not getting notifications?" because the hash put it on the wrong channel.

In short: I was reinventing a worse version of Kafka, inside my own application, on top of a system that already had a perfectly good primitive for exactly this problem.

The right model is simpler: one channel per room, LISTEN only when you have local clients, UNLISTEN when the last one leaves. Postgres handles the rest. The complexity budget is better spent elsewhere.

11. The Performance Turning Point: Batched Fanout + One Batch Send per Room per Tick

I expected the Postgres NOTIFY path to be the first bottleneck. It wasn't.

The first bottleneck was the fanout loop itself. Every notification triggered an immediate synchronous loop over all connected sockets in the room. With 300 receivers, that's 300 socket.send() calls in a single event loop tick. While that loop ran, everything else (incoming frames, heartbeats, new connections) waited.

Two changes fixed it.

First: batch messages before sending. Rather than flushing one WebSocket frame per NOTIFY, each notification appends its payload to a per-room buffer. A shared 30ms timer flushes all pending buffers, sending a single new_message_batch frame per room per tick. Under high insert rates this collapses many individual sends into one, dramatically reducing the number of socket.send() calls per second.

private enqueueRoomMessage(roomId: string, payload: unknown): void {
  const buf = this.roomMessageBatchBuffers.get(roomId);
  if (buf) buf.push(payload);
  else this.roomMessageBatchBuffers.set(roomId, [payload]);

  if (this.roomMessageBatchFlushTimer) return;
  this.roomMessageBatchFlushTimer = setTimeout(() => {
    this.roomMessageBatchFlushTimer = null;
    this.flushRoomMessageBatches();
  }, ChatWsService.ROOM_MESSAGE_BATCH_WINDOW_MS);
}

Second: yield once with setImmediate, then send to all eligible sockets. After the batch window closes, we JSON.stringify the batch once, filter to sockets that are open and not backpressured, then hand off to setImmediate. The actual send loop runs on the next event loop tick, meaning the current tick finishes cleanly before any socket I/O begins.

private broadcastToRoom(
  roomId: string,
  message: { event: string; data: unknown },
): void {
  const sockets = this.roomSockets.get(roomId);
  if (!sockets?.size) return;

  const payload = JSON.stringify(message);
  const openSockets = Array.from(sockets).filter(
    (socket) =>
      socket.readyState === WebSocket.OPEN &&
      socket.bufferedAmount <= WS_BACKPRESSURE_THRESHOLD_BYTES,
  );

  if (!openSockets.length) return;

  // setImmediate yields control back to the event loop before sending.
  // Without this, the send loop blocks incoming frames, heartbeats,
  // and new connections for the entire duration of the broadcast.
  setImmediate(() => {
    for (const socket of openSockets) {
      try {
        socket.send(payload);
      } catch {
        // ignore per-socket send errors
      }
    }
  });
}

Presence and typing notifications go out unbatched since they are low frequency and latency-sensitive. Only chat messages use the buffer and timer.

12. Proof: Load Test Results

The test script runs two concurrent scenarios: senders ramp from 0 to 120 VUs over 15 seconds, hold for 2 minutes 30 seconds, then ramp back down. Receivers hold at 300 constant VUs for the full 3 minutes. Peak concurrency is 420 VUs hitting the same single room, deliberately the worst-case fanout scenario. All tests ran locally on an Apple M4 Pro (10 cores), 5 app nodes behind Nginx. Results will vary on actual production environments.

Run	Delivery rate	Latency p95	Msgs received/s
1	99.76%	744ms	~25.6k
2	99.58%	167ms	~25.4k
3	98.93%	594ms	~25.0k
4	99.83%	310ms	~13.9k
5 (threshold tripped)	99.35%	1.11s	~12.6k

On the variance in p95 latency: the 30ms batch window is a tradeoff, not a constant. Shorter windows mean more flushes and more event loop pressure. Longer windows mean larger bursts when the timer fires. In a single-hot-room test like this, you can see the system “catch a bad rhythm” (GC, timer alignment, bursty flushes), and p95 will swing even when the underlying design is unchanged. The numbers that stayed consistent across every single run: zero duplicate sequences, zero out-of-order delivery. The database side of the story held perfectly.

On running locally vs cloud: the M4 Pro is not a server. There is no network hop between Docker containers, no cloud storage latency, no noisy neighbors. A properly provisioned cloud deployment with dedicated instances and low-latency networking should perform comparably on the fanout path. Postgres write throughput will depend on your storage class. These numbers are a floor, not a ceiling.

On why delivery isn't 100%: this is the most aggressive test case possible: one room, 300 receivers, 120 senders, all on the same fanout path. Real traffic spreads across rooms and varies over time. The few missed deliveries in each run are sender echo-back misses under event-loop/backpressure pressure, not “Postgres lost messages.”

About the “failed” run (threshold tripped): This isn’t the architecture failing—it’s the system exposing its bottleneck: WebSocket fanout on a single Node.js event loop.

When p95 latency crosses the threshold, it’s not because ordering breaks or Postgres falls behind. It happens when a hot-room burst aligns poorly with event loop scheduling, causing fanout delays and triggering backpressure.

That’s exactly what stress tests are supposed to reveal: not whether the system works, but where it stops being predictable.

At that point, the decision becomes explicit—either distribute load across rooms, relax latency expectations for worst-case hotspots, or move fanout beyond a single event loop (e.g. multiple processes or workers).

13. Tradeoffs (The Part You Should Actually Read)

This architecture is not universally correct. It's deliberately narrow, and it earns its simplicity by accepting real constraints.

When Postgres-as-bus makes sense

Your events come from rows you're already writing. Messages, presence changes, typing indicators: these all exist in the DB anyway. NOTIFY is free.
You want transactional coupling between write and publish. If the insert fails, the notification never fires. You can't accidentally notify clients about a write that rolled back.
You want a small operational footprint. One database, many stateless nodes. No broker cluster to provision, monitor, or debug.
Your consumers are currently connected clients. NOTIFY is not a durable queue. If no one is listening, the event is gone. That's fine for real-time fanout to live connections. It's not fine for offline consumers or audit pipelines.
Presence fits naturally. Online/offline state is a row in a presence table, swept by a periodic cleanup job. If a single room grows to 10k+ users, you would move presence aggregation to a materialized view or a dedicated summary table rather than broadcasting raw rows. But the underlying mechanism stays the same.

When you should reach for Kafka (or friends)

You need durable, replayable event streams that exist independently of your database tables.
Consumers need to catch up from arbitrary offsets without querying the DB directly.
You have multiple heterogeneous consumers (analytics pipelines, audit logs, downstream services) each with their own consumption rate and retry semantics.
You need retention and backpressure as first-class features, not something you're building yourself on top of a messages table.
You're at a scale where Postgres becoming the coordination point is genuinely risky, where a slow query or a checkpoint stall has downstream consequences across the entire system.

Kafka is not a real-time messaging tool. Kafka is a distributed, durable log with consumer groups. If that's what you need, reach for it. If you're reaching for it because "that's what everyone does for real-time," slow down and check whether Postgres is already doing 80% of the work.

14. The Takeaway

Most systems don't have a broker problem.

They have an "I don't know my database well enough" problem.

Postgres is not just a place to put rows. It's a concurrency control engine, a transactional boundary, a durable store, and if you let it be, a perfectly capable event source for a whole class of real-time products. Triggers fire in the same transaction as your writes. LISTEN/NOTIFY delivers to every connected node. Per-room sequences give clients a reliable cursor with no gaps. A connection pool handles your queries. One dedicated client handles your subscriptions. That's the entire infrastructure story.

The lesson from this project isn’t “never use Redis.” It’s to understand your tools before you add more of them.

In fast-moving teams with tight deadlines, adding Redis or a message broker early is often the right call because it reduces risk and gives predictable scaling. There’s nothing wrong with that.

But if you always reach for those tools by default, you never learn where your existing stack actually breaks. You end up solving problems you might not have yet and carrying the complexity anyway.

Build the boring version when you can. Push it until it shows real limits. Then add complexity deliberately, not reflexively.

Premature Optimization Is Bad, But Your App Is Just Slow Because You're Lazy

Adam - The Developer — Thu, 12 Mar 2026 04:21:42 +0000

"Premature optimization is the root of all evil."

Donald Knuth wrote that in 1974. It is one of the most cited lines in all of software engineering, and it has been used to justify more genuinely terrible code than almost any other idea in the field.

The quote is correct. The way most developers apply it is not.

There is a difference between premature optimization and basic engineering competence. Somewhere along the way, the industry collapsed that distinction, and the result is production systems that make users wait for things that should be instant.

What Knuth Actually Said

Here is the full sentence, which almost nobody quotes:

"We should forget about small efficiencies, about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%."

He was talking about micro-optimizations. Loop unrolling. Manual register allocation. Squeezing cycles out of hot paths before you know which paths are hot. He was not giving you permission to write N+1 queries, load 400KB of JavaScript on a login page, or fetch entire database tables into memory to filter them in your application layer.

The "premature optimization" shield has been stretched so far beyond its original meaning that developers now invoke it to defend code that is simply slow by design.

The Difference Between Optimization and Competence

There are two entirely different things that get lumped together under "performance":

Premature optimization is spending three days hand-tuning a sorting algorithm before you know if sorting is even on the critical path. It is rewriting a function in assembly before you have profiled anything. It is trading code clarity for speed gains that may not matter.

Basic competence is not making obviously expensive choices when obviously cheaper ones exist at the same level of effort.

These are not the same thing. One requires you to know the future. The other just requires you to know your tools.

Writing a loop that queries the database on every iteration is not a performance decision you deferred for later. It is a mistake you made right now. Selecting every column with SELECT * when you need two fields is not an optimization you skipped. It is unnecessary work you added.

Nobody calls it premature optimization when a carpenter pre-drills a hole before driving a screw. That is just knowing what you are doing.

The Patterns That Are Just Laziness

Let us be specific. These are not edge cases or nuanced tradeoffs. These are patterns that slow applications down and have no corresponding benefit.

The N+1 Query

const posts = await db.query('SELECT * FROM posts');

for (const post of posts) {
  post.author = await db.query(
    'SELECT * FROM users WHERE id = ?', [post.author_id]
  );
}

If you have 200 posts, this runs 201 queries. If your database round trip takes 2ms, that is 402ms of pure waiting added to every single request, for the lifetime of the application, for every user, forever.

The fix is not an optimization. It is a JOIN, which is what relational databases were designed to do in 1970.

SELECT posts.*, users.name, users.avatar
FROM posts
JOIN users ON users.id = posts.author_id;

One query. Done. This is not a performance tradeoff. There is no version of the world where 201 queries is better than 1.

Selecting Everything

const user = await db.query('SELECT * FROM users WHERE id = ?', [id]);
return { name: user.name, email: user.email };

You fetched every column including the password hash, the encrypted recovery codes, the full address, the preferences blob, and the thirty other fields your schema has accumulated over three years. You used two of them.

Every extra column is bytes over the network, memory allocated, time spent serializing and deserializing. More importantly, SELECT * means your application will silently break or leak data if someone adds a sensitive column to that table later.

Select what you need. Always.

Synchronous Work That Does Not Need To Be Synchronous

// These two things don't depend on each other
const user = await getUser(userId);
const settings = await getSettings(userId);
const permissions = await getPermissions(userId);

Each await waits for the previous call to finish before starting the next. If each takes 50ms, you have spent 150ms doing work that could have been done in 50ms.

const [user, settings, permissions] = await Promise.all([
  getUser(userId),
  getSettings(userId),
  getPermissions(userId)
]);

Three concurrent requests, one round of waiting. This is not a micro-optimization. It is understanding how asynchronous code works, which is a baseline expectation for anyone writing it.

Rendering Thousands of DOM Nodes Because It Is Easy

A dropdown with 8,000 options. A table with 50,000 rows. A chat window that mounts every message since 2019 into the DOM.

The browser has to create, style, layout, and paint every one of those nodes. Then it has to keep them in memory. Scrolling becomes janky. Interactions stutter. The user experience becomes noticeably bad.

Virtualization, pagination, and windowing exist. They are not heroic performance engineering. They are the correct default for lists of unbounded size.

Not Caching Things That Never Change

// Called on every request
const countries = await db.query('SELECT * FROM countries ORDER BY name');

There are 195 countries. The list has not changed meaningfully in decades. You are hitting the database for it on every single page load.

A cache with a 24-hour TTL, or even just an in-memory constant loaded at startup, costs essentially nothing and eliminates the query entirely. This is not premature. This is reading the data and making an obvious decision about it.

Why This Keeps Happening

The honest answer is that slow code usually still works. The user experiences a delay. The developer does not feel the delay because they are testing on localhost against a database with 50 rows. The feature ships. The slowness becomes someone else's problem later.

There is also a subtler force at play. Modern frameworks and ORMs make it extremely easy to write slow code. ActiveRecord's lazy loading, GraphQL resolvers that each hit the database, React components that fetch independently of their siblings. These tools are excellent. They also make it trivially easy to produce N+1 queries without ever writing a single explicit loop.

The tools do not save you from understanding what they are doing on your behalf. That is still your job.

The Standard Worth Holding

"We'll optimize it later" is a reasonable thing to say about caching strategies, query tuning, and infrastructure scaling. It is not a reasonable thing to say about selecting fewer columns, batching database calls, or running independent tasks in parallel.

The bar is not "did the feature ship." The bar is "does the feature ship without obvious waste."

Profiling before optimizing is correct. Knowing what your code does before you write it is also correct. These are not in conflict. You do not need a profiler to know that 200 database queries is more than 1.

A Practical Filter

When you are writing code and wondering whether something is premature optimization or basic competence, ask one question:

Do I need to measure anything to know this is slower?

If the answer is yes, finish the feature and measure later. That is the Knuth principle in action.

If the answer is no, if the slower choice is obviously slower by construction and the better choice takes the same amount of time to write, then shipping the slow version is not a principled stance on optimization. It is just not doing the work.

Your users feel the difference. The profiler just helps you find it on a map.

The Unwritten Laws Running Your Code

Adam - The Developer — Thu, 05 Mar 2026 17:14:58 +0000

Nobody warns you about these in bootcamp. You learn them the hard way, usually at 11pm before a release, wondering how a three-week feature became a three-month saga. These are the laws, theorems, and hard-won principles that quietly govern every software project. The sooner you internalize them, the sooner you stop being surprised.

1. Brooks's Law

Adding developers to a late project makes it later.

Fred Brooks figured this out in 1975. We are still ignoring it in 2025.

The intuition is straightforward: if a project is behind, hire more people. The reality is brutal. Every new developer needs onboarding. Every existing developer loses productivity doing that onboarding. Communication overhead grows not linearly but combinatorially. With 4 people you have 6 communication channels. With 8 you have 28.

The deeper lesson is that software development is not like digging a ditch. You cannot divide a nine-month baby into nine one-month babies by assigning nine mothers. Some work is inherently sequential, and coordination is never free.

When a project is late, the first instinct should be to cut scope, not add headcount.

2. Conway's Law

Your architecture mirrors your org chart.

Mel Conway wrote this in 1968. It still lands like a punch every time a team discovers it firsthand.

Teams build systems that reflect their communication structures. If your frontend, backend, and data teams barely talk to each other, you will end up with a frontend, a backend, and a data layer that barely talk to each other. Want a clean service boundary? First, create a clean team boundary. Want a monolith? Put everyone in the same room.

This is not a coincidence or a failure of engineering discipline. It is gravity. You cannot fight it by drawing better architecture diagrams. You fight it by changing how people coordinate.

The inverse, sometimes called the Inverse Conway Maneuver, is actionable: design your team structure to match the architecture you want to build.

3. Parkinson's Law

Work expands to fill the time available for its completion.

Cyril Northcote Parkinson was writing about British bureaucracy in 1955. He may as well have been writing about sprint planning.

Give a developer two weeks for a two-day task and they will find ways to fill those two weeks. Not out of laziness, but out of a very human tendency to explore, refine, and polish when there is room to do so. Deadlines create focus. Vague timelines create scope creep, gold plating, and endless bike-shedding.

This is not an argument for crunch. It is an argument for honest estimation, enforced constraints, and clear definitions of done. A deadline without consequences is not a deadline. It is a suggestion.

4. Hofstadter's Law

It always takes longer than you expect, even when you take Hofstadter's Law into account.

Douglas Hofstadter wrote this in 1979, and the recursive phrasing is entirely intentional. The law mocks itself, and you, and all of us.

Human brains are optimism machines. We plan for the best case. We forget about testing, code review, integration issues, vague requirements, and the three meetings that will interrupt every afternoon this week. We have been burned before. We add buffer. We are still wrong.

The practical response is not despair. It is padding estimates by more than feels reasonable, tracking estimation accuracy over time, and treating deadlines as planning tools rather than commitments to the millisecond.

Accept the inevitability. Build in slack. Communicate early when things slip.

5. Gall's Law

A complex system that works evolved from a simple system that worked.

John Gall wrote this in 1975 and it is perhaps the most violated law in this entire list.

The failure mode is familiar: a team designs the full, complete, production-ready, infinitely scalable, perfectly abstracted system from day one. They spend months building something nobody has validated. It collapses under its own weight before a single user touches it.

The alternative is to start with the smallest working system possible, prove it solves the actual problem, and grow complexity only as the problem demands it. You cannot design your way to a working complex system. You can only grow one.

Microservices extracted too early. Event-driven architectures added before the data model is stable. Kubernetes before you have five users. These are Gall's Law violations. Start simple. Earn complexity.

6. Goodhart's Law

When a measure becomes a target, it ceases to be a good measure.

Economist Charles Goodhart articulated this in 1975. It is a trap every engineering team eventually falls into.

Lines of code per day. Story points per sprint. Test coverage percentage. Deployment frequency. Every one of these is a useful signal until it becomes a goal. Once it becomes a goal, people optimize for the metric rather than the outcome. Coverage climbs because developers write tests for easy code paths. Velocity climbs because teams inflate estimates. Deployment frequency climbs because you split changes into trivial one-liners.

The metric looked healthy. The system got worse.

Metrics are diagnostic tools. Treat them as health checks, not scorecards. Rotate them before they calcify into targets. Be suspicious of any number that is trending in the right direction too smoothly.

7. Hyrum's Law

With a sufficient number of users, all observable behaviors of your system will be depended upon by somebody.

Hyrum Wright articulated this from hard experience at Google. If you have ever tried to deprecate anything in a widely-used API, you already understand it viscerally.

It does not matter what your documentation says is the contract. Users observe the actual behavior of your system, and they build on top of it. Response time. Error message wording. Undocumented side effects. The order that items appear in a list. If it is observable and consistent, someone is depending on it.

This means that at scale, there are no truly internal implementation details. Any change to observable behavior is a breaking change for someone. Design public interfaces with this in mind. Version carefully. Deprecate slowly. Test the behavior your users actually rely on, not just the behavior you intended.

8. The Law of Leaky Abstractions

All non-trivial abstractions, to some degree, are leaky.

Joel Spolsky wrote this in 2002 and it remains one of the most useful mental models for debugging and system design.

Abstractions are supposed to hide complexity. TCP hides the unreliability of IP. ORMs hide the complexity of SQL. HTTP clients hide the messiness of network sockets. And they do hide it, until they do not.

The leak happens when the abstraction fails in a way that exposes the complexity underneath. Your ORM generates a query so inefficient that you have to write raw SQL to fix it. Your HTTP client fails with a timeout and you have to understand TCP keepalives to diagnose it. You cannot escape the underlying system. You only get to ignore it until something goes wrong.

The practical implication is that you must understand what your abstractions are built on, even if you never intend to touch that layer. When the leak appears, and it will, ignorance of the underlying system is expensive.

9. The 90/90 Rule

The first 90% of the code accounts for the first 90% of the development time. The remaining 10% accounts for the other 90% of the development time.

Tom Cargill is credited with this. It is darkly funny because everyone has lived it.

The feature works in the demo. It handles the happy path. It impresses stakeholders. Then comes the edge cases, the error handling, the accessibility, the performance under load, the mobile layout, the security review, the QA pass, the localization, the migration script, the rollback plan, the documentation, and the six bugs that only appear in production.

This is why "it is basically done" is a phrase that should make project managers nervous. The last stretch of software development is where assumptions meet reality, and reality usually has more surface area than you planned for.

Budget for it. Estimate it honestly. Do not promise shipping dates based on the completion of a working prototype.

10. No Solutions, Only Tradeoffs

There is no perfect solution. Every architectural decision trades one set of problems for another.

This one does not have a single author because it is a hard-won truth that every experienced engineer arrives at independently.

Microservices give you independent deployability and team autonomy. They also give you distributed systems complexity, network latency, and operational overhead. Caching improves read performance and introduces stale data problems. Type systems catch bugs at compile time and slow down iteration in early stages. Consistency guarantees in databases trade throughput. Premature optimization trades readability.

Every time you hear someone describe a technology as universally better than another, be skeptical. Better for what workload, at what scale, with what team, under what constraints? The question is never which option is good and which is bad. The question is which set of tradeoffs fits your current situation.

Senior engineers are not people who know more answers. They are people who have learned to ask better questions about tradeoffs.

Putting It Together

These laws do not predict the future. They describe the shape of problems you are already walking toward. Brooks's Law will not stop you from overstaffing a late project, but knowing it might make you pause before you do. Goodhart's Law will not prevent a manager from turning metrics into targets, but it might help you push back when you see it happening.

The goal is pattern recognition. The more fluent you are in these principles, the earlier you can name what is happening, and naming it is the first step to navigating it.

You will still ship late. You will still underestimate. You will still inherit a complex system that nobody planned to build. But you will be a little less surprised, and in this industry, that counts for a lot.

When Past Team Failures Become Your Team's Problem

Adam - The Developer — Sat, 28 Feb 2026 06:32:58 +0000

I want to tell you about a situation I'm currently navigating as a team lead.

A senior technical leader came in with a reputation. Not a bad one on paper. He'd been around, worked with multiple teams, seen projects go sideways. On paper, experience. In practice, something else entirely.

Before we'd shipped a single line of code together, before he'd sat in a single planning session with us, before he'd looked at our past projects or asked a single question about how we work, he had already decided what kind of team we were.

We were a risk. A liability. Another team waiting to fail.

And everything that followed came from that assumption.

What It Actually Looks Like

It doesn't always show up as open hostility. That would almost be easier to deal with. Instead it looks like this:

We ran benchmark tests. We ran load tests. We brought results to the table, clean numbers, real data. They got acknowledged with a nod and then quietly set aside in favor of a gut feeling shaped by teams he'd worked with before us.

We raised concerns about the proposed tech stack. An enterprise-grade setup for what was, honestly, a straightforward application. The kind of choice that makes sense when you're building for ten thousand concurrent users on day one, not for where we actually were. The concerns got labeled as resistance. Not curiosity. Not due diligence. Resistance.

We pointed to the projects we'd delivered. The ones that worked, the ones users actually used, the ones we were proud of. One past mistake we'd made got surfaced and held up like evidence. The wins were invisible. The one stumble was apparently the whole story.

And my job as a team lead, the part where I'm supposed to protect my team, advocate for their talent, fight for their credibility, started to feel like pushing against a wall that had already decided it wasn't going to move.

What's Actually Happening Here

Here's what I've come to understand about this pattern, because it is a pattern and I doubt I'm the only one who's lived it.

When a leader has been burned by failing teams, they don't always process that experience cleanly. Instead of asking "what specifically went wrong and why," they develop a generalized threat response. Teams become a category of risk rather than a collection of individual people with individual track records. The heuristic becomes: teams fail, therefore this team might fail, therefore I should treat them as if they already have.

It's not entirely irrational. Being responsible for projects that collapse is genuinely painful. But it is deeply unfair, and more importantly, it is actively counterproductive. Because what you get when you lead from that place is not a safer project. You get a demoralized team, bloated architecture chosen for fear rather than fit, and an environment where raising a legitimate concern gets read as insubordination.

The irony is that the conditions most likely to make a team fail are exactly the ones this kind of leadership creates. No trust, no psychological safety, no ownership. That's not protection from failure. That's a setup for it.

The Unfair Inheritance

What stings most about this situation is that my team didn't fail those previous projects. We weren't there. We have nothing to answer for.

We showed up with our actual work. Our actual results. Our actual expertise. And none of it was enough to override the story that had already been written about us before we walked in the room.

That's the unfair inheritance. You didn't cause the trauma but you're expected to carry it. You have to prove a negative. You have to somehow demonstrate that you are not a team that doesn't even exist anymore, that failed on projects you weren't part of, for reasons that had nothing to do with you.

There is no benchmark result good enough for that. There is no track record clean enough. Because the goalpost isn't really about your team at all.

How I've Been Pushing Back

I'm still in this. I won't pretend I've figured it all out. But here's what I've been doing and what I'd tell any team lead in the same position.

Document everything publicly. Results, decisions, concerns raised, responses received. Not as ammunition but as a paper trail of reality. When the narrative tries to rewrite your team's story, having a clear record of what actually happened and what you actually said matters. It protects your team and it protects you.

Name the pattern directly, calmly, and early. The first time a past team's failure gets used to justify a decision about your team, address it in the room. Something like: "I want to make sure we're evaluating this based on our team's actual work and results, not on experiences with previous teams." It doesn't always land. But it puts it on record and it signals that you're paying attention.

Keep bringing the data anyway. Even when it feels like it's being ignored, keep presenting benchmark results, load test outcomes, delivery history. Not because it will always change minds in the moment, but because it builds a consistent record that your team's decisions are evidence-based. Over time, that record speaks.

Advocate for your team visibly. When a team member raises a valid concern and it gets dismissed, back them up explicitly. "I think this is worth taking seriously" goes further than silence. Your team needs to see you fight for them, even when the fight is hard.

Know when to escalate. This one is uncomfortable but real. If a leader's fear-based decisions are putting your team, your project, or your people's wellbeing at genuine risk, that is a leadership problem, not a technical one, and it belongs in front of whoever owns leadership accountability in your organization. Protecting your team sometimes means going over someone's head and being honest about why.

To Any Team Lead Reading This

If you've worked with teams that failed, I genuinely empathize. That is hard. The weight of a collapsed project doesn't go away quickly.

But the team in front of you right now is not that team. They deserve to be evaluated on their own merits, their own track record, and their own results. When you walk in having already decided what they are, you don't protect the project. You just make it harder for the people trying to actually deliver it.

Your scar tissue is yours to process. It is not your new team's problem to solve.

The situation I'm in isn't resolved yet. But I'm not staying quiet about it, and my team isn't invisible. Some battles are worth fighting even when the outcome is uncertain, and advocating for people who trust you to have their back is one of them.

If you've been here too, you already know. And if you're heading here for the first time, I hope something in this helps you feel a little less alone in it.

Why Most Teams Shouldn't Build a Chat System

Adam - The Developer — Tue, 24 Feb 2026 03:22:58 +0000

There's a moment in every product team's life. The roadmap is open, someone's had a coffee too many - going all out of their mind, and from across the table comes those 5 fateful words:

"What if we added chat?"

The room nods. It feels obvious. It feels modern. It has that new-feature smell. Nobody asks why. Nobody does the math. The ticket gets created, the estimate gets wildly undershot, and six months later a senior dev is at 2am debugging why messages arrive out of order for users in Southeast Asia.

This post is an intervention.

🎭 Chat Looks Simple. Chat Is Not Simple.

The cruel trick of chat is that the demo takes about four hours to build. WebSockets, a text input, a message list, boom, you're basically WhatsApp. Ship it.

Then your first real user shows up.

Suddenly you need to handle:

📬 Message delivery guarantees: Did the message actually arrive, or did the socket drop silently? Better build acknowledgment logic. Better handle retries. Better make sure the retry doesn't deliver the same message twice. Users don't love getting "hey are you free?" four times.
⏱️ Message ordering: Two users send messages at roughly the same time. Whose arrives "first"? Clocks on distributed systems disagree more than you'd expect. Congratulations, you've stumbled into the world of distributed clocks and causal ordering (vector clocks, Lamport timestamps), and you did not pack for this trip.
🟢 Presence and online state: "Is this person online?" sounds like one database column. It is actually a distributed state synchronization problem that has made grown engineers weep.
✅ Read receipts: Those two little checkmarks? Each one is a fan-out write to every participant in the conversation. At scale, this gets spicy fast.
🔔 Notification systems: Push notifications, email fallbacks, in-app badges, do-not-disturb windows, notification preferences per conversation. This is its own product.
📜 Message history: How far back do you store? How do you paginate it efficiently? What happens when someone scrolls up through 50,000 messages in a support channel? Your database will have opinions about this.
🛡️ Moderation: Someone will say something terrible. What's your plan? Who sees flagged messages? Can you delete from all clients simultaneously? Can you audit it?
📎 Attachments: "Can we send files?" Yes, and now you've also built a file storage system, a media preview system, and a malware scanning pipeline. Congrats.
🧑‍💼 The admin dashboard nobody warned you about: Here's one that sneaks up on teams: the moment chat exists, management will want to see it. Who's messaging whom? Can we flag certain conversations? Can we pull reports? Can we suspend a user mid-conversation? What started as a feature request for your users has now spawned an entirely separate internal tool for your ops team. That's two products for the price of one sprint estimate.

And this is before you've handled any of the product complexity: group chats, thread replies, reactions, message editing, search, slash commands, bots...

The iceberg doesn't even have the decency to show you its bottom.

😤 It's Literally One Google Search Away

Here's what genuinely stings. This isn't obscure knowledge buried in a whitepaper or locked behind a paywall. You don't need to have built three chat systems to know what you're walking into. You just need to type "challenges of building a chat system" into a search bar and spend fifteen minutes reading.

The tradeoffs are documented. The war stories are public. Engineers have been writing about WebSocket pitfalls, message ordering nightmares, and presence system complexity for years, for free, on the internet. Stack Overflow threads, engineering blogs from Slack, Discord, Telegram, all out there, all findable in under a minute.

And yet, somehow, some leads and managers will sit in a planning meeting, confidently wave away engineering concerns, and greenlight a chat feature with a two-week estimate without doing that one search. Not because they're malicious. But because the decision felt obvious, and obvious decisions don't feel like they need research.

This is the part that frustrates engineers the most, and fairly so. It's not that hard problems exist. It's that the hard problems were completely avoidable with the smallest amount of due diligence. When you raise concerns in the meeting and get "I think you're overcomplicating it" from someone who has never had to implement fan-out writes at 3am, something inside you dies a little.

So if you're a lead or manager reading this: please, before the next time you greenlight a real-time feature, spend fifteen minutes googling it. Not to become an expert. Just to have a basic respect for what you're asking your team to build. Your engineers will notice, and they will appreciate you for it more than you know.

🤔 Why Teams Default to Chat Anyway

The reasons are understandable, even if they're not good:

"Our competitors have it." Ah yes, the product manager's favorite reasoning: copy the thing, inherit the success. Your competitors also have years of infrastructure investment, dedicated teams, and hard-won scar tissue from all the times it broke. You have a sprint and a prayer.

"Users expect it." Sometimes true. Usually a projection. Have you asked users? Or did you assume they'd want it because you use Slack?

"It'll increase engagement." It will increase something, yes. Mostly support tickets and infrastructure costs until it's done properly.

"It can't be that hard." Reader, it was that hard.

💡 What You Probably Actually Need

Here's the thing: underneath every "let's add chat" request is a real product problem. The question is whether chat is actually the right solution to it.

If the goal is customer support: You don't need to build this. Intercom, Crisp, Zendesk, Freshchat. These exist, they're good, and they have entire teams working on the thing you'd be building from scratch. Integrate one in a day and ship literally anything else.

If the goal is async team collaboration: Are you sure that's in scope for your product? Unless communication tooling is your product, you're about to build a worse Slack inside your app and charge people for the privilege of using it instead of the Slack tab they already have open.

If the goal is real-time multiplayer or collaboration (think Google Docs-style, or a game): this is one of the few cases where building real-time infrastructure is genuinely core to the product. Go for it, but go in with eyes open and allocate real engineering time to it.

If the goal is notifications or status updates: You almost certainly want a notification feed, not a chat system. Much simpler data model, no two-way communication complexity, still feels interactive and alive.

❓The Three Questions Worth Asking First

Before your team writes a single line of WebSocket code, answer these honestly:

What specific behavior changes for users who have chat vs. users who don't? If you can't answer this with a concrete user story, you don't have a product reason. You have a feature hunch.
What's the cheapest non-chat solution to the same problem? Could an email thread solve it? A comment section? A shared document? A third-party embed? If something much simpler solves 80% of the problem, that's probably the right call for now.
Can your product survive chat being broken for 24 hours? If chat becomes core infrastructure and it goes down, what's the blast radius? Real-time systems fail in ways batch systems don't. Plan accordingly.

✅ When You Should Build Chat

To be fair: some teams absolutely should build a chat system. If real-time communication is your core value proposition, if users come to your product because of the messaging, then yes, invest in it properly, staff it appropriately, and build it to last.

The keyword there is properly. Not "let's add a websocket endpoint and see how it goes." Proper investment, proper architecture, proper on-call rotation for when things inevitably get weird at 2am.

But if chat is a side feature, a nice-to-have, a "might be cool," reconsider hard. Your engineering team's time is finite and precious. Every week spent on chat infrastructure is a week not spent on the thing that actually differentiates your product.

🧾 The Bottom Line

Chat is seductive because it looks like a feature and is a platform. You don't stumble into building Slack. You commit to it, staff for it, and treat it as a first-class technical investment.

Most teams don't need to do that. Most teams need to solve the underlying problem that made chat sound appealing in the first place, and there's usually a simpler, faster, cheaper path to solving it.

So next time someone says "what if we added chat?" don't nod along. Ask why. Poke at the real problem. Reach for the third-party embed.

And if the answer is still "no really, we need to build this ourselves," great. Now at least you know what you're signing up for.

Good luck. You'll need it around the read receipts.

Only The Strong Survive: Breaking Into Software Engineering in 2026

Adam - The Developer — Mon, 09 Feb 2026 07:28:20 +0000

I don't have a solution to offer you. I wish I did. But I do have a warning.

If you're trying to break into software engineering right now, in 2026, in the age of AI coding assistants, you need to understand something: the bar hasn't just moved. It's been launched into orbit.

The Entry-Level Paradox

Here's what's happening: the work that used to train junior developers is being automated away. Not the sexy work. The grunt work. The tedious, boring, character-building work that taught you how systems actually worked.

Five years ago, a junior developer would spend months:

Writing unit tests for legacy code
Fixing minor bugs to learn the codebase
Converting data formats
Implementing straightforward features from specs
Doing code reviews on simple PRs

This wasn't glamorous. But it was education disguised as work. You got paid to break things, to understand edge cases, to see how production systems actually behave under stress.

Today? Claude does that. Copilot does that. Cursor does that. Faster, cheaper, and often with fewer syntax errors.

The work that used to be your training ground is now a cost-cutting opportunity.

The Gate

Picture the software engineering career as a walled garden. There's a gate you need to pass through.

In 2015, that gate was maybe 6 feet tall. If you had a CS degree or a bootcamp certificate, some personal projects, and decent problem-solving skills, you could climb over. Not easily (it was always competitive) but it was climbable.

In 2026, that gate is 20 feet tall. With barbed wire. And it's on fire.

Only the exceptionally strong are making it over. The naturally talented. The people who are so driven they built three production apps before they even applied. The ones who contribute to open source, who blog about system design, who can talk fluently about distributed systems despite never having worked on one professionally.

And even many of them are struggling.

Why This Is Different

"Every generation thinks the next one has it harder." I hear you. And yes, tech has always been competitive.

But this is structurally different.

Before, the path was:

Get hired as a junior
Do boring work that teaches you fundamentals
Gradually take on more complex work
Become senior

Now, step 2 doesn't exist. Companies are using AI for the boring work. Which means:

Get hired as a junior (but why would they?)
???
Become senior

The learning ladder has been removed while you were climbing it.

What "Exceptionally Hard" Looks Like

If you're starting out now, here's the brutal truth about what you're competing against:

You're not competing against other juniors anymore. You're competing against AI that can write boilerplate faster than you, and against seniors who use AI as a force multiplier.

To even get in the door, you need to demonstrate things that used to take years to develop:

Systems thinking (understanding how components interact)
Debugging skills (finding the needle in the haystack)
Code reading ability (parsing what someone else—or some AI—wrote)
Architectural judgment (knowing when the "correct" solution is wrong)

These used to be senior skills. Now they're entry requirements.

The Talent Filter Has Become Extreme

The industry is bifurcating:

Tier 1: The Super-Seniors
Developers with 10+ years of experience who use AI to move faster. They review AI output, catch its mistakes, and know when to override it. They're more productive than ever.

Tier 1.5: The Squeezed Middle
Mid-level engineers with 3-7 years of experience who understand systems, use AI effectively, and could be solving senior-level problems—but don't have the title, authority, or years to be trusted with architectural decisions yet.

These engineers are at a critical inflection point. They're good enough that AI makes them dangerous (in a good way), but not senior enough to have full autonomy.

Watch this group closely. Some will break upward fast, becoming the next generation of super-seniors. Others will find themselves competing with AI-augmented juniors from below and entrenched seniors from above, stuck in a shrinking middle that's being automated away from underneath and credential-checked from above.

Tier 2: The Struggling Aspirants
People trying to break in who are told to "just learn to code" and "build projects," only to find that:

Their projects look identical to everyone else's (because everyone uses the same AI prompts)
Interviews now expect you to already know what you were supposed to learn on the job
The "junior" positions require 2-3 years of experience (because why hire a true junior when AI can do that work?)

The middle is hollowing out. The path from Tier 2 to Tier 1 is becoming nearly impassable.

So What Do You Do?

I told you I don't have a solution. I don't. But I have observations about the people who are making it:

1. They Go Deeper, Not Wider

They don't just "know React." They understand why React re-renders, what the virtual DOM actually does, how reconciliation works. When AI generates code, they can spot the performance footgun.

2. They Build Things AI Can't (Yet)

Not another todo app. Not another weather app. Something weird. Something that required real problem-solving. Something where they had to make architectural decisions with trade-offs.

3. They Become Exceptional Debuggers

They can look at a stack trace and work backwards. They can read unfamiliar code and understand it. They can take AI-generated code that "doesn't work" and figure out why.

4. They Treat AI as a Sparring Partner, Not a Crutch

They use AI to go faster, but they read every line it generates. They ask "why did it do it this way?" They refactor it. They treat it like a junior developer they're mentoring, not a senior they're deferring to.

5. They Network Relentlessly

Because when the gate is this high, sometimes you need someone on the other side to open it for you.

The Uncomfortable Truth

Here's what I think is happening, and no one wants to say it out loud:

The industry is in the process of deciding it doesn't need as many entry-level code producers.

Not because the work is going away (there's more software being built than ever). But because the productivity multiplier from AI means the nature of what humans do is changing. We need fewer people writing boilerplate and more people making judgment calls that AI can't make yet.

The problem isn't that there's no room for new engineers. It's that the entry point now assumes capabilities that used to take years to develop.

The ones who make it through will be exceptional. The bar for "good enough" has been raised to "excellent."

And if you're not willing to push yourself to that level—or if you don't have the resources, time, or support to do so—the gate might be too high.

This Isn't Fair

Let me be clear: this isn't meritocratic, and it isn't fair.

The people who make it through won't just be the most talented. They'll be the ones who:

Could afford to spend months building projects without income
Had mentors or networks to guide them
Didn't have to work two jobs while learning
Had the background knowledge to even know what "good" looks like

We're creating a system where the already-privileged have an even bigger advantage. The gate isn't just high—it's placed at the top of a hill that not everyone can reach.

Where Do We Go From Here?

I don't know. I genuinely don't.

Maybe companies will realize they need to invest in new pathways—teaching people to audit AI, to be "forensic coders" who understand what the machine hallucinated and why.

Maybe bootcamps and universities will adapt, focusing less on syntax and more on systems thinking and debugging.

Maybe the industry will correct, and we'll realize that a generation of "vibe coders" who can prompt but not debug is a house of cards waiting to collapse.

Or maybe the gate stays high, and the software engineering career becomes something only the exceptionally driven or exceptionally privileged can access.

My Advice, For What It's Worth

If you're trying to break in right now:

Go harder than you think is reasonable. The normal path won't cut it anymore.

Learn to read code better than you write it. AI will outwrite you. But it can't yet out-understand you.

Build real things that solve real problems. Even small ones. Especially small ones that you actually deploy and maintain.

Find people who will open the gate. Mentorship, networking, contributing to open source: these aren't optional anymore.

Don't just learn to use AI. Learn to use it critically. Treat every line it generates with suspicion. Understand why it made those choices.

And if you make it through, if you're one of the ones who climbs that 20-foot burning gate, remember how hard it was. Remember the people who didn't make it not because they weren't smart or hardworking, but because the system made it nearly impossible.

And maybe, when you're on the other side, you'll help lower the gate for the next person.

A Final Note on What This Advice Means

Everything I just told you to do—go harder, build more, network relentlessly—is descriptive, not prescriptive. I'm telling you what I see working, not what should be required.

If you don't make it through that gate, it is not a moral failing.

It doesn't mean you're not smart enough, or you didn't work hard enough, or you're not "cut out for this." It means the gate is absurdly, unjustifiably high, and the system is broken in ways that have nothing to do with your worth as a person or even your potential as an engineer.

Some of the most talented developers I know didn't make it through—not because they couldn't, but because they couldn't afford to spend a year building projects for free, or they had family obligations, or they just got unlucky with timing.

Your survival of this system is not a measure of your value. It's a measure of the system's cruelty.

I'm giving you the map as it exists, not as it should be. Use it if you can. But if you can't, or if you try and it doesn't work out, know that the failure is the system's, not yours.

[Boost]

Adam - The Developer — Mon, 09 Feb 2026 04:57:34 +0000

NorthernDev

Feb 5

The Junior Developer is Extinct (And we are creating a disaster)

#discuss #career #ai #future

193

Comments 192

2 min read

Sharding Databases with Spring Boot: Patterns, Pitfalls, and Failure Modes

Adam - The Developer — Fri, 06 Feb 2026 02:51:29 +0000

Sharding is one of those topics that sounds intimidating until you actually understand what's happening under the hood. If you're a senior Java developer encountering sharding for the first time, or if you need a refresher on implementing it with Spring Boot, this guide covers everything from basic concepts to production ready implementations.

What is Database Sharding?
Why Shard?
Core Sharding Concepts
- The Shard Key
- Sharding Strategies
- Range Based Sharding
- Hash Based Sharding
- Directory Based Sharding
- Geographic Sharding
Hash Function Deep Dive
- Choosing a Hash Function
- The Modulo Problem
- Consistent Hashing to the Rescue
Spring Boot Implementation Patterns
- Setting Up Multiple DataSources
- DataSource Configuration
- Shard Resolver Service
- Sharded JDBC Template
Query Patterns and Challenges
- Single Shard Queries
- Cross Shard Queries (Scatter Gather)
- Joins Across Shards
Transactions Across Shards
- Two Phase Commit with Spring
- Better Approach: Saga Pattern
Monitoring and Observability with Spring Boot Actuator
- Health Check Endpoint
Rebalancing and Resharding
- Adding New Shards
- Zero Downtime Migration with Dual Writes
Real World Example: Complete User Service with Spring Boot
Common Pitfalls and Solutions
- Pitfall 1: Wrong Shard Key Choice
- Pitfall 2: Not Planning for Growth
- Pitfall 3: Connection Pool Exhaustion
- Pitfall 4: Not Handling Shard Failures
- Pitfall 5: No Backup Strategy Per Shard
Conclusion

What is Database Sharding?

Database sharding is a horizontal partitioning strategy where you split your data across multiple database instances (shards). Instead of having one massive database trying to handle all your traffic and data, you distribute the load across multiple smaller databases.

Think of it like this: instead of one gigantic library holding every book ever written, you have multiple libraries where books are distributed based on some criteria (maybe by author's last name, subject, or publication date).

Why Shard?

Before diving into implementation, let's talk about when you actually need sharding:

Vertical scaling hits a wall. You can only add so much RAM, CPU, and faster disks to a single database server before you hit physical or cost limitations.

Query performance degrades. Even with proper indexing, queries slow down as your dataset grows. A query on a 10TB table is inherently slower than the same query on a 100GB table.

Write throughput becomes a bottleneck. A single database can only handle so many writes per second. When your application needs to process millions of writes per minute, one database won't cut it.

High availability requirements. Sharding can improve availability because a failure in one shard doesn't take down your entire system.

Core Sharding Concepts

The Shard Key

The shard key is the field you use to determine which shard holds a particular piece of data. This is absolutely critical to get right because changing it later is extremely painful.

Common shard key choices:

User ID
Tenant ID (for multi tenant applications)
Geographic region
Timestamp ranges (for time series data)

The shard key should have these properties:

High cardinality: Many unique values to distribute data evenly
Query friendly: Your most common queries should be able to target a single shard
Stable: The value shouldn't change frequently (or at all)

Sharding Strategies

There are several ways to distribute data across shards:

1. Range Based Sharding

You divide data based on ranges of the shard key.

Shard 1: user_id 1 to 1,000,000
Shard 2: user_id 1,000,001 to 2,000,000
Shard 3: user_id 2,000,001 to 3,000,000

Pros: Simple to implement, range queries are efficient
Cons: Can lead to hotspots if data isn't uniformly distributed

2. Hash Based Sharding

You apply a hash function to the shard key and use the result to determine the shard.

public int getShard(Long userId, int numShards) {
    return Math.abs(userId.hashCode()) % numShards;
}

Pros: Excellent data distribution
Cons: Range queries require hitting all shards

3. Directory Based Sharding

You maintain a lookup table that maps shard keys to specific shards.

user_id -> shard mapping stored in a lookup service
1234 -> shard_2
5678 -> shard_1
9012 -> shard_3

Pros: Flexible, can rebalance without changing keys
Cons: Lookup table becomes a single point of failure, adds latency

4. Geographic Sharding

Data is sharded based on geographic location.

Users in North America -> shard_us
Users in Europe -> shard_eu
Users in Asia -> shard_asia

Pros: Reduces latency for geo distributed users, helps with data residency compliance
Cons: Can lead to uneven distribution

Hash Function Deep Dive

Let's get into the details of how hash based sharding actually works in Java.

Choosing a Hash Function

You need a hash function that:

Distributes keys uniformly across shards
Is fast to compute
Has minimal collisions
Is deterministic (same input always gives same output)

Common choices:

MD5: Fast and provides good distribution

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.math.BigInteger;

public class ShardingUtils {

    public static int md5HashShard(String key, int numShards) {
        try {
            MessageDigest md = MessageDigest.getInstance("MD5");
            byte[] messageDigest = md.digest(key.getBytes());
            BigInteger bigInt = new BigInteger(1, messageDigest);
            return Math.abs(bigInt.intValue()) % numShards;
        } catch (NoSuchAlgorithmException e) {
            throw new RuntimeException("MD5 algorithm not found", e);
        }
    }
}

MurmurHash: Faster than MD5, excellent distribution

import com.google.common.hash.Hashing;
import java.nio.charset.StandardCharsets;

public class ShardingUtils {

    public static int murmurHashShard(String key, int numShards) {
        int hash = Hashing.murmur3_32_fixed()
            .hashString(key, StandardCharsets.UTF_8)
            .asInt();
        return Math.abs(hash) % numShards;
    }
}

CRC32: Very fast, good enough for most use cases

import java.util.zip.CRC32;
import java.nio.charset.StandardCharsets;

public class ShardingUtils {

    public static int crc32HashShard(String key, int numShards) {
        CRC32 crc = new CRC32();
        crc.update(key.getBytes(StandardCharsets.UTF_8));
        return (int) (Math.abs(crc.getValue()) % numShards);
    }
}

The Modulo Problem

The simple modulo approach has a major flaw: when you add or remove shards, almost all keys get reassigned to different shards.

// With 3 shards
hash(123) % 3 = 0  // Goes to shard 0

// Add a shard, now 4 shards
hash(123) % 4 = 3  // Now goes to shard 3!

This means massive data migration when scaling.

Consistent Hashing to the Rescue

Consistent hashing solves the resharding problem by minimizing the number of keys that need to move when shards are added or removed.

Here's how it works conceptually:

Imagine a ring (circle) with values from 0 to 2^32
Each shard is assigned multiple points on this ring (virtual nodes)
Each key is hashed to a point on the ring
The key belongs to the first shard found moving clockwise from the key's position

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.*;

public class ConsistentHash<T> {

    private final int virtualNodes;
    private final TreeMap<Long, T> ring;
    private final MessageDigest md;

    public ConsistentHash(int virtualNodes) {
        this.virtualNodes = virtualNodes;
        this.ring = new TreeMap<>();
        try {
            this.md = MessageDigest.getInstance("MD5");
        } catch (NoSuchAlgorithmException e) {
            throw new RuntimeException("MD5 algorithm not found", e);
        }
    }

    public void addNode(T node) {
        for (int i = 0; i < virtualNodes; i++) {
            String virtualKey = node.toString() + ":" + i;
            long hash = hash(virtualKey);
            ring.put(hash, node);
        }
    }

    public void removeNode(T node) {
        for (int i = 0; i < virtualNodes; i++) {
            String virtualKey = node.toString() + ":" + i;
            long hash = hash(virtualKey);
            ring.remove(hash);
        }
    }

    public T getNode(String key) {
        if (ring.isEmpty()) {
            return null;
        }

        long hash = hash(key);

        // Find first node with hash >= key hash
        Map.Entry<Long, T> entry = ring.ceilingEntry(hash);

        // If not found, wrap around to first node
        if (entry == null) {
            entry = ring.firstEntry();
        }

        return entry.getValue();
    }

    private long hash(String key) {
        md.reset();
        md.update(key.getBytes());
        byte[] digest = md.digest();

        // Take first 8 bytes to create a long
        long hash = 0;
        for (int i = 0; i < 8; i++) {
            hash = (hash << 8) | (digest[i] & 0xFF);
        }
        return hash;
    }

    public int size() {
        return ring.size() / virtualNodes;
    }
}

// Usage
ConsistentHash<String> ch = new ConsistentHash<>(150);
ch.addNode("shard_1");
ch.addNode("shard_2");
ch.addNode("shard_3");

String shard = ch.getNode("user_12345");
System.out.println("Key belongs to: " + shard);

With consistent hashing, adding or removing a shard only affects approximately 1/N of the keys (where N is the number of shards).

Spring Boot Implementation Patterns

Setting Up Multiple DataSources

First, let's configure multiple datasources in Spring Boot:

// application.yml
spring:
  datasource:
    shard0:
      jdbc-url: jdbc:postgresql://db1.example.com:5432/shard_0
      username: app_user
      password: secret
      driver-class-name: org.postgresql.Driver
      hikari:
        maximum-pool-size: 10
        minimum-idle: 2
    shard1:
      jdbc-url: jdbc:postgresql://db2.example.com:5432/shard_1
      username: app_user
      password: secret
      driver-class-name: org.postgresql.Driver
      hikari:
        maximum-pool-size: 10
        minimum-idle: 2
    shard2:
      jdbc-url: jdbc:postgresql://db3.example.com:5432/shard_2
      username: app_user
      password: secret
      driver-class-name: org.postgresql.Driver
      hikari:
        maximum-pool-size: 10
        minimum-idle: 2

DataSource Configuration

import com.zaxxer.hikari.HikariDataSource;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.jdbc.DataSourceBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.sql.DataSource;
import java.util.HashMap;
import java.util.Map;

@Configuration
public class DataSourceConfig {

    @Bean
    @ConfigurationProperties("spring.datasource.shard0")
    public DataSource shard0DataSource() {
        return DataSourceBuilder.create()
            .type(HikariDataSource.class)
            .build();
    }

    @Bean
    @ConfigurationProperties("spring.datasource.shard1")
    public DataSource shard1DataSource() {
        return DataSourceBuilder.create()
            .type(HikariDataSource.class)
            .build();
    }

    @Bean
    @ConfigurationProperties("spring.datasource.shard2")
    public DataSource shard2DataSource() {
        return DataSourceBuilder.create()
            .type(HikariDataSource.class)
            .build();
    }

    @Bean
    public Map<String, DataSource> shardDataSources(
            DataSource shard0DataSource,
            DataSource shard1DataSource,
            DataSource shard2DataSource) {

        Map<String, DataSource> dataSources = new HashMap<>();
        dataSources.put("shard_0", shard0DataSource);
        dataSources.put("shard_1", shard1DataSource);
        dataSources.put("shard_2", shard2DataSource);
        return dataSources;
    }
}

Shard Resolver Service

import org.springframework.stereotype.Service;

@Service
public class ShardResolver {

    private final ConsistentHash<String> consistentHash;

    public ShardResolver() {
        this.consistentHash = new ConsistentHash<>(150);
        consistentHash.addNode("shard_0");
        consistentHash.addNode("shard_1");
        consistentHash.addNode("shard_2");
    }

    public String resolveShardId(Long userId) {
        return consistentHash.getNode(userId.toString());
    }

    public String resolveShardId(String key) {
        return consistentHash.getNode(key);
    }
}

Sharded JDBC Template

import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Component;

import javax.sql.DataSource;
import java.util.Map;

@Component
public class ShardedJdbcTemplate {

    private final Map<String, JdbcTemplate> shardTemplates;
    private final ShardResolver shardResolver;

    public ShardedJdbcTemplate(
            Map<String, DataSource> shardDataSources,
            ShardResolver shardResolver) {

        this.shardResolver = shardResolver;
        this.shardTemplates = new HashMap<>();

        shardDataSources.forEach((shardId, dataSource) -> {
            this.shardTemplates.put(shardId, new JdbcTemplate(dataSource));
        });
    }

    public JdbcTemplate getTemplate(Long userId) {
        String shardId = shardResolver.resolveShardId(userId);
        return shardTemplates.get(shardId);
    }

    public JdbcTemplate getTemplate(String shardId) {
        return shardTemplates.get(shardId);
    }

    public Map<String, JdbcTemplate> getAllTemplates() {
        return new HashMap<>(shardTemplates);
    }
}

Query Patterns and Challenges

Single Shard Queries

When your query includes the shard key, you can route directly to one shard.

@Service
public class UserService {

    private final ShardedJdbcTemplate shardedJdbcTemplate;

    public UserService(ShardedJdbcTemplate shardedJdbcTemplate) {
        this.shardedJdbcTemplate = shardedJdbcTemplate;
    }

    public User getUserById(Long userId) {
        JdbcTemplate template = shardedJdbcTemplate.getTemplate(userId);

        return template.queryForObject(
            "SELECT id, email, name, created_at FROM users WHERE id = ?",
            new Object[]{userId},
            (rs, rowNum) -> new User(
                rs.getLong("id"),
                rs.getString("email"),
                rs.getString("name"),
                rs.getTimestamp("created_at")
            )
        );
    }

    public void createUser(User user) {
        JdbcTemplate template = shardedJdbcTemplate.getTemplate(user.getId());

        template.update(
            "INSERT INTO users (id, email, name, created_at) VALUES (?, ?, ?, ?)",
            user.getId(),
            user.getEmail(),
            user.getName(),
            new Timestamp(System.currentTimeMillis())
        );
    }
}

Cross Shard Queries (Scatter Gather)

When your query doesn't include the shard key, you need to query all shards and merge results.

import org.springframework.stereotype.Service;
import java.util.List;
import java.util.ArrayList;
import java.util.concurrent.*;

@Service
public class UserService {

    private final ShardedJdbcTemplate shardedJdbcTemplate;
    private final ExecutorService executorService;

    public UserService(ShardedJdbcTemplate shardedJdbcTemplate) {
        this.shardedJdbcTemplate = shardedJdbcTemplate;
        this.executorService = Executors.newFixedThreadPool(10);
    }

    public List<User> findUsersByEmail(String email) {
        Map<String, JdbcTemplate> templates = shardedJdbcTemplate.getAllTemplates();
        List<Future<List<User>>> futures = new ArrayList<>();

        // Query all shards in parallel
        for (Map.Entry<String, JdbcTemplate> entry : templates.entrySet()) {
            Future<List<User>> future = executorService.submit(() -> {
                return entry.getValue().query(
                    "SELECT id, email, name, created_at FROM users WHERE email = ?",
                    new Object[]{email},
                    (rs, rowNum) -> new User(
                        rs.getLong("id"),
                        rs.getString("email"),
                        rs.getString("name"),
                        rs.getTimestamp("created_at")
                    )
                );
            });
            futures.add(future);
        }

        // Collect results from all shards
        List<User> results = new ArrayList<>();
        for (Future<List<User>> future : futures) {
            try {
                results.addAll(future.get(5, TimeUnit.SECONDS));
            } catch (InterruptedException | ExecutionException | TimeoutException e) {
                throw new RuntimeException("Error executing scatter gather query", e);
            }
        }

        return results;
    }
}

Joins Across Shards

This is where sharding gets painful. Joins across shards are expensive and complex.

Solution 1: Denormalize
Store redundant data to avoid cross shard joins.

-- Instead of joining users and orders across shards
-- Store user info in the orders table
CREATE TABLE orders (
    id BIGINT,
    user_id BIGINT,
    user_name VARCHAR(255),  -- Denormalized
    user_email VARCHAR(255),  -- Denormalized
    total DECIMAL(10,2)
)

Solution 2: Application Level Joins
Fetch from multiple shards and join in memory.

public class OrderWithUser {
    private Order order;
    private User user;

    // getters and setters
}

@Service
public class OrderService {

    private final ShardedJdbcTemplate shardedJdbcTemplate;
    private final UserService userService;

    public OrderWithUser getOrderWithUser(Long orderId, Long userId) {
        // Get order from its shard
        JdbcTemplate orderTemplate = shardedJdbcTemplate.getTemplate(orderId);
        Order order = orderTemplate.queryForObject(
            "SELECT * FROM orders WHERE id = ?",
            new Object[]{orderId},
            (rs, rowNum) -> mapOrder(rs)
        );

        // Get user from their shard
        User user = userService.getUserById(userId);

        OrderWithUser result = new OrderWithUser();
        result.setOrder(order);
        result.setUser(user);
        return result;
    }
}

Solution 3: Shard by Relationship
Co locate related data on the same shard.

// Users and their orders always on the same shard
// Use userId as shard key for both tables
@Service
public class OrderService {

    public List<OrderWithUser> getUserOrders(Long userId) {
        JdbcTemplate template = shardedJdbcTemplate.getTemplate(userId);

        // Both tables on the same shard, joins work normally
        return template.query(
            "SELECT u.*, o.* FROM users u " +
            "JOIN orders o ON u.id = o.user_id " +
            "WHERE u.id = ?",
            new Object[]{userId},
            (rs, rowNum) -> mapOrderWithUser(rs)
        );
    }
}

Transactions Across Shards

Distributed transactions are complex and slow. ACID guarantees across shards typically require two phase commit (2PC).

Two Phase Commit with Spring

import org.springframework.stereotype.Service;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.TransactionStatus;
import org.springframework.transaction.support.DefaultTransactionDefinition;

import java.util.*;

@Service
public class DistributedTransactionService {

    private final Map<String, PlatformTransactionManager> transactionManagers;
    private final ShardedJdbcTemplate shardedJdbcTemplate;

    public DistributedTransactionService(
            Map<String, PlatformTransactionManager> transactionManagers,
            ShardedJdbcTemplate shardedJdbcTemplate) {
        this.transactionManagers = transactionManagers;
        this.shardedJdbcTemplate = shardedJdbcTemplate;
    }

    public void executeTwoPhaseCommit(Map<String, Runnable> shardOperations) {
        Map<String, TransactionStatus> transactions = new HashMap<>();

        try {
            // Phase 1: Prepare (execute all operations)
            for (Map.Entry<String, Runnable> entry : shardOperations.entrySet()) {
                String shardId = entry.getKey();
                PlatformTransactionManager tm = transactionManagers.get(shardId);

                TransactionStatus status = tm.getTransaction(
                    new DefaultTransactionDefinition()
                );
                transactions.put(shardId, status);

                try {
                    entry.getValue().run();
                } catch (Exception e) {
                    throw new RuntimeException("Failed to execute on shard: " + shardId, e);
                }
            }

            // Phase 2: Commit all
            for (Map.Entry<String, TransactionStatus> entry : transactions.entrySet()) {
                String shardId = entry.getKey();
                PlatformTransactionManager tm = transactionManagers.get(shardId);
                tm.commit(entry.getValue());
            }

        } catch (Exception e) {
            // Rollback all prepared transactions
            for (Map.Entry<String, TransactionStatus> entry : transactions.entrySet()) {
                try {
                    String shardId = entry.getKey();
                    PlatformTransactionManager tm = transactionManagers.get(shardId);
                    tm.rollback(entry.getValue());
                } catch (Exception rollbackException) {
                    // Log rollback failure
                }
            }
            throw new RuntimeException("Distributed transaction failed", e);
        }
    }
}

Problems with 2PC:

Slow (multiple network round trips)
Coordinator failure can leave system in limbo
Locks held for long duration

Better Approach: Saga Pattern

import org.springframework.stereotype.Service;
import java.util.*;

@Service
public class SagaTransactionService {

    private final ShardedJdbcTemplate shardedJdbcTemplate;
    private final SagaLogRepository sagaLogRepository;

    public void transferMoney(Long fromUserId, Long toUserId, BigDecimal amount) {
        String sagaId = UUID.randomUUID().toString();

        try {
            // Step 1: Debit from user
            JdbcTemplate fromTemplate = shardedJdbcTemplate.getTemplate(fromUserId);
            fromTemplate.update(
                "UPDATE accounts SET balance = balance - ? WHERE user_id = ?",
                amount, fromUserId
            );
            sagaLogRepository.logStep(sagaId, "debit_complete", fromUserId);

            // Step 2: Credit to user
            JdbcTemplate toTemplate = shardedJdbcTemplate.getTemplate(toUserId);
            toTemplate.update(
                "UPDATE accounts SET balance = balance + ? WHERE user_id = ?",
                amount, toUserId
            );
            sagaLogRepository.logStep(sagaId, "credit_complete", toUserId);

            sagaLogRepository.markComplete(sagaId);

        } catch (Exception e) {
            compensateTransfer(sagaId, fromUserId, toUserId, amount);
            throw new RuntimeException("Transfer failed, compensation executed", e);
        }
    }

    private void compensateTransfer(
            String sagaId, 
            Long fromUserId, 
            Long toUserId, 
            BigDecimal amount) {

        List<SagaStep> completedSteps = sagaLogRepository.getCompletedSteps(sagaId);

        for (SagaStep step : completedSteps) {
            if ("debit_complete".equals(step.getStepName())) {
                // Reverse the debit
                JdbcTemplate template = shardedJdbcTemplate.getTemplate(fromUserId);
                template.update(
                    "UPDATE accounts SET balance = balance + ? WHERE user_id = ?",
                    amount, fromUserId
                );
            }
        }

        sagaLogRepository.markCompensated(sagaId);
    }
}

Monitoring and Observability with Spring Boot Actuator

import io.micrometer.core.instrument.MeterRegistry;
import io.micrometer.core.instrument.Tags;
import org.springframework.stereotype.Component;

import java.util.Map;

@Component
public class ShardMetricsCollector {

    private final ShardedJdbcTemplate shardedJdbcTemplate;
    private final MeterRegistry meterRegistry;

    public ShardMetricsCollector(
            ShardedJdbcTemplate shardedJdbcTemplate,
            MeterRegistry meterRegistry) {
        this.shardedJdbcTemplate = shardedJdbcTemplate;
        this.meterRegistry = meterRegistry;
    }

    @Scheduled(fixedRate = 60000) // Every minute
    public void collectShardMetrics() {
        Map<String, JdbcTemplate> templates = shardedJdbcTemplate.getAllTemplates();

        for (Map.Entry<String, JdbcTemplate> entry : templates.entrySet()) {
            String shardId = entry.getKey();
            JdbcTemplate template = entry.getValue();

            // Collect row count
            Long rowCount = template.queryForObject(
                "SELECT COUNT(*) FROM users",
                Long.class
            );

            meterRegistry.gauge(
                "shard.row.count",
                Tags.of("shard", shardId),
                rowCount != null ? rowCount : 0
            );

            // Collect table size
            Long tableSize = template.queryForObject(
                "SELECT pg_total_relation_size('users')",
                Long.class
            );

            meterRegistry.gauge(
                "shard.table.size.bytes",
                Tags.of("shard", shardId),
                tableSize != null ? tableSize : 0
            );
        }
    }

    public Map<String, ShardMetrics> getShardMetrics() {
        Map<String, ShardMetrics> metrics = new HashMap<>();
        Map<String, JdbcTemplate> templates = shardedJdbcTemplate.getAllTemplates();

        for (Map.Entry<String, JdbcTemplate> entry : templates.entrySet()) {
            String shardId = entry.getKey();
            JdbcTemplate template = entry.getValue();

            ShardMetrics shardMetrics = new ShardMetrics();
            shardMetrics.setShardId(shardId);
            shardMetrics.setRowCount(
                template.queryForObject("SELECT COUNT(*) FROM users", Long.class)
            );

            metrics.put(shardId, shardMetrics);
        }

        return metrics;
    }
}

@Data
class ShardMetrics {
    private String shardId;
    private Long rowCount;
    private Long tableSizeBytes;
    private Double queriesPerSecond;
    private Double avgQueryTimeMs;
}

Health Check Endpoint

import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.stereotype.Component;

@Component
public class ShardHealthIndicator implements HealthIndicator {

    private final ShardedJdbcTemplate shardedJdbcTemplate;

    public ShardHealthIndicator(ShardedJdbcTemplate shardedJdbcTemplate) {
        this.shardedJdbcTemplate = shardedJdbcTemplate;
    }

    @Override
    public Health health() {
        Map<String, JdbcTemplate> templates = shardedJdbcTemplate.getAllTemplates();
        Map<String, String> shardStatus = new HashMap<>();
        boolean allHealthy = true;

        for (Map.Entry<String, JdbcTemplate> entry : templates.entrySet()) {
            String shardId = entry.getKey();
            try {
                entry.getValue().queryForObject("SELECT 1", Integer.class);
                shardStatus.put(shardId, "UP");
            } catch (Exception e) {
                shardStatus.put(shardId, "DOWN");
                allHealthy = false;
            }
        }

        if (allHealthy) {
            return Health.up().withDetails(shardStatus).build();
        } else {
            return Health.down().withDetails(shardStatus).build();
        }
    }
}

Rebalancing and Resharding

Eventually, you'll need to add more shards or rebalance data.

Adding New Shards

@Service
public class ShardManagementService {

    private final ConsistentHash<String> consistentHash;
    private final Map<String, DataSource> shardDataSources;
    private final ShardedJdbcTemplate shardedJdbcTemplate;

    public void addShard(String newShardId, DataSource newDataSource) {
        // Add to consistent hash ring
        consistentHash.addNode(newShardId);

        // Add to datasources
        shardDataSources.put(newShardId, newDataSource);

        // Migrate affected keys
        migrateKeysToNewShard(newShardId);
    }

    private void migrateKeysToNewShard(String newShardId) {
        Map<String, JdbcTemplate> templates = shardedJdbcTemplate.getAllTemplates();

        for (Map.Entry<String, JdbcTemplate> entry : templates.entrySet()) {
            String oldShardId = entry.getKey();

            if (oldShardId.equals(newShardId)) {
                continue;
            }

            JdbcTemplate oldTemplate = entry.getValue();

            // Get all users from old shard
            List<User> users = oldTemplate.query(
                "SELECT id, email, name, created_at FROM users",
                (rs, rowNum) -> new User(
                    rs.getLong("id"),
                    rs.getString("email"),
                    rs.getString("name"),
                    rs.getTimestamp("created_at")
                )
            );

            JdbcTemplate newTemplate = shardedJdbcTemplate.getTemplate(newShardId);

            for (User user : users) {
                // Check if user should be on new shard
                String targetShard = consistentHash.getNode(user.getId().toString());

                if (targetShard.equals(newShardId)) {
                    // Move to new shard
                    newTemplate.update(
                        "INSERT INTO users (id, email, name, created_at) VALUES (?, ?, ?, ?)",
                        user.getId(),
                        user.getEmail(),
                        user.getName(),
                        user.getCreatedAt()
                    );

                    // Delete from old shard
                    oldTemplate.update(
                        "DELETE FROM users WHERE id = ?",
                        user.getId()
                    );
                }
            }
        }
    }
}

Zero Downtime Migration with Dual Writes

@Service
public class DualWriteService {

    private final ShardedJdbcTemplate shardedJdbcTemplate;
    private final Set<String> dualWriteShards = new ConcurrentHashSet<>();

    public void enableDualWrite(String oldShardId, String newShardId) {
        dualWriteShards.add(oldShardId + ":" + newShardId);
    }

    public void createUser(User user) {
        String primaryShard = resolvePrimaryShard(user.getId());
        JdbcTemplate primaryTemplate = shardedJdbcTemplate.getTemplate(primaryShard);

        // Write to primary shard
        primaryTemplate.update(
            "INSERT INTO users (id, email, name, created_at) VALUES (?, ?, ?, ?)",
            user.getId(), user.getEmail(), user.getName(), user.getCreatedAt()
        );

        // Check if dual write is enabled
        String dualWriteKey = findDualWriteConfig(primaryShard);
        if (dualWriteKey != null) {
            String secondaryShard = extractSecondaryShard(dualWriteKey);
            JdbcTemplate secondaryTemplate = shardedJdbcTemplate.getTemplate(secondaryShard);

            // Async write to secondary shard
            CompletableFuture.runAsync(() -> {
                try {
                    secondaryTemplate.update(
                        "INSERT INTO users (id, email, name, created_at) VALUES (?, ?, ?, ?)",
                        user.getId(), user.getEmail(), user.getName(), user.getCreatedAt()
                    );
                } catch (Exception e) {
                    // Log error but don't fail the primary write
                    logger.error("Dual write failed for user: " + user.getId(), e);
                }
            });
        }
    }
}

Real World Example: Complete User Service with Spring Boot

// Domain Model
@Data
@AllArgsConstructor
@NoArgsConstructor
public class User {
    private Long id;
    private String email;
    private String name;
    private Timestamp createdAt;
}

// Repository Interface
public interface UserRepository {
    void save(User user);
    Optional<User> findById(Long id);
    List<User> findByEmail(String email);
    boolean update(User user);
    void delete(Long id);
    Map<String, Long> countPerShard();
}

// Sharded Repository Implementation
@Repository
public class ShardedUserRepository implements UserRepository {

    private final ShardedJdbcTemplate shardedJdbcTemplate;
    private final ExecutorService executorService;

    public ShardedUserRepository(ShardedJdbcTemplate shardedJdbcTemplate) {
        this.shardedJdbcTemplate = shardedJdbcTemplate;
        this.executorService = Executors.newFixedThreadPool(10);
    }

    @Override
    public void save(User user) {
        JdbcTemplate template = shardedJdbcTemplate.getTemplate(user.getId());

        template.update(
            "INSERT INTO users (id, email, name, created_at) VALUES (?, ?, ?, ?)",
            user.getId(),
            user.getEmail(),
            user.getName(),
            new Timestamp(System.currentTimeMillis())
        );
    }

    @Override
    public Optional<User> findById(Long id) {
        JdbcTemplate template = shardedJdbcTemplate.getTemplate(id);

        try {
            User user = template.queryForObject(
                "SELECT id, email, name, created_at FROM users WHERE id = ?",
                new Object[]{id},
                (rs, rowNum) -> new User(
                    rs.getLong("id"),
                    rs.getString("email"),
                    rs.getString("name"),
                    rs.getTimestamp("created_at")
                )
            );
            return Optional.ofNullable(user);
        } catch (EmptyResultDataAccessException e) {
            return Optional.empty();
        }
    }

    @Override
    public List<User> findByEmail(String email) {
        Map<String, JdbcTemplate> templates = shardedJdbcTemplate.getAllTemplates();
        List<Future<List<User>>> futures = new ArrayList<>();

        // Query all shards in parallel
        for (JdbcTemplate template : templates.values()) {
            Future<List<User>> future = executorService.submit(() -> 
                template.query(
                    "SELECT id, email, name, created_at FROM users WHERE email = ?",
                    new Object[]{email},
                    (rs, rowNum) -> new User(
                        rs.getLong("id"),
                        rs.getString("email"),
                        rs.getString("name"),
                        rs.getTimestamp("created_at")
                    )
                )
            );
            futures.add(future);
        }

        // Collect results
        List<User> results = new ArrayList<>();
        for (Future<List<User>> future : futures) {
            try {
                results.addAll(future.get(5, TimeUnit.SECONDS));
            } catch (Exception e) {
                throw new RuntimeException("Error in scatter gather query", e);
            }
        }

        return results;
    }

    @Override
    public boolean update(User user) {
        JdbcTemplate template = shardedJdbcTemplate.getTemplate(user.getId());

        int rowsAffected = template.update(
            "UPDATE users SET email = ?, name = ? WHERE id = ?",
            user.getEmail(),
            user.getName(),
            user.getId()
        );

        return rowsAffected > 0;
    }

    @Override
    public void delete(Long id) {
        JdbcTemplate template = shardedJdbcTemplate.getTemplate(id);
        template.update("DELETE FROM users WHERE id = ?", id);
    }

    @Override
    public Map<String, Long> countPerShard() {
        Map<String, JdbcTemplate> templates = shardedJdbcTemplate.getAllTemplates();
        Map<String, Long> counts = new HashMap<>();

        for (Map.Entry<String, JdbcTemplate> entry : templates.entrySet()) {
            Long count = entry.getValue().queryForObject(
                "SELECT COUNT(*) FROM users",
                Long.class
            );
            counts.put(entry.getKey(), count != null ? count : 0L);
        }

        return counts;
    }
}

// Service Layer
@Service
public class UserService {

    private final UserRepository userRepository;

    public UserService(UserRepository userRepository) {
        this.userRepository = userRepository;
    }

    public void createUser(Long id, String email, String name) {
        User user = new User(id, email, name, null);
        userRepository.save(user);
    }

    public User getUser(Long id) {
        return userRepository.findById(id)
            .orElseThrow(() -> new UserNotFoundException("User not found: " + id));
    }

    public List<User> findByEmail(String email) {
        return userRepository.findByEmail(email);
    }

    public void updateUserName(Long id, String newName) {
        User user = getUser(id);
        user.setName(newName);

        if (!userRepository.update(user)) {
            throw new RuntimeException("Failed to update user: " + id);
        }
    }

    public Map<String, Long> getShardDistribution() {
        return userRepository.countPerShard();
    }
}

// REST Controller
@RestController
@RequestMapping("/api/users")
public class UserController {

    private final UserService userService;

    public UserController(UserService userService) {
        this.userService = userService;
    }

    @PostMapping
    public ResponseEntity<Void> createUser(@RequestBody CreateUserRequest request) {
        userService.createUser(request.getId(), request.getEmail(), request.getName());
        return ResponseEntity.status(HttpStatus.CREATED).build();
    }

    @GetMapping("/{id}")
    public ResponseEntity<User> getUser(@PathVariable Long id) {
        User user = userService.getUser(id);
        return ResponseEntity.ok(user);
    }

    @GetMapping("/search")
    public ResponseEntity<List<User>> searchByEmail(@RequestParam String email) {
        List<User> users = userService.findByEmail(email);
        return ResponseEntity.ok(users);
    }

    @PutMapping("/{id}")
    public ResponseEntity<Void> updateUser(
            @PathVariable Long id,
            @RequestBody UpdateUserRequest request) {
        userService.updateUserName(id, request.getName());
        return ResponseEntity.ok().build();
    }

    @GetMapping("/metrics/distribution")
    public ResponseEntity<Map<String, Long>> getDistribution() {
        Map<String, Long> distribution = userService.getShardDistribution();
        return ResponseEntity.ok(distribution);
    }
}

// DTOs
@Data
class CreateUserRequest {
    private Long id;
    private String email;
    private String name;
}

@Data
class UpdateUserRequest {
    private String name;
}

Common Pitfalls and Solutions

Pitfall 1: Wrong Shard Key Choice

Problem: Choosing a low cardinality key or one that creates hotspots.

// Bad: Status field as shard key
int shard = hash(orderStatus) % numShards;
// Most orders are "completed", creating a hot shard

Solution: Choose high cardinality keys that distribute evenly.

// Good: User ID or Order ID
int shard = hash(userId) % numShards;

Pitfall 2: Not Planning for Growth

Problem: Hardcoding number of shards makes it hard to scale.

// Bad: Hardcoded modulo
int shard = hash(userId) % 4;  // What happens when you need 8 shards?

Solution: Use consistent hashing from day one.

Pitfall 3: Connection Pool Exhaustion

Problem: Not properly configuring connection pools per shard.

// Good: Configure Hikari properly per shard
spring.datasource.shard0.hikari.maximum-pool-size=10
spring.datasource.shard0.hikari.minimum-idle=2
spring.datasource.shard0.hikari.connection-timeout=30000
spring.datasource.shard0.hikari.idle-timeout=600000
spring.datasource.shard0.hikari.max-lifetime=1800000

Pitfall 4: Not Handling Shard Failures

Problem: No circuit breaker or retry logic for failing shards.

Solution: Use Resilience4j

@Configuration
public class ResilienceConfig {

    @Bean
    public CircuitBreakerRegistry circuitBreakerRegistry() {
        CircuitBreakerConfig config = CircuitBreakerConfig.custom()
            .failureRateThreshold(50)
            .waitDurationInOpenState(Duration.ofMillis(1000))
            .slidingWindowSize(10)
            .build();

        return CircuitBreakerRegistry.of(config);
    }
}

@Service
public class ResilientUserRepository {

    private final ShardedUserRepository delegate;
    private final CircuitBreakerRegistry circuitBreakerRegistry;

    public Optional<User> findById(Long id) {
        String shardId = resolveShardId(id);
        CircuitBreaker circuitBreaker = circuitBreakerRegistry
            .circuitBreaker("shard_" + shardId);

        return circuitBreaker.executeSupplier(() -> delegate.findById(id));
    }
}

Pitfall 5: No Backup Strategy Per Shard

Problem: Treating all shards as one unit for backups.

Solution: Each shard needs its own backup strategy.

@Service
public class ShardBackupService {

    private final Map<String, DataSource> shardDataSources;
    private final ExecutorService executorService;

    public void backupAllShards() {
        List<Future<String>> futures = new ArrayList<>();

        for (Map.Entry<String, DataSource> entry : shardDataSources.entrySet()) {
            String shardId = entry.getKey();

            Future<String> future = executorService.submit(() -> {
                String timestamp = LocalDateTime.now()
                    .format(DateTimeFormatter.ISO_LOCAL_DATE_TIME);
                String backupFile = String.format("backup_%s_%s.sql", shardId, timestamp);

                // Execute pg_dump or similar
                ProcessBuilder pb = new ProcessBuilder(
                    "pg_dump",
                    "--host=localhost",
                    "--dbname=" + shardId,
                    "--file=" + backupFile
                );
                Process process = pb.start();
                process.waitFor();

                return backupFile;
            });

            futures.add(future);
        }

        // Wait for all backups to complete
        for (Future<String> future : futures) {
            try {
                String backupFile = future.get();
                logger.info("Backup completed: " + backupFile);
            } catch (Exception e) {
                logger.error("Backup failed", e);
            }
        }
    }
}

Conclusion

Database sharding with Spring Boot is a powerful technique for scaling beyond what a single database can handle, but it comes with significant complexity. The key takeaways:

Choose your shard key carefully based on your query patterns
Use consistent hashing to make resharding easier
Leverage Spring Boot's multi datasource capabilities
Design your schema to minimize cross shard queries
Monitor shard balance and performance with Actuator
Have a clear migration and rollback strategy
Accept eventual consistency where possible to avoid distributed transactions
Use connection pooling wisely per shard

Sharding is not something to implement until you need it, but when you do need it, understanding these fundamentals will help you build a scalable, maintainable Spring Boot application.

Remember: the best sharding strategy is one that aligns with your specific access patterns and business requirements. There's no one size fits all solution.

Now go build something that scales!

Forem: Adam - The Developer

Your Caching Strategy Is Not a Strategy

Table of Contents

Caching Is a Contract

The Three Classic Patterns (And When to Use Each)

Build a Cache Client Worth Using

Cache Key Design: More Important Than It Looks

Cache Invalidation: The Part Everyone Skips

TTL vs. Event Driven Invalidation

The Thundering Herd and How to Solve It

Solution 1: Jitter

Solution 2: Stale While Revalidate

Solution 3: Distributed Lock on Cache Miss

Observability: Know What Is Actually Happening

The Decision Checklist Before You Add a Cache

Putting It All Together

Closing

Episode 2: I Was a Junior Developer and I Must Be Stopped

The import() Method, Which Does Not Import

The Rollback That Has Nothing to Roll Back

User ID 1. Just User ID 1. Only Ever User ID 1.

The if Block That Tried So Hard and Was Immediately Betrayed

Six Queries. Same Table. In a Row. Like a Person Who Has Completely Lost the Plot.

The Room Number Is Random. I Need You To Understand That The Room Number Is Random.

Nine Queries Per Guest. Inside The Loop. Per Guest. Nine. Per Guest. Inside The Loop.

$message_index, a Variable That Means $i - 1 and Wants You to Know It

The Ghost of PHP 4, Given the Keys to a Laravel Application

The Final Tally

The Verdict

Episode 1: I Was a Junior Developer and I Must Be Stopped

The Mysterious $r_arr Variable

Sir, This Is an Int

The ?? null Epidemic

The Comment That Explains Nothing

The Round Trip to Nowhere

The Database Loop of Sadness

The Method That Was Right There the Whole Time

The Final Tally

Resume-Driven Development Is Quietly Killing Your Product

What It Actually Looks Like

The Honest Reason This Happens

RDD Is Not the Actual Problem

What It Costs When That Conversation Doesn't Happen

The Boring Stack Works

When New Technology Is Actually the Right Call

The Conversation That Makes the Difference

No Redis. No Kafka. Just Postgres. I Built Chat.

Table of Contents

1. The Bet: "Just Postgres" Is Enough

Why not Redis pub/sub?

2. The Core Mechanism: Triggers + LISTEN/NOTIFY as the Message Bus

3. Schema Decision: Per-Room Sequences for Gapless Ordering

4. Schema Decision: NOTIFY on Insert, With the Full Payload

5. Node Mechanics: A Dedicated LISTEN Client + In-Process Event Routing

6. Reference-Counted Room Subscriptions

7. Exponential Backoff Reconnect for the LISTEN Connection

8. Horizontal Scaling: Stateless Nodes + Nginx least_conn + Postgres Fanout

9. The Unsexy Part: Postgres Settings That Mattered

10. The Detour I Didn't Keep: Sharding LISTEN/NOTIFY

11. The Performance Turning Point: Batched Fanout + One Batch Send per Room per Tick

12. Proof: Load Test Results

13. Tradeoffs (The Part You Should Actually Read)

When Postgres-as-bus makes sense

When you should reach for Kafka (or friends)

14. The Takeaway

Premature Optimization Is Bad, But Your App Is Just Slow Because You're Lazy

What Knuth Actually Said

The Difference Between Optimization and Competence

The Patterns That Are Just Laziness

The N+1 Query

Selecting Everything

Synchronous Work That Does Not Need To Be Synchronous

Rendering Thousands of DOM Nodes Because It Is Easy

Not Caching Things That Never Change

Why This Keeps Happening

The Standard Worth Holding

A Practical Filter

The Unwritten Laws Running Your Code

1. Brooks's Law

2. Conway's Law

The `import()` Method, Which Does Not Import

The `if` Block That Tried So Hard and Was Immediately Betrayed

`$message_index`, a Variable That Means `$i - 1` and Wants You to Know It

The Mysterious `$r_arr` Variable

The `?? null` Epidemic