Forem: rishabh pahwa

Why "No Rollback" Breaks Production

rishabh pahwa — Fri, 15 May 2026 08:44:38 +0000

Most data migration strategies focus on getting to the new state. But your actual success metric isn't "migration complete," it's "can we revert this change without data loss?" A robust rollback mechanism isn't a luxury; it's the only way to guarantee business continuity when migrations inevitably hit a snag.

Why "No Rollback" Breaks Production

Imagine your team deploys a new feature requiring a crucial schema change—say, adding a user_preferences JSONB column with a NOT NULL constraint. You run the migration, deploy the new application code, and for the first 10 minutes, everything looks green. Then, an edge case surfaces: existing users with implicit empty preference data (handled by old app logic) start seeing 500 errors because the new application expects a specific, non-null JSON structure. Revenue instantly drops by 15%, and PagerDuty is screaming.

Without a safe rollback strategy, you're in a nightmare scenario:

Roll forward with a hotfix: Rushing a fix under pressure is a recipe for more bugs, especially if the underlying data is already corrupted or partially transformed.
Restore from backup: This means hours of downtime and guaranteed data loss since the backup was taken. Any new data written in the last few hours is gone.
Manual data repair: An error-prone, slow process for critical data, often involving direct database manipulation, leading to further inconsistency.

All options are unacceptable in a production system handling high traffic or sensitive data.

Designing for Zero-Data-Loss Rollback: The Phased Migration

The core idea for safe rollbacks is to ensure your old system can continue to operate correctly throughout the migration, especially writing data, even as you transition to a new schema or database. This allows you to revert to the old application version without data loss if something breaks.

This typically involves a phased approach often called "dual write" or "shadow write."

           +--------------------+
           |                    |
           |   Application v1   |
           |  (Reads/Writes Old)|
           |                    |
           +----------+---------+
                      |
                      | Reads/Writes (Old Schema)
                      v
            +-------------------+
            |                   |
            |    Old Database   |
            |    (Old Schema)   |
            |                   |
            +-------------------+

Phase 1: Dual Write Introduction (No Read Change)

Your new application version (v2) is deployed alongside v1. Critically, v2 writes to both the old schema and the new schema. Reads continue to come from the old schema by both v1 and v2. This ensures the old path is always kept up-to-date and valid.

           +--------------------+      +--------------------+
           |    Application v1  |      |    Application v2  |
           | (Reads/Writes Old) |      | (Writes Old & New) |
           |                    |      | (Reads Old)        |
           +----------+---------+      +----------+---------+
                      |                             |
                      | Reads/Writes (Old Schema)   | Writes (New Schema)
                      v                             v
            +-------------------+           +-------------------+
            |                   |           |                   |
            |    Old Database   |<----------|    New Database   |
            |    (Old Schema)   |           |    (New Schema)   |
            |                   |           |                   |
            +-------------------+           +-------------------+

Phase 2: Backfill Historical Data

While dual writes ensure new data is captured in both places, existing historical data only lives in the old schema. An asynchronous job is run to backfill and transform this data from the old schema into the new schema. This must be idempotent and carefully handle concurrent writes from Phase 1.

Phase 3: Read Switchover (Still Dual Writing)

Once the backfill is complete and verified, you update Application v2 to read primarily from the new schema. Application v1 continues to read and write to the old schema. Dual writes from v2 continue, ensuring both databases remain synchronized.

           +--------------------+      +--------------------+
           |    Application v1  |      |    Application v2  |
           | (Reads/Writes Old) |      | (Writes Old & New) |
           |                    |      | (Reads New)        |
           +----------+---------+      +----------+---------+
                      |                             |
                      | Reads/Writes (Old Schema)   | Writes (New Schema)
                      v                             v
            +-------------------+           +-------------------+
            |                   |           |                   |
            |    Old Database   |<----------|    New Database   |
            |    (Old Schema)   |           |    (New Schema)   |
            |                   |           |                   |
            +-------------------+           +-------------------+

Rollback Point: If at any point during Phases 1-3 an issue arises, you can instantly rollback Application v2 to Application v1. Since Application v1 was always writing to the old schema, and Application v2 was also writing to it, the critical data for your production system remains intact and consistent in the old schema. The new schema might contain inconsistent or orphaned data, but your core business operations are unaffected.

Phase 4: Cutover and Cleanup

Once confidence is high (e.g., after weeks of monitoring with no issues), you can remove the dual writes from v2 and eventually deprecate/drop the old schema or database.

Real-world Application: Stripe's Data Migrations

Stripe, processing billions of API calls daily, cannot afford data loss or significant downtime. Their approach to critical data migrations (e.g., changing how PaymentIntent objects are stored, or migrating customer data between sharded databases) heavily relies on phased strategies for zero-downtime, zero-data-loss transitions.

When migrating to new data models or infrastructure, Stripe often employs a variation of the dual-write pattern, sometimes extended with a "shadow-read" phase. For instance, if migrating a service to a new database or schema, they might:

Replicate data: Stream existing data from the old system to the new, ensuring eventual consistency.
Dual-write: All new writes go to both the old and new systems. This is critical for rollback: the old system always has the latest state.
Shadow-read/Verify: New application code starts reading from the new system but compares the result with the old system. If there's a discrepancy, it logs an error but serves the response from the old system. This acts as a "dark launch" validation, catching data inconsistencies before they impact users.
Phased Read Cutover: Once shadow-reads are validated (e.g., 99.999% consistency over days), reads are progressively switched to the new system, starting with a small percentage of traffic (canary deployment) and gradually increasing.
Remove Dual-write: Once all traffic is routed to the new system and it's stable, the dual-write logic is removed.
Decommission: The old system is eventually decommissioned.

This process can take weeks or even months for critical systems, providing an extremely long window for verification and instant rollback at any stage before the old system is retired. The overhead of writing twice (or reading twice) is a recognized trade-off for business continuity.

Common Mistakes Engineers Make

Forgetting Data Integrity Constraints: Focusing only on changing column types but neglecting the NOT NULL constraints or unique indexes. If you add NOT NULL to a column that has existing NULL values, your migration will fail unless you've backfilled defaults before applying the constraint. This seems basic, but it's a frequent cause of production failures.
Prematurely Dropping Old Data or Indices: Convinced the migration is "done" after a few hours, engineers drop old columns, tables, or indices. If a hidden bug emerges days later, a rollback becomes a partial data restoration from backup (data loss) or a manual, complex data reconstruction task. Keep old structures around for weeks or months if possible, even if unused, until full confidence is achieved.
Inadequate Monitoring on the Old Path: During dual-write, the focus often shifts entirely to the new path. If the old path's writes (which are critical for rollback) start failing due to unexpected application interactions or database load, and you don't monitor it, your safety net is silently compromised. Monitor both paths comprehensively, especially write success rates and latencies.

Interview Angle

Interviewers love to probe into data migration because it exposes your understanding of trade-offs and production resilience.

Question: "You need to add a new status column (enum type) to a critical orders table that processes thousands of transactions per second. Describe a zero-downtime, zero-data-loss migration strategy and how you'd handle a rollback."

Strong Answer Breakdown:

Phase 1: Safe Schema Evolution. Start by adding the new status column as NULLABLE and with no default. This ensures existing rows remain valid. Deploy this schema change without application code changes.
Phase 2: Dual Write with Backfill.
- Deploy a new version of your application (v2) that, when writing or updating an order, writes to both the old and new status columns. For existing orders, backfill the status column based on existing logic or a reasonable default value using an asynchronous, idempotent job.
- Application v1 continues to operate as normal, reading/writing only the old columns.
- Rollback Safety: At this stage, if v2 has issues, you can roll back to v1. All critical data (including the old status representation) is preserved in the original format. The new status column might become stale or inconsistent, but it doesn't impact v1.
Phase 3: Phased Read Switchover.
- Once backfill is complete and the dual-write period has passed without issues, deploy an updated v2 that reads the status from the new column first. If it's NULL (indicating an un-migrated row or an old version), fall back to inferring status from the old logic. Continue dual-writing.
- Use feature flags to gradually roll out this read change to a small percentage of users, carefully monitoring for errors and data discrepancies.
Phase 4: Enforce Constraint and Cleanup.
- Once confident, add a NOT NULL constraint to the status column.
- Finally, remove the old status logic and column, typically after a significant soak period (weeks).
Key Mitigations and Trade-offs:
- Data Inconsistency: Validate data written to the new column against the old. Use eventual consistency patterns.
- Performance Overhead: Dual writes add latency and database load. Monitor this closely.
- Complexity: More application code paths, more deployment steps. Mitigate with automated testing and clear operational runbooks.
- Rollback: Emphasize that the existence of the old, valid data and the ability for the old application version to function means you can always revert to a known good state without data loss.

Need help designing robust migration strategies or preparing for your next system design interview?

Book a 1:1 session with me on Topmate to discuss your challenges and level up your skills.

Want to Go Deeper?

I do 1:1 sessions on system design, backend architecture, and interview prep.
If you're preparing for a Staff/Senior role or cracking FAANG rounds — book a session here.

The Production Problem with Async Dual Writes

rishabh pahwa — Wed, 13 May 2026 15:00:19 +0000

Many "zero-downtime" data migration strategies involving dual writes promise seamless transitions, but often hide insidious data consistency traps. Without careful handling, you're not just moving data; you're silently corrupting or losing it, only to discover the issue months after cutover.

The Production Problem with Async Dual Writes

Imagine you're an engineer at a rapidly growing SaaS company. Your users table needs to be sharded or migrated to a new database technology. To avoid downtime, you implement a dual-write strategy: all new writes go to both the old and new users tables. Reads initially come from the old table, then eventually switch to the new one. This sounds solid.

Now, picture this: A user updates their profile. Your application sends two write requests: one to OldDB.users and one to NewDB.users. The write to OldDB succeeds, returning HTTP 200. But the write to NewDB fails due to a network timeout, a transient database hiccup, or a schema validation error specific to the new system. What does your application do? If it immediately returns success because the OldDB write worked, you now have an inconsistency: the user's profile is updated in the old system but stale in the new. Over days or weeks, these small, non-atomic failures accumulate, leading to widespread data divergence. When you finally cut over to reading solely from NewDB, users start seeing outdated profiles, missing orders, or incorrect balances. Your "zero-downtime" migration just became a "zero-consistency" disaster.

The Expand-Contract Pattern and Dual Writes

The Expand-Contract pattern is a common strategy for zero-downtime schema migrations. It involves phases:

Expand: Modify your application to read from the old schema and write to both the old and new schemas.
Migrate Data: Backfill historical data from the old schema to the new.
Validate: Continuously compare data between old and new.
Contract: Switch reads to the new schema, then remove the old schema and dual-write logic.

Here's how the dual-write phase typically works, and where consistency issues arise:

                  +-----------------------------------+
                  |            Application            |
                  |  (v1.1 - Dual-Write/Read Old)     |
                  +-----------------------------------+
                       |        ^         ^
                       | Write  | Read    | Write
                       v        |         |
      +---------------------+   |         |   +---------------------+
      | Old Database (v1.0) |<--+---------+-->| New Database (v1.1) |
      | (e.g., MySQL)       |                 | (e.g., PostgreSQL)  |
      +---------------------+                 +---------------------+
                                  ^
                                  | Backfill / Sync Job
                                  | (e.g., Debezium, custom scripts)

In this setup:

Reads: Go to the Old Database (or read from both and merge, with old as authoritative).
Writes: Go to both Old Database and New Database.
Backfill: A separate job continuously copies existing data from Old to New.

The fundamental challenge is that writing to two separate databases (or even two different tables in the same database) is not an atomic operation. Without a distributed transaction across both write operations, there's always a window where one succeeds and the other fails, leading to divergence.

How Stripe Maintains Sanity at Scale

Stripe, processing billions in transactions, performs hundreds of schema changes monthly. Their approach to zero-downtime data migration heavily relies on dual writes but is backed by extensive reconciliation. When migrating critical financial data, they recognize that non-atomic dual writes are a reality.

Instead of assuming perfect consistency, Stripe engineers build systems that detect and fix discrepancies. Their strategy often includes:

Shadow Writes: Before dual-writing, they might "shadow write" to the new schema. The new system receives a copy of write traffic, but these writes aren't considered authoritative and are often discarded. This allows testing the performance and correctness of the new schema under production load without impacting the old system or risking data integrity.
Idempotency and Retries: Application logic ensures that write operations are idempotent, meaning they can be safely retried. When a dual write occurs, if one database write fails, the application logs the failure and often retries later or enqueues it for asynchronous processing.
Continuous Reconciliation: This is the most crucial part. After dual writes are enabled, Stripe runs continuous, automated reconciliation jobs. These jobs scan both the old and new databases, compare records based on a unique identifier, and identify discrepancies. If a difference is found (e.g., a record exists in OldDB but not NewDB, or attributes differ), the reconciliation job logs it, potentially attempts to fix it (e.g., by re-applying the change to NewDB), or flags it for manual review. For example, a reconciliation job might compare 100 million customer records daily, flagging any divergence beyond a 0.0001% threshold. This background process ensures eventual consistency and acts as a safety net against non-atomic dual-write failures.

This rigorous validation and reconciliation process is what turns a risky dual-write strategy into a production-grade, zero-downtime migration.

Common Mistakes When Implementing Dual Writes

Assuming Atomicity Across Databases: Many engineers treat a dual-write operation (e.g., db1.save() and db2.save()) as a single atomic unit. It's not. If your application code just calls two database clients, success from one and failure from the other leads to data divergence. You need explicit error handling, retries, and compensation logic, or rely on eventual consistency with strong reconciliation.
Inadequate Read Strategy During Transition: During the dual-write phase, how do you read?
- Read-Old: Reading only from the old system is safer for consistency during the transition, but means data written to the new system isn't immediately visible, and requires a hard cutover for reads.
- Read-New-Fallback-Old: Reading from the new, falling back to old if not found, can lead to inconsistencies if the new system is incomplete or subtly different.
- Read-Both-Merge: Reading from both and merging requires complex conflict resolution and can be slow. Most get this wrong by not clearly defining the source of truth for reads at each stage.
Neglecting Reconciliation and Observability: Simply setting up dual writes and a backfill job isn't enough. Without robust monitoring to track dual-write success rates, latency for each write, and, critically, continuous data validation (reconciliation) between the old and new systems, you're flying blind. Silent data loss is guaranteed without it. Many engineers skip this crucial, complex step, leading to post-cutover data integrity nightmares.

Interview Angle: What Interviewers Ask

Interviewers will probe your understanding beyond the basic concept. Expect questions like:

"How do you ensure data consistency during a dual-write phase if one database write succeeds and the other fails?"
- Strong Answer: "Since distributed transactions are rarely feasible or desirable, I wouldn't assume atomicity. Instead, I'd implement a compensation mechanism. For writes, I'd typically wrap the dual-write logic in a transaction within the application or use an idempotent message queue. The application would first publish the data change to a reliable queue (e.g., Kafka). A consumer would then attempt to write to both databases. If one write fails, the message could be retried with backoff. If persistent failures occur, it lands in a dead-letter queue for manual intervention or triggers an alert. Ultimately, even with retries, you need a continuous, asynchronous reconciliation job that scans both databases for discrepancies and fixes them, ensuring eventual consistency. This shifts the complexity from transactional guarantees to robust error handling and eventual repair."
"When would you use a 'shadow write' versus a 'dual write'?"
- Strong Answer: "Shadow writes are primarily for testing the new system with production-like load and data, without letting it impact the live system. You write to both the old authoritative system and the new system, but the new system's writes are often ignored or merely logged for validation. This is low-risk. Dual writes, however, mean both systems are authoritative for writes during a transitional period, with the intent to eventually cut over reads to the new system. It's a higher-risk strategy because data consistency is paramount. I'd use shadow writes for initial performance testing or schema validation of the new system, and dual writes when I'm confident in the new system's write path and am preparing for a full cutover, backed by strong reconciliation."

Moving critical data without disruption is hard. Do it right, and your systems evolve gracefully. Cut corners, and you'll spend weeks on data recovery.

Need to refine your system design skills for your next interview? Book a 1:1 session with me to discuss real-world system challenges and effective design patterns.

Want to Go Deeper?

I do 1:1 sessions on system design, backend architecture, and interview prep.
If you're preparing for a Staff/Senior role or cracking FAANG rounds — book a session here.

Your "Cache Invalidation is Hard" Answer Misses the Real Horror

rishabh pahwa — Sun, 10 May 2026 08:42:41 +0000

Your "Cache Invalidation is Hard" Answer Misses the Real Horror

Most engineers parrot "cache invalidation is hard" as a standard interview response, but few understand why it's hard or the real-world horrors it introduces. It's not just about stale data; it's about financial losses, broken business logic, and cascading failures when eventual consistency hits critical paths.

The Production Nightmare: Financial Impact of Stale Data

Imagine a ride-sharing platform like Uber. A user updates their payment method because the old card expired. The update is written to the database successfully. However, due to an aggressive cache TTL or a failed invalidation, the dispatch service still sees the old, expired card for the next 5 minutes. The user tries to book a ride, it fails. They try again, it fails. Frustrated, they switch to a competitor.

This isn't just "stale data"; it's a direct loss of revenue, a degraded user experience, and a hit to brand loyalty. In banking, showing an incorrect account balance, even for seconds, can trigger compliance violations and massive reputational damage. In e-commerce, a product showing "in stock" when it's sold out leads to cancelled orders and angry customers. The problem isn't theoretical; it's financial and operational.

Beyond TTLs: Active Invalidation in Distributed Systems

The naive approach to cache invalidation often relies on Time-To-Live (TTL) or a simple write-through/write-around policy. While these have their place, critical systems demand more robust strategies that aim for stronger consistency than basic eventual consistency can provide, especially when data is updated from multiple sources.

Consider an active invalidation strategy:

+------------+       +------------+       +------------+       +-------------+
|    User    |       |  Frontend  |       |  Backend   |       |   Database  |
| (API Client)|       |    Service |       |    Service |       |  (Postgres) |
+------------+       +------------+       +------------+       +-------------+
      |                   |                      |                      |
      | 1. Update Profile |                      |                      |
      +------------------>|                      |                      |
      |                   | 2. Call Update API   |                      |
      |                   +--------------------->|                      |
      |                   |                      | 3. Update DB         |
      |                   |                      +--------------------->|
      |                   |                      | (DB transaction ACK) |
      |                   |                      |<---------------------+
      |                   |                      |                      |
      |                   |                      | 4. Publish Invalidation Event to Message Bus
      |                   |                      +--------------------->+
      |                   |                      | (e.g., Kafka)        |
      |                   |                      |                      |
      |                   |                      |                      |
      |                   |                      |                      |
      |                   |                      |                      |
      |                   |                      |                      |
      |                   |                      |                      |
+------------+       +------------+       +------------+       +-------------+
|  Cache     |       | Invalidator|       |  Message   |
| (Redis)    |       |  Service   |       |    Bus     |
+------------+       +------------+       +------------+
      ^                   ^                      ^
      |                   | 5. Consume Invalidation Event
      |                   |<---------------------+
      |                   |                      |
      | 6. Invalidate Key |                      |
      |<------------------+                      |
      | (Cache ACK)       |                      |
      |                   |                      |

In this flow, after the database is updated (step 3), an invalidation event is published to a message bus (step 4). An Invalidator Service consumes this event (step 5) and then explicitly deletes or updates the corresponding key in the cache (step 6). This decouples the write path from cache invalidation, improving write latency, but introduces eventual consistency. The critical aspect is making this event propagation and consumption reliable and fast.

Meta's Approach to Consistent Caching at Scale

At companies like Meta (Facebook), operating some of the world's largest caches, simple TTLs aren't enough. They can't afford to show stale profile data, friend lists, or post engagement for minutes. Their "Cache Made Consistent" initiatives aim to solve the very race conditions and inconsistencies that plague distributed caching.

They've moved beyond basic invalidation to sophisticated systems that ensure stronger consistency guarantees. One approach involves using transaction logs (like binlogs in MySQL) from the database to drive invalidation. A service tails these logs, filters relevant updates, and publishes specific invalidation messages to a distributed system. Cache nodes then subscribe to these messages. This pushes the consistency window from minutes (TTL) down to milliseconds, closely following database writes.

This system is built for extreme scale: potentially hundreds of thousands of updates per second across petabytes of data. It's not just about sending an invalidate(key) command; it's about guaranteeing delivery, handling partial failures (what if a cache node is down?), and ensuring that all relevant dependent caches (e.g., user profile, friend count, feed items) are consistently updated or invalidated.

Common Mistakes Engineers Make

Over-relying on TTL for critical data: While great for performance, a 5-minute TTL on a user's payment method or an item's stock count is a ticking time bomb. It trades consistency for availability in places where consistency is paramount. For high-stakes data, TTLs should be very short (seconds) and coupled with active invalidation, or the cache should be bypassed entirely for reads requiring strong consistency.
Ignoring cache dependency graphs: Invalidating a single key like user:123 is often insufficient. What about other cached entities that depend on user:123's data, such as user_profile_page:123 or feed_for_user:123? If you don't invalidate the entire dependency tree, you'll still show stale data. Building and maintaining this dependency graph is complex and often overlooked until production issues arise.
Not building resilient invalidation pipelines: Active invalidation introduces its own distributed system problems. What happens if the message bus is down? What if an invalidation message is lost? What if a cache node fails to receive an invalidation? Without retries, dead-letter queues, and eventual reconciliation mechanisms, your cache will drift indefinitely. This is where cache invalidation is hard actually holds true – building a reliable invalidation mechanism.

The Interview Angle: Beyond the Buzzwords

When an interviewer asks about cache invalidation, they're looking for more than "it's hard, use TTL." They want to understand your appreciation for:

Consistency models and trade-offs: When would you tolerate eventual consistency? When do you need strong consistency, and how would you achieve it with a cache? (e.g., using a write-through cache with a transactional database, or bypassing the cache for critical reads).
Failure modes: What happens if invalidation fails? How do you detect it? How do you recover? Strong answers discuss monitoring cache hit ratios, consistency checks between cache and DB, and fallback mechanisms like circuit breakers.
Complexity at scale: How do you invalidate data across hundreds or thousands of cache nodes? How do you handle fan-out invalidation for dependent data? Think about event-driven architectures, distributed transactions (though rare for caches), and sophisticated messaging patterns.

For instance, if asked, "How would you design a caching system for a bank account balance?", a strong answer would emphasize strong consistency. You might propose a very short TTL (e.g., 1 second) coupled with immediate, transactional invalidation for updates, or even suggest not caching the balance at all for reads that require absolute accuracy, fetching directly from the database to avoid any risk of stale data. The cost of an inconsistent balance outweighs the latency benefit of a cache.

Need to level up your system design skills?

Book a 1:1 session with me to deep dive into real-world system challenges and ace your next interview. Let's build your expertise together.

Want to Go Deeper?

I do 1:1 sessions on system design, backend architecture, and interview prep.
If you're preparing for a Staff/Senior role or cracking FAANG rounds — book a session here.