Forem: HarshKumar Jha

From Weeks to 15 Minutes: How We Built a Data Migration System That Changed Everything - Part 2

HarshKumar Jha — Mon, 22 Dec 2025 03:30:00 +0000

Part 2: The Architecture That Actually Works (Not Just Survives)

The Problem We Had (And You Probably Have Too)

Remember when I said our system crashed at 200K records?

That wasn't a random failure. It was architecture screaming for help.

Our original code looked "fine" on paper. But when you feed it a million records, it doesn't just slow down—it breaks in weird ways.

Database connections maxed out (too many tiny queries)
Memory kept growing (loading everything at once)
CPU pegged at 100% (no async processing)
Redis filled up (no compression)
No way to recover (crashes meant starting over)
No monitoring (couldn't see what was breaking)

We'd look at the logs and think: "Which part is actually broken?"

Spoiler: It wasn't the code. It was the thinking behind the code.

We were solving yesterday's problems (50K records) with yesterday's architecture.

The Breakthrough: Stop Thinking Sequential

Here's the thing nobody tells you:

When you scale 10x, you can't just optimize code. You have to rethink architecture.

Optimization gets you 10-20% improvement. Architecture gets you 100x.

We realized: Our system wasn't designed for scale. It was designed for convenience.

So we asked five questions:

What if we never loaded all data into memory at once?
What if we queried the database only once, not a million times?
What if we processed everything asynchronously, not blocking the user?
What if the system could recover from any failure automatically?
What if we could see exactly what's happening in real-time?

The answer to all five: A completely different system.

The Five-Layer Architecture (Simplified)

Imagine moving houses. You don't carry everything at once, right?

Bad way: Load all boxes, drive to new house, unload all at once.

Your truck breaks (too heavy)
You get tired (single person)
New house overwhelmed (can't fit everything)
If truck breaks, you've lost everything
No idea how much is left to move

Good way: Load smart, drive with helpers, unload strategically, track everything.

Small trips per person
Multiple helpers work in parallel
New house gets organized flow
If one trip fails, others continue
Always know: "23 of 50 boxes done"

Our system works the same way:

┌─────────────────────────────────────────┐
│         USER UPLOADS BIG FILE           │
└──────────────────┬──────────────────────┘
                   │
        ┌──────────┴──────────┐
        │                     │
   ┌────▼────┐          ┌────▼─────┐
   │  AWS S3 │          │PostgreSQL│
   │ (Inbox) │          │(Metadata)│
   └────┬────┘          └────┬─────┘
        │                    │
        └─────────┬──────────┘
                  │
         ┌────────▼────────┐
         │     Redis       │
         │ (Staging Area)  │
         │  - Raw data     │
         │  - Transformed  │
         │  - Checkpoints  │ ← NEW: Recovery points
         └────────┬────────┘
                  │
         ┌────────▼────────┐
         │     Celery      │
         │   (Workers)     │
         │  - 5-8 at once  │
         │  - Auto-scale   │ ← NEW: Grows/shrinks
         └────────┬────────┘
                  │
         ┌────────▼────────┐
         │ Monitoring      │ ← NEW: Real-time dashboard
         │ - Progress      │
         │ - Alerts        │
         │ - Health        │
         └────────┬────────┘
                  │
         ┌────────▼────────┐
         │  Live Database  │
         │  (Final Dest)   │
         └─────────────────┘

Why this works:

S3: Infinitely scalable (file sits there, doesn't hurt)
PostgreSQL: Stores only metadata (tiny, never gets hammered)
Redis: Fast temp storage + checkpoints for recovery
Celery: Parallel workers (8 people working, not 1)
Monitoring: See everything, catch problems early
Live DB: Only gets clean, validated data

Each layer does ONE thing. Does it well.

Layer 1: The S3 Inbox (Never Crashes)

When someone uploads a 184MB file, we don't process it immediately.

We put it in S3. Done.

# That's literally it
file.save_to_s3(bucket='migrations', timeout=1_hour)

Why this matters:

Your server isn't tied up. If the file is 1MB or 1GB, S3 handles it. You just point and move on.

Compare this to:

# Old way (BAD)
df = pd.read_csv(file)  # File must fit in memory. Dies at 184MB on small server.

Layer 2: The Intelligence Layer (One Query, Infinite Uses)

This is where most teams fail.

The typical mistake:

for row in million_rows:
    # Check if consumer exists
    consumer = Consumer.objects.get(consumer_no=row['consumer_no'])

What happens:

Row 1: Query database (50ms)
Row 2: Query database (50ms)
...
Row 1,000,000: Query database (50ms)

Total time: 1,000,000 × 50ms = 13.8 hours

And your database dies at row 50,000 anyway.

We do it differently:

class SmartCache:
    """
    Load valid values ONCE. Reuse forever.
    """

    def __init__(self):
        self.valid_consumers = None  # Will cache here
        self.valid_meters = None
        self.valid_plans = None
        self.valid_territories = None

    def preload_all_relationships_once(self):
        """
        Query database for ALL relationships upfront
        This is the magic that prevents N+1 queries
        """
        print("🔄 Preloading all relationships...")

        # Query 1: Load ALL consumers (one query)
        self.valid_consumers = set(
            Consumer.objects.values_list('consumer_no', flat=True)
        )
        print(f"✅ Loaded {len(self.valid_consumers)} consumers")

        # Query 2: Load ALL meters (one query)
        self.valid_meters = set(
            Meter.objects.values_list('meter_number', flat=True)
        )
        print(f"✅ Loaded {len(self.valid_meters)} meters")

        # Query 3: Load ALL plans (one query)
        self.valid_plans = {
            plan.name: plan.id
            for plan in Plan.objects.all()
        }
        print(f"✅ Loaded {len(self.valid_plans)} plans")

        # Query 4: Load ALL territories (one query)
        self.valid_territories = {
            t.code: t.id
            for t in Territory.objects.all()
        }
        print(f"✅ Loaded {len(self.valid_territories)} territories")

        print("✅ Preloading complete. Ready to validate 1M rows.")

    def is_consumer_valid(self, consumer_no):
        """
        Check instantly (zero database queries)
        O(1) lookup in set
        """
        return consumer_no in self.valid_consumers

    def is_meter_valid(self, meter_number):
        """O(1) lookup"""
        return meter_number in self.valid_meters

    def validate_all_relationships(self, rows):
        """
        Validate 1 million rows with ZERO database queries
        """
        errors = []

        for idx, row in enumerate(rows):
            # Check consumer exists (in-memory, instant)
            if not self.is_consumer_valid(row['consumer_no']):
                errors.append({
                    "row": idx + 1,
                    "field": "consumer_no",
                    "value": row['consumer_no'],
                    "error": "Consumer not found in system"
                })

            # Check meter exists (in-memory, instant)
            if not self.is_meter_valid(row['meter_number']):
                errors.append({
                    "row": idx + 1,
                    "field": "meter_number",
                    "value": row['meter_number'],
                    "error": "Meter not found in system"
                })

        return errors

What happens now:

Load all relationships: 4 queries (5 seconds total)
Validate 1,000,000 rows: Zero queries (12 seconds, all in-memory)

Impact:

Before: 1,000,000 queries × 50ms = 13.8 hours + database crash
After: 4 queries + 1M in-memory checks = 17 seconds total
Improvement: 2,929x faster

The key insight: Your database is not a lookup service. It's a data store.

Query once. Validate a million times in memory.

Layer 3: Async Processing (The Game Changer)

Here's a question: When someone uploads a file, do they want to wait 25 minutes?

No.

So why would you make them?

# Old way (USER BLOCKED)
def migrate_data(file):
    raw_data = load_csv(file)
    transformed = transform_all(raw_data)  # 5 mins, user waits
    validated = validate_all(transformed)  # 5 mins, user waits
    final = insert_to_db(validated)        # 10 mins, user waits

    return success  # Finally!

Better way (USER NOT BLOCKED):

def migrate_data(file):
    """
    User uploads file. Immediately gets response.
    Processing happens in background.
    """
    # Store file path
    job = create_migration_job(file)

    # Send to background queue (Celery)
    process_migration_async.delay(job.id)

    # User immediately gets: "Processing started!"
    return {"status": "queued", "job_id": job.id}

# Meanwhile, in background...
@celery_app.task
def process_migration_async(job_id):
    """
    This runs on a worker, not the web server
    """
    raw_data = load_csv(job.file)
    transformed = transform_all(raw_data)
    validated = validate_all(transformed)
    final = insert_to_db(validated)

    # Email user when done: "Your data is ready!"

User experience:

User: "I'll upload this"
System: "Got it! Check back in 5 mins"
User: Goes back to work, gets email when ready
Everyone happy

System benefit:

Web server free to handle other requests
Processing happens on separate workers
Can scale workers independently
If one worker crashes, others keep going

The Magic: Celery Workers (Multiple Hands)

Imagine you have 1 person processing 1 million boxes.

They work 24 hours straight. Still takes 2 weeks.

Now imagine 8 people working together.

Same amount of work. 8x faster.

That's Celery.

# Start 5 workers (each can process a chunk)
celery -A myapp worker --concurrency=5

# Each worker processes independently
Worker 1: Processing rows 1-10,000
Worker 2: Processing rows 10,001-20,000
Worker 3: Processing rows 20,001-30,000
Worker 4: Processing rows 30,001-40,000
Worker 5: Processing rows 40,001-50,000

# All in parallel

What makes it work:

# The worker knows exactly what to do
@app.task(bind=True, max_retries=3)
def process_chunk(self, chunk_index):
    """
    One worker, one chunk at a time
    Automatically retries on failure
    """
    try:
        # Get chunk from cache (already there)
        chunk = redis.get(f"chunk:{chunk_index}")

        # Transform it
        transformed = [transform_row(row) for row in chunk]

        # Validate it
        validated = [v for v in transformed if is_valid(v)]

        # Save it
        db.bulk_insert(validated)

        # Done! Next worker takes next chunk
        return {"rows_processed": len(validated)}

    except Exception as e:
        # Retry with exponential backoff
        raise self.retry(
            exc=e,
            countdown=2 ** self.request.retries  # 2s, 4s, 8s
        )

Real numbers:

1 worker: 1,000,000 rows in 40 minutes
8 workers: 1,000,000 rows in 5 minutes

That's not a typo. 8x faster, roughly.

The Secret Nobody Talks About: Dynamic Chunk Sizing

Here's where it gets smart.

You don't use fixed chunk sizes. That's dumb.

Why?

Because:

Small server (2GB RAM) + big chunks = CRASH
Large server (32GB RAM) + small chunks = WASTE
High CPU load + big chunks = OVERHEAT

We calculate chunk size on the fly:

import psutil

class AdaptiveChunker:
    """
    Dynamically adjust chunk size based on system health
    This is what prevents crashes at scale
    """

    def __init__(self):
        self.base_chunk_size = 5000
        self.min_chunk_size = 1000
        self.max_chunk_size = 10000

    def calculate_how_much_i_can_handle(self):
        """
        Ask system: "How much work can you do right now?"
        """
        memory = psutil.virtual_memory()
        cpu = psutil.cpu_percent(interval=0.1)

        # Get current metrics
        memory_percent = memory.percent
        available_memory_gb = memory.available / (1024**3)

        # Start with base size
        chunk_size = self.base_chunk_size

        # Rule 1: Low memory? Reduce chunk size
        if memory_percent > 80:  # Over 80% memory used
            chunk_size = self.min_chunk_size
            logger.warning(
                f"⚠️ Memory high ({memory_percent}%), "
                f"reducing chunk to {chunk_size} rows"
            )

        elif memory_percent > 60:  # Over 60% memory used
            chunk_size = int(self.base_chunk_size * 0.6)
            logger.info(f"Memory at {memory_percent}%, chunk={chunk_size}")

        # Rule 2: High CPU? Give it time to cool down
        if cpu > 85:  # CPU pegged
            chunk_size = min(chunk_size, 2000)
            logger.warning(
                f"⚠️ CPU high ({cpu}%), "
                f"reducing chunk to {chunk_size} rows"
            )

        # Rule 3: System healthy? Maximize throughput
        elif memory_percent < 40 and cpu < 50:
            chunk_size = self.max_chunk_size
            logger.info(f"✅ System healthy, chunk={chunk_size}")

        # Ensure within bounds
        return max(
            self.min_chunk_size,
            min(chunk_size, self.max_chunk_size)
        )

    def process_intelligently(self, session_id, total_rows):
        """
        Process with dynamic chunking
        """
        processed_rows = 0

        while processed_rows < total_rows:
            # Calculate optimal chunk size EVERY iteration
            chunk_size = self.calculate_how_much_i_can_handle()

            # Process this chunk
            chunk = get_chunk(session_id, processed_rows, chunk_size)
            result = process_chunk(chunk)

            processed_rows += len(chunk)

            # Log performance
            logger.info(
                f"✅ Processed {processed_rows}/{total_rows} "
                f"(chunk_size={chunk_size})"
            )

What happens:

Small server struggles:
- Chunk 1: Memory 40%, CPU 45% → use 8,000 rows ✅
- Chunk 2: Memory 75%, CPU 60% → use 3,000 rows ✅
- Chunk 3: Memory 85%, CPU 80% → use 1,000 rows (careful!) ✅
- Chunk 4: Memory 60%, CPU 55% → use 5,000 rows ✅

Never crashes. Just adjusts on the fly.

Large server thrives:
- All chunks: Memory 20%, CPU 30% → use 10,000 rows ✅
- Finishes 3x faster than small server
- Both complete successfully

Layer 4: The Checkpoint System (Never Lose Progress)

Here's a nightmare scenario:

You're migrating 2 million records.

At row 1,847,392, your network fails.

Old system: Start from scratch. Waste 40 minutes.

Our system: Resume from row 1,840,000. Continue in 2 minutes.

How?

class CheckpointManager:
    """
    Save progress every 10,000 rows
    Resume from exact failure point
    """

    def __init__(self, redis_client):
        self.redis = redis_client
        self.checkpoint_interval = 10000  # Save every 10K rows

    def should_create_checkpoint(self, processed_rows):
        """
        Should we save progress now?
        """
        return processed_rows % self.checkpoint_interval == 0

    def create_checkpoint(self, session_id, processed_rows, state):
        """
        Save current progress
        """
        checkpoint = {
            "session_id": session_id,
            "processed_rows": processed_rows,
            "timestamp": datetime.utcnow().isoformat(),
            "state": state,
            "can_resume": True,
            "chunk_size_at_checkpoint": state.get('current_chunk_size'),
            "workers_active": state.get('active_workers'),
        }

        key = f"session:{session_id}:checkpoint"

        # Store in Redis with 2-hour expiry
        self.redis.setex(
            key,
            7200,  # 2 hours
            json.dumps(checkpoint)
        )

        logger.info(
            f"✅ Checkpoint saved: {processed_rows:,} rows completed"
        )

    def get_last_checkpoint(self, session_id):
        """
        Get last saved checkpoint
        """
        key = f"session:{session_id}:checkpoint"
        checkpoint_data = self.redis.get(key)

        if not checkpoint_data:
            return None

        return json.loads(checkpoint_data)

    def resume_from_checkpoint(self, session_id):
        """
        Resume migration from last successful checkpoint
        """
        checkpoint = self.get_last_checkpoint(session_id)

        if not checkpoint:
            logger.info("No checkpoint found, starting from beginning")
            return 0

        processed_rows = checkpoint['processed_rows']

        logger.info(
            f"🔄 Resuming from checkpoint: "
            f"row {processed_rows:,} "
            f"(saved at {checkpoint['timestamp']})"
        )

        return processed_rows

# Usage in migration
def process_with_checkpoints(session_id, total_rows):
    """
    Process with automatic checkpoints
    """
    checkpoint_mgr = CheckpointManager(redis_client)

    # Try to resume from checkpoint
    start_row = checkpoint_mgr.resume_from_checkpoint(session_id)

    processed_rows = start_row

    while processed_rows < total_rows:
        # Process chunk
        chunk = get_chunk(session_id, processed_rows, 5000)
        process_chunk(chunk)

        processed_rows += len(chunk)

        # Save checkpoint every 10K rows
        if checkpoint_mgr.should_create_checkpoint(processed_rows):
            state = {
                'current_chunk_size': len(chunk),
                'active_workers': get_active_worker_count(),
            }
            checkpoint_mgr.create_checkpoint(
                session_id,
                processed_rows,
                state
            )

Real-World Scenario:

Migration starts: 2,000,000 rows to process

10,000 rows ✅ → Checkpoint saved
20,000 rows ✅ → Checkpoint saved
...
800,000 rows ✅ → Checkpoint saved
810,000 rows ✅ → Checkpoint saved

💥 NETWORK FAILURE at row 847,392

System detects failure, checks checkpoint:
  Last checkpoint: 840,000 rows
  Rows lost: 7,392 (less than 1 chunk)

System resumes:
  Starting from: 840,000
  Remaining: 1,160,000 rows
  Time to complete: 7 minutes

VS without checkpoints:
  Starting from: 0
  Remaining: 2,000,000 rows
  Time to complete: 42 minutes wasted

The economics:

Without checkpoints:
- Network blip at 90% complete = Start over
- Power outage at row 1.8M = Start over
- Worker crash at any point = Start over

With checkpoints:
- Network blip = Resume in seconds
- Power outage = Resume from last 10K
- Worker crash = Another worker picks up

Layer 5: Real-Time Monitoring (See Everything)

You can't fix what you can't see.

The Dashboard Users See:

class MigrationMetrics:
    """
    Track everything in real-time
    Users see this dashboard while migration runs
    """

    def __init__(self, session_id):
        self.session_id = session_id
        self.redis = get_redis_connection()

    def track_progress(self):
        """
        Calculate and cache real-time metrics
        """
        session = MigrationSession.objects.get(id=self.session_id)

        # Calculate all metrics
        elapsed_time = (datetime.utcnow() - session.started_at).total_seconds()

        metrics = {
            "session_id": self.session_id,
            "status": session.status,

            # Progress metrics
            "total_rows": session.total_rows,
            "processed_rows": session.processed_rows,
            "progress_percent": round(
                (session.processed_rows / session.total_rows) * 100, 2
            ),

            # Performance metrics
            "elapsed_time": elapsed_time,
            "elapsed_formatted": self._format_duration(elapsed_time),
            "rows_per_second": self._calculate_throughput(session),
            "estimated_time_remaining": self._calculate_eta(session),


            # Quality metrics
            "validation_errors": session.error_count,
            "success_rate": round(
                ((session.processed_rows - session.error_count) / 
                 session.processed_rows * 100) if session.processed_rows > 0 else 0,
                2
            ),

            # Last activity
            "last_activity": session.last_activity.isoformat(),
            "last_checkpoint": session.last_checkpoint_at,
        }

        # Cache for dashboard (1 minute TTL)
        self.redis.setex(
            f"session:{self.session_id}:metrics",
            60,
            json.dumps(metrics)
        )

        return metrics

    def _calculate_throughput(self, session):
        """Calculate current processing speed"""
        if session.processed_rows == 0:
            return 0

        elapsed = (datetime.utcnow() - session.started_at).total_seconds()
        return round(session.processed_rows / elapsed) if elapsed > 0 else 0

    def _calculate_eta(self, session):
        """Estimate time remaining"""
        throughput = self._calculate_throughput(session)

        if throughput == 0:
            return None

        remaining_rows = session.total_rows - session.processed_rows
        seconds_remaining = remaining_rows / throughput

        return {
            "seconds": int(seconds_remaining),
            "formatted": self._format_duration(seconds_remaining)
        }

    def _format_duration(self, seconds):
        """Human-readable duration"""
        if seconds < 60:
            return f"{int(seconds)}s"
        elif seconds < 3600:
            minutes = int(seconds / 60)
            secs = int(seconds % 60)
            return f"{minutes}m {secs}s"
        else:
            hours = int(seconds / 3600)
            minutes = int((seconds % 3600) / 60)
            return f"{hours}h {minutes}m"

    def _get_active_worker_count(self):
        """Count active Celery workers"""
        from celery import current_app
        inspect = current_app.control.inspect()
        stats = inspect.stats()
        return len(stats) if stats else 0

What the user sees (live updating dashboard):

┌─────────────────────────────────────────────────────────────┐
│  MIGRATION SESSION: abc-123-def                             │
│  Data Type: Consumers                                       │
│  Status: Processing ●                                       │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────┬───────────────────────────────────────┐
│  Progress           │  Performance                          │
├─────────────────────┼───────────────────────────────────────┤
│  1,247,892          │  Throughput: 2,847 rows/sec           │
│  ████████████░░░░░  │  Elapsed: 7m 18s                      │
│  62.4% complete     │  ETA: 4m 42s                          │
│                     │                                       │
│  Errors: 31,247     │  Active Workers: 8                    │
│  Success: 98.5%     │  Memory: 156MB / 8GB (2%)             │ 
└─────────────────────┴───────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│  Recent Activity (Live)                                     │
├─────────────────────────────────────────────────────────────┤
│  ✅ Chunk 247 complete: 5,000 rows (1.2s)                   │
│  ✅ Chunk 248 complete: 5,000 rows (1.1s)                   │
│  💾 Checkpoint saved at 1,250,000 rows                      │
│  ✅ Chunk 249 complete: 5,000 rows (1.3s)                   │
│  ⚠️  Validation error: Row 1,247,892 (invalid email)        │
│  ✅ Chunk 250 complete: 5,000 rows (1.2s)                   │
└─────────────────────────────────────────────────────────────┘

Why this matters:

User confidence - "I can see it's actually working"
Problem detection - "Why did throughput drop?"
Resource awareness - "Should I scale up workers?"
Time planning - "I'll grab coffee, 5 mins left"

Layer 7: Three-Layer Validation (Catch Errors Early)

Most systems validate one way. We validate three.

Layer 1: Does the file make sense?

required_fields = ['consumer_no', 'first_name', 'email', 'phone']

if not all(field in csv_headers for field in required_fields):
    return "Missing required columns. Upload failed."

Layer 2: Are the values correct?

# Check email format
if not is_valid_email(row['email']):
    errors.append("Row 5: Invalid email")

# Check phone is numeric
if not is_numeric(row['phone']):
    errors.append("Row 12: Phone must be numbers")

Layer 3: Do relationships make sense?

# Does this consumer actually exist?
if row['consumer_no'] not in valid_consumers:
    errors.append("Row 89: Consumer not found in system")

# Is this meter already assigned?
if meter_already_mapped(row['meter_no']):
    errors.append("Row 145: Meter already assigned to another consumer")

Why three layers?

Each layer catches different mistakes.

Layer 1: Catches file corruption
Layer 2: Catches bad data
Layer 3: Catches business logic violations

Together? They catch 98% of problems before they touch production.

One More Thing: Redis Compression (The Underrated Hero)

We store 1 million records in Redis temporarily.

Uncompressed? 850MB. Your Redis crashes.

Compressed? 127MB. Still plenty of space.

import zlib
import pickle

def compress_data(data):
    """
    Compress before storing in Redis
    6.7:1 compression ratio
    """
    serialized = pickle.dumps(data)
    compressed = zlib.compress(serialized, level=6)
    return compressed

def decompress_data(compressed):
    """
    Decompress when retrieving
    Takes 400ms for 5K rows (negligible)
    """
    serialized = zlib.decompress(compressed)
    data = pickle.loads(serialized)
    return data

# Usage
cache.set(f"chunk:{i}", compress_data(chunk_data))
chunk = decompress_data(cache.get(f"chunk:{i}"))

The math:

Compression ratio: 6.7:1
Memory saved: 85%
Decompression time: 400ms per 5K chunk (negligible)
Cost saved: Huge (16GB Redis → 4GB Redis)

It's one of those "boring" optimizations that's actually genius.

Real Performance: What Does This Actually Achieve?

We tested with real data. Real numbers.

The Setup:

1,000,000 consumer records
25 fields per record
Real-world messy data
Standard server (8GB RAM, 4 cores)

The Results:

STAGE                           TIME
─────────────────────────────────────
File upload to S3               1m 30s
Load to Redis (with compress)   1m 45s
Preload relationships (4 queries) 5s
Fuzzy mapping (find columns)    18s
Transform all rows              3m 20s
Validate 3 layers               4m 50s
Insert to database (8 workers)  8m 40s
─────────────────────────────────────
TOTAL                          20m 28s

For 1 million records. On a standard server. Zero crashes.

With:

Real-time monitoring dashboard
Automatic checkpoints every 10K rows
Smart alerts catching problems
Auto-scaling workers (5 → 8 → 5)
Adaptive chunk sizing (1K - 10K)
Zero database overload (4 queries total)

Before our changes? Would crash around 200K records.

Why This Beats Everything Else

Let's be honest. You could use a migration tool from the market.

But here's what you get with this architecture:

Works at any scale (1K to 10M records)
Auto-adjusts (chunk size, workers, everything)
Never crashes (validation catches errors first)
Fast feedback (user not blocked waiting)
Easy to debug (real-time dashboard shows everything)
Recovers from failures (checkpoints every 10K rows)
Catches problems early (smart alerts)
Costs almost nothing (runs on existing infrastructure)

The migration tools? They work. But they're generic.

This? This is built for your data. Your system. Your scale.

The Moment It All Clicked

We had a customer with 2 million records.

Old system: "This will take 2 weeks and probably crash."

New system: "25 minutes, guaranteed, no downtime."

They were skeptical. We showed them the architecture.

More importantly, we showed them the dashboard:

They said: "Wait, I can actually SEE what's happening?"

We said: "Yes. And if anything goes wrong, you'll know immediately."

Ran the migration. 25 minutes 42 seconds. Success.

They asked: "How are you doing this so fast AND so safely?"

We said: "We're not smarter. We just thought differently."

What You Need to Remember

If you're building something that scales:

Separate concerns — Files go to S3, metadata to DB, temp data to Redis
Query once, reuse forever — Cache valid values upfront (4 queries, not 4M)
Process async — Don't block users waiting
Use workers — One person can't do million tasks; 8 people can
Adjust dynamically — Chunk size based on system health
Validate ruthlessly — Three layers catch 98% of errors
Compress everything — 6:1 compression is free performance
Save checkpoints — Resume from failure, never start over
Monitor everything — Real-time dashboard shows what's happening
Alert proactively — Catch problems before they become disasters

Do these things. You'll handle any scale.

Next Up: Part 3

Next week, we dive into:

What actually happens at 2 million records (with REAL production numbers)
How auto-scaling actually works (code + scenarios)
The bulk insert strategy (45 mins → 2.5 mins)
The lessons we learned (some painful)
Economics: How much does this cost?

Hint: The answer will surprise you.

Questions?

Drop them below. I genuinely love talking about this.

And if you built something similar, tell me about it. Would love to hear your story.

Share this with your team if you're dealing with scale. Nobody should have to learn these lessons the hard way.

Next: Part 3: When 2 Million Records Hit — The Reality Check

Written with ❤️ for engineers who believe that scale shouldn't mean pain, and monitoring shouldn't be an afterthought.

If this helped, a share would mean a lot. Your colleague is probably struggling with migrations right now.

500K records in 15 minutes (was 4-6 weeks). See how we killed the N+1 query problem, optimized with caching, and scaled data migrations 20x faster.

HarshKumar Jha — Sun, 23 Nov 2025 07:43:37 +0000

HarshKumar Jha

Nov 22 '25

From Weeks to 15 Minutes: How We Built a Data Migration System That Changed Everything

#data #datamigration #technology #startup

Comments

3 min read

From Weeks to 15 Minutes: How We Built a Data Migration System That Changed Everything

HarshKumar Jha — Sat, 22 Nov 2025 15:55:13 +0000

Part 1: The Business Story – Why We Built This

The Problem Nobody Talks About

Imagine this: It's Monday morning at a utility company. They've just signed up for SMART360—an amazing platform that will transform how they manage customers, meters, and billing. They're excited. Their leadership is excited.

Then reality hits.

They have 500,000+ customer records sitting in an old system. They need to move it all over. The spreadsheets are messy. Column names don't match. Phone numbers are formatted differently. Some data is missing. Some is duplicated.

Their options?

Manual mapping: Hire a team to manually match columns, clean data, and load it in batches. Timeline? 4-6 weeks. Cost? Thousands in labor.
Custom scripts: Build one-off ETL pipelines. It works, but breaks with the next dataset. Timeline? 2-3 weeks per data type.
Hope and pray: Let someone use Postman to upload rows one by one. Timeline? As long as it takes. Accuracy? Not great.

Sound familiar?

The Real Cost of the Problem

Let's be honest—this isn't just a technical inconvenience.

For Operations Teams:

They can't start operations until data is migrated
Delayed go-live means delayed revenue
Every week of delay costs real money

For Support Teams:

One support engineer is stuck helping customers migrate data
That's one less person handling actual support tickets
Customers are frustrated waiting for assistance

For Finance Teams:

Billing can't start until consumer data is accurate
Meter readings don't have a home
Payment history is scattered or missing

For Customers:

They want to go live now, not "eventually"
Manual migrations feel unprofessional
Data errors = billing errors = unhappy customers

We saw this happening. Over and over. Customers would onboard, hit the data migration wall, and suddenly it's week 3 and nothing's moved. Our support team was drowning. Our reputation was taking a hit.

That's when we decided to solve it properly.

What We Built

We didn't build just another CSV uploader. We built something that understands the problem at its core.

Here's what it does:

Intelligently maps your messy data to our system fields automatically (not 100%, but ~95% accuracy)
Transforms data on the fly (fixes formatting, splits names, cleans addresses, standardizes everything)
Validates everything before it touches production (3 layers of validation)
Processes massive files fast (500K+ records in 15 minutes, not weeks)
Gives you control (you can override any mapping, any transformation)
Saves templates (reuse mappings for next time)

The Impact

Here's what changed:

Before:

Customers waiting 4-6 weeks to go live
Support team spending 30% of time on migrations
High error rates = data cleanup nightmare
One customer type at a time

After:

Customers go live in 1-2 weeks (migration is ~15 mins to 1 hour)
Support team reclaimed (no more manual migrations)
Error rates dropped by 85% (validation catches issues before they happen)
We can handle Consumers, Meters, Payments, Assets, and more—all in the same flow

Numbers That Matter

500,000+ records migrated in a single session
50MB+ file sizes handled without breaking a sweat
Support for 1 million+ records in infrastructure
95%+ automatic field mapping accuracy
15 minutes average processing time (was 4-6 weeks)

Translation?

Customers go live 20x faster
Support team saves ~100+ hours per month
Zero more "data migration support tickets"
Happy customers = more referrals

Why This Matters to You

Whether you're a product manager thinking about customer success, an engineer building data pipelines, or a CTO looking at operational efficiency:

This is what happens when you take a real problem and actually solve it. Not with a Band-Aid. With a proper system.

It's not about the technology stack. It's about understanding that migrations are how customers experience your platform first. If that experience is painful, nothing else matters.

I Harsh Jha, Software Engineer at Bynry Inc., spearheaded the implementation of Data Migration. I work alongside Tech Lead Gaurav Dagde to continuously improve the quality and reliability of the Smart360 platform.

to read more please visit

Know more about Bynry:

Homepage → https://bynry.com
Product page → https://bynry.com/products/smart360
Billing solution → https://bynry.com/solutions/billing
Asset management → https://bynry.com/solutions/asset-management
Metering solution → https://bynry.com/solutions/metering

🔥 From Chaos to Confidence: How We Slashed Bug Reports by 80% at Bynry Inc.

HarshKumar Jha — Sat, 03 May 2025 09:27:27 +0000

HarshKumar Jha

May 3 '25

Building a Rock-Solid Testing Framework at Bynry:

#testing #django #tdd #architecture

Comments 1

7 min read

Building a Rock-Solid Testing Framework at Bynry:

HarshKumar Jha — Sat, 03 May 2025 08:49:15 +0000

Implementation (Part 2)

In Part 1 of this series, I shared how and why we embraced Test-Driven Development (TDD) at Bynry Inc. to improve the quality and reliability of our Smart360 utility management platform. Now, let's dive into the technical details of our implementation—how we designed our testing framework, the architectural choices we made, and the real-world results we've achieved.

Setting the Technical Foundation

When Gaurav and I set out to build our testing framework, we had several key requirements:

Support for API testing: Since Smart360 is built on microservices, we needed robust API testing capabilities
Easy to write tests: Developers should find the framework intuitive and straightforward
Comprehensive coverage: The framework should help identify untested code paths
Clear feedback: When tests fail, developers should quickly understand why
Maintainable test code: Test code should be as clean and maintainable as production code
CI/CD integration: Tests should run automatically with each code change

After evaluating various options, we chose pytest as our foundation due to its flexibility, extensive plugin ecosystem, and excellent support for fixtures and parametrization.

The Architecture of Our Testing Framework

Our testing framework for Smart360 is organized with a clear separation of concerns:

📦 tests
├─ data/                     # Test data organized by module
│  ├─ meter_data.py          # Meter test data
│  ├─ route_data.py          # Route test data
│  └─ ...
├─ views/                    # Tests for API views
│  ├─ test_meter_view.py
│  ├─ test_bulk_meter_upload_view.py
│  └─ ...
├─ models/                   # Tests for models
├─ utils/                    # Utilities for testing
│  ├─ dummy_wrapper_utils/   # Mock external services
│  │  ├─ auth_wrapper.py
│  │  ├─ onboarding_wrapper.py
│  │  └─ ...
│  ├─ base_test_api_view.py  # Base test class
│  └─ ...
└─ conftest.py               # Global pytest fixtures

This structure reflects our philosophy that test code should be as organized and maintainable as production code. Let's explore each component in detail.

The Heart of Our Framework: BaseTestApiView

At the core of our testing framework is the BaseTestApiView class. This base class provides a rich set of assertions and utilities specifically designed for API testing:

class BaseTestApiView:
    def setup_method(self):
        self.smart360_api_client()

    def assertEqual(self, first, second, msg=None):
        if first != second:
            standard_msg = f"{first!r} != {second!r}"
            self.fail(msg or standard_msg)

    def assertIn(self, member, container, msg=None):
        if member not in container:
            standard_msg = f"{member!r} not found in {container!r}"
            self.fail(msg or standard_msg)

    # ... more assertions ...

    def assert_response_contains_keys(self, response, expected_keys):
        """Assert that a response contains certain keys"""
        data = self._get_response_data(response)
        # ... implementation ...

    def assert_valid_response(self, response_data, schema):
        """Assert that a response is valid according to the expected schema"""
        # ... implementation ...

By inheriting from this base class, test classes get access to a rich set of assertions specifically designed for API testing. These assertions provide clear, detailed error messages when tests fail, making it easy to diagnose issues.

Data-Driven Testing

One of our key innovations was adopting a data-driven approach to testing. Rather than hardcoding test inputs and expected outputs in test methods, we store them in dedicated data files:

# In tests/data/meter_data.py

meter_test_cases = {
    "get_meter_by_id": {
        "request": {
            "method": "GET",
            "url": "/api/mx/meter/{id}/",
            "data": {"remote_utility_id": 10000}
        },
        "response": {
            "status_code": 200,
            "content_type": "application/json",
            "data_checks": {
                "meter_number": "M12345",
                "status": 0
            }
        }
    },
    # More test cases...
}

This separation of test data from test logic offers several benefits:

Readability: Test methods focus on behavior, not data
Maintainability: Changes to expected behavior only require updating data files
Reusability: The same test data can be used across multiple tests
Documentation: Test data files serve as documentation of expected API behavior

Mocking External Services

Smart360's microservice architecture means services often depend on other services. To test in isolation, we created a system of "dummy wrapper utilities" that mimic the behavior of external services:

# In tests/utils/dummy_wrapper_utils/onboarding_wrapper.py
class DummyOnboardingWrapper:
    def get_premise_info(self, unique_values, remote_utility_id, type_json=None):
        """Return predefined mock response based on the type_json"""
        if type_json == "category_json":
            return [
                {
                    'code': 'Category-1',
                    'hierarchy_names': {'category': 'Mock Category Name'},
                    '_': {'name': 'Mock Category Name'}
                }
            ]
        else:
            return [
                {
                    'code': 'Test Premise',
                    'hierarchy_names': {
                        'area': 'Mock Area Name',
                        'sub_area': 'Mock Sub Area Name',
                        'premise': 'Mock Premise Name'
                    }
                }
            ]

Test classes then use these dummy wrappers to mock external service calls:

def setup_service_mocks(self):
    """Setup mocks for all external services"""
    self.onboarding_patcher = patch('meter.utils.wrapper_utils.onboarding_wrapper.OnboardingWrapper')
    self.mock_onboarding = self.onboarding_patcher.start()
    self.mock_onboarding_instance = MagicMock()
    self.mock_onboarding.return_value = self.mock_onboarding_instance

    # Configure responses using dummy wrapper
    self.mock_onboarding_instance.get_premise_info.return_value = \
        dummy_onboarding_wrapper.get_premise_info(["Test Premise"], 10000)

This approach ensures tests are isolated, deterministic, and not dependent on external services being available or behaving consistently.

Schema Validation

For API testing, validating the structure and content of responses is critical. We developed a schema validation system that verifies:

Required fields are present
Fields have the correct types
Status codes map to the appropriate display values
Pagination logic is consistent

meter_detail_schema = {
    "pagination_fields": {
        "count": {"type": int, "required": True},
        "next": {"type": (str, type(None)), "required": True},
        "previous": {"type": (str, type(None)), "required": True},
    },
    "result_key": "results",
    "result_fields": {
        "id": {"type": int, "required": True},
        "meter_number": {"type": str, "required": True},
        "status": {"type": int, "required": True},
        "status_display": {"type": str, "required": True},
    },
    "status_mappings": {
        "status": {
            0: "Unassigned",
            1: "Assigned",
            2: "Expired"
        }
    }
}

Our assert_valid_response method performs comprehensive validation against these schemas, ensuring APIs maintain consistent response formats and adhere to their documented contracts.

Standard Test Structure

We established a consistent pattern for all test classes:

class TestMeterDetailAPIView(BaseTestApiView):
    """Test suite for the MeterDetailAPIView"""

    def setup_method(self):
        """Set up test data and client for each test"""
        super().setup_method()

        # Define URL and utility ID
        self.url = '/api/mx/meter-detail/'
        self.utility_id = 10000

        # Create necessary database objects
        self.setup_database_objects()

        # Setup mocks for external services
        self.setup_service_mocks()

    def setup_database_objects(self):
        """Create all database objects from test data"""
        # Create meters from test data
        self.meters = []
        for data in meter_setup_data["existing_meters"]:
            meter = Meter.objects.create(**data)
            self.meters.append(meter)

    def setup_service_mocks(self):
        """Setup mocks for all external services"""
        # Set up mock for onboarding service
        # ... implementation ...

    def test_get_meter_by_id(self):
        """Test retrieving meter by ID"""
        meter = self.meters[0]
        response = self.client.get(f"{self.url}{meter.id}/", {"remote_utility_id": self.utility_id})

        self.assertEqual(response.status_code, 200)
        self.assertEqual(response.data["id"], meter.id)

This consistent structure makes tests easy to read and maintain. Each test follows the Arrange-Act-Assert pattern:

Arrange: Set up test data and conditions
Act: Perform the action being tested
Assert: Verify expected outcomes

Test Running and Reporting

To make our testing framework easy to use, we created a custom test runner script:

#!/bin/bash
# Script to run tests with comprehensive reporting

# Function to display help
show_help() {
    echo "Usage: $0 [OPTIONS]"
    echo ""
    echo "Test Selection:"
    echo "  -f, --failed           Run only previously failed tests"
    echo "  -k PATTERN             Run tests matching pattern"
    # ... more options ...
}

# Process options
# ... implementation ...

# Run the tests
echo "Running tests..."
eval "$COMMAND"

# Display summary information
echo "Test Execution Summary"
echo "========================"
echo "Duration: $duration_str"
echo "Coverage HTML: tests/reports/coverage/index.html"

This script provides a user-friendly interface for running tests, with options for running specific tests, generating coverage reports, and more.

Understanding Test Coverage

A critical aspect of our testing framework is coverage reporting. Test coverage measures how much of your code is executed during testing, helping identify untested code paths.

We configured pytest-cov to generate detailed coverage reports:

[run]
source = meter
omit = 
    */migrations/*
    */tests/*
    # ... more exclusions ...

[report]
# Show missing lines in report
show_missing = True

# Fail the command if coverage is below this threshold
fail_under = 80

[html]
directory = tests/reports/coverage
title = Smart360 Test Coverage Report

These reports help us identify which parts of the codebase need more testing, focusing our efforts where they'll have the most impact.

Handling Exceptions in Django Views

One subtle but important aspect of our framework is how we test exception handling in Django views. In Django, when an exception occurs in a view, Django catches it and returns an appropriate HTTP response. This means we can't use pytest's raises context manager to test exceptions in views.

Instead, we test the response:

# INCORRECT: Django views catch exceptions and return responses
with pytest.raises(CustomException):  # This will fail
    self.client.get('/your-url/')

# CORRECT: Check the response indicates an error
response = self.client.get('/your-url/')
self.assertEqual(response.status_code, 400)
self.assertIn("error message", response.data["message"])

This approach accurately tests how the API behaves from the client's perspective.

Real-World Results

After implementing our TDD framework, we've seen dramatic improvements across the board:

Code Quality Metrics

Test coverage: Increased from <30% to >80%
Bug reports: Decreased by 70-80%
Production incidents: Reduced by 65%
Rework: Decreased by 50%

Developer Experience

Onboarding time: New developers become productive faster
Collaboration: Improved understanding across services
Confidence: Developers make changes with less fear of breaking things
Critical thinking: More emphasis on edge cases and error handling

Business Impact

Release predictability: More features delivered on time and bug-free
Customer satisfaction: Fewer service disruptions
Developer retention: Less burnout from firefighting
Innovation pace: More time for new features, less time fixing bugs

Lessons Learned

Our TDD journey at Bynry wasn't without challenges. Here are some key lessons we learned:

Start with the right mindset: TDD is as much about thinking differently as it is about writing tests
Invest in infrastructure: A good testing framework pays dividends by making tests easier to write
Make tests easy to read: Tests serve as documentation—they should be clear and understandable
Balance testing and pragmatism: 100% coverage isn't always necessary; focus on critical paths
Lead by example: Nothing convinces skeptics like seeing TDD prevent real bugs

Looking Forward

Our testing framework continues to evolve. We're currently exploring:

Property-based testing: Generating test cases automatically to find edge cases
Performance testing: Identifying performance regressions early
Mutation testing: Verifying the quality of our tests by introducing faults
Contract testing: Formalizing the contracts between microservices

Conclusion

Implementing TDD at Bynry has transformed not just our code but our entire approach to software development. We've moved from reactive firefighting to proactive quality assurance, from hoping things work to knowing they do.

The investment in our testing framework has paid off many times over in reduced bugs, improved developer satisfaction, and more reliable software. For other organizations considering a similar journey, our advice is simple: start today. The cost of not testing is far greater than the cost of testing.

At Bynry, we're proud of the culture of quality we've built. Our testing framework is more than just code—it's a manifestation of our commitment to engineering excellence and delivering the best possible experience to our customers.

I Harsh Jha, Software Developer Engineer at Bynry Inc., spearheaded the implementation of Test-Driven Development across the company's engineering teams. I work alongside Tech Lead Gaurav Dagde to continuously improve the quality and reliability of the Smart360 platform.

to read more please visit

Know more about Bynry:

Homepage → https://bynry.com
Product page → https://bynry.com/products/smart360
Billing solution → https://bynry.com/solutions/billing
Asset management → https://bynry.com/solutions/asset-management
Metering solution → https://bynry.com/solutions/metering

Learn how Software Quality is transformed in early Startups

HarshKumar Jha — Sun, 20 Apr 2025 10:11:29 +0000

HarshKumar Jha

Apr 19 '25

Transforming Software Quality at Bynry: Our Test-Driven Development Journey (Part 1)

#testing #python #django #programming

Comments 2

6 min read

Transforming Software Quality at Bynry: Our Test-Driven Development Journey (Part 1)

HarshKumar Jha — Sat, 19 Apr 2025 10:46:32 +0000

The Breaking Point

Picture this: It's 2 AM on a Tuesday. Our on-call engineer's phone lights up with alerts. Smart360, our flagship utility management platform, is experiencing downtime. Again. Another unexpected edge case has reared its head in production, and our customers are feeling the impact.

This wasn't the first time we'd encountered such issues, but it marked a turning point for us at Bynry. As we huddled around our monitors, frantically debugging the production system, a question hung in the air: "How did we miss this?"

The answer was simple, yet difficult to confront: our development approach was fundamentally flawed.

The Reality Check

At Bynry Inc., we build Smart360, a comprehensive AI-powered SaaS platform that helps utility companies manage their customers, assets, billing, meters, and more. We operate in a fast-paced environment with aggressive release cycles and constantly evolving requirements. Like many growing tech companies, we had prioritized speed above all else.

The results were predictable in hindsight:

Frequent production issues: Edge cases were consistently overlooked
Growing technical debt: Quick fixes that inevitably led to more problems
Increasing downtime: Directly impacting our customers' operations
Developer burnout: Constant firefighting drained our team's energy and creativity
Lost focus on core principles: Our developers were forgetting fundamental software engineering practices

Most concerningly, we realized our microservice architecture was becoming a liability rather than an asset. Changes in one service would unexpectedly break functionality in others. Something had to change.

The Seed of an Idea

During one particularly tough debug session, my Tech Lead Gaurav Dagde and I found ourselves discussing the root cause of our recurring issues.

"We're writing code, but we're not really thinking about how it will be used," Gaurav noted. "Everyone's building in isolation."

That observation sparked a realization: we weren't properly testing the interactions between our services. Developers were focused on making their individual components work in isolation, without considering the broader impact of their changes.

The answer seemed clear: we needed to embrace Test-Driven Development (TDD).

Understanding Test-Driven Development

For those unfamiliar, Test-Driven Development follows a simple but powerful cycle:

Write a test that defines a function or improvements to a function
Run the test, which should fail because the function isn't implemented yet
Write the simplest code that passes the test
Refactor the code to meet quality standards
Repeat with the next test case

This "red-green-refactor" cycle fundamentally changes how developers approach problem-solving. Rather than writing code first and testing later (if at all), TDD forces you to think about requirements, edge cases, and interfaces before writing a single line of implementation code.

The Traditional Approach vs. TDD

Before implementing TDD, our development process looked something like this:

Receive requirements for a new feature
Write code to implement the feature
Manually test the happy path
Deploy to production
Wait for bug reports to identify edge cases
Fix bugs reactively
Repeat steps 5-6 indefinitely

The problems with this approach were numerous:

Reactive bug fixing: We were always playing catch-up
Incomplete testing: Manual testing rarely covered all edge cases
Poor documentation: New developers had to reverse-engineer how systems worked
Impact blindness: Developers couldn't see how their changes affected other services
Technical debt accumulation: Quick fixes led to more complex problems later

With TDD, we envisioned a completely different workflow:

Receive requirements for a new feature
Break down the feature into testable components
Write tests that define how each component should behave
Implement the minimal code needed to pass those tests
Refactor for maintainability
Deploy with confidence
Catch regressions automatically with each change

The Decision

After that late-night debugging session, Gaurav and I put together a proposal for our engineering leadership team. We didn't just want to add more tests—we wanted to fundamentally transform how we build software at Bynry.

We highlighted the costs of our current approach:

Customer dissatisfaction due to downtime
Developer time wasted on debugging
Increasingly brittle codebase
Slowing pace of innovation as we dealt with technical debt

And we outlined the benefits of TDD:

Improved code quality and reliability
Better understanding of requirements
Natural documentation through tests
Early detection of bugs and design flaws
Confidence in making changes

To our relief, the leadership team immediately recognized the value. We got the green light to implement TDD across all our development teams.

First Steps: The Mindset Shift

Implementing TDD isn't just about writing tests—it requires a fundamental shift in how developers think about their work. We knew that for TDD to succeed at Bynry, we needed to change our engineering culture.

Our first step was education. We organized a series of workshops to teach the principles of TDD and help developers understand why it matters. We focused on the benefits:

Design improvement: Writing tests first forces you to think about your API design
Documentation: Tests serve as executable documentation for how code should behave
Confidence: Well-tested code can be changed with less fear of breaking things
Focus: Working on one test at a time keeps development targeted and efficient

We also addressed common concerns:

"It will slow us down": Initially yes, but it pays dividends in reduced debugging time
"It's overkill for simple changes": Even simple changes can have unexpected impacts
"We don't have time for this": We don't have time to keep fixing the same bugs

Early Challenges

Not everyone was immediately convinced. We faced resistance from some team members who saw TDD as bureaucratic overhead or "testing for testing's sake." This is natural—changing ingrained habits is difficult.

We took several approaches to address this resistance:

Lead by example: Gaurav and I started implementing TDD in our own work
Pair programming: We put TDD-experienced developers from Bynry together with people who were unsure about it.
Celebrate wins: We highlighted cases where TDD caught bugs before they reached production
Share metrics: We tracked and shared data showing reduced bug counts in TDD-implemented features

One particularly effective tactic was our "Bug Museum"—a wall where we documented production bugs that could have been prevented with proper testing. This visual reminder helped reinforce why we were making this change.

Building the Framework

While the cultural shift was underway, we also needed to build the technical infrastructure to support TDD. We needed a testing framework that would:

Work seamlessly with our Python/Django microservices
Support API testing across service boundaries
Integrate with our CI/CD pipeline
Provide clear feedback when tests failed
Generate coverage reports to track our progress

After evaluating several options, we decided to build a custom testing framework based on pytest. This framework would need to handle the complexities of our microservice architecture and provide developers with tools to write effective tests.

In Part 2 of this series, I'll dive deep into the technical details of this framework—how we designed it, how it works, and how it transformed our development process at Bynry.

The Transformation Begins

As we rolled out TDD across our teams, we started to see small changes. Developers began thinking more critically about their code. Discussions shifted from "How do we implement this?" to "How will we know if this works correctly?"

Code reviews became more substantive, focusing on test cases as much as implementation. "Did you test what happens if this parameter is missing?" "What about this edge case?" These questions became standard.

Most importantly, our developers started to take ownership of quality. Testing was no longer seen as something separate from development—it was an integral part of the development process itself.

Early Results

Within the first few months, we started to see encouraging results:

Fewer production incidents: Our monitoring showed a 40% decrease in critical alerts
More predictable releases: Features were more complete at launch
Better collaboration: Tests served as a common language between teams
Improved onboarding: New developers could understand system behavior by reading tests

But the most significant change was harder to measure: our developers were growing. They were thinking more deeply about their work, considering edge cases, and building more robust systems. The focus had shifted from "making it work" to "making it work reliably."

Looking Ahead

In Part 2 of this series, I'll explore the technical implementation of our testing framework, including:

Our pytest-based architecture
Our approach to mocking external services
Our custom assertions for API testing
Our test data organization strategy
Our CI/CD integration

I'll also share concrete metrics on how TDD transformed our development process, with before-and-after comparisons of key indicators like bug rates, development velocity, and code maintainability.

Stay tuned for a deep dive into the technical side of our TDD journey at Bynry!