Forem: Eddy Adegnandjou

Eventual Consistency Guarantees Correctness. Your job is to Make Users Believe It.

Eddy Adegnandjou — Sat, 21 Mar 2026 11:10:31 +0000

Your system says the transaction succeeded. Your user says it didn't.

A user transfers money.

They see:

Transaction successful

They refresh their balance.

Nothing changed.

They refresh again.
Still nothing.

Now doubt sets in:

Was I charged?
Did it fail?
Should I try again?

From the user's perspective, the system is broken.

From the system's perspective... everything is working perfectly.

Two realities, one system

Behind the scenes, your system already knows the truth:

The transaction has been recorded
The operation is valid
The system state is correct

But what the user sees is different:

The balance hasn't updated
The UI reflects stale data
The confirmation feels unreliable

You now have two realities:

1. The system truth (what actually happened)
2. The perceived truth (what the user sees)

And they are temporarily out of sync.

This is Eventual consistency

Modern systems are rarely fully synchronous.

To scale and remain resilient, they rely on:

Asynchronous processing
Distributed data stores
Replication across services

Which leads to:

The system will become consistent... eventually

But that "eventually" is where problems begin.

What really happens under the hood

Let's break it down:

User initiates transaction
API writes to the primary database
An event is published
A background worker processes it
A read replica updates later

At any given moment:

One part of the system says: "done"
Another part says: "not yet"

Both are technically correct.

Correct systems are not enough. Users must perceive correctness.

Models

The models define the core domain of the system and illustrate how the backend maintains correctness while the user sees a delayed view.

# models.py
class Account(models.Model):
    user = models.ForeignKey(User, on_delete=models.CASCADE)
    balance = models.DecimalField(max_digits=12, decimal_places=2, default=0)


class Transaction(models.Model):
    account = models.ForeignKey(Account, on_delete=models.CASCADE)
    amount = models.DecimalField(max_digits=10, decimal_places=2)
    status = models.CharField(max_length=20, default="pending")
    created_at = models.DateTimeField(auto_now_add=True)

class LedgerEntry(models.Model):
    account = models.ForeignKey(Account, on_delete=models.CASCADE, related_name="ledger_entries")
    transaction = models.ForeignKey(Transaction, on_delete=models.CASCADE)
    amount = models.DecimalField(max_digits=12, decimal_places=2)
    created_at = models.DateTimeField(auto_now_add=True)

Account

Represents a user's financial account.
balance shows the current amount, but it may be temporarily outdated due to async processing.

Transaction

Represents a single user-initiated transaction.
Starts with status pending, showing the action was received by the system.
Updated to completed after async processing finishes.

LedgerEntry

Provides an immutable record of all transactions applied to an account.
Ensures an auditable source of truth for balances.
Even if the user's view is delayed, the ledger guarantees correctness.
Reinforces eventual consistency and auditability.

Celery Task

This task handles the asynchronous processing, demonstrating how backend truth is updated safely while users perceive a delay.

# tasks.py (Celery)
from django.db import transaction
from celery import shared_task
from .models import Transaction, LedgerEntry

@shared_task 
def process_transaction(transaction_id):
    with transaction.atomic():
        tx = Transaction.objects.select_for_update().get(id=transaction_id)
        account = tx.account

        LedgerEntry.objects.create(
            account=account,
            transaction=tx,
            amount=tx.amount
        )

        account.balance += tx.amount
        account.save()

        tx.status = "completed"
        tx.save()

process_transaction

Runs asynchronously, updating the transaction and account balance.
Uses select_for_update within a transaction to prevent race conditions when multiple transactions happen simultaneously.
Creates a LedgerEntry to record the transaction.
Updates account balance and marks the transaction completed.

View

The view shows how the system immediately communicates to the user while the actual processing happens in the background.

# views.py
from django.http import JsonResponse
from django.shortcuts import get_object_or_404
from .models import Transaction, Account
from .tasks import process_transaction

def create_transaction(request):
    account_id = request.POST.get("account_id")
    amount = float(request.POST.get("amount"))

    account = get_object_or_404(Account, id=account_id, user=request.user)

    tx = Transaction.objects.create(account=account, amount=amount)

    process_transaction.delay(tx.id)

    return JsonResponse({
        "transaction_id": tx.id,
        "status": tx.status,
        "message": "Transaction received and processing..."
    })

create_transaction

Receives account_id and amount from the request.
Ensures the account belongs to the requesting user.
Creates a Transaction in pending state and triggers the async task.
Returns immediate feedback to the user.

Closing Thought

Eventual consistency ensures correctness in distributed systems. But trust is a UX problem.

Show pending states
Use optimistic UI

The system is correct. Users must believe it.

Mitigating the Two Generals Problem in Django with Idempotence

Eddy Adegnandjou — Sat, 28 Feb 2026 18:19:28 +0000

A Fintech Architecture Perspective

When building payment systems, you eventually face a terrifying question:

Did we charge the customer or not?

The worst part?

Sometimes, the system genuinely doesn't know.

This is not a bug. It's a distributed systems reality described by the Two Generals Problem: two parties can not guarantee agreement over an unreliable network.

In fintech, this becomes:

Merchant --> Backend --> Payment Provider
Charge succeds
Network drops
Backend never receives confirmation
Client retries

Now what?

If you retry blindly, you double charge.

If you don't retry, you might lose revenue.

This is where idempotency become a core architectural pattern.

Why this matters in Fintech

In international payment systems:

Mobile networks are unstable
Clients double-click
Workers crash mid-transaction

If your backend is not idempotent, you will:

Double charge customers
Create accounting inconsistencies
Trigger reconciliation nightmares
Lose merchant trust

Large providers like Stripe formalized idempotency because this problem is structural, not incidental.

The Goal: Deterministic Retries

We cannot guarantee exactly-once execution over HTTP.

What we can guarantee:

The same request key always produces the same result.

That's the architectural shift. Instead of solving uncertainty, we make retries safe.

Designing Idempotency in Django

This is not about adding a if exists check. This is about enforcing correctness at the database boundary.

1. Require an Idempotency Key

Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000

This key represents one logical business operation.

2. Persist the Key with a Unique Constraint

# models.py

class IdempotencyRecord(models.Model):
    key = models.CharField(max_length=255, unique=True)
    request_hash = models.CharField(max_length=64)
    response_body = models.JSONField(null=True, blank=True)
    response_status = models.IntegerField(null=True, blank=True)
    created_at = models.DateTimeField(auto_now_add=True)

Why this matters:

unique=True protects against concurrency.
The database becomes the source of truth.
Race conditions move from application logic to atomic constraints.

This is architectural, not syntax.

3. Hash the Request Payload

import hashlib

def hash_request(request):
    return hashlib.sha256(request.body).hexdigest()

Why ?

If someone reuses the same key with a different payload, that's dangerous. We must detect:

Same key
Different business intent

And reject it.

The hash is generated on the server when Django receives the request.

request payload -> server-side hash computation 
                -> database storage 
                -> hash comparison on retries 
                -> deterministic decision (replay or reject)

4. Atomic Handling

from django.db import transaction, IntegrityError
from django.http import JsonResponse

@transaction.atomic 
def process_payment(request):
    key = request.headers.get("Idempotency-Key")
    if not key:
        return JsonResponse({"error": "Missing Idempotency-Key"}, status=400)

    request_hash = hash_request(request)

    try:
        record = IdempotencyRecord.objects.create(
            key=key,
            request_hash=request_hash
        )
        first_request = True
    except IntegrityError:
        record = IdempotencyRecord.objects.select_for_update().get(key=key)
        first_request = False

    if not first_request:
        if record.request_hash != request_hash:
            return JsonResponse(
            {"error": "Payload mismatch"},
            status=409
            )
        return JsonResponse(record.response_body, status=record.response_status)

    # External side effect (charge)
    charge = external_payment_call()

    response = {"payment_id": charge.id}

    record.response_body = response
    record.response_status = 200
    record.save()

    return JsonResponse(response, status=200)

What this guarantees:

First request executes side effects
Duplicate request returns stored result
No duplicate charges
Safe client retries

What This Achieves Architecturally

This pattern:

Simulates exactly-once semantics
Accepts eventual consistency
Moves correctness into a deterministic replay
Makes the system resilient to network failures

It does not eliminate distributed uncertainty. It engineers around it.

Critical Fintech Edge Cases

To operate at international scale, you must also consider:

1. DB Crash After External Charge

If the charge succeeds but DB commit fails, you still have inconsistency.

Mitigation:

Use provider webhooks
Reconciliation jobs
Ledger-based accounting

2. Key Expiration

Idempotency tables grow.

Add TTL cleanup:

24-48h retention for payment operations
Longer for financial transfers

3. Observability

In real fintech systems:

Every idempotency key is tracable
Every retry is logged
Metrics track replay frequency

Idempotency without observability is incomplete.

Closing Thought

In fintech, correctness is not about writing clean code. It is about designing systems that behave deterministically under failure.

Networks will drop.
Clients will retry.
Workers will crash.

Idempotency is how you make your Django system financially safe in an unreliable world.