Forem: Akarshan Gandotra

I Dug Up My 10-Year-Old Android App, Dusted It Off With AI, and Put It Back on the Play Store

Akarshan Gandotra — Sun, 22 Feb 2026 06:44:45 +0000

Last week I did something I didn't expect to enjoy as much as I did — I resurrected a side project I hadn't touched in a decade.

Meet Drivelert: an anti-drowsiness app I built ages ago to help drivers stay alert on long trips. The Play Store had quietly pulled it down due to ancient target SDK, zero maintenance, the usual graveyard story. I'd moved on. The app hadn't.

Then, for reasons I can't fully explain (nostalgia? a slow weekend? some stubborn refusal to let past-me's work die?), I decided to bring it back.

What Even Was This Thing?

Drivelert was a simple but genuinely useful idea: monitor signs of driver fatigue and alert them before it becomes dangerous. I built it back when I was younger, more idealistic, and apparently not bothered by shipping something and completely abandoning it.

The code was... a time capsule. Deprecated APIs, patterns I wouldn't touch today, some choices that made me genuinely wince. But the idea was solid. The bones were good.

A decade later: smarter, more experienced, and AI finally made the revival real

This time around, I wasn't flying solo. I brought AI into the workflow for unpicking outdated patterns, modernizing chunks of logic, and accelerating the parts that would've taken me days to slog through manually.

It's a weird experience, honestly. You're reading code written by a younger version of yourself, and then having an AI help you translate it into something modern. Felt a bit like co-writing a letter to the past.

What made it work was the mix of experience and capability.
I had the wisdom and context of why things were built a certain way. AI brought the momentum and precision to rebuild them better.
That blend of hindsight, experience, and AI support turned out to be surprisingly powerful.

It's Live Again

After a week of evenings, Drivelert is back up on the Play Store:

👉 Drivelert on Google Play

There's something quietly satisfying about seeing an old project breathe again not just preserved, but actually improved. Younger-me would probably be pleased. Slightly jealous of the tools available now, but pleased.

What I Actually Learned

Old side projects aren't necessarily dead, sometimes they just need the right moment and a better toolkit.
AI assistance genuinely changes the calculus on revival projects. The "is this worth the effort?" math shifts when you can move faster.
Shipping something imperfect 10 years ago > never shipping the perfect version. Past-me understood this intuitively. I'd forgotten.

It's a second innings for a scrappy little app. Let's see how it goes. 🏏

Testing Redis Circuit Breaker with Toxiproxy

Akarshan Gandotra — Tue, 03 Feb 2026 07:28:46 +0000

Building resilient distributed systems requires thorough testing of failure scenarios. While unit tests are great for business logic, they can't simulate the complex network failures that happen in production. This is where Toxiproxy comes in—a powerful tool for testing how your application handles real-world network chaos.

In this tutorial, we'll explore how to test a Redis circuit breaker implementation using Toxiproxy to simulate various failure modes, from complete outages to subtle network degradation.

What is Toxiproxy?

Toxiproxy is a TCP proxy developed by Shopify that simulates network and system conditions for chaos and resiliency testing. Unlike traditional testing approaches that mock dependencies, Toxiproxy sits between your application and its dependencies, allowing you to inject real network failures:

Connection failures (service down)
Latency (slow networks)
Bandwidth limitations (network congestion)
Connection resets (unstable networks)
Data corruption (packet loss)

This makes it ideal for testing circuit breakers, retry logic, timeouts, and other resilience patterns.

Understanding Circuit Breakers

Before we dive into testing, let's briefly review the circuit breaker pattern. A circuit breaker prevents an application from repeatedly trying to execute an operation that's likely to fail, giving the failing service time to recover.

The circuit breaker has three states:

Closed: Normal operation, requests pass through
Open: Too many failures detected, requests are blocked or fail-fast
Half-Open: Testing if the service has recovered

For our Redis implementation, we'll configure the circuit breaker to:

Open after 5 consecutive failures
Either fail-open (allow requests through) or fail-closed (block requests)
Log when state transitions occur

Setup

Installing Toxiproxy

First, install Toxiproxy on your system:

macOS:

brew install toxiproxy

Starting the Toxiproxy Server

Start the Toxiproxy server in the background:

toxiproxy-server &

The server starts on localhost:8474 by default. You can verify it's running:

curl http://localhost:8474/version

Creating a Proxy for Redis

Now create a proxy that sits between your auth service and Redis:

# Create proxy: listens on 6380, forwards to Redis on 6379
toxiproxy-cli create redis-proxy \
  -l localhost:6380 \
  -u localhost:6379

This creates a proxy named redis-proxy that:

Listens on port 6380 (your application will connect here)
Forwards traffic to Redis on port 6379

Verify the proxy was created:

toxiproxy-cli list

Output:

Name        Listen          Upstream        Enabled
============================================================
redis-proxy localhost:6380  localhost:6379  true

Configuring Your Application

Point your auth service to use the Toxiproxy port instead of connecting directly to Redis:

export REDIS_HOST=localhost
export REDIS_PORT=6380  # Toxiproxy port, not 6379

# Restart your service

Now all Redis traffic flows through Toxiproxy, allowing you to inject failures without modifying your application code.

Testing Scenarios

Let's explore different failure scenarios and verify our circuit breaker behaves correctly.

Scenario 1: Simulate Redis Down (Circuit Opens)

The most critical test—what happens when Redis becomes completely unavailable?

Disable the proxy:

toxiproxy-cli toggle redis-proxy

This simulates Redis being down. Your application will start getting connection failures.

Expected Behavior:

First 4 requests fail but circuit remains closed
5th consecutive failure → Circuit breaker opens
Logs should show:

   {"level":"warn","msg":"Circuit breaker opened - Redis failures exceeded threshold","service":"auth"}

Subsequent requests are either:
- Fail-open: Allowed through (degraded mode, no Redis caching)
- Fail-closed: Rejected immediately (fail-fast)

Monitoring the circuit state:

While the circuit is open, check your application metrics or logs. The circuit should remain open for the configured recovery period.

Re-enable the proxy:

toxiproxy-cli toggle redis-proxy

After re-enabling, the circuit should transition to half-open on the next request, then back to closed if the request succeeds.

Verification checklist:

✅ Circuit opens after 5 failures
✅ Log message appears with correct timestamp
✅ Requests are handled according to fail-open/fail-closed policy
✅ Circuit recovers when service returns
✅ Application continues functioning (degraded or failed requests)

Scenario 2: Add Latency (Slow Redis)

Network latency is often more insidious than complete failures. It can cause timeouts, thread pool exhaustion, and cascading failures.

Add 5-second latency:

toxiproxy-cli toxic add redis-proxy \
  -t latency \
  -a latency=5000

This adds a 5000ms (5 second) delay to all requests passing through the proxy.

Expected Behavior:

If your Redis client timeout is less than 5 seconds (e.g., 2 seconds), requests will timeout and count as failures. After 5 consecutive timeouts, the circuit should open.

Logs you should see:

{"level":"error","msg":"Redis operation timeout","error":"context deadline exceeded"}
{"level":"warn","msg":"Circuit breaker opened - Redis failures exceeded threshold"}

Testing different latency levels:

# Moderate latency (1 second)
toxiproxy-cli toxic update redis-proxy \
  -n latency_downstream \
  -a latency=1000

# Extreme latency (10 seconds)
toxiproxy-cli toxic update redis-proxy \
  -n latency_downstream \
  -a latency=10000

Remove the latency toxic:

toxiproxy-cli toxic remove redis-proxy -n latency_downstream

Key insights:

Latency above your timeout threshold behaves like a failure
Helps verify your timeouts are properly configured
Tests thread pool behavior under slow dependencies

Scenario 3: Connection Reset (Network Errors)

Simulate unstable network connections that reset mid-request:

Add reset_peer toxic:

toxiproxy-cli toxic add redis-proxy \
  -t reset_peer \
  -a timeout=500

This closes the connection 500ms after receiving data, simulating network instability.

Expected Behavior:

Connections will be abruptly closed, causing errors like:

connection reset by peer
unexpected EOF
broken pipe

These should count as failures and eventually open the circuit.

Remove the toxic:

toxiproxy-cli toxic remove redis-proxy -n reset_peer_downstream

When to use this test:

Verifying connection pool recovery
Testing retry logic
Ensuring proper cleanup of broken connections

Scenario 4: Bandwidth Limit (Network Congestion)

Simulate network congestion with bandwidth restrictions:

Limit to 1KB/s:

toxiproxy-cli toxic add redis-proxy \
  -t bandwidth \
  -a rate=1

This restricts throughput to 1 kilobyte per second, simulating a severely congested network.

Expected Behavior:

Small Redis operations (GET, SET) might still work but slowly
Large operations (fetching big values, pipeline operations) will timeout
Gradual degradation rather than immediate failure

Test different bandwidth levels:

# Moderate congestion (10 KB/s)
toxiproxy-cli toxic update redis-proxy \
  -n bandwidth_downstream \
  -a rate=10

# Severe congestion (100 bytes/s)
toxiproxy-cli toxic update redis-proxy \
  -n bandwidth_downstream \
  -a rate=0.1

Remove the toxic:

toxiproxy-cli toxic remove redis-proxy -n bandwidth_downstream

What this tests:

Application behavior under sustained degradation
Whether your timeouts are appropriate for typical data sizes
If your circuit breaker is sensitive enough to detect slow failures

Scenario 5: Jitter (Variable Latency)

Real networks don't have consistent latency—they fluctuate. Simulate this with jitter:

Add latency with jitter:

toxiproxy-cli toxic add redis-proxy \
  -t latency \
  -a latency=1000 \
  -a jitter=500

This creates latency ranging from 500ms to 1500ms (1000ms ± 500ms).

Expected Behavior:

Some requests complete quickly
Others timeout randomly
Circuit breaker sees intermittent failures
Tests the circuit breaker's threshold logic

Why this matters:

More realistic than fixed latency
Reveals issues with retry timing
Tests how your system handles sporadic failures

Scenario 6: Slow Close (Graceful Degradation)

Simulate a service slowly dying:

Add slice toxic:

toxiproxy-cli toxic add redis-proxy \
  -t slice \
  -a average_size=64 \
  -a size_variation=32 \
  -a delay=100

This slices data into small chunks with delays between them.

Expected Behavior:

Operations take progressively longer
Eventually exceed timeouts
Allows testing gradual degradation vs sudden failure

Advanced Testing Patterns

Combining Multiple Toxics

You can apply multiple toxics simultaneously to simulate complex scenarios:

# Add both latency and packet loss
toxiproxy-cli toxic add redis-proxy -t latency -a latency=200
toxiproxy-cli toxic add redis-proxy -t slow_close -a delay=100

This simulates a network that's both slow AND unstable.

Automated Testing Script

Create a bash script to run your test scenarios automatically:

#!/bin/bash

echo "Starting Redis circuit breaker tests..."

# Test 1: Complete failure
echo "Test 1: Simulating Redis down"
toxiproxy-cli toggle redis-proxy
sleep 5
curl http://localhost:8080/health
toxiproxy-cli toggle redis-proxy

# Test 2: High latency
echo "Test 2: Adding high latency"
toxiproxy-cli toxic add redis-proxy -t latency -a latency=5000
sleep 5
curl http://localhost:8080/health
toxiproxy-cli toxic remove redis-proxy -n latency_downstream

# Test 3: Reset connections
echo "Test 3: Reset connections"
toxiproxy-cli toxic add redis-proxy -t reset_peer -a timeout=500
sleep 5
curl http://localhost:8080/health
toxiproxy-cli toxic remove redis-proxy -n reset_peer_downstream

echo "Tests complete!"

Key Metrics to Track

Circuit state: closed, open, half-open
Failure count: Number of consecutive failures
Success rate: Percentage of successful requests
Latency percentiles: p50, p95, p99
Request volume: Total requests during the test

Best Practices

1. Test All Failure Modes

Don't just test complete outages. Real production issues include:

Partial failures (some operations succeed, others fail)
Slow failures (latency-induced timeouts)
Intermittent failures (flaky networks)

2. Verify Recovery

Always test that your circuit breaker recovers properly:

# Cause failure
toxiproxy-cli toggle redis-proxy

# Wait for circuit to open
sleep 10

# Restore service
toxiproxy-cli toggle redis-proxy

# Verify recovery
curl http://localhost:8080/health

3. Test Under Load

Run toxics while your service is under load to see realistic behavior:

# Start load test
hey -z 60s -c 10 http://localhost:8080/api/endpoint &

# Inject failure mid-test
sleep 20
toxiproxy-cli toxic add redis-proxy -t latency -a latency=3000

4. Clean Up Between Tests

Always remove toxics and reset state:

# Remove all toxics
toxiproxy-cli toxic remove redis-proxy --all

# Or reset the entire proxy
toxiproxy-cli delete redis-proxy
toxiproxy-cli create redis-proxy -l localhost:6380 -u localhost:6379

Real-World Example

Here's a complete test scenario simulating a production incident:

#!/bin/bash

echo "=== Simulating Production Incident ==="

# Phase 1: Normal operation
echo "Phase 1: Normal operation (30s)"
sleep 30

# Phase 2: Redis starts slowing down
echo "Phase 2: Redis latency increases (1s → 3s)"
toxiproxy-cli toxic add redis-proxy -t latency -a latency=1000
sleep 10
toxiproxy-cli toxic update redis-proxy -n latency_downstream -a latency=3000
sleep 10

# Phase 3: Redis becomes unstable
echo "Phase 3: Connections start resetting"
toxiproxy-cli toxic add redis-proxy -t reset_peer -a timeout=1000
sleep 10

# Phase 4: Complete outage
echo "Phase 4: Redis goes down completely"
toxiproxy-cli toggle redis-proxy
sleep 20

# Phase 5: Redis recovers
echo "Phase 5: Redis comes back online"
toxiproxy-cli toggle redis-proxy
toxiproxy-cli toxic remove redis-proxy --all
sleep 20

echo "=== Incident simulation complete ==="
echo "Check logs and metrics to verify circuit breaker behavior"

Expected outcome:

Phase 1-2: Elevated latency, some timeouts
Phase 3: Circuit might start opening/closing intermittently
Phase 4: Circuit should open and stay open
Phase 5: Circuit should recover to closed state

Conclusion

Testing with Toxiproxy transforms abstract resilience patterns into concrete, verifiable behaviors. By simulating real network failures, you can:

Validate that your circuit breaker opens at the correct threshold
Verify that your application handles failures gracefully
Discover edge cases before they occur in production
Build confidence in your system's resilience

The key is to test not just complete failures, but the spectrum of degradation that happens in production: latency spikes, intermittent errors, bandwidth constraints, and gradual deterioration.

Remember: A circuit breaker that's never tested is just technical debt in disguise.

Resources

Have you used Toxiproxy for testing? What failure scenarios have you discovered that surprised you? Share your experiences in the comments! 💬

LRU vs TinyLFU: Choosing the Right Cache Eviction Strategy 🚀

Akarshan Gandotra — Sat, 03 Jan 2026 06:30:51 +0000

When building high-performance applications, choosing the right cache eviction policy can make or break your system's performance. While both TinyLRU and TinyLFU aim to keep your hot data cached, they take fundamentally different approaches. Understanding when to use each can save you from performance disasters and wasted CPU cycles.

The Fundamentals 📚

What is LRU?

LRU (Least Recently Used) is one of the most intuitive cache eviction policies. The principle is simple: when the cache is full and you need to make room for new data, evict the item that was accessed longest ago.

Think of it like organizing books on a small shelf. Every time you read a book, you put it at the front. When the shelf fills up, you remove the book from the back that you haven't touched in the longest time.

LRU Cache Operations:
┌─────────────────────────┐
│ Get(key)                │
│  └─ Move to front       │
│                         │
│ Set(key, value)         │
│  └─ Add to front        │
│  └─ Evict oldest if full│
└─────────────────────────┘

What is TinyLFU?

TinyLFU (Least Frequently Used with Frequency sketch) takes a different approach. Instead of just tracking when items were accessed, it tracks how often they've been accessed. Before admitting a new item, TinyLFU asks: "Is this item more valuable than what I'd have to evict?"

TinyLFU Cache Operations:
┌─────────────────────────────┐
│ Get(key)                    │
│  └─ Increment frequency     │
│  └─ Return if cached        │
│                             │
│ Set(key, value)             │
│  └─ Compare frequencies     │
│  └─ Admit if worthy         │
│  └─ Reject if not valuable  │
└─────────────────────────────┘

The Critical Difference: Recency vs Frequency 🎯

The fundamental distinction between these algorithms is what they optimize for:

LRU optimizes for:  RECENCY (when was it last used?)
TinyLFU optimizes for: FREQUENCY (how often is it used?)

This difference seems subtle but has massive implications for real-world systems.

Visual Comparison 📊

LRU Behavior

Timeline of Cache Operations:

Initial: [A] ← → [B] ← → [C]
         ↑               ↑
        MRU             LRU

Access B: [B] ← → [A] ← → [C]

Add D (full): [D] ← → [B] ← → [A]
              ↑               ↑
             MRU    Evicted → C

Result: C removed (oldest), regardless of access count

TinyLFU Behavior

Cache State:
Item A: frequency = 500
Item B: frequency = 300  
Item C: frequency = 100 ← LFU victim

New Item D arrives (frequency = 5)
    ↓
Compare: D(5) vs C(100)
    ↓
Decision: REJECT ❌
    ↓
Result: D never enters cache
        C stays (more valuable)

New Item E arrives (frequency = 150)
    ↓
Compare: E(150) vs C(100)
    ↓
Decision: ADMIT ✅
    ↓
Result: E enters cache
        C evicted

The Real-World Problem: JWT Authentication 🔐

Let's examine a production scenario that perfectly illustrates why the choice matters. This example is drawn from real JWT authentication systems handling 100,000 requests per second.

The Setup

You're running an auth gateway. Every incoming request carries a JWT. Verifying that JWT costs 200 microseconds. At 100,000 requests per second, that's 20 full CPU cores doing nothing but crypto.

So you add caching. Problem solved... until it isn't.

Your Traffic Pattern

🔥 Hot Tokens (70% of traffic)
   └─ Long-lived sessions
   └─ Service accounts: 20 req/sec each
   └─ Browser sessions: 50 req/min each
   └─ Thousands of hits per 20-min lifetime

❄️  Cold Tokens (25% of traffic)
   └─ One-off API calls
   └─ Batch jobs with unique tokens
   └─ Each token: 1-2 hits, never again

🎲 Noise Tokens (5% of traffic)
   └─ Scanners
   └─ Malformed requests
   └─ Expired tokens from broken clients
   └─ Pure garbage

The LRU Failure Mode

Time: 3:00 AM
Event: Batch job spawns 1000 workers
Each worker: unique JWT, single request

LRU Decision Tree:
┌─────────────────────────────┐
│ New token arrives           │
│ Cache is full               │
└──────────┬──────────────────┘
           │
           ▼
    ┌──────────────┐
    │ Admit token  │  ← Always admits
    │ Evict oldest │  ← Evicts hot tokens
    └──────────────┘
           │
           ▼
┌─────────────────────────────┐
│ Cache now full of 1000      │
│ tokens that will NEVER      │
│ be seen again               │
└─────────────────────────────┘

Impact:
├─ Service account tokens evicted
├─ Active browser sessions evicted
├─ Hit rate: 92% → 61%
└─ CPU usage doubles

As the author of the TinyLFU JWT authentication analysis notes, this scenario transforms a stable caching system into an unpredictable performance disaster. Hot tokens that should remain cached for their entire 20-minute lifetime get evicted by cold tokens that will never be seen again.

The TinyLFU Solution

Same scenario: 1000 batch job tokens arrive

TinyLFU Decision for Each Token:
┌─────────────────────────────┐
│ New batch token             │
│ Frequency: 1                │
└──────────┬──────────────────┘
           │
           ▼
    ┌──────────────┐
    │ Compare with │
    │ LFU victim   │
    │ (freq: 500)  │
    └──────┬───────┘
           │
           ▼
    ┌──────────────┐
    │ 1 < 500      │
    │ REJECT ❌    │
    └──────────────┘
           │
           ▼
┌─────────────────────────────┐
│ Token never enters cache    │
│ Hot tokens remain cached    │
│ Hit rate stays: 92%         │
└─────────────────────────────┘

TinyLFU's admission gate prevents cache pollution. Batch tokens never achieve sufficient frequency to compete with legitimate hot tokens.

Data Structure Comparison 🏗️

LRU Implementation

Components:
┌─────────────────────────────────┐
│ HashMap: key → node             │
│ (O(1) lookups)                  │
└─────────────────────────────────┘
           │
           ▼
┌─────────────────────────────────┐
│ Doubly Linked List              │
│ (maintains access order)        │
│                                 │
│ [Head] ← → [Node] ← → [Tail]    │
│  (MRU)                   (LRU)  │
└─────────────────────────────────┘

Operations:
Get:  HashMap lookup + move to head  = O(1)
Set:  HashMap insert + add to head   = O(1)
      (evict tail if full)

Memory: ~100 bytes/entry overhead

TinyLFU Implementation

Components:
┌─────────────────────────────────┐
│ Main Cache (LRU)                │
│ 99% of capacity                 │
└─────────────────────────────────┘
           │
           ▼
┌─────────────────────────────────┐
│ Admission Window (tiny LRU)     │
│ 1% of capacity                  │
│ (for cold-start protection)     │
└─────────────────────────────────┘
           │
           ▼
┌─────────────────────────────────┐
│ Count-Min Sketch                │
│ (frequency tracking)            │
│ 10x cache size                  │
│ ~4 bits per counter             │
└─────────────────────────────────┘

Operations:
Get:  Increment sketch + cache lookup    = O(1)
Set:  Compare frequencies + conditional admit = O(1)

Memory: ~500 KB for 100k cache with 1M counters

Count-Min Sketch: The Magic Behind TinyLFU 🎩

The Count-Min Sketch is the secret sauce that makes TinyLFU practical. It's a probabilistic data structure that tracks frequencies for millions of items using only a tiny amount of memory.

The Problem It Solves

Imagine you need to track access frequencies for every JWT token you've ever seen:

Naive Approach:
HashMap: token → counter

Problems:
├─ Need to store every unique token
├─ 1 million tokens × 64 bytes = 64 MB just for keys
├─ Plus counter storage
└─ Total: ~100 MB+ for frequency tracking alone ❌

Count-Min Sketch solves this with a clever tradeoff: instead of exact counts, it provides approximate counts with strong guarantees.

How Count-Min Sketch Works

Think of it as a 2D grid of counters with multiple hash functions:

Count-Min Sketch Structure:

Hash1 →  [ 3][ 1][ 5][ 2][ 8][ 0][ 4]...
Hash2 →  [ 2][ 4][ 1][ 7][ 3][ 1][ 2]...
Hash3 →  [ 1][ 2][ 6][ 3][ 1][ 4][ 5]...
Hash4 →  [ 4][ 1][ 3][ 5][ 2][ 1][ 8]...
         ↑
    Counters (4 bits each)

Dimensions:
├─ Width (w): Number of counters per row
├─ Depth (d): Number of hash functions
└─ Total memory: w × d × 4 bits

Operations Visualized

Incrementing a Token:

Token: "jwt_user_123"
    ↓
Apply 4 hash functions:
├─ Hash1(token) = 42  → Increment row1[42]
├─ Hash2(token) = 157 → Increment row2[157]
├─ Hash3(token) = 891 → Increment row3[891]
└─ Hash4(token) = 23  → Increment row4[23]

Before:
Row1: ...[ 3][ 5]...  (position 42)
Row2: ...[10][ 2]...  (position 157)
Row3: ...[ 7][ 1]...  (position 891)
Row4: ...[ 2][ 8]...  (position 23)

After:
Row1: ...[ 4][ 5]...  ← Incremented
Row2: ...[11][ 2]...  ← Incremented
Row3: ...[ 8][ 1]...  ← Incremented
Row4: ...[ 3][ 8]...  ← Incremented

Time: O(1) - Just 4 increments

Querying Frequency:

Token: "jwt_user_123"
    ↓
Apply same 4 hash functions:
├─ Hash1(token) = 42  → Read row1[42] = 4
├─ Hash2(token) = 157 → Read row2[157] = 11
├─ Hash3(token) = 891 → Read row3[891] = 8
└─ Hash4(token) = 23  → Read row4[23] = 3

Estimated frequency: min(4, 11, 8, 3) = 3 ✅

Why minimum?
└─ Counters can be inflated by collisions
    Take the minimum = most conservative estimate

Time: O(1) - Just 4 lookups + min operation

Why It Works: Hash Collisions

Different tokens can hash to the same positions (collisions), but the Count-Min Sketch handles this elegantly:

Collision Example:

Token A and Token B both hash to position 42 in Row1

Row1[42] stores: countA + countB = 15
    ↓
When querying Token A:
├─ Row1: 15 (includes Token B) ← Overestimate
├─ Row2: 12 (includes Token C) ← Overestimate
├─ Row3: 9  (no collision)     ← True count
└─ Row4: 11 (includes Token D) ← Overestimate

Result: min(15, 12, 9, 11) = 9

Key Property:
├─ Estimates can be HIGHER than true count
├─ Estimates are NEVER LOWER than true count
└─ Using multiple rows reduces overestimation

The Guarantee

Count-Min Sketch provides mathematical guarantees:

True frequency: f
Estimated frequency: f'

Guarantee: f ≤ f' ≤ f + ε × N

Where:
├─ ε (epsilon): Error parameter (e.g., 0.01 = 1%)
├─ N: Total number of items seen
└─ f' never underestimates

Example with ε = 0.01:
├─ True count: 100
├─ Total items: 1,000,000
├─ Max error: 0.01 × 1,000,000 = 10,000
└─ Estimated: 100 to 10,100

For TinyLFU caching decisions, this is perfect because we're making relative comparisons:

Admission Decision:
Token A frequency: ~500 (might be 500-600)
Token B frequency: ~50  (might be 50-100)

Decision: A > B is correct regardless of small errors ✅

Memory Efficiency

Compare memory usage for tracking 1 million tokens:

Exact HashMap:
├─ 1M entries × 64 bytes (key)
├─ 1M entries × 8 bytes (counter)
└─ Total: ~72 MB

Count-Min Sketch (w=10M, d=4):
├─ 10M counters × 4 rows
├─ 40M counters × 4 bits each
├─ 160M bits = 20 MB
└─ 72% memory savings ✅

Even Better - 4-bit Counters:
├─ Each counter: 0 to 15
├─ When counter reaches 15, halve all counters
├─ Maintains relative frequencies
└─ Prevents overflow

Counter Decay in Action

TinyLFU uses periodic decay to keep frequencies fresh:

Before Decay:
Row1: [15][12][ 8][15]...
Row2: [13][15][ 6][ 9]...
Row3: [15][ 7][11][15]...
Row4: [10][15][15][ 4]...

Decay Event (right shift by 1 bit = divide by 2):
Row1: [ 7][ 6][ 4][ 7]...
Row2: [ 6][ 7][ 3][ 4]...
Row3: [ 7][ 3][ 5][ 7]...
Row4: [ 5][ 7][ 7][ 2]...

Benefits:
├─ Recent activity matters more
├─ Old patterns fade away
├─ Prevents counter overflow
└─ Extremely fast (bit shift operation)

Real-World Example: JWT Cache

Scenario: 100k req/sec, tracking 5M unique tokens

Configuration:
├─ Width: 10M counters
├─ Depth: 4 hash functions
├─ Counter size: 4 bits
└─ Total memory: 20 MB

Token Types:
┌─────────────────────────────────────┐
│ Service Account (Token A)          │
│ True frequency: 2000                │
│ Sketch estimate: 2003               │
│ Error: 0.15%                        │
└─────────────────────────────────────┘

┌─────────────────────────────────────┐
│ Batch Job Token (Token B)          │
│ True frequency: 1                   │
│ Sketch estimate: 1                  │
│ Error: 0%                           │
└─────────────────────────────────────┘

┌─────────────────────────────────────┐
│ Random Attack Token (Token C)      │
│ True frequency: 1                   │
│ Sketch estimate: 2 (collision)      │
│ Error: 100% (but still tiny!)       │
└─────────────────────────────────────┘

Admission Decision:
├─ Token A (2003) vs Token B (1) → A wins ✅
├─ Token A (2003) vs Token C (2) → A wins ✅
└─ Small errors don't affect decisions

Why This Matters for TinyLFU

The Count-Min Sketch enables TinyLFU to:

Track millions of items with minimal memory
Make instant decisions with O(1) operations
Handle high throughput without becoming a bottleneck
Adapt to changing patterns through decay
Tolerate approximation because relative comparisons still work

Without Count-Min Sketch, TinyLFU would need exact frequency counters for every item ever seen, making it impractical for high-scale systems. The sketch transforms TinyLFU from a theoretical idea into a production-ready cache algorithm.

The Trade-off:
├─ Give up: Exact counts
├─ Gain: 70-90% memory savings
├─ Guarantee: Never underestimate
└─ Result: Practical frequency-based caching ✅

Attack Resilience 🛡️

LRU Under Attack

Attacker Strategy: 50k random JWTs/sec

LRU Response:
┌─────────────────────────────────┐
│ Random Token 1 → ADMIT          │
│ Random Token 2 → ADMIT          │
│ Random Token 3 → ADMIT          │
│ ... (evicting legitimate tokens)│
│ Random Token 50000 → ADMIT      │
└─────────────────────────────────┘

Result:
├─ Cache fills with garbage
├─ Legitimate tokens evicted
├─ Hit rate collapses
└─ CPU usage spikes

TinyLFU Under Attack

Same Attack: 50k random JWTs/sec

TinyLFU Response:
┌─────────────────────────────────┐
│ Random Token 1 (freq=1) → REJECT│
│ Random Token 2 (freq=1) → REJECT│
│ Random Token 3 (freq=1) → REJECT│
│ ... (legitimate tokens safe)    │
│ Random Token 50000 → REJECT     │
└─────────────────────────────────┘

Meanwhile:
├─ Legitimate Token A (freq=2000) → stays cached
├─ Service Account B (freq=1500) → stays cached
└─ Hit rate remains: 92%

As demonstrated in the JWT authentication case study, the attack actually makes TinyLFU's cache more resilient because random tokens can never achieve the frequency needed to displace legitimate hot tokens.

Configuration Deep Dive ⚙️

LRU Configuration

Simple Parameters:

cache := NewLRU(100000)  // Just capacity

Optional:
├─ TTL per entry
├─ Expiration callback
└─ Memory limit (bytes)

Rule of Thumb:
Size = ActiveUsers × AverageSessionRequests × SafetyMargin
     = 10,000 × 50 × 2
     = 1,000,000 entries

TinyLFU Configuration

Critical Parameters:

cache := NewTinyLFU(TinyLFUConfig{
    MaxSize:      100000,    // Main cache capacity
    NumCounters:  1000000,   // Frequency sketch size (10x)
    BufferItems:  1000,      // Admission window (1%)
})

Parameter Breakdown:

NumCounters:
├─ Rule: 10x cache size
├─ Why: Reduces hash collisions
├─ Memory: counters × 4 bits
└─ Example: 1M counters = 500 KB

BufferItems:
├─ Rule: 1% of MaxSize
├─ Purpose: Cold-start protection
├─ Behavior: New items land here first
└─ Graduation: Build frequency → main cache

MaxCost (optional):
├─ For variable-sized entries
├─ Memory-bounded eviction
└─ Example: 50 MB budget for mixed sizes

Conclusion 🎯

Both TinyLRU and TinyLFU have their place in modern systems:

TinyLRU is your default choice. It's simple, fast, and works well for most caching scenarios. Use it when recency correlates with value in your workload.

TinyLFU is your specialist tool. It shines when you have frequency-skewed workloads, face burst traffic, or need resilience against cache pollution. The JWT authentication scenario demonstrates how critical this becomes at scale where poor cache decisions directly translate to wasted CPU cores and degraded latency.

The right choice depends on understanding your access patterns. Monitor your cache behavior, measure hit rates under realistic load, and choose the algorithm that matches your workload characteristics.

Further Reading:

📄 TinyLFU: A Highly Efficient Cache Admission Policy
🔐 TinyLFU in JWT Authentication Systems
🔧 Ristretto: A fast, concurrent TinyLFU cache
📚 golang-lru: Simple LRU implementation
📊 Count-Min Sketch: The probabilistic data structure powering TinyLFU

Choose wisely, cache efficiently, and may your hit rates stay high! 🚀

TinyLFU: Why Your JWT Auth Cache Needs Better Eviction 🔐

Akarshan Gandotra — Mon, 22 Dec 2025 05:29:20 +0000

You're running an auth gateway. Every incoming request carries a JWT. Verifying that JWT checking the signature, validating claims, maybe hitting a revocation list costs you 200 microseconds. At 100,000 requests per second, that's 20 full CPU cores doing nothing but crypto. 💸

So you cache verified tokens. Problem solved. ✅

Except three weeks later, your P99 latency spikes during a product launch. Cache hit rate drops from 92% to 61%. CPU usage doubles. You scale horizontally, and the problem... gets worse. 📉

This isn't a configuration issue. It's not a bug. It's LRU doing exactly what it was designed to do, which happens to be the wrong thing for your workload.

The Problem With LRU in Auth Systems 🚨

LRU (Least Recently Used) evicts whatever you touched longest ago. In theory, this keeps "hot" data in cache. In practice, at high throughput with diverse traffic patterns, LRU makes decisions that actively hurt you.

Here's why: your JWT traffic isn't uniform. You have three categories of tokens hitting your gateway:

Your Token Traffic Breakdown 📊

🔥 Hot Tokens (70% of traffic)
   └─ Long-lived sessions
   └─ Browsers, mobile apps, service accounts
   └─ Single user = 50 req/min
   └─ Service account = 20 req/sec
   └─ Thousands of hits per 20-min lifetime

❄️  Cold Tokens (25% of traffic)
   └─ One-off API calls
   └─ Scheduled jobs
   └─ Infrequent users
   └─ Each token appears 1-2 times, never again

🎲 Random Tokens (5% of traffic)
   └─ Scanners
   └─ Malformed requests
   └─ Expired tokens from broken clients
   └─ Attackers probing endpoints
   └─ Pure noise

The LRU Failure Mode:

Time: 3:00 AM
Event: Batch job spawns 1000 workers
Each worker: unique JWT, single request

LRU Decision Tree:
┌─────────────────────────────┐
│ New token arrives           │
│ Cache is full              │
└──────────┬──────────────────┘
           │
           ▼
    ┌──────────────┐
    │ Admit token  │  ← Always admits
    │ Evict oldest │  ← Evicts your hot tokens
    └──────────────┘
           │
           ▼
┌─────────────────────────────┐
│ Result: Cache now full of   │
│ 1000 tokens that will       │
│ NEVER be seen again         │
└─────────────────────────────┘

Now your hot tokens miss the cache. You're verifying the same legitimate user tokens over and over, burning CPU, while your cache is full of one-time tokens that will never be seen again.

Your hit rate collapses. Your latency spikes. And because this happens gradually as the cache churns, you don't see a clean error you see slow, expensive drift. 🐌

The Core Problem 🎯

LRU optimizes for recency, but recency is the wrong signal when traffic has massive frequency skew.

What TinyLFU Actually Changes 🔄

TinyLFU splits caching into two distinct decisions: admission and eviction.

LRU vs TinyLFU Flow

Traditional LRU:

Token arrives → Miss → ALWAYS admit → Evict something old
                                           ↓
                                    No questions asked

TinyLFU:

Token arrives → Miss → Is this token valuable enough?
                              ├─ Yes → Admit & evict
                              └─ No  → Reject, verify once, move on

TinyLFU adds a gate. When a JWT arrives and misses the cache, TinyLFU asks:

"Is this token more valuable than what I'd have to evict?"

If the answer is no, the token doesn't enter the cache at all. You verify it, return the response, and move on.

The Signal Shift 📡

LRU measures: Recency (when was it last accessed?)
TinyLFU measures: Frequency (how often has it been accessed?)

For auth workloads, frequency is the correct signal:

Token A: 5000 accesses  >  Token B: 1 access
    ↑                           ↑
  More valuable              Less valuable

(regardless of which one you saw most recently)

TinyLFU Without Jargon 🧮

TinyLFU keeps approximate frequency counts for every item it's seen recently. Not exact counts approximate. This matters for performance and memory.

The Admission Algorithm

JWT arrives
    ↓
1. Increment counter for this token
    ↓
2. Already in cache? → HIT ✅ (done)
    ↓
3. Not in cache? → MISS
    ↓
4. Compare frequencies:
   ┌─────────────────────────────────────┐
   │ New token frequency: 3              │
   │ Victim token frequency: 500         │
   │                                     │
   │ Decision: REJECT (3 < 500)          │
   │ Action: Verify once, don't cache    │
   └─────────────────────────────────────┘

   OR

   ┌─────────────────────────────────────┐
   │ New token frequency: 520            │
   │ Victim token frequency: 480         │
   │                                     │
   │ Decision: ADMIT (520 > 480)         │
   │ Action: Cache new, evict victim     │
   └─────────────────────────────────────┘

Counter Decay ⏰

Counters decay over time to keep frequency reflecting recent history:

Counter Values Before Decay:
Token A: 1000
Token B: 500
Token C: 200

[Decay event: halve all counters]

Counter Values After Decay:
Token A: 500
Token B: 250
Token C: 100

Why this matters: A token that was hot yesterday but hasn't been seen in an hour shouldn't still dominate. Decay is fast (just bit-shifting memory) and keeps the frequency signal fresh.

At 100k TPS, "recent history" typically means the last few seconds to a minute. That's enough to distinguish genuinely hot tokens from noise, but short enough that stale patterns don't stick around.

Applying TinyLFU to a JWT Cache 🎯

Let's trace what happens to each category of token:

🔥 Hot Tokens (Browsers, Service Accounts)

Timeline of a Browser Token:

Request #1  → Miss, frequency=1   → Verify, reject admission
Request #2  → Miss, frequency=2   → Verify, reject admission
Request #3  → Miss, frequency=3   → Verify, reject admission
...
Request #8  → Miss, frequency=8   → Verify, WIN admission ✅
Request #9  → HIT ✅
Request #10 → HIT ✅
Request #50 → HIT ✅ (frequency=50+)
...
Request #3000 → HIT ✅ (frequency=3000+)

Result: Cached for entire 20-min lifetime

Hot tokens naturally win admission after a few misses and stay cached because every access keeps their frequency elevated. 🎉

❄️ Cold Tokens (One-off Requests)

Timeline of a Batch Job Token:

Request #1 → Miss, frequency=1 → Verify
             ↓
         Compare for admission:
         New token frequency: 1
         Victim frequency: 500
         Decision: REJECT ❌
             ↓
         Cache unchanged
             ↓
         Token never seen again

Result: Never pollutes the cache

Cold tokens never build enough frequency to compete. They bounce off the admission gate. ✋

🎲 Random/Attack Tokens

Attacker floods 50k requests/sec with random JWTs:

Random Token 1: frequency=1 → REJECT ❌
Random Token 2: frequency=1 → REJECT ❌
Random Token 3: frequency=1 → REJECT ❌
...
Random Token 50000: frequency=1 → REJECT ❌

Meanwhile:
Legitimate Token A: frequency=2000 → ADMIT ✅ (stays cached)

Attackers can't displace your hot tokens because random tokens never achieve sufficient frequency. The attack makes your cache MORE resilient, not less. 🛡️

The Key Insight 💡

TinyLFU doesn't need to be told what's hot. It infers value from access patterns:

Service account polling every 100ms → naturally high frequency → stays cached
Batch job with 1000 one-time tokens → 1000 low-frequency items → never admitted

The policy emerges from the data.

Admission Decisions in Practice ⚖️

Scenario: Cache holds 100k tokens. It's full. New JWT arrives.

┌─────────────────────────────────────────┐
│ New JWT arrives (miss)                  │
│ Token: eyJhbGc...                       │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│ Lookup frequency                        │
│ New token: 3 hits                       │
│ LFU victim (least frequent cached): 480 │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│ 3 < 480                                 │
│ Decision: REJECT ❌                     │
│ Action: Verify token, return response   │
│         Cache remains unchanged          │
└─────────────────────────────────────────┘

Another token arrives:

┌─────────────────────────────────────────┐
│ New JWT arrives (miss)                  │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│ Lookup frequency                        │
│ New token: 520 hits                     │
│ LFU victim: 480                         │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│ 520 > 480                               │
│ Decision: ADMIT ✅                      │
│ Action: Evict victim, cache new token   │
└─────────────────────────────────────────┘

This comparison happens on every miss. It's fast TinyLFU uses a probabilistic data structure (Count-Min Sketch) to track frequencies in O(1) time with tiny memory footprint.

Why This Matters Under Attack 🛡️

Attack scenario: Adversary sends 50k req/sec with random JWTs

Cache Type	Result
LRU	❌ Random tokens churn cache, evict legitimate tokens, cache hit rate collapses
TinyLFU	✅ Each random token has frequency=1, legitimate tokens have frequency=100s-1000s, random tokens never win admission, cache remains stable

Burst scenario: Batch job spawns 5000 workers at midnight

Cache Type	Result
LRU	❌ 5000 one-time tokens flush out service accounts and browser sessions
TinyLFU	✅ Batch tokens never achieve sufficient frequency, hot tokens remain cached

Why Configuration Matters ⚙️

TinyLFU has a few knobs. They're not arbitrary, they map directly to your workload.

🎛️ NumCounters

What it controls: Size of the frequency sketch

Rule of thumb: 10x your cache size

Example:

Cache capacity: 100k tokens
NumCounters: 1 million

Why?
├─ Sketch uses hash functions to map tokens → counters
├─ Too few counters = frequent collisions
│  └─ Different tokens share same counter
│  └─ Frequency estimates degrade
└─ Enough counters = rare collisions = accurate tracking

Memory cost: 1M counters × 4 bits = 500 KB ✅

At 100k TPS with 20-min token lifetimes:

You might see 5 million distinct tokens in an hour
Using 1-2M counters gives low collision rates
Total memory cost: ~1 MB (cheap!)

🎛️ BufferItems

What it is: TinyLFU's admission buffer (small LRU window)

Typical size: 1% of total cache size

Why it exists: Protects against cold-start problem

Problem without buffer:
New token arrives → frequency=0 → loses every admission battle

Solution with buffer:
New token arrives → lands in admission buffer
                 → builds frequency over next few hits
                 → graduates to main cache once proven

Critical for bursts:

1000 browser clients log in simultaneously
    ↓
Their tokens start cold (frequency=0)
    ↓
Buffer holds them while they accumulate hits
    ↓
Once they've proven frequency → graduate to main cache

🎛️ MaxCost

What it controls: Memory-bounded eviction

When it matters: Variable-sized cache entries

Scenario 1: Uniform token size
├─ Most JWTs: ~500 bytes
├─ MaxCost = 100k tokens × 500 bytes = 50 MB
└─ Simple: count-based eviction works fine

Scenario 2: Variable token size + metadata
├─ Small JWT: 300 bytes
├─ JWT + user roles: 2 KB
├─ JWT + org context: 10 KB
├─ MaxCost = 50 MB total budget
└─ Cost-based eviction keeps memory bounded
    (might cache 100k small tokens OR 5k large ones)

For most auth caches, token size is uniform, so MaxCost is just token_count × avg_size. But if you're caching metadata with each token, costs matter.

What TinyLFU Does Not Guarantee ⚠️

❌ Eventual Consistency of `Set`

Your code:
cache.Set("token123", verifiedJWT)
value := cache.Get("token123")  // Might be nil!

Why? Admission can fail.

TinyLFU might drop your Set on the floor if admission fails. This is by design a tradeoff for resilience.

In practice, this doesn't matter for auth:

JWTs are ephemeral
Verification is deterministic
If a token misses cache → verify it → return response
User doesn't know the difference
Over a few seconds, hot tokens accumulate frequency and stabilize

The system converges to the right steady state. ✅

📊 Approximate Counts

Count-Min Sketch uses hash collisions:

Counts can be overestimated
Counts are never underestimated

True frequency: 100
Reported frequency: 105 or 110 (due to collision)

Impact: None
└─ You're making relative comparisons
└─ Error is small and consistent
└─ Doesn't affect admission quality

⏰ No Explicit TTL Tracking

TinyLFU doesn't track TTLs that happens independently.

Your auth cache setup:
├─ TinyLFU: controls admission + frequency-based eviction
└─ TTL: handles expiration (20 min = JWT lifetime)

Result:
├─ Tokens naturally age out via TTL
└─ TinyLFU ensures what stays cached is worth keeping

How to tell if you need it:

Have you ever seen your cache hit rate drop during traffic bursts, despite having enough memory? That's LRU churning under cold traffic. TinyLFU fixes it. 🔧

Final Mental Model 🧠

Think of TinyLFU as a nightclub bouncer. 🕴️

LRU Bouncer 👎

Policy: Let everyone in, kick out whoever's been 
        standing around longest

Bus of tourists arrives
    ↓
All tourists get in
    ↓
Regulars get kicked out to make room
    ↓
Tourists leave after 5 minutes
    ↓
Regulars are outside
    ↓
Club full of strangers who don't want to be there

TinyLFU Bouncer 👍

Policy: Check how many times you've visited this month
        Regulars get in. Tourists stay out.

Bus of tourists arrives
    ↓
Bouncer checks visit history:
├─ Regulars: 50+ visits → Come in ✅
└─ Tourists: 0 visits → Sorry, no room ❌
    ↓
Even if there's technically space, letting tourists 
in means kicking out regulars
    ↓
Club stays full of people who actually want to be there

For a JWT Auth Cache 🔐

🔥 "Regulars" = Active user sessions, service accounts
❄️  "Tourists" = One-off tokens, batch jobs, scanner garbage

TinyLFU keeps regulars in and tourists out.

The result:

✅ Stable hit rates
✅ Predictable CPU usage
✅ Cache does what you wanted: keep hot stuff hot, let cold stuff stay cold

TL;DR 📝

TinyLFU isn't magic. It's a policy that matches the reality of how traffic actually behaves.

LRU:     Designed for workloads where recency = value
TinyLFU: Designed for workloads where frequency = value

In modern high-throughput systems, frequency equals value. TinyLFU makes that tradeoff explicit, and your cache gets better as a result.

If you're running an auth service at scale, it's worth the switch. 🚀

Further Reading:

📄 Original TinyLFU paper: "TinyLFU: A Highly Efficient Cache Admission Policy"
🔧 Popular implementation: github.com/dgraph-io/ristretto
📊 Count-Min Sketch: The probabilistic data structure behind frequency tracking

The Delete Button Dilemma: When to Soft Delete vs Hard Delete

Akarshan Gandotra — Mon, 24 Nov 2025 07:33:15 +0000

Every developer has been there. You're building a feature, the user clicks "delete," and you pause. Should this record actually disappear from the database, or should you just mark it as deleted? It seems like a simple question, but get it wrong and you'll either compromise user privacy, break your analytics, or wake up to a panicked Slack message about accidentally deleted data that's gone forever.

Let me save you from those 2 AM emergencies.

The Tale of Two Deletes

Soft delete means you're not really deleting anything. You're marking a record with a deleted_at timestamp or an is_deleted flag, then hiding it from normal queries. The data stays in your database, living its best ghost life.

Hard delete is the real deal. Execute that SQL DELETE statement and the record is gone. Unless you've got backups (and please tell me you have backups), there's no coming back from this.

When Soft Deletes Save Your Day

Here’s the thing. In many systems, deleting data feels straightforward. Something’s no longer needed, so you remove it permanently. Clean and simple.

Until it isn’t.

Picture a company that manages a product catalog. Users can delete products that are discontinued. The team implemented hard delete because it felt efficient.

Fast forward a few months. The analytics team tries to generate annual revenue reports. Suddenly, every past order linked to a deleted product starts showing up as "Unknown." Historical reports break. Finance panics. Leadership demands answers. The engineering team scrambles to restore data from backups, patch broken queries, and rebuild what used to work.

All because they chose hard delete without thinking about long-term consequences.

Soft delete would have preserved history, maintained referential integrity, and saved everyone a lot of pain.

What this really means is: sometimes the data you think you’ll never need again becomes the data you desperately need later. Soft delete keeps the door open. Hard delete slams it shut.

Use soft deletes when:

1. Money is Involved

Anything touching financial transactions should probably stick around. Tax audits don't care that you deleted that invoice three years ago, they want to see it. Orders, invoices, payment records, subscription history... soft delete all of it. Your accountant will thank you.

2. You Need to Tell a Story

Analytics and business intelligence thrive on historical data. If you're trying to understand why customer churn increased last quarter, you need to see those deleted accounts. If you're analyzing which products perform best, you need the full lifecycle (including when they were discontinued).

3. Relationships Matter

Your database is a web of relationships. When a user deletes their account, you've got comments, posts, orders, reviews, and a dozen other records pointing to that user. Hard delete the user and suddenly you've got orphaned records everywhere.

Soft deletes preserve referential integrity. The data relationships stay intact, your foreign keys don't break, and you can still run joins without your queries exploding.

4. Oops, Can I Have That Back?

Users delete things by accident. Constantly. If you've ever worked in customer support, you know the pain of "I deleted my account/post/file by mistake, can you restore it?"

With soft deletes, you're a hero who recovers their data in seconds. With hard deletes, you're digging through backups (if you're lucky) or delivering bad news (if you're not).

When Hard Deletes Are Non-Negotiable

But here's the thing: soft deletes aren't always the answer. Sometimes they're actually the wrong choice legally, ethically, and practically.

1. Privacy Laws Aren't Suggestions

Let’s break it down. There are situations where soft delete is absolutely the wrong choice, and privacy regulations are at the top of that list.

Consider global data protection laws like GDPR in Europe or CCPA in California. They include a legally enforceable right to erasure. When someone requests their personal data to be removed, regulators expect that data to be truly gone (not just flagged as inactive in a table).

Some organizations assumed a soft delete was enough and kept the data sitting quietly in the database. Then came audits. Regulators discovered that “deleted” records were still accessible internally. The penalties that followed were painful, both financially and reputationally.

The lesson here: compliance isn’t optional, and soft delete won’t save you when the law requires complete removal. When privacy is on the line, hard delete is the only correct answer.

2. Security Isn't Negotiable

Authentication tokens, session data, password reset codes this stuff needs to actually disappear. Soft deleting a compromised password or an expired auth token is a security vulnerability waiting to happen.

Same goes for credit card data. PCI-DSS compliance requires that you don't store card data longer than necessary. "Soft deleted" card numbers are still stored card numbers. That's a compliance violation and a security risk.

3. Your Database Isn't Infinite

Early-stage startups often soft delete everything because it's easier and safer. Fast forward two years: their database is 80% soft-deleted records, queries are slow, backups take forever, and storage costs are through the roof.

4. Some Data Is Just Temporary

Shopping cart items, temporary file uploads, notification read status this is ephemeral data. It exists to serve an immediate purpose, then it's done. There's no business value in keeping soft-deleted shopping cart items from three years ago. Just actually delete them.

The Hybrid Approach: Best of Both Worlds

Here's what I recommend for most applications: two-stage deletion.

When a user clicks delete:

Soft delete immediately - Mark it as deleted, hide it from the UI
Grace period - Keep it recoverable for 30-90 days
Hard delete automatically - Purge old soft-deletes with a scheduled job

This gives you:

Instant recovery for accidental deletions
Time for users to change their minds
Analytics on recently deleted items
Eventual compliance with privacy laws
Controlled database growth

We implemented this for a social media platform. Users could delete posts, which were soft deleted for 30 days. During that period, they could undo the deletion. After 30 days, a nightly job permanently removed posts deleted more than 30 days ago. Accidental deletion complaints dropped to nearly zero, and we stayed compliant with privacy regulations.

A Real Decision Tree

Implementation Tips That'll Save You Headaches

If you're implementing soft deletes:

Use timestamps, not booleans. deleted_at is better than is_deleted because you automatically capture when deletion happened. NULL means active, timestamp means deleted. Simple.

Index that column. You'll be filtering on it constantly. WHERE deleted_at IS NULL should be fast.

Create a database view for active records. Instead of every query needing WHERE deleted_at IS NULL, create a view or base filter that filters it automatically.

Be consistent. If you're soft deleting users, soft delete everything related to users. Mixing approaches in related tables is a recipe for confusion and bugs.

Don't forget to cascade. When soft deleting a parent record, decide whether children should also be soft deleted. An active comment on a deleted post creates awkward edge cases.

The Bottom Line

There's no one-size-fits-all answer. Soft deletes are powerful for business data, analytics, and preventing mistakes. Hard deletes are essential for privacy, security, and keeping your database lean.

Most applications need both. The key is being deliberate about which deletion strategy serves each type of data best.

And whatever you choose, document it. Future you (or the developer who inherits your code) will want to know why customer accounts are soft deleted but session tokens are hard deleted.

Trust me on this one. I've been the developer digging through undocumented deletion logic at midnight, trying to figure out why some data disappeared and other data didn't. Don't be the person who creates that mystery.

What's your deletion strategy? Have you been burned by choosing the wrong one? I'd love to hear your war stories in the comments.

Implementing Multitenancy in Auth0 Using Organizations: A Complete Guide

Akarshan Gandotra — Wed, 12 Nov 2025 10:18:42 +0000

Introduction

What is Multitenancy?

Multitenancy is an architectural pattern where a single instance of software serves multiple customers (tenants). Each tenant's data and configuration remain isolated from others, creating the illusion of having their own dedicated system. For SaaS applications, multitenancy is essential for scalability, cost-efficiency, and maintaining separate customer environments.

The Challenge

When building modern SaaS applications, you often face complex authentication requirements:

Tenant Isolation: Different organizations need separate user pools and authentication policies
Flexible Authentication: Each organization may require different login methods (social, enterprise SSO, username/password)
Branding: Organizations want customized login experiences reflecting their brand
Security: Strict isolation between tenants to prevent data leakage
Scalability: Managing hundreds or thousands of organizations without infrastructure overhead

Why Auth0 Organizations?

Auth0 Organizations provide a native solution for multitenancy within a single Auth0 tenant. Instead of creating multiple Auth0 tenants (expensive and hard to manage) or building custom isolation logic, Organizations offer:

Built-in tenant isolation with proper security boundaries
Flexible connection management per organization
Simplified user management with organization membership
Customizable branding for each organization
Single infrastructure to manage, reducing operational complexity

Architecture Overview

Auth0 Organizations create logical groupings within your Auth0 tenant. Each organization acts as an isolated container for users and authentication methods, while sharing the underlying Auth0 infrastructure.

High-Level Architecture

Key Components

Auth0 Tenant: Your main Auth0 account
Organizations: Logical containers representing each customer/tenant
Connections: Authentication methods (database, social, enterprise) available to organizations
Users: Individuals who belong to one or more organizations
Applications: Your SaaS app(s) that authenticate users

Benefits Over Alternative Approaches

Approach	Pros	Cons
Multiple Auth0 Tenants	Complete isolation	Expensive, complex management, no shared infrastructure
Single Tenant + Custom Logic	Full control	Security risks, maintenance burden, reinventing the wheel
Auth0 Organizations	Native isolation, scalable, flexible, cost-effective	Requires proper implementation, learning curve

Implementation Steps

Step 1: Creating Machine-to-Machine (M2M) Application

Machine-to-Machine applications enable your backend services to interact with Auth0 Management API programmatically. This is essential for automating organization creation, user management, and configuration.

What are M2M Applications?

M2M applications use the OAuth 2.0 Client Credentials flow to obtain access tokens without user interaction. They authenticate using a Client ID and Client Secret, making them perfect for server-to-server communication.

Creating an M2M Application

Via Auth0 Dashboard:

Navigate to Auth0 Dashboard → Applications → Applications
Click Create Application
Enter application name (e.g., "Organization Management Service")
Select Machine to Machine Applications
Choose the Auth0 Management API as the authorized API
Select required scopes:
- read:organizations
- create:organizations
- update:organizations
- delete:organizations
- read:organization_members
- create:organization_members
- read:organization_connections
- create:organization_connections
- update:organization_connections
- read:connections
Click Authorize

Obtaining Credentials:

After creation, navigate to the application's Settings tab to find:

Client ID: Public identifier for your M2M application
Client Secret: Secret key (keep this secure!)
Domain: Your Auth0 domain (e.g., your-tenant.auth0.com)

M2M Authentication Flow

sequenceDiagram
    participant App as Your Backend Service
    participant Auth0 as Auth0 Token Endpoint
    participant API as Auth0 Management API

    App->>Auth0: POST /oauth/token<br/>(client_id, client_secret, audience)
    Auth0->>Auth0: Validate credentials
    Auth0-->>App: Access Token (JWT)
    App->>API: API Request<br/>(Authorization: Bearer {token})
    API->>API: Validate token
    API-->>App: API Response

Example: Authenticating with M2M Credentials

Python:

import requests
import os

def get_management_api_token():
    """
    Obtain an access token for Auth0 Management API
    """
    auth0_domain = os.getenv('AUTH0_DOMAIN')
    client_id = os.getenv('AUTH0_M2M_CLIENT_ID')
    client_secret = os.getenv('AUTH0_M2M_CLIENT_SECRET')

    token_url = f'https://{auth0_domain}/oauth/token'

    payload = {
        'client_id': client_id,
        'client_secret': client_secret,
        'audience': f'https://{auth0_domain}/api/v2/',
        'grant_type': 'client_credentials'
    }

    response = requests.post(token_url, json=payload)
    response.raise_for_status()

    return response.json()['access_token']

# Usage
token = get_management_api_token()
print(f"Access token obtained: {token[:20]}...")

Step 2: Creating Organizations and Linking Login Methods

Organizations are the core entities that represent your tenants. Each organization can have its own set of enabled authentication connections, branding, and metadata.

Understanding Auth0 Organizations

An organization in Auth0 represents a group of users who share common access to applications. Key properties include:

Organization ID: Unique identifier (e.g., org_abc123xyz)
Display Name: Human-readable name shown to users
Enabled Connections: Authentication methods available for this organization
Metadata: Custom key-value pairs for your application logic
Branding: Colors, logos, and styling for login pages

Creating Organizations

Via Auth0 Dashboard (Manual):

Navigate to Organizations in the Auth0 Dashboard
Click Create Organization
Enter Organization Name and Display Name
(Optional) Add metadata
Click Create

Via Management API (Programmatic):

import requests

def create_organization(token, name, display_name, metadata=None):
    """
    Create a new organization in Auth0
    """
    auth0_domain = os.getenv('AUTH0_DOMAIN')
    url = f'https://{auth0_domain}/api/v2/organizations'

    headers = {
        'Authorization': f'Bearer {token}',
        'Content-Type': 'application/json'
    }

    payload = {
        'name': name,  # Must be unique, URL-friendly
        'display_name': display_name,
        'metadata': metadata or {}
    }

    response = requests.post(url, json=payload, headers=headers)
    response.raise_for_status()

    org = response.json()
    print(f"Organization created: {org['id']}")
    return org

# Usage
token = get_management_api_token()
org = create_organization(
    token,
    name='acme-corp',
    display_name='Acme Corporation',
    metadata={
        'plan': 'enterprise',
        'industry': 'technology'
    }
)

Linking Authentication Connections

Connections define how users authenticate. Auth0 supports:

Database Connections: Username/password authentication
Social Connections: Google, Microsoft, Facebook, LinkedIn, etc.
Enterprise Connections: SAML, OIDC, ADFS, Azure AD, etc.

Enabling Connections for an Organization:

def enable_connection_for_org(token, org_id, connection_id, assign_membership_on_login=True):
    """
    Enable an authentication connection for an organization
    """
    auth0_domain = os.getenv('AUTH0_DOMAIN')
    url = f'https://{auth0_domain}/api/v2/organizations/{org_id}/enabled_connections'

    headers = {
        'Authorization': f'Bearer {token}',
        'Content-Type': 'application/json'
    }

    payload = {
        'connection_id': connection_id,
        'assign_membership_on_login': assign_membership_on_login
    }

    response = requests.post(url, json=payload, headers=headers)
    response.raise_for_status()

    return response.json()

# Enable Google Social connection
enable_connection_for_org(token, 'org_abc123xyz', 'con_google123')

# Enable Database connection
enable_connection_for_org(token, 'org_abc123xyz', 'con_database456')

Getting Available Connections:

def get_connections(token):
    """
    Retrieve all available connections
    """
    auth0_domain = os.getenv('AUTH0_DOMAIN')
    url = f'https://{auth0_domain}/api/v2/connections'

    headers = {
        'Authorization': f'Bearer {token}'
    }

    response = requests.get(url, headers=headers)
    response.raise_for_status()

    connections = response.json()
    for conn in connections:
        print(f"Connection: {conn['name']} (ID: {conn['id']}, Strategy: {conn['strategy']})")

    return connections

Step 3: User Authentication with Organization Context

The final step is directing users to authenticate within the context of their organization. This ensures they only see login methods available to their organization and that tokens include organization information.

The /authorize Endpoint

Auth0's /authorize endpoint initiates the authentication flow. When you include the organization parameter, Auth0:

Filters available login methods to only those enabled for that organization
Includes organization claims in the returned tokens
Applies organization-specific branding (if configured)

Building the Authentication URL

Required Parameters:

organization: The organization ID (e.g., org_abc123xyz)
client_id: Your application's Client ID
redirect_uri: Where Auth0 redirects after authentication
response_type: Typically code for authorization code flow
scope: Requested scopes (e.g., openid profile email)

Example Authentication URL:

https://your-tenant.auth0.com/authorize
  ?organization=org_abc123xyz
  &client_id=your_app_client_id
  &redirect_uri=https://yourapp.com/callback
  &response_type=code
  &scope=openid profile email
  &state=random_state_value

Complete Authentication Flow

Implementation Example

Python (Flask):

from flask import Flask, redirect, request, session, url_for
from authlib.integrations.flask_client import OAuth
import os

app = Flask(__name__)
app.secret_key = os.getenv('FLASK_SECRET_KEY')

oauth = OAuth(app)

auth0 = oauth.register(
    'auth0',
    client_id=os.getenv('AUTH0_CLIENT_ID'),
    client_secret=os.getenv('AUTH0_CLIENT_SECRET'),
    api_base_url=f'https://{os.getenv("AUTH0_DOMAIN")}',
    access_token_url=f'https://{os.getenv("AUTH0_DOMAIN")}/oauth/token',
    authorize_url=f'https://{os.getenv("AUTH0_DOMAIN")}/authorize',
    client_kwargs={'scope': 'openid profile email'}
)

@app.route('/login/<org_id>')
def login(org_id):
    """
    Initiate login for a specific organization
    """
    # Store org_id in session for validation later
    session['expected_org_id'] = org_id

    return auth0.authorize_redirect(
        redirect_uri=url_for('callback', _external=True),
        organization=org_id
    )

@app.route('/callback')
def callback():
    """
    Handle the Auth0 callback
    """
    # Exchange authorization code for tokens
    token = auth0.authorize_access_token()

    # Get user info from ID token
    user_info = token.get('userinfo')

    # Validate organization
    org_id = user_info.get('org_id')
    expected_org_id = session.get('expected_org_id')

    if org_id != expected_org_id:
        return "Organization mismatch!", 403

    # Store user session
    session['user'] = {
        'email': user_info.get('email'),
        'name': user_info.get('name'),
        'org_id': org_id
    }

    return redirect('/dashboard')

@app.route('/dashboard')
def dashboard():
    """
    Protected route - requires authentication
    """
    if 'user' not in session:
        return redirect(url_for('login', org_id='default'))

    user = session['user']
    return f"Welcome {user['name']} from organization {user['org_id']}!"

Extracting Organization Information from Tokens

After successful authentication, the ID token will contain organization claims:

{
  "sub": "auth0|123456789",
  "email": "user@example.com",
  "name": "John Doe",
  "org_id": "org_abc123xyz",
  "org_name": "Acme Corporation",
  "iat": 1635360000,
  "exp": 1635363600
}

Token Validation:

import jwt
from jwt import PyJWKClient

def validate_token(token, expected_org_id):
    """
    Validate JWT token and verify organization claim
    """
    auth0_domain = os.getenv('AUTH0_DOMAIN')
    jwks_url = f'https://{auth0_domain}/.well-known/jwks.json'

    # Get signing key
    jwks_client = PyJWKClient(jwks_url)
    signing_key = jwks_client.get_signing_key_from_jwt(token)

    # Decode and verify token
    payload = jwt.decode(
        token,
        signing_key.key,
        algorithms=['RS256'],
        audience=os.getenv('AUTH0_CLIENT_ID'),
        issuer=f'https://{auth0_domain}/'
    )

    # Verify organization claim
    if payload.get('org_id') != expected_org_id:
        raise ValueError('Organization ID mismatch')

    return payload

Additional Resources

Auth0 Documentation: https://auth0.com/docs/manage-users/organizations
Management API Reference: https://auth0.com/docs/api/management/v2/organizations
Auth0 Community: https://community.auth0.com
GitHub Examples: Search for "auth0-organizations" for community examples

Questions or need help? Join the Auth0 community forums or reach out to Auth0 support. Share your implementation experiences and help others building multitenant applications!

FastAPI: request.state vs Context Variables - When to Use What? 🚀

Akarshan Gandotra — Wed, 30 Jul 2025 17:52:28 +0000

When building FastAPI applications, you'll often need to share data across different parts of your request lifecycle i.e. middleware, dependencies, and route handlers. FastAPI provides two primary mechanisms for this: request.state and context variables (contextvars). But when should you use each one?

Understanding request.state 📦

request.state is a simple namespace object attached to each FastAPI request. It's designed to store arbitrary data that needs to be shared during the processing of a single request.

Basic Usage 🔧

from fastapi import FastAPI, Request, Depends

app = FastAPI()

@app.middleware("http")
async def add_user_info(request: Request, call_next):
    # Store data in request.state
    request.state.user_id = "user_123"
    request.state.request_time = time.time()

    response = await call_next(request)
    return response

@app.get("/profile")
async def get_profile(request: Request):
    # Access data from request.state
    user_id = request.state.user_id
    return {"user_id": user_id, "message": "User profile"}

Pros of request.state ✅

✅ Simple and straightforward
✅ Explicitly tied to the request object
✅ Easy to understand and debug
✅ No additional imports needed
✅ Works well with dependency injection

Cons of request.state ❌

❌ Requires passing the Request object around
❌ Not accessible in deeply nested functions without explicit passing
❌ Can become verbose in complex applications

Understanding Context Variables 🌍

Context variables (contextvars) provide a way to store data that's automatically available throughout the execution context of a request, without explicitly passing it around.

Basic Usage 🚀

from contextvars import ContextVar
from fastapi import FastAPI, Request
import time

# Define context variables
user_id_var: ContextVar[str] = ContextVar('user_id')
request_time_var: ContextVar[float] = ContextVar('request_time')

app = FastAPI()

@app.middleware("http")
async def add_user_info(request: Request, call_next):
    # Set context variables
    user_id_var.set("user_123")
    request_time_var.set(time.time())

    response = await call_next(request)
    return response

@app.get("/profile")
async def get_profile():
    # Access context variables from anywhere
    user_id = user_id_var.get()
    request_time = request_time_var.get()
    return {
        "user_id": user_id, 
        "request_time": request_time,
        "message": "User profile"
    }

Pros of Context Variables ✅

✅ Available anywhere in the execution context
✅ No need to pass request object around
✅ Cleaner function signatures
✅ Automatic cleanup after request completion
✅ Thread-safe and async-safe

Cons of Context Variables ❌

❌ Less explicit than request.state
❌ Harder to debug and trace
❌ Requires additional setup and imports
❌ Can make testing more complex

Advanced Patterns 🎨

Using Context Variables with Dependencies 🔗

from contextvars import ContextVar
from fastapi import FastAPI, Depends, HTTPException
from typing import Optional

current_user_var: ContextVar[Optional[dict]] = ContextVar('current_user', default=None)

app = FastAPI()

async def get_current_user() -> dict:
    user = current_user_var.get()
    if not user:
        raise HTTPException(status_code=401, detail="User not authenticated")
    return user

@app.middleware("http")
async def auth_middleware(request: Request, call_next):
    # Simulate user authentication
    token = request.headers.get("authorization")
    if token:
        user = {"id": "123", "name": "John Doe"}  # Simulate user lookup
        current_user_var.set(user)

    response = await call_next(request)
    return response

@app.get("/protected")
async def protected_route(user: dict = Depends(get_current_user)):
    return {"message": f"Hello {user['name']}!"}

Combining Both Approaches 🤝

from contextvars import ContextVar
from fastapi import FastAPI, Request
import uuid

# Context var for request ID (global access)
request_id_var: ContextVar[str] = ContextVar('request_id')

app = FastAPI()

@app.middleware("http")
async def request_middleware(request: Request, call_next):
    # Generate request ID and store in both places
    request_id = str(uuid.uuid4())

    # Store in context var for global access
    request_id_var.set(request_id)

    # Store in request.state for explicit access
    request.state.request_id = request_id
    request.state.start_time = time.time()

    response = await call_next(request)
    return response

Performance Considerations ⚡

Memory Usage

request.state: Minimal overhead, data is garbage collected with the request 🗑️
contextvars: Slightly higher overhead due to context management, but still very efficient 💪

When to Use What? 🤔

Use request.state when:

🎯 You want explicit, traceable data flow
🎯 Building simple applications
🎯 You're already passing Request objects around
🎯 You need to store request-specific metadata
🎯 Debugging and testing simplicity is important

Use context variables when:

🎯 You have deeply nested function calls
🎯 You want cleaner function signatures
🎯 Building complex applications with many layers
🎯 You need global access to request-scoped data
🎯 Working with third-party libraries that need access to request data

Real-World Example: Logging System 📝

Here's how you might implement a request logging system with both approaches:

With Context Variables 🌐

from contextvars import ContextVar
import logging
import uuid

request_id_var: ContextVar[str] = ContextVar('request_id')
logger = logging.getLogger(__name__)

class RequestLogger:
    @staticmethod
    def info(message: str):
        request_id = request_id_var.get("unknown")
        logger.info(f"[{request_id}] {message}")

@app.middleware("http")
async def logging_middleware(request: Request, call_next):
    request_id_var.set(str(uuid.uuid4()))
    RequestLogger.info(f"Request started: {request.method} {request.url}")

    response = await call_next(request)

    RequestLogger.info(f"Request completed: {response.status_code}")
    return response

@app.get("/users/{user_id}")
async def get_user(user_id: int):
    RequestLogger.info(f"Fetching user {user_id}")
    # Your business logic here
    return {"user_id": user_id}

With request.state 📋

import logging
import uuid

logger = logging.getLogger(__name__)

def log_with_request_id(request: Request, message: str):
    request_id = getattr(request.state, 'request_id', 'unknown')
    logger.info(f"[{request_id}] {message}")

@app.get("/users/{user_id}")
async def get_user(user_id: int, request: Request):
    log_with_request_id(request, f"Fetching user {user_id}")
    # Your business logic here
    return {"user_id": user_id}

Have you used both approaches in your FastAPI projects? Share your experiences in the comments below! 👇

🔌 Building Resilient Database Operations with Async SQLAlchemy + CircuitBreaker

Akarshan Gandotra — Mon, 14 Jul 2025 04:16:58 +0000

When building production applications, database failures are inevitable. Network issues, high load, or database maintenance can cause your application to hang or crash. The Circuit Breaker pattern provides an elegant solution to handle these failures gracefully while maintaining system stability. 🛡️

⚡ What is the Circuit Breaker Pattern?

The Circuit Breaker pattern prevents your application from repeatedly attempting operations that are likely to fail. Just like an electrical circuit breaker, it monitors failures and "trips" when a threshold is reached, preventing further calls until the system recovers. 🔄

🚦 Circuit Breaker States

🟢 Closed: Normal operation, requests flow through
🔴 Open: Failures exceeded threshold, requests fail immediately
🟡 Half-Open: Testing if the system has recovered

🗄️ Why Use Circuit Breakers for Database Operations?

Database operations are particularly vulnerable to:

⏰ Network timeouts
🏊 Connection pool exhaustion
📈 Database server overload
🔧 Temporary unavailability during maintenance

Without circuit breakers, your application might:

💥 Exhaust connection pools
🌊 Create cascading failures
😤 Provide poor user experience with long timeouts
💸 Waste resources on doomed operations

🏗️ Implementation: Database Circuit Breaker in Python

Let's build a robust database layer with circuit breaker protection using SQLAlchemy and asyncio.

🔧 Core Circuit Breaker Component

from aiobreaker import CircuitBreaker, CircuitBreakerError
from datetime import timedelta
import logging

class DatabaseCircuitBreakerError(Exception):
    """Custom exception for circuit breaker failures."""
    def __init__(self, message: str = None):
        self.message = message or "Database service is temporarily unavailable"
        self.error_code = "DB_CIRCUIT_BREAKER_OPEN"
        super().__init__(self.message)

class DatabaseCircuitBreaker:
    """Circuit breaker for database operations."""

    def __init__(self, failure_threshold: int = 5, recovery_timeout: int = 60):
        self.circuit_breaker = CircuitBreaker(
            fail_max=failure_threshold,
            timeout_duration=timedelta(seconds=recovery_timeout),
            exclude=self._get_excluded_exceptions(),
        )

    def _get_excluded_exceptions(self) -> tuple:
        """Exclude application errors, include infrastructure errors."""
        return (
            IntegrityError,        # Data validation errors
            InvalidRequestError,   # SQL syntax errors
            ValueError,           # Application logic errors
            TypeError,            # Programming errors
        )

    async def call(self, func, *args, **kwargs):
        try:
            return await self.circuit_breaker.call_async(func, *args, **kwargs)
        except CircuitBreakerError as e:
            logger.warning("Circuit breaker is open, cannot execute database operation")
            raise DatabaseCircuitBreakerError() from e

🔧 Understanding Circuit Breaker Configuration

Let's break down the key parameters of the CircuitBreaker initialization:

CircuitBreaker(
    fail_max=failure_threshold,
    timeout_duration=timedelta(seconds=recovery_timeout),
    exclude=self._get_excluded_exceptions(),
)

fail_max - The failure threshold that triggers the circuit breaker:

After this many consecutive failures, the circuit breaker "trips" and opens
Example: With fail_max=5, the circuit breaker opens after 5 database connection failures

timeout_duration - How long the circuit breaker stays open:

When open, it waits this duration before testing recovery (half-open state)
Example: With timeout_duration=60, it waits 60 seconds before retrying

exclude - Exceptions that should NOT count toward failures:

These are "application errors" rather than infrastructure failures
Infrastructure failures (network timeouts, connection refused) should trip the breaker
Application failures (bad SQL, invalid data) should not trip the breaker

🚦 Circuit Breaker State Flow

🟢 CLOSED (Normal operation)
    ↓ (5 consecutive failures)
🔴 OPEN (Blocking all requests)
    ↓ (60 seconds timeout)
🟡 HALF-OPEN (Testing recovery)
    ↓ (Success) → 🟢 CLOSED
    ↓ (Failure) → 🔴 OPEN

🎯 Session Manager with Circuit Breaker

from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker, AsyncSession
from contextlib import asynccontextmanager
import asyncio

class SessionManager:
    """Database session manager with circuit breaker protection."""

    def __init__(self):
        self.engine = None
        self.session_factory = None
        self.circuit_breaker = DatabaseCircuitBreaker()
        self._is_initialized = False
        self._engine_lock = asyncio.Lock()

    async def init_db(self):
        """Initialize database engine and session factory."""
        async with self._engine_lock:
            if self._is_initialized:
                return

            # Database URL construction
            database_url = self._build_database_url()

            # Create engine with optimized settings
            self.engine = create_async_engine(
                database_url,
                pool_size=10,
                max_overflow=20,
                pool_pre_ping=True,
                pool_recycle=3600,
                pool_timeout=30,
                connect_args={
                    "server_settings": {
                        "statement_timeout": "5000"  # 5 second timeout
                    }
                }
            )

            self.session_factory = async_sessionmaker(
                bind=self.engine,
                expire_on_commit=False,
                autoflush=False,
                class_=AsyncSession,
            )

            await self._health_check()
            self._is_initialized = True
            logger.info("Database initialized successfully")

    async def _health_check(self):
        """Verify database connectivity."""
        try:
            async with self.engine.begin() as conn:
                await conn.execute(text("SELECT 1"))
            logger.info("Database health check passed")
        except Exception as e:
            logger.error(f"Database health check failed: {e}")
            raise RuntimeError(f"Database health check failed: {e}") from e

    @asynccontextmanager
    async def get_session(self):
        """Get a circuit breaker protected session."""
        session = await self.circuit_breaker.call(self._create_session)
        protected_session = CircuitBreakerSession(session, self.circuit_breaker)

        try:
            yield protected_session
        except Exception as e:
            await session.rollback()
            logger.exception(f"Exception in DB session, rolling back: {e}")
            raise
        finally:
            await session.close()

    async def _create_session(self) -> AsyncSession:
        if not self._is_initialized:
            await self.init_db()

        if not self.session_factory:
            raise RuntimeError("Database session factory is not initialized.")

        return self.session_factory()

🛡️ Protected Session Wrapper

The CircuitBreakerSession wraps the original SQLAlchemy session and applies circuit breaker protection to all database operations. This ensures that every database call goes through the circuit breaker, providing consistent protection across your application.

🔍 Key Design Principles

Selective Protection: Only async operations that can fail due to infrastructure issues are protected:

execute() - Running SQL queries
commit() - Persisting transactions
rollback() - Reverting transactions
flush() - Synchronizing with database
refresh() - Reloading object state

Synchronous Operations: Methods like add() and delete() are not protected because they only modify in-memory state and don't communicate with the database until a flush/commit occurs.

🏗️ Implementation Details

class CircuitBreakerSession:
    """Session wrapper that applies circuit breaker to all operations."""

    def __init__(self, session: AsyncSession, circuit_breaker: DatabaseCircuitBreaker):
        self._session = session
        self._circuit_breaker = circuit_breaker
        self._is_closed = False

    def _ensure_not_closed(self):
        """Prevent operations on closed sessions."""
        if self._is_closed:
            raise RuntimeError("Session is already closed")

    async def execute(self, statement, **kwargs):
        """Execute statement with circuit breaker protection."""
        self._ensure_not_closed()

        async def _execute():
            return await self._session.execute(statement, **kwargs)

        return await self._circuit_breaker.call(_execute)

    async def commit(self):
        """Commit transaction with circuit breaker protection."""
        self._ensure_not_closed()
        return await self._circuit_breaker.call(self._session.commit)

    async def rollback(self):
        """Rollback transaction with circuit breaker protection."""
        self._ensure_not_closed()
        return await self._circuit_breaker.call(self._session.rollback)

    # Synchronous operations don't need circuit breaker
    def add(self, instance):
        """Add object to session (in-memory only)."""
        self._ensure_not_closed()
        return self._session.add(instance)

    def delete(self, instance):
        """Mark object for deletion (in-memory only)."""
        self._ensure_not_closed()
        return self._session.delete(instance)

🔄 How It Works

Wrapping Pattern: The session wrapper intercepts all method calls
State Checking: Each method verifies the session isn't closed
Selective Protection: Only async database operations get circuit breaker protection
Transparent Interface: Behaves exactly like a regular SQLAlchemy session
Error Propagation: Circuit breaker errors are raised when the breaker is open

🚀 Usage with FastAPI

from fastapi import FastAPI, Depends, HTTPException

app = FastAPI()
sessionmanager = SessionManager()

async def get_db():
    """FastAPI dependency for database sessions."""
    async with sessionmanager.get_session() as session:
        yield session

@app.get("/users/{user_id}")
async def get_user(user_id: int, db = Depends(get_db)):
    try:
        result = await db.execute(
            select(User).where(User.id == user_id)
        )
        user = result.scalar_one_or_none()

        if not user:
            raise HTTPException(status_code=404, detail="User not found")

        return user

    except DatabaseCircuitBreakerError:
        raise HTTPException(
            status_code=503, 
            detail="Database service temporarily unavailable"
        )

@app.on_event("startup")
async def startup():
    await sessionmanager.init_db()

@app.on_event("shutdown")
async def shutdown():
    await sessionmanager.close()

⚙️ Configuration Best Practices

🌍 Environment Variables

# Circuit breaker settings
DB_CIRCUIT_BREAKER_FAILURE_THRESHOLD = 5  # Trip after 5 failures
DB_CIRCUIT_BREAKER_RECOVERY_TIMEOUT = 60   # Wait 60 seconds before retry

# Connection pool settings
POOL_SIZE = 10
MAX_OVERFLOW = 20
POOL_RECYCLE = 3600
POOL_TIMEOUT = 30

⚖️ Benefits and Trade-offs

✅ Benefits

🛡️ Improved resilience: Prevents cascading failures
😊 Better user experience: Fast failure instead of timeouts
🔐 Resource protection: Prevents connection pool exhaustion
🔄 Automatic recovery: Tests system health periodically

⚠️ Trade-offs

🔧 Added complexity: More moving parts to monitor
❌ False positives: May block valid requests during recovery
⚙️ Configuration overhead: Requires tuning for your workload

✅ Demo Video

🎯 Conclusion

The Circuit Breaker pattern is essential for building resilient database layers in production applications. By implementing circuit breakers around your database operations, you can:

🚫 Prevent cascading failures
💪 Improve system stability
🎯 Provide better error handling
🛡️ Protect system resources

The implementation shown here provides a solid foundation that you can adapt to your specific needs. Remember to monitor circuit breaker behavior and tune the configuration based on your application's requirements. 📊

What patterns do you use for database resilience in your applications? Share your experiences in the comments below! 💬

Supercharging Your API Cache: A Deep Dive into Serialization Performance 🚀

Akarshan Gandotra — Thu, 10 Jul 2025 08:16:38 +0000

When milliseconds matter, choosing the right serialization format can make or break your application's performance.

The Performance Problem We All Face

Picture this: Your API is handling thousands of requests per second, and your cache is working overtime. But here's the catch – every time you serialize and deserialize data for caching, you're eating into precious microseconds that add up to real performance bottlenecks.

As developers, we often default to JSON for caching without questioning whether it's the optimal choice. But what if I told you that switching serialization formats could give you a 3-10x performance boost with minimal code changes?

The Great Serialization Showdown

I recently implemented support for multiple serialization formats in our API caching layer and ran comprehensive performance tests. The results were eye-opening! Here's what we compared:

JSON (stdlib) - The trusty default
orjson - The speed demon
Pickle - Python's native powerhouse

Implementation Architecture

The Multi-Format Cache Manager

First, let's look at how we structured the enhanced caching layer:

from enum import Enum
from typing import Any, Optional
import json
import pickle
import time

class SerializationFormat(Enum):
    JSON = "json"
    ORJSON = "orjson"
    PICKLE = "pickle"

class PerformanceMetrics:
    """Tracks performance metrics for different serialization formats."""

    def __init__(self):
        self.metrics = {
            format.value: {
                "serialize_times": [],
                "deserialize_times": [],
                "sizes": [],
                "total_operations": 0
            }
            for format in SerializationFormat
        }

    def record_operation(self, format: SerializationFormat, 
                        serialize_time: float, deserialize_time: float, 
                        size: int):
        format_key = format.value
        self.metrics[format_key]["serialize_times"].append(serialize_time)
        self.metrics[format_key]["deserialize_times"].append(deserialize_time)
        self.metrics[format_key]["sizes"].append(size)
        self.metrics[format_key]["total_operations"] += 1

The Serialization Manager

The heart of our implementation is the SerializationManager class that handles all formats with built-in performance tracking:

class SerializationManager:
    def __init__(self):
        self.metrics = PerformanceMetrics()

    def serialize_orjson(self, data: Any) -> bytes:
        """Serialize using orjson - the performance champion."""
        import orjson
        start_time = time.time()
        serialized = orjson.dumps(data)
        serialize_time = time.time() - start_time
        return serialized, serialize_time

    def deserialize_orjson(self, data: bytes) -> Any:
        """Deserialize using orjson."""
        import orjson
        start_time = time.time()
        deserialized = orjson.loads(data)
        deserialize_time = time.time() - start_time
        return deserialized, deserialize_time

    # Similar methods for JSON and Pickle...

Cache Manager with Format Selection

class APICacheManager:
    def __init__(self, default_format: SerializationFormat = SerializationFormat.JSON):
        self.default_format = default_format
        self.serialization_manager = SerializationManager()

    async def set(self, key: str, value: Any, ttl: int = 3600, 
                  format: Optional[SerializationFormat] = None) -> bool:
        """Cache data with specified serialization format."""
        format = format or self.default_format

        # Serialize based on format
        if format == SerializationFormat.ORJSON:
            serialized_data, _ = self.serialization_manager.serialize_orjson(value)
        elif format == SerializationFormat.PICKLE:
            serialized_data, _ = self.serialization_manager.serialize_pickle(value)
        # ... handle other formats

        # Store in cache (Redis/Memcached/etc.)
        return await self._store_in_cache(key, serialized_data, ttl)

The Performance Results That Will Blow Your Mind 🤯

I tested three different data sizes across 1,000 operations each:

Small Data (242 bytes) - Typical API Response

Format	Serialization	Deserialization	Data Size
orjson ⭐	0.000000s	0.000001s	221 bytes
JSON	0.000003s	0.000002s	242 bytes
Pickle	0.000001s	0.000001s	232 bytes

Medium Data (561 bytes) - Complex API Response

Format	Serialization	Deserialization	Data Size
orjson ⭐	0.000001s	0.000001s	516 bytes
JSON	0.000005s	0.000004s	561 bytes
Pickle	0.000002s	0.000002s	521 bytes

Large Data (3,346 bytes) - Data-Heavy Response

Format	Serialization	Deserialization	Data Size
orjson ⭐	0.000003s	0.000006s	2,963 bytes
JSON	0.000030s	0.000022s	3,346 bytes
Pickle ⭐	0.000009s	0.000009s	2,536 bytes

The Clear Winner: orjson 🏆

orjson absolutely dominates in almost every category:

3-10x faster serialization than standard JSON
2-4x faster deserialization than standard JSON
Smallest data size for small and medium payloads
Drop-in replacement for standard JSON

Here's the math that'll make you happy:

1,000 cache operations per second
Standard JSON: 30ms total serialization time
orjson: 3ms total serialization time
Savings: 27ms per 1,000 operations = 27 seconds per million operations!

When to Use Each Format

🚀 Use orjson when:

Performance is critical
You're dealing with high-throughput APIs
You want easy migration from standard JSON
You need language-agnostic serialization

# Easy migration example
@cache_response(
    prefix="users",
    ttl=3600,
    format=SerializationFormat.ORJSON  # Just change this line!
)
async def get_user(user_id: int):
    return await fetch_user_from_db(user_id)

🐍 Use Pickle when:

You're in a Python-only environment
You have very large, complex data structures
You need the smallest possible data size for large objects
You don't need cross-language compatibility

🔧 Use Standard JSON when:

Cross-language compatibility is essential
You're working with legacy systems
You need human-readable cached data for debugging
Performance is acceptable for your use case

Real-World Implementation Tips

1. Gradual Migration Strategy

# Start with high-traffic endpoints
cache_mgr_fast = APICacheManager(default_format=SerializationFormat.ORJSON)
cache_mgr_compatible = APICacheManager(default_format=SerializationFormat.JSON)

# Use fast cache for internal APIs
@cache_response(manager=cache_mgr_fast, ttl=3600)
async def internal_user_data(user_id: int):
    pass

# Use compatible cache for external APIs
@cache_response(manager=cache_mgr_compatible, ttl=3600)
async def public_api_endpoint():
    pass

2. Performance Monitoring

def log_cache_metrics():
    metrics = serialization_manager.get_performance_summary()
    for format, data in metrics.items():
        logger.info(f"{format}: avg_serialize={data['avg_serialize_time']:.6f}s, "
                   f"avg_deserialize={data['avg_deserialize_time']:.6f}s")

3. Fallback Strategies

async def robust_cache_get(key: str, formats: List[SerializationFormat]):
    """Try multiple formats for backward compatibility."""
    for format in formats:
        try:
            data = await cache_mgr.get(key, format=format)
            if data:
                return data
        except Exception as e:
            logger.warning(f"Failed to deserialize {key} with {format}: {e}")
    return None

The Bottom Line

After extensive testing, here's my recommendation hierarchy:

orjson - Use this for 80% of your caching needs
Pickle - Use for Python-only, data-heavy scenarios
JSON - Use when you need maximum compatibility

Getting Started

Want to implement this in your project? Here's the minimal setup:

pip install orjson

from enum import Enum

class SerializationFormat(Enum):
    ORJSON = "orjson"
    JSON = "json"

# Your existing cache code here, just add format parameter
async def cache_set(key: str, value: any, format: SerializationFormat = SerializationFormat.ORJSON):
    if format == SerializationFormat.ORJSON:
        import orjson
        serialized = orjson.dumps(value)
    else:
        import json
        serialized = json.dumps(value).encode()

    # Store in your cache system
    await your_cache.set(key, serialized)

Conclusion

The numbers don't lie – orjson is a game-changer for API caching performance. With minimal code changes, you can achieve significant performance improvements that directly translate to better user experience and lower infrastructure costs.

The beauty of this approach is that you can implement it incrementally, starting with your highest-traffic endpoints and gradually migrating your entire caching layer.

What serialization format are you currently using? Have you measured its performance impact? Drop a comment below and let's discuss your caching optimization strategies!

Found this helpful? Give it a ❤️ and follow me for more performance optimization content!

Building a Robust Redis Client: Async Redis + Tenacity + Circuit Breaker

Akarshan Gandotra — Sun, 06 Jul 2025 20:08:34 +0000

Redis is the backbone of many high-performance applications, but network hiccups and temporary outages can wreak havoc on your system. What if I told you there's a way to make your Redis operations virtually bulletproof? 🛡️

Today, we'll dive deep into building a production-ready Redis client with comprehensive retry logic, circuit breaker patterns, and connection pooling that can handle the chaos of distributed systems.

🚨 The Problem: When Redis Goes Down

Picture this: Your application is humming along perfectly, handling thousands of requests per second. Suddenly, Redis experiences a brief network hiccup. Your application starts throwing exceptions, users see errors, and your monitoring dashboard lights up like a Christmas tree. 🎄💥

Common Redis failure scenarios:

Network timeouts 🌐⏰
Connection drops 🔌❌
Redis server restarts 🔄
Memory pressure 💾⚠️
Temporary unavailability 🚫

🏗️ The Solution: A Resilient Redis Client

Our solution combines several battle-tested patterns:

1. Exponential Backoff Retry with Tenacity 📈

We leverage the powerful Tenacity library for sophisticated retry logic. Tenacity provides a clean, declarative way to handle retries with various strategies:

from tenacity import AsyncRetrying, retry_if_exception_type, stop_after_attempt, wait_exponential

@dataclass(frozen=True)
class RedisRetryConfig:
    max_attempts: int = settings.REDIS_MAX_ATTEMPTS
    base_delay: float = settings.REDIS_BASE_DELAY
    max_delay: float = settings.REDIS_MAX_DELAY
    exponential_base: float = settings.REDIS_EXPONENTIAL_BASE
    retry_exceptions: tuple = field(
        default_factory=lambda: (
            redis.ConnectionError,
            redis.TimeoutError,
            redis.RedisError,
            ConnectionRefusedError,
            TimeoutError,
        )
    )

Why Tenacity? 🎯

Declarative syntax - Easy to read and configure
Multiple retry strategies - Exponential, linear, random jitter
Exception filtering - Retry only on specific exceptions
Async support - Perfect for modern Python applications
Battle-tested - Used in production by thousands of applications

2. Circuit Breaker Pattern 🔌

class CircuitBreaker:
    """Simple circuit breaker for Redis operations."""

    def __init__(self, threshold: int, timeout: float):
        self.threshold = threshold
        self.timeout = timeout
        self.failure_count = 0
        self.last_failure_time = 0
        self.state = "closed"  # closed, open, half-open

3. Connection Pooling 🏊‍♂️

self._connection_pool = ConnectionPool.from_url(
    settings.REDIS_URL,
    max_connections=getattr(settings, "REDIS_CONNECTION_POOL", 20),
    retry_on_timeout=True,
    socket_keepalive=True,
    health_check_interval=30,
)

📚 Key Dependencies

This implementation leverages several powerful Python libraries:

🔄 Tenacity - The star of our retry logic! Provides declarative retry decorators with exponential backoff, jitter, and exception filtering
🔴 Redis-py - Official async Redis client for Python
⚙️ Asyncio - For async/await support and thread-safe operations
📊 Dataclasses - Clean, immutable configuration objects

pip install tenacity redis[hiredis]

🎯 Key Features

✨ Automatic Retry Logic

The redis_retry decorator automatically wraps Redis operations with intelligent retry logic using Tenacity:

def redis_retry(
    config: Optional[RedisRetryConfig] = None,
    operation_name: Optional[str] = None,
    use_circuit_breaker: bool = True,
) -> Callable:
    """Decorator for Redis operations with retry logic and circuit breaker."""

    retry_config = config or RedisRetryConfig()

    def decorator(func: Callable[..., T]) -> Callable[..., T]:
        @wraps(func)
        async def wrapper(*args, **kwargs) -> T:
            async def execute_with_retry():
                try:
                    # Tenacity's AsyncRetrying provides the retry logic
                    async for attempt in AsyncRetrying(
                        stop=stop_after_attempt(retry_config.max_attempts),
                        wait=wait_exponential(
                            multiplier=retry_config.base_delay,
                            max=retry_config.max_delay,
                            exp_base=retry_config.exponential_base,
                        ),
                        retry=retry_if_exception_type(retry_config.retry_exceptions),
                        reraise=True,
                    ):
                        with attempt:
                            return await func(*args, **kwargs)
                except RetryError as e:
                    logger.error(f"Failed after {retry_config.max_attempts} attempts")
                    raise e.last_attempt.exception() from e

            if circuit_breaker:
                return await circuit_breaker.call(execute_with_retry)
            else:
                return await execute_with_retry()

        return wrapper
    return decorator

Tenacity Features in Action: 🚀

stop_after_attempt() - Limits total retry attempts
wait_exponential() - Implements exponential backoff with jitter
retry_if_exception_type() - Only retries on specific exceptions
AsyncRetrying - Async-first design for modern Python apps

🔄 Circuit Breaker States

Closed 🟢: Normal operation, requests pass through
Open 🔴: Failing fast, requests immediately rejected
Half-Open 🟡: Testing if service recovered

🏊‍♂️ Enhanced Redis Client

class RedisWithRetry(redis.Redis):
    """Enhanced Redis client with comprehensive retry logic and circuit breaker."""

    def __getattribute__(self, name: str) -> Any:
        """Override attribute access to automatically add retry logic."""
        attr = super().__getattribute__(name)

        if (
            name in self._core_methods
            and callable(attr)
            and asyncio.iscoroutinefunction(attr)
        ):
            retry_config = super().__getattribute__("retry_config")
            return redis_retry(retry_config, name)(attr)

        return attr

This clever __getattribute__ override automatically wraps all Redis operations with retry logic! 🎩✨

📊 Health Monitoring

The client includes comprehensive health checking:

async def health_check(self) -> Dict[str, Any]:
    """Comprehensive health check for Redis connection."""
    try:
        start_time = time.time()
        await self.ping()
        ping_time = time.time() - start_time

        info = await self.info()

        return {
            "status": "healthy",
            "ping_time_ms": round(ping_time * 1000, 2),
            "connected_clients": info.get("connected_clients", 0),
            "used_memory": info.get("used_memory", 0),
            "used_memory_human": info.get("used_memory_human", "0B"),
            "redis_version": info.get("redis_version", "unknown"),
        }
    except Exception as e:
        return {
            "status": "unhealthy",
            "error": str(e),
            "error_type": type(e).__name__,
        }

🔧 Advanced Tenacity Configurations

Tenacity offers incredible flexibility for different retry scenarios:

Custom Wait Strategies ⏱️

from tenacity import wait_fixed, wait_random, wait_combine

# Fixed delay
wait=wait_fixed(2)  # Always wait 2 seconds

# Random jitter to prevent thundering herd
wait=wait_random(min=1, max=3)

# Combine strategies
wait=wait_combine(
    wait_exponential(multiplier=1, max=10),
    wait_random(min=0, max=2)  # Add jitter
)

Stop Conditions 🛑

from tenacity import stop_after_delay, stop_never

# Stop after total time
stop=stop_after_delay(30)  # Stop after 30 seconds total

# Stop after attempts OR time
stop=(stop_after_attempt(5) | stop_after_delay(30))

Custom Retry Conditions 🎯

from tenacity import retry_if_result, retry_if_not_result

# Retry based on return value
@retry(retry=retry_if_result(lambda x: x is None))
async def get_data():
    return await redis_client.get("key")

# Retry on specific exception attributes
@retry(retry=retry_if_exception(
    lambda e: isinstance(e, redis.ConnectionError) and "timeout" in str(e)
))
async def redis_operation():
    # Your Redis operation here
    pass

🔧 Usage Examples

Basic Usage

# Get the singleton client
client = await get_redis_client()

# All operations now have automatic retry logic!
await client.set("key", "value")
value = await client.get("key")

Pipeline Operations

async with redis_pipeline() as pipeline:
    pipeline.set("key1", "value1")
    pipeline.set("key2", "value2")
    results = await pipeline.execute()  # Retries included!

Health Monitoring

health = await redis_health_check()
print(f"Redis status: {health['status']}")
print(f"Ping time: {health['ping_time_ms']}ms")

📈 Performance Benefits

Scenario	Without Retry	With Retry + Circuit Breaker
Network blip (100ms)	❌ Immediate failure	✅ Transparent recovery
Redis restart (5s)	❌ 5s of errors	✅ Fast failure → recovery
Sustained outage	❌ Continuous timeouts	✅ Fail fast after threshold

🛠️ Configuration Options

# Custom retry configuration
config = RedisRetryConfig(
    max_attempts=5,
    base_delay=0.1,
    max_delay=60.0,
    exponential_base=2.0,
    circuit_breaker_threshold=5,
    circuit_breaker_timeout=30.0
)

client = await get_redis_client(config)

🔐 Thread Safety & Singleton Pattern

The RedisClientManager ensures thread-safe singleton behavior:

class RedisClientManager:
    """Thread-safe singleton manager for Redis client."""

    def __init__(self):
        self._client: Optional[RedisWithRetry] = None
        self._lock = asyncio.Lock()

    async def get_client(self) -> RedisWithRetry:
        if self._client is None:
            async with self._lock:
                if self._client is None:
                    await self._initialize_client()
        return self._client

🚀 Production Tips

1. Monitor Circuit Breaker States 📊

# Add metrics collection
if circuit_breaker.state == "open":
    metrics.counter("redis.circuit_breaker.open").increment()

2. Tune Retry Parameters ⚙️

Start with conservative values
Monitor success/failure rates
Adjust based on your Redis setup

3. Connection Pool Sizing 🏊‍♂️

# Rule of thumb: 2-3x your concurrent request count
max_connections = concurrent_requests * 2.5

4. Health Check Integration 🏥

# Add to your application health endpoint
@app.get("/health")
async def health_check():
    redis_health = await redis_health_check()
    return {"redis": redis_health}

🎭 Error Handling Hierarchy

📝 Conclusion

Building resilient systems requires more than just hoping nothing goes wrong. With this Redis client, you get:

🔄 Automatic retries with exponential backoff
🔌 Circuit breaker protection
🏊‍♂️ Connection pooling for performance
📊 Health monitoring for observability
🛡️ Production-ready error handling

Your Redis operations will now gracefully handle network issues, temporary outages, and other distributed system challenges. No more 3 AM alerts for transient Redis hiccups! 🌙😴

Full Code Snippet

Building Bulletproof Authentication: A Modern Stateless Session Guide 🔐

Akarshan Gandotra — Tue, 01 Jul 2025 19:05:12 +0000

Building Secure, Scalable Authentication with Auth0 Integration

🎯 The Problem: Traditional Sessions Don't Scale

Picture this: Your app hits 100K users, and suddenly your Redis session store becomes the bottleneck. Users get logged out randomly when load balancers route them to different servers. Sound familiar?
We faced this exact problem and decided to go completely stateless. Here's how we built a bulletproof authentication system that scales horizontally without breaking a sweat.

🧠 Understanding Stateless Session Management

🤔 What Is It?

Stateless session management is like having a self-contained passport for every request. Instead of the server remembering who you are (like a hotel keeping your room key), all the information travels with you in a secure, tamper-proof token.

🔍 Key Characteristics

Aspect	Description
🗄️ Storage	No server-side session storage
📦 Data Location	Session data embedded in tokens
🔄 Request Context	Each request carries complete authentication context
🔐 Security	Cryptographic signatures ensure token integrity

⭐ Why Go Stateless?

🚀 1. Horizontal Scalability

Think of it like a food truck festival - any truck can serve any customer because the order is written on the receipt!

Benefit	Impact
No Sticky Sessions	Requests can be handled by any server instance
Load Balancer Simplicity	No need for session affinity
Auto-Scaling Friendly	New instances can immediately handle requests

⚡ 2. Performance Optimization

Metric	Improvement
Database Calls	Eliminated session lookups
Memory Footprint	Reduced server memory usage
Processing Speed	Direct token validation

Traditional Sessions: Request → DB Lookup → Validation → Response

Stateless Sessions: Request → Token Validation → Response

🛠️ 3. Operational Excellence

Area	Advantage
Deployment	No session replication concerns
Disaster Recovery	No session state to backup/restore
Microservices	Tokens can be validated independently

🔒 4. Security Benefits

Security Feature	Benefit
Reduced Attack Surface	No centralized session store to compromise
Token-Based Security	Cryptographic validation ensures integrity
Fine-Grained Expiration	Individual token control vs global session invalidation

🛡️ Security Arsenal

Our multi-layered security approach protects against various attack vectors:

🛡️ 1. Token Theft Protection

Device Fingerprinting

Device fingerprinting creates a unique digital signature for each device, like a fingerprint for your browser/device combination.

def generate_device_fingerprint(request):
    """Creates a unique device signature"""
    fingerprint_data = {
        'user_agent': request.headers.get('User-Agent'),
        'accept_language': request.headers.get('Accept-Language'),
        'screen_resolution': request.headers.get('X-Screen-Resolution'),
        'timezone': request.headers.get('X-Timezone'),
        'ip_subnet': get_ip_subnet(request.client.host)
    }
    return hashlib.sha256(json.dumps(fingerprint_data, sort_keys=True).encode()).hexdigest()

Challenges with Device Fingerprinting:

Scenario	Impact	Frequency
Browser updates	User-Agent changes	Weekly/Monthly
Network changes	IP address changes	Daily
Mobile networks	IP changes frequently	Constant
Browser settings	Accept headers change	Occasional
Corporate proxies	Headers modified	Common

Advanced Fingerprint Validation:

class AdvancedFingerprintValidator:
    """Smart fingerprint validation with flexibility"""

    def __init__(self):
        self.weights = {
            'user_agent': 0.4,        # Most important
            'screen_resolution': 0.3,  # Hardware specific
            'timezone': 0.2,          # Geographic
            'accept_language': 0.1    # User preference
        }

        # Different thresholds for different scenarios
        self.thresholds = {
            'strict': 0.95,   # Banking/financial
            'normal': 0.85,   # Regular apps
            'relaxed': 0.70   # Public/demo apps
        }

    def calculate_similarity(self, stored_fp, current_fp):
        """Calculate similarity score between fingerprints"""
        total_score = 0
        for component, weight in self.weights.items():
            if stored_fp.get(component) == current_fp.get(component):
                total_score += weight
        return total_score

    def is_valid_device(self, stored_fp, current_fp, security_level='normal'):
        """Determine if device fingerprint is valid"""
        similarity = self.calculate_similarity(stored_fp, current_fp)
        threshold = self.thresholds[security_level]

        return {
            'valid': similarity >= threshold,
            'similarity_score': similarity,
            'threshold_used': threshold,
            'risk_level': self.get_risk_level(similarity)
        }

Implementation Features:

✅ Embed device fingerprint in JWT claims
✅ Validate fingerprint on each request
✅ Automatic logout on fingerprint mismatch
✅ Gradual fingerprint validation (allow minor changes)

🛡️ How Fingerprinting Prevents Token Theft

⏰ Short Access Token TTL

Token Type	Duration	Purpose
Access Tokens	15-30 minutes	Short-lived for security
Refresh Tokens	7-30 days	Based on risk assessment
Sliding Window	Dynamic	For active sessions

🛡️ 2. Replay Attack Prevention

We only track JTI in cases with extremely sensitive operations. In normal API usage, the same access token is used repeatedly until it expires.

🎯 JTI (JWT ID) Implementation

def generate_jti():
    """Creates unique token identifier"""
    return str(uuid.uuid4())

def validate_jti(jti, user_id):
    """Prevents token reuse"""
    cache_key = f"jti_blacklist:{user_id}:{jti}"
    if redis_client.exists(cache_key):
        raise InvalidTokenError("Token already used")

    # Add to blacklist with TTL equal to token expiry
    redis_client.setex(cache_key, token_ttl_seconds, "used")

🔒 Security Measures

Feature	Description
Unique JTI	Each token is generated with a unique identifier (jti)
Distributed Blacklist	Redis cluster tracks revoked/expired tokens across nodes
Automatic Cleanup	Expired tokens automatically removed from blacklist
Rate Limiting	Tracks token usage frequency and enforces limits

🛡️ 3. Long-term Compromise Protection

🗝️ Key Rotation Strategy

Step-by-Step Key Rotation Process:

Step	When	Action	Result
1. Create New Key	June 1	Generate Key A	New secret password created
2. Sign New Tokens	June 1-July 1	Use Key A for all tokens	All new tokens use Key A
3. Create Key B	July 1	Generate Key B	Prepare for rotation
4. Grace Period	July 1-3	Accept both keys	Smooth transition
5. Retire Key A	July 3	Remove Key A	Only Key B tokens work

⏰ Rotation Schedule

Key Type	Frequency	Emergency
Signing Keys	Monthly	1 hour
Key Retirement	48-hour overlap	Immediate

🔑 Key ID (kid) in JWTs

The kid (Key ID) is a header claim in JWT that tells the verifier which key was used to sign the token.

JWT Header Example:

{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "key-2025-06"
}

🛡️ 4. Token Farming Attack Prevention

Scenario: Attacker collects multiple refresh tokens and rapidly exchanges them for access tokens to harvest credentials or maintain persistent access.

The Token Bucket Rate Limiter prevents abuse of refresh token endpoints by controlling the request processing rate.

Key Features:

Burst Tolerance: Allows legitimate users to make multiple requests quickly
Smooth Rate Limiting: Tokens refill at a steady rate over time
Per-User Isolation: Each user has their own token bucket
Memory Efficient: Automatic cleanup and minimal memory footprint
Configurable: Easy to adjust capacity, refill rate, and intervals

Algorithm Flow:

Request Arrives → Check Bucket → Has Tokens? → Process Request
                                     ↓ No
                              Reject with 429 Error

🛡️ 5. Brute Force and Bot Attack Prevention with Auth0

Auth0 provides built-in capabilities to protect authentication endpoints.

Brute Force Protection

How it works:

Monitors failed login attempts per user
After 10 failed attempts, account is blocked for 1 minute (configurable)
Block duration increases progressively with persistent failures
All blocked attempts are logged in Auth0 Dashboard

Benefits:

Protects against brute-force password guessing
Reduces risk of account enumeration

Bot Detection

How it works:

Analyzes login requests for suspicious patterns
Uses reputation intelligence and anomaly detection
Challenges or blocks requests automatically
May show CAPTCHA for suspicious logins

Benefits:

Stops automated attacks before reaching authentication pipeline
Reduces system load from malicious traffic

🧾 Example JWT Structures

🎫 Access Token

JWT Header:

{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "key-2025-06"
}

JWT Payload:

{
  "iss": "https://iam.yourcompany.com/",
  "sub": "user_123456",
  "aud": "your-app-client-id",
  "exp": 1717500000,
  "iat": 1717496400,
  "jti": "0d2ae315-01de-42e0-9f21-3b4e4212f41f",
  "scope": "read:profile write:profile",
  "device_fingerprint": "9f6a3cbcf56d2f2b5a4bfb59e5fcbba9f2...",
  "session_id": "session_abc123"
}

🔄 Refresh Token

JWT Header:

{
  "alg": "RS256",
  "typ": "JWT",
  "kid": "key-2025-06"
}

JWT Payload:

{
  "iss": "https://iam.yourcompany.com/",
  "sub": "user_123456",
  "aud": "your-app-client-id",
  "exp": 1719500000,
  "iat": 1717496400,
  "jti": "67c4d2f2-317b-4a83-b5cb-8e7d5ad732ba",
  "device_fingerprint": "9f6a3cbcf56d2f2b5a4bfb59e5fcbba9f2..."
}

Key Fields

Field	Description
kid	Identifies which signing key was used
alg	Algorithm used to sign the token (RS256)
iss	Who issued the token
sub	User ID
aud	Intended recipient
exp	Expiration time
iat	Issued-at timestamp
jti	Unique identifier for the token
scope	Permissions granted
device_fingerprint	Unique signature of the device
session_id	Session reference

🔄 Architecture Flows

🚀 1. Login Flow

The login flow orchestrates the complete authentication process, from Auth0 validation to secure token generation with comprehensive security checks.

🔄 2. Refresh Token Flow

The refresh flow implements secure token rotation with comprehensive validation to prevent token replay attacks and unauthorized access.

👋 3. Logout Flow

The logout flow ensures complete session termination with comprehensive cleanup to prevent any residual access or security vulnerabilities.

📊 Session Activity Management

Session activity management is a critical component that monitors user engagement, detects anomalies, and automatically handles session lifecycle events including inactivity-based logouts. This system ensures optimal security while maintaining user experience through intelligent session management.

Session activity management is our digital fitness tracker 📱 for user sessions! It monitors engagement, detects anomalies, and automatically handles session lifecycle events.

🧩 How It Works

📈 Activity Tracking Pipeline

Stage	Description	Example
1. Event Capture	Every user interaction recorded via WebSocket	Mouse click, API call, typing
2. Weight Assignment	Each event gets a relevance score	Keyboard = 0.3, API call = 0.7
3. Score Calculation	Activity score measures engagement	Higher = more active session
4. Session Update	Redis record gets refreshed	Timestamp, score, device info
5. Timeout Reset	TTL gets reset to idle_timeout	30 minutes from last activity

⚙️ Configuration Matrix

Setting	Value	Description
Idle Timeout	30 minutes	Session expires after inactivity
Max Concurrent	5 sessions	Per user limit
Activity Buffer	100 events	Rolling activity log

🎯 Activity Event Weights

Event Type	Weight	Rationale
Mouse	0.1	Basic interaction
Keyboard	0.3	Active engagement
Navigation	0.5	Intentional movement
API Calls	0.7	System interaction
Sensitive Operations	1.0	Critical actions

Event Types Catalog

Event	Description	Trigger
activity_update	New user action extended session	Any qualifying activity
session_expired	Session expired due to inactivity	Timeout reached
session_terminated	Admin or policy terminated session	Manual/automatic termination
warning_threshold_reached	User approaching idle timeout	5 minutes before expiry

Example Client Payload

{
  "activity_type": "keyboard_input",
  "is_sensitive_operation": false,
  "is_automated": false
}

🛑 Redis Outage Impact and Fallback

Impacts of Redis Outage

Component	Impact
Session Cap Enforcement	❌ Cannot track or enforce maximum sessions per user
Token Revocation	❌ Cannot check blacklist, revoked tokens still accepted
Session Timeout Tracking	❌ Cannot auto-expire sessions based on inactivity
Real-time Notifications	❌ No WebSocket notifications for session changes
Refresh Token Rate Limiter	❌ No Token Bucket maintained per user
Token Validation (Signature)	✅ Still works (JWT signature validation is stateless)

Fallback Strategy: Stateless-Only Mode

When Redis is down, fall back to purely stateless validation:

What We Keep Doing ✅

Validate token signatures using signing key
Validate expiry (exp claim)
Validate claims (sub, aud, etc.)
Users can log in and receive new tokens

What We Temporarily Lose ❌

Enforcement of maximum sessions per user
Token revocation (can't check if jti is blacklisted)
Inactivity-based session expiry
Token Rate Limiter

📚 Glossary

Technical Terms

Term	Definition
JWT (JSON Web Token)	A self-contained token that carries user information and is cryptographically signed
JTI (JWT ID)	A unique identifier for each JWT to prevent replay attacks
Device Fingerprinting	Technique to uniquely identify devices based on browser/system characteristics
TTL (Time To Live)	How long a token or session remains valid before expiring
Token Rotation	Security practice of regularly changing cryptographic keys
Replay Attack	Security attack where valid tokens are reused maliciously
WebSocket	Real-time, bidirectional communication protocol between client and server
Redis	In-memory data store used for caching and session management
Stateless	Architecture where servers don't store client session information
Auth0	Third-party authentication service for identity management

Security Concepts

Concept	Explanation
Defense in Depth	Multiple layers of security controls working together
Principle of Least Privilege	Users get minimum access needed for their role
Risk-Based Authentication	Security measures adjust based on calculated risk levels
Session Sliding Window	Session timeout resets with each user activity
Concurrent Session Limiting	Restricting number of simultaneous user sessions

Conclusion

This stateless session management approach provides a robust, scalable, and secure foundation for our FastAPI IAM Platform. By combining Auth0's authentication capabilities with our comprehensive security measures, we achieve both excellent user experience and enterprise-grade security.

Setting Up Telemetry in Golang

Akarshan Gandotra — Thu, 03 Apr 2025 07:47:09 +0000

Introduction

Observability is a key aspect of modern applications, enabling developers to monitor and troubleshoot their systems effectively. OpenTelemetry (OTel) provides a standardized framework for tracing and metrics collection. This blog post covers the setup of telemetry in a Golang application using OpenTelemetry, focusing on traces and metrics.

By integrating OpenTelemetry into your application, you can:

Collect detailed traces of API calls and internal function executions.
Monitor system performance through structured metrics.
Gain insight into potential bottlenecks and optimize resource utilization.

This guide will walk you through setting up telemetry for a Golang service, including trace and metric collection, middleware integration, and best practices.

Initializing Telemetry Objects

1. `serviceResource`

The serviceResource object defines metadata about the service, such as its name. It uses OpenTelemetry's semantic conventions to describe the running application:

var serviceResource = resource.NewWithAttributes(
    semconv.SchemaURL,
    semconv.ServiceNameKey.String("pth-auth-service"),
)

This helps identify telemetry data originating from this service when viewed in monitoring tools like Grafana or Jaeger.

2. `initTracer`

This function initializes the OpenTelemetry Tracer Provider, which enables distributed tracing. Distributed tracing is useful for tracking requests across different microservices and identifying latency issues.

func initTracer(ctx context.Context, otlpEndpoint string) error {
    exporter, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpoint(otlpEndpoint), otlptracehttp.WithInsecure())
    if err != nil {
        return fmt.Errorf("failed to create OTLP trace exporter: %w", err)
    }

    traceProvider = trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(serviceResource),
    )

    otel.SetTracerProvider(traceProvider)
    return nil
}

This function sets up an HTTP-based OpenTelemetry exporter, which sends trace data to an OpenTelemetry collector or a monitoring backend. The WithBatcher(exporter) option ensures that traces are batched before being exported to improve performance.

3. `initMetricProvider`

This function sets up the Metric Provider, allowing the application to collect and export metrics for monitoring performance:

func initMetricProvider(ctx context.Context, otelMetricsEndpoint string) error {
    exporter, err := otlpmetrichttp.New(ctx, otlpmetrichttp.WithEndpoint(otelMetricsEndpoint), otlpmetrichttp.WithInsecure())
    if err != nil {
        return fmt.Errorf("failed to create OTLP metric exporter: %w", err)
    }

    MeterProvider = sdkmetric.NewMeterProvider(
        sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exporter)),
        sdkmetric.WithResource(serviceResource),
    )
    otel.SetMeterProvider(MeterProvider)

    return nil
}

Metrics are crucial for tracking application behavior over time. Examples include HTTP request durations, memory usage, and active database connections.

4. `registerMetrics`

This function registers various HTTP server metrics to track API performance:

func registerMetrics(meter metric.Meter) {
    metrics := []struct {
        name        string
        description string
        register    func(metric.Meter) (interface{}, error)
    }{
        {HTTPServerDuration, "Measures the duration of inbound HTTP requests", func(m metric.Meter) (interface{}, error) { return m.Float64Histogram(HTTPServerDuration) }},
        {HTTPServerActiveRequests, "Number of concurrent HTTP requests in flight", func(m metric.Meter) (interface{}, error) { return m.Int64UpDownCounter(HTTPServerActiveRequests) }},
        {HTTPServerRequestSize, "Measures the size of HTTP request messages", func(m metric.Meter) (interface{}, error) { return m.Float64Histogram(HTTPServerRequestSize) }},
        {HTTPServerResponseSize, "Measures the size of HTTP response messages", func(m metric.Meter) (interface{}, error) { return m.Float64Histogram(HTTPServerResponseSize) }},
    }

    for _, metricDef := range metrics {
        if _, err := metricDef.register(meter); err != nil {
            fmt.Printf("[Telemetry] Failed to create metric %s: %v\n", metricDef.name, err)
        }
    }
}

These metrics help monitor key aspects of API performance, such as latency and data transfer sizes.

5. `InitTelemetry`

This function ties everything together by initializing both tracing and metrics. It reads necessary configurations from environment variables and ensures that telemetry is properly set up before the application starts handling requests.

func InitTelemetry(ctx context.Context) error {
    if strings.ToLower(os.Getenv("OTEL_SDK_DISABLED")) == "true" {
        return nil
    }

    otelEndpoint := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    if otelEndpoint == "" {
        return fmt.Errorf("failed to create OTLP trace exporter: endpoint not set")
    }

    if err := initTracer(ctx, otelEndpoint); err != nil {
        return err
    }

    appLogger.Debug().Msg("[Telemetry] Tracer initialized")

    otelMetricsEndpoint := os.Getenv("OTEL_EXPORTER_OTLP_METRICS_ENDPOINT")
    if otelMetricsEndpoint == "" {
        otelMetricsEndpoint = os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
    }

    if otelMetricsEndpoint == "" {
        return fmt.Errorf("failed to create OTLP metric exporter: endpoint not set")
    }

    if err := initMetricProvider(ctx, otelMetricsEndpoint); err != nil {
        return err
    }

    appLogger.Debug().Msg("[Telemetry] Metric Provider initialized")

    meter := MeterProvider.Meter("pth-auth-service")
    registerMetrics(meter)

    return nil
}

Integrating Telemetry with the Router

To enable telemetry in the router, we use OpenTelemetry middleware, which ensures all incoming requests are instrumented for tracing and metrics:

func SetupRouter() *gin.Engine {
    router := gin.Default()

    router.Use(otelgin.Middleware("pth-auth-service"))
    if telemetry.MeterProvider != nil {
        router.Use(middlewares.RequestMetricsMiddleware(telemetry.MeterProvider.Meter("pth-auth-service")))
    }

    router.Use(middlewares.LogRequestResponseMiddleware)

    router.GET("/401", controllers.UnauthorizedResponse)
    router.GET("/403", controllers.ForbiddenResponse)

    router.Use(middlewares.TenantValidation())
    router.Use(middlewares.TokenVerification())

    router.POST("/auth", controllers.Auth)
    router.GET("/health", controllers.Health)
    router.GET("/version", controllers.Version)

    return router
}

By adding OpenTelemetry middleware, we ensure that all HTTP requests are traced, allowing developers to debug issues more effectively.

References

This setup ensures observability for our Golang service, helping developers gain insights into system behavior and performance.