Forem: Dylan Dumont

The RED Method: Request Rate, Errors, and Duration as Your Core SLIs

Dylan Dumont — Sun, 19 Apr 2026 12:39:38 +0000

"Noise drowns out signal; focus on the three metrics that actually indicate system health."

What We're Building

We are instrumenting a Go-based HTTP handler to expose the three Request Rate, Errors, and Duration metrics required to calculate Service Level Indicators (SLIs). This scope excludes internal tracing spans or database metrics, focusing strictly on the surface API gateway to ensure consistency across a distributed backend. The goal is to replace legacy monitoring scripts with a structured metrics export that feeds directly into a Prometheus stack.

Step 1 — Instrument the Middleware

The first step is intercepting incoming requests before they reach the application logic. You need a middleware function that wraps the handler and captures the timing start point.

type RequestInfo struct {
    Start time.Time
}

func RequestMetricsMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        reqInfo := RequestInfo{Start: time.Now()}

        // Wrap the original handler logic here
        next.ServeHTTP(w, r)

        // Extract duration
        duration := time.Since(reqInfo.Start)
    })
}

This separation ensures the application logic remains clean while observability concerns are handled at the infrastructure boundary.

Step 2 — Aggregate Request Counts

Counters track the total volume of requests. You should maintain separate counters for 4xx errors and 5xx errors to distinguish client failures from server failures.

var (
    totalRequests = prometheus.NewCounter(prometheus.CounterOpts{
        Name: "api_total_requests_total",
        Help: "Total number of API requests.",
    })

    error5xx = prometheus.NewCounter(prometheus.CounterOpts{
        Name: "api_errors_5xx_total",
        Help: "Server-side errors.",
    })
)

Counters are essential for calculating Request Rate per second, which helps determine capacity planning thresholds.

Step 3 — Classify Error Labels

Do not just count errors; label them. Use status codes (2xx, 4xx, 5xx) as labels to allow you to query specific failure modes later.

func recordError(status int) {
    if status >= 500 {
        error5xx.Inc()
        // Record 4xx in a similar gauge or counter with a label
    }
}

This specificity allows you to distinguish between a rate-limiting issue (429) and a database crash (500) during incident response.

Step 4 — Measure Latency Histograms

Duration needs more than an average. A histogram with percentiles (p50, p95, p99) is required to understand the tail latency that impacts user experience.

duration := time.Since(reqInfo.Start)
apiDurationHistogram.Observe(duration.Seconds())

Histograms normalize for request volume, preventing a flood of requests from skewing the average latency significantly.

Step 5 — Export Metrics via HTTP Endpoint

The final step is exposing these values so a collector like Prometheus can scrape them every 15 seconds. Ensure your server does not block during the write phase.

func startServer() {
    mux := http.NewServeMux()
    mux.Handle("/metrics", prometheus.Handler())
    http.ListenAndServe(":8080", mux)
}

Standard HTTP endpoints provide the necessary protocol compliance for cloud-native observability stacks.

Key Takeaways

Request Rate provides visibility into traffic volume and helps identify capacity saturation points in real-time.
Errors must be labeled by status code to allow engineers to differentiate between client and server failures.
Duration histograms are superior to averages because they reveal the tail latency that causes actual user complaints.
Instrumentation should happen at the edge, ensuring that metrics reflect the contract presented to the client, not internal implementation details.
SLOs derived from these RED metrics drive meaningful alerts rather than noise from every internal dependency failure.

What's Next?

Next, define Service Level Objectives (SLOs) based on the 99.9th percentile of the Duration histogram. You should calculate error budgets to determine how much failure is acceptable before slowing down feature deployment. Finally, implement alerting rules that trigger on sustained spikes in error5xx over 5xx rates exceeding your threshold for one minute.

Building a Job Queue in Rust: Persistent Tasks With Retry Logic

Dylan Dumont — Fri, 17 Apr 2026 12:40:11 +0000

"Transient failures are inevitable; durable execution requires state to survive the crash."

What We're Building

We are constructing a resilient worker service in Rust that processes background tasks from a persistent queue. This example prioritizes data durability over peak throughput, ensuring that failed jobs are never lost but eventually succeed or move to a dead letter queue. We will use async Rust with SQL for storage, demonstrating how to structure state transitions that survive application restarts. The focus is on architectural correctness over raw performance, building a foundation for long-running background processing systems.

Step 1 — Define the Job State Machine

The worker must track a job's lifecycle without relying on volatile memory alone. We start by defining an enum that explicitly tracks every state transition, ensuring the logic is exhaustive.

pub enum JobStatus {
  Pending,
  Running,
  Succeeded,
  Failed,
  DeadLetter,
}

This choice matters because explicit states prevent silent state drifts that often plague long-running daemon processes. By forcing the developer to handle every case, we reduce the chance of forgetting to update a database column after a panic.

Step 2 — Persist Job State in Storage

A transient failure of the application worker must not result in data loss. We model the job table to include columns for status, retry count, and last attempt timestamp, creating a source of truth that survives restarts.

#[derive(sqlx::FromRow)]
pub struct Job {
  pub id: uuid::Uuid,
  pub status: JobStatus,
  pub retry_count: i32,
  pub created_at: DateTime<Utc>,
  pub last_attempted: Option<DateTime<Utc>>,
}

Storing metadata here allows us to query for pending work and ensures we can resume processing from exactly where the application died. We use UUIDs for the ID to maintain uniqueness and avoid accidental collisions.

Step 3 — Implement Exponential Backoff Logic

When a job fails, we must wait before retrying to prevent database overload. We generate a delay based on the current retry count, using a tokio::time::sleep to enforce a pause before the next attempt.

pub fn calculate_delay(retry_count: i32) -> Duration {
  // Start with 1 second delay and double it with each retry
  let base_duration = Duration::from_secs(1);
  let max_duration = Duration::from_secs(30);

  let raw_delay = base_duration * (1 << (retry_count as u32));
  let capped_delay = raw_delay.min(max_duration);

  // Add jitter to prevent thundering herd issues
  let jitter = Duration::from_millis(rand::random::<u64>() % 100);

  Duration::from_secs(capped_delay.as_secs() + jitter.as_secs())
}

Using exponential backoff instead of a fixed delay ensures that transient network issues resolve without overwhelming the system resources. The jitter component is critical for preventing multiple workers from retrying at the exact same second, which can cause spikes in database load.

Step 4 — Handle Permanent Failures in a DLQ

A job should not be retried infinitely if the error is irrecoverable. If the retry count exceeds a threshold, we transition the state to DeadLetter to prevent an infinite loop and allow operators to manually inspect or discard the job.

pub fn should_retry(job: &Job, error: &Error) -> bool {
  if job.retry_count >= MAX_RETRIES {
    // Mark as DeadLetter
    return false;
  }
  true
}

This separation isolates error handling from success paths, adhering to the principle of separation of concerns. The DeadLetter state acts as a final repository for problematic jobs, ensuring the system doesn't block on them.

Takeaways

Building a durable job queue requires treating state as an external truth source rather than application memory. By defining a strict state machine and persisting it in a relational database, we ensure that no work is ever lost even if the worker process crashes. The retry logic with exponential backoff protects system health, while the dead letter queue allows for manual intervention on permanent failures. This pattern scales well for any background processing system that values correctness over speed. The separation of concerns—logic for success, logic for retry, logic for failure—ensures that the code remains maintainable and the architecture remains robust against transient failures.

To expand on this pattern, consider adding concurrency controls to process jobs in parallel without overloading the database write locks. Investigate how postgres connection pooling interacts with long-running transactions when processing large payloads. Finally, review the logging strategies for tracking job lifecycle events in a distributed system context to ensure observability aligns with operational expectations. You might also consider implementing a metrics pipeline to track average processing times per job type.

Reading

Books

Designing Data-Intensive Applications (Kleppmann): Covers the tradeoffs between durability and availability that inform our database schema choices.
A Philosophy of Software Design (Ousterhout): The chapter on coupling applies to how we separate the retry logic from the processing logic.

Log-Structured Merge Trees: The Data Structure That Powers Modern Databases

Dylan Dumont — Thu, 16 Apr 2026 12:43:31 +0000

LSM trees optimize write performance by buffering changes in memory before flushing to disk sequentially.

What We're Building

We are implementing a simplified LSM tree architecture to understand the mechanics behind high-throughput databases like Cassandra and RocksDB. This scope focuses on the core trade-off between write speed and storage durability. We will explore how write-heavy workloads are decoupled from read-heavy operations by leveraging sequential disk access rather than random seeking. This pattern is essential for modern backend systems handling massive logging or ingestion streams.

Step 1 — In-Memory Write Buffer

The core innovation of LSM trees is the in-memory buffer called a memtable. Incoming writes are appended to this vector rather than hitting the disk immediately. This drastically reduces the number of expensive seek operations required to update the dataset. In a production context, this allows the system to absorb burst traffic by queuing updates until the buffer capacity is reached.

struct MemTable {
    entries: Vec<(String, String)>,
    capacity_limit: usize,
}

impl MemTable {
    fn new(capacity: usize) -> Self {
        MemTable {
            entries: Vec::new(),
            capacity_limit: capacity,
        }
    }

    fn append(&mut self, key: String, value: String) {
        self.entries.push((key, value));
    }
}

Rust vectors provide contiguous memory allocation, which aligns perfectly with how operating systems handle sequential writes to storage media.

Step 2 — Flushing to SSTables

Once the memtable fills its capacity limit, the buffer is frozen and flushed to the disk as a Sorted String Table (SSTable). This process is asynchronous and ensures that no data is lost by moving the buffer content into an immutable file. The file is written sequentially, which minimizes random read/write amplification that plagues traditional B-Trees under heavy update loads.

fn flush_memtable_to_disk(entries: &MemTable) -> String {
    // Simulate serialization to SSTable format
    let data = serde_json::to_vec(&entries.entries).unwrap();
    // In reality, this would be written to disk with compression
    String::from("sst-000001") 
}

Immutable files allow multiple versions to coexist without locking, enabling readers to see consistent snapshots of the database state.

Step 3 — Indexing for Random Reads

Storing all data in memtables would be inefficient for reads. We must maintain indexes to locate keys quickly. We utilize bloom filters or inverted indices to determine if a key exists in an SSTable before loading the full file into memory. This check is fast and requires minimal disk I/O, making the system efficient for large datasets.

use std::collections::HashSet;

fn check_bloom_filter(sst_key: &str) -> bool {
    // Check bit array to determine existence probability
    sst_key.contains("valid") 
}

The bloom filter returns a probabilistic false positive but never a false negative, ensuring that valid keys are always found.

Step 4 — Asynchronous Compaction

Over time, the number of SSTable files grows, consuming excessive disk space. A background compaction process merges these files, removing duplicates and reclaiming free space. This ensures that the storage usage remains proportional to the active data lifespan.

fn compact_sstables(sst_files: &[String]) -> Vec<String> {
    // Simulate merging files into a new sorted file
    let mut merged = Vec::new();
    for file in sst_files {
        merged.push(format!("merged-{}", file));
    }
    merged
}

This process runs on a separate thread to ensure that write throughput is not degraded by background cleanup tasks.

Key Takeaways

Write Amplification: LSM trees reduce write amplification by grouping updates and writing them sequentially.
Sequential I/O: By avoiding random seeks, the system utilizes the high bandwidth of modern storage.
Durability: Data is persisted as soon as it is written to the immutable SSTable on disk.
Concurrency: The write buffer allows high concurrency without contention on the storage device.

What's Next?

Explore immutable snapshots for point-in-time recovery.
Investigate how memtables handle concurrent reads with read amplification.
Consider the trade-offs when implementing bloom filters in low-memory environments.
Compare LSM tree implementations across different database engines like Redis and RocksDB.

Exactly-Once Delivery Is a Lie: How Systems Approximate It Anyway

Dylan Dumont — Tue, 14 Apr 2026 12:44:01 +0000

Reliability in distributed systems does not mean eliminating failures; it means ensuring failures do not corrupt state.

What We're Building

We are implementing a financial order service that processes thousands of transactions per second across multiple availability zones. The requirement is strict consistency regarding money. We cannot charge a customer twice, even if the network retries. We must handle network timeouts without creating ghost orders or losing revenue. This architecture assumes the network is eventually consistent and focuses on managing the retries safely rather than pretending they never happen.

Step 1 — The Idempotency Key

Every incoming request must carry a unique identifier. When the service receives a payment instruction, it checks if that identifier has already been processed. If a duplicate arrives, the service rejects it immediately without executing business logic. This prevents double-charging when a client resends a failed request. The code ensures the identifier is hashed and stored before processing the payload.

func (s *Service) CreateOrder(ctx context.Context, req *OrderRequest) error {
    id := req.IdempotencyKey

    // Check existing state in a separate store
    existing := s.db.GetProcessedKey(ctx, id)
    if existing != nil {
        return nil // Return previously created order, don't process again
    }
    // ... process order
    return s.db.InsertOrder(ctx, order)
}

Step 2 — The Outbox Pattern

Database transactions and message queue deliveries often fail together. We separate these by writing to a local table during the same transaction as the database record. A separate worker process reads this table and sends the message to the external queue. This guarantees the order event is logged before it is sent, ensuring durability without requiring complex two-phase commits across services.

// Inside a transaction
tx := db.Begin()
db.InsertOrderRecord(tx, order) // Write data
db.InsertOutboxEntry(tx, id)   // Write event log
tx.Commit()
// Worker reads 'id' and sends to MQ asynchronously

Step 3 — State Machine Validation

We track the lifecycle of each order. States include Created, Paid, and Completed. A transition from Created to Paid is idempotent. A retry on a Paid order is ignored. If the database is down, the state remains Created. Once the database recovers, we replay the log. The state machine prevents moving backward or skipping steps, which would violate accounting principles. We use a finite state machine to enforce valid transitions at every request boundary.

Step 4 — Compensating Transactions

If a payment fails after an order is created, we must undo the side effects. This is a compensating transaction. We record a cancellation event with the same unique identifier logic as the creation request. If a cancellation arrives, we check the current state. If the state is still Created, we update it to Cancelled and refund the account. This ensures we never hold funds indefinitely without a valid order. The worker processes cancellations with the same priority as creation.

func (s *Service) CancelOrder(ctx context.Context, key string) error {
    order, err := s.db.FindByOrderKey(ctx, key)
    if order == nil {
        return errors.New("not found")
    }
    if order.State != "CREATED" {
        return nil // State is already processed
    }
    return s.db.UpdateOrderState(ctx, order.ID, "CANCELLED")
}

Key Takeaways

Idempotency keys protect against duplicate requests but require client support. The Outbox pattern ensures messages are not lost during DB failures. State machines validate valid lifecycle paths and prevent invalid updates. Compensating transactions clean up partial failures and maintain financial integrity. Together, these patterns approximate exactly-once semantics by acknowledging network unreliability and designing for eventual consistency.

What's Next?

The final step is monitoring. We track the ratio of retries to successes. If the retry rate spikes, we check the queue depth. We also monitor the deduplication store for memory usage. High cardinality of keys can cause performance degradation. We plan to shard the state storage to handle large volumes of concurrent requests. Next, we will explore distributed tracing to observe these flows across service boundaries.

Architecture Patterns Series

Part of the Architecture Patterns series.

Serde Deep Dive: Custom Serialization for Wire Protocols

Dylan Dumont — Tue, 14 Apr 2026 06:13:38 +0000

"When standard serialization attributes can't satisfy your wire protocol constraints, you must descend to manual implementation for precision."

What We're Building

We are constructing a production-grade Rust module that bridges an internal domain model with an external network protocol. The goal is to enforce strict field ordering, handle optional legacy fields, and ensure binary-compatible output without the rigidity of default JSON serialization. This pattern addresses scenarios where you cannot use generic JSON and need to optimize for specific wire protocols like gRPC, FlatBuffers, or custom binary streams. We will use serde for its flexibility while demonstrating where the derive macros fall short, necessitating a custom approach to maintain architectural integrity.

Step 1 — Define the Internal Domain Model

First, establish the clean Rust type that represents your business logic. This structure should contain only the fields strictly necessary for your application logic. The wire protocol will be an abstraction built around this, not a replacement for it. You need to isolate the core state from the transport concerns to ensure your business logic remains decoupled from network specifics.

#[derive(Debug)]
pub struct UserUpdate {
    pub id: i32,
    pub name: String,
    pub metadata: Option<String>,
}

Step 2 — Override Serialization Attributes

Default Serialize implementations serialize optional fields as null or skip them entirely depending on the variant. For wire protocols requiring specific field ordering or skipping nulls entirely, you must implement custom serialization logic. Use serialize_with or implement the trait manually to enforce strict formatting rules required by your downstream consumers.

#[derive(Serialize)]
struct UserUpdate {
    #[serde(skip_serializing_if = "Option::is_none")]
    metadata: Option<String>,
}

Step 3 — Manage Optional and Deprecated Fields

Wire protocols often require deprecated fields to be removed from payloads or sent as specific boolean flags rather than null. This prevents version drift and ensures forward compatibility. By marking optional fields with skip_serializing_if or custom flags, you control the exact byte stream sent over the network without altering the internal struct definition.

Step 4 — Implement Custom Deserialization Logic

Deserialization is where validation usually fails silently in default implementations. You should customize the error output to be actionable for the API client. Creating a custom Deserialize implementation allows you to map unknown fields to a safe default instead of failing with a generic type error, which improves resilience when handling version mismatches.

Step 5 — Control Versioning and Protocols

Versioning in wire protocols is often managed via a header or a specific enum tag. In Rust, this maps to serde::flatten or tagged unions for handling different schema versions within the same payload. Implementing a VersionedMessage wrapper struct allows you to serialize multiple schema versions into a single wire packet structure.

Key Takeaways

Type Safety: Custom serialization ensures your internal state maps exactly to your wire format without unexpected fields leaking into the network stream.
Protocol Agnosticism: This pattern allows you to switch underlying transport mechanisms (HTTP, UDP, TCP) by swapping the serialization context without changing the business logic.
Backward Compatibility: By explicitly managing how optional fields are handled, you prevent breaking changes when deploying new service versions.
Error Granularity: Custom deserialization provides precise error messages that help clients fix issues quickly rather than facing generic serialization failures.
Code Maintainability: Centralizing wire protocol logic in custom Serialize implementations keeps business logic clean and separated from transport concerns.
Performance: Custom serialization allows you to avoid allocating unnecessary types during the serialization process, leading to more efficient wire formats.

What's Next?

Explore how to integrate serde with binary protocols like Protobuf by generating Rust structs from .proto definitions. Investigate how to handle compression transparently using zstd wrappers over the serialized payload. Finally, consider implementing serde-like abstractions in Go using encoding/json or msgp to compare architectural trade-offs between the two ecosystems.

Cooperative Cancellation: Propagating Shutdown Signals Through Async Trees

Dylan Dumont — Mon, 13 Apr 2026 07:09:37 +0000

Fire-and-forget concurrency leaves dangling resources until you explicitly stop the engine.

What We're Building

We are implementing a worker tree where a shutdown signal from the root propagates to all leaf nodes without dropping tasks or leaking memory. In this scope, we define a control structure that traverses down an async call stack. The design ensures that when the parent task terminates, child tasks receive a clean interrupt to run finalizers. We focus on Rust using tokio because its ownership model handles cleanup deterministically. The goal is a robust shutdown pattern applicable to gRPC servers, message queues, or high-throughput data pipelines where abrupt termination causes data loss.

Step 1 — Establish a Broadcast Channel

You need a mechanism to send a single shutdown command to many tasks simultaneously. Instead of passing mutable references, create a tokio::sync::broadcast channel. The receiver (Receiver) lives with the task logic, while the sender (Sender) moves into the shutdown function at the root level.

let (shutdown_tx, mut shutdown_rx) = tokio::sync::broadcast::channel(1);
// ... pass shutdown_tx to parent ...
// ... pass shutdown_rx to children ...

Rust's channel model prevents data races by enforcing move semantics, ensuring the sender isn't copied inadvertently.

Step 2 — Distribute Tokens to Children

When spawning a worker, pass a clone of the shutdown_rx handle along with the task payload. This allows the child task to become an independent subscriber to the same control stream. Do not clone the receiver for every logical sub-operation; pass one per task instance.

let worker = async move {
    // Process data...
    tokio::select! {
        result = do_work() => result,
        _ = shutdown_rx.recv() => Ok(()),
    }
};
tokio::spawn(worker);

Cloning receivers is cheap, but passing them maintains the logical tree structure required for propagation.

Step 3 — Listen in a Select Statement

To react immediately to a shutdown signal, wrap your core logic in a tokio::select! block. This combinator chooses between the successful completion of work and the reception of a cancellation message. This pattern prioritizes the interrupt over normal flow, preventing the task from getting stuck in a blocking operation.

tokio::select! {
    Ok(data) = task_data_channel.recv() => process(data),
    _ = shutdown_rx.recv() => break Loop,
    _ = timer.tick() => panic!("Timeout"),
}

select! provides a non-blocking path for interruption, which is critical for maintaining liveness in a tree.

Step 4 — Implement Cleanup Closures

A shutdown signal must trigger resource release, such as closing file handles or dropping database connections. Wrap the logic in a closure or match arm that executes cleanup code upon break Loop. This ensures that Drop traits are called even if the task was interrupted mid-process.

let worker = async move {
    match do_work().await {
        Ok(_) => println!("Work done"),
        Err(e) => handle_error(e),
    }
};
// On shutdown, ensure resources are released
if let Err(e) = shutdown_logic.await {
    eprintln!("Interrupted: {}", e);
}

Handling errors explicitly avoids silently swallowing the reason for cancellation.

Step 5 — Trigger from the Root

The parent task holds the Sender. When application exit is imminent, call shutdown_tx.send(true). Because it is a broadcast channel, all registered children receive the signal instantly. The propagation stops at the leaf nodes where tasks exit their loops. This top-down approach prevents "zombie" threads in an async runtime.

async fn start_server() {
    let (tx, mut rx) = tokio::sync::broadcast::channel(1);
    spawn_task(&tx);
    // ...
    shutdown_tx.send(true).unwrap();
}

A single broadcast ensures a consistent state where no child task is left unaware of the shutdown decision.

Key Takeaways

Broadcast Channels — Single signals can interrupt multiple independent tasks simultaneously without needing shared mutable state.
Select Patterns — Using tokio::select! prioritizes shutdown interrupts over long-running blocking operations.
Ownership Safety — Move semantics in Rust ensure shutdown handles are consumed, preventing accidental double-sending.
Cleanup Hooks — Explicit error matching allows you to finalize resources rather than relying solely on Drop.
Tree Propagation — Top-down shutdown prevents straggling tasks that waste CPU cycles on orphaned logic.

What's Next?

Extend this pattern to handle network I/O that times out automatically. Investigate how to implement exponential backoff if a task fails immediately after a shutdown signal. Consider adding metrics to track how long each level takes to drain during shutdown. Finally, integrate this into your CI pipeline to ensure no resource leaks occur under load.

Rate Limiting Deep Dive: Token Bucket vs Leaky Bucket vs Sliding Window

Dylan Dumont — Sun, 12 Apr 2026 07:09:14 +0000

Protect your backend infrastructure from resource exhaustion by controlling traffic admission with precision.

What We're Building

This article guides you through implementing robust rate limiting strategies in Go. We explore the core mechanics of Token Bucket, Leaky Bucket, and Sliding Window algorithms, explaining their trade-offs in handling bursts versus sustained load. We also address persistence for distributed systems and selection criteria to ensure your system remains stable under pressure.

Step 1 — Define the Rate Limit Contract

Before selecting an algorithm, you must establish a data structure to track time and tokens. This contract ensures type safety and serialization compatibility. In Go, you typically use an atomic integer for simple counters or a sync.Map for complex state. For a Token Bucket, you track tokens and lastRefillTime. For a Sliding Window, you track a map of timeWindow to count.

type RateLimiter struct {
    tokens       int64
    capacity     int64
    lastRefill   time.Time
    refillRate   int64 // tokens per second
}

The RateLimiter struct encapsulates the state required for a single-node implementation. By centralizing the state here, you ensure that every request goes through a consistent logic path. This also simplifies debugging. If the rate limit is too strict, you can increase capacity or refillRate without changing the structural layout. This approach supports testing by isolating the logic.

Step 2 — Implement Token Bucket for Burst Tolerance

This algorithm simulates a bucket of tokens that fills up at a fixed rate. A request consumes one token. If empty, the request is rejected. This allows bursts up to the bucket capacity. When no requests arrive for 1 second, the bucket refills to capacity.

func (rl *RateLimiter) Allow() bool {
    now := time.Now()
    // Refill logic
    elapsed := now.Sub(rl.lastRefill)
    tokens := elapsed.Microseconds() / (1000000 / rl.refillRate)
    newTokens := rl.tokens + tokens
    if newTokens > rl.capacity {
        newTokens = rl.capacity
    }
    rl.tokens = newTokens
    rl.lastRefill = now

    if rl.tokens > 0 {
        rl.tokens--
        return true
    }
    return false
}

The refill calculation uses elapsed to determine how many tokens to add. This handles sudden traffic spikes better than fixed windows, which reset abruptly at the end of a period. For example, if you allow 100 requests per second, a sudden burst of 50 requests is allowed, then the system waits for the remaining capacity to refill before allowing the next batch. This reduces latency for legitimate high-demand periods.

      Tokens
      ^
      |
      |      [ Bucket Fills ]
      |     /
      |    /
      |---/----------------> Time
      |  /    [ Burst Allowed ]
      | /
      |/

Step 3 — Implement Leaky Bucket for Traffic Smoothing

Unlike the Token Bucket, the Leaky Bucket processes requests at a constant rate regardless of arrival rate. Requests enter a queue. If the bucket is full, new requests are dropped. The "leak" rate represents the backend processing speed. This prevents downstream overload from upstream bursts.

func (lb *LeakyBucket) Enqueue(req Request) bool {
    if lb.queueDepth > lb.maxDepth {
        return false // Drop request
    }
    lb.queueDepth++
    // Simulate leak
    go func() {
        time.Sleep(time.Second / lb.leakRate)
        if lb.queueDepth > 0 {
            lb.queueDepth--
        }
    }()
    return true
}

The go routine simulates the leak. In production, this leak logic should run on the same thread to avoid concurrency issues or use a dedicated worker pool. The core logic is the comparison of queueDepth against maxDepth. When depth exceeds the limit, the request is rejected with a 429 status. This algorithm is ideal for protecting a specific downstream service from high variance in incoming traffic.

Step 4 — Implement Sliding Window for Precision

The Token Bucket and Leaky Bucket are approximations. The Sliding Window counts requests in a fixed time interval (e.g., 1 second) and checks the count. It tracks a sliding window of logs within a window. Code checks the count since start time. The window shifts forward with every tick.

func (sw *SlidingWindow) Check(req Request) bool {
    window := req.timeWindow
    count := window.counts[req.time]
    if count < sw.limit {
        count++
        window.counts[req.time] = count
        return true
    }
    return false
}

This approach is accurate because it counts exactly how many requests have arrived in the current window. However, it is more complex to implement efficiently without a database or cache. A naive implementation uses a slice or map of timestamps. For high throughput, you might combine a Token Bucket with a counter reset. The benefit is that the reset is gradual rather than abrupt. This prevents the "sawtooth" pattern where all limits are reached just before the reset.

Step 5 — Persist State for Distributed Systems

Single-node implementations fail under multi-node load. You must use Redis for multi-node state. The key is to store the current state in Redis. When a request arrives, the application atomically checks and updates the Redis key. This avoids a single point of failure.

# Example Redis Lua Script
local tokens = tonumber(redis.call('GET', KEYS[1]) or "100")
local now = tonumber(redis.call('SET', KEYS[1], tokens - 1, 'PX', 1000))
return now

The Lua script ensures atomicity. If you used separate GET and SET commands, a concurrent request might read a stale value and succeed, bypassing the limit. The PX parameter sets an expiration on the key. This ensures memory leaks are managed. This setup is required for microservices where no central authority controls the rate limit.

Step 6 — Select the Right Algorithm

Your choice depends on traffic patterns and downstream constraints.

Pattern	Algorithm	Reason
Burst Tolerance	Token Bucket	Allows spikes up to capacity.
Constant Rate	Leaky Bucket	Smooths output for fragile APIs.
Precision	Sliding Window	Accurate counting without abrupt resets.
Global Limits	Redis + Lua	Shared state across distributed instances.

If you need to protect a specific user or IP, use the Sliding Window. If you want to smooth out traffic spikes for a shared resource, use the Leaky Bucket. For general API protection, Token Bucket is the default choice. The decision matrix ensures you match the algorithm to the specific problem.

Takeaways

State is Key: You must maintain state to track limits.
Atomicity: Use Lua scripts for Redis to prevent race conditions.
Algorithm Choice: Pick based on whether you want burst tolerance (Token) or smoothing (Leaky).
Complexity vs. Accuracy: Sliding Window is most accurate but computationally heavier.
Persistence: Use Redis for distributed setups to share state across instances.
Retry Logic: Clients should retry with exponential backoff if rate limited.

What's Next

Extend this by implementing adaptive rate limiting where the limit adjusts based on response times. You can monitor the tail latency of your backend and increase the limit if the error rate drops. This ensures you utilize available capacity while protecting against overload. You can also add custom headers to inform clients of their current quota.

Two-Phase Commit Demystified: When Distributed Transactions Are Unavoidable

Dylan Dumont — Sat, 11 Apr 2026 12:45:53 +0000

When a banking payment must update inventory in one database and capture funds in another, no single-database transaction can guarantee atomicity across both — that is exactly where two-phase commit becomes necessary.

What We're Building

We are implementing a core logic layer for a high-value e-commerce platform where data integrity is paramount. This system must reserve inventory in one database and capture payment in another simultaneously. If the inventory is reserved but payment fails, refunding becomes a complex operational nightmare involving manual reconciliation and customer dissatisfaction. We need atomicity across distinct database tables that do not share a transaction ID, forcing us to rely on a distributed transaction protocol. A two-phase commit (2PC) ensures that either the order is processed fully or rolled back completely. This architecture assumes a strong consistency requirement, such as financial ledgers or critical reservation slots, justifying the latency cost of synchronous coordination between distributed nodes.

Step 1 — Define the Transaction Coordinator

The coordinator acts as the conductor of the distributed orchestra, managing the state of the transaction across all participants. It holds the write lock for the global transaction ID (XID) and sends instructions to resources. In our design, the coordinator is an internal RPC service written in Go, utilizing context.Context for cancellation. This choice matters because Go's goroutine scheduling and cancellation tokens naturally model the synchronous wait for participant readiness without spawning heavy threads. We define the ResourceManager interface to abstract the underlying database drivers, ensuring loose coupling between the coordinator and specific storage engines.

type ResourceManager interface {
        Prepare(context.Context, string) (string, error)
        Commit(context.Context, string) error
        Rollback(context.Context, string) error
}

Step 2 — Request Prepared States

The coordinator sends a PREPARE message to each registered resource, asking if they can safely commit the changes requested. This phase involves the resources checking internal locks, network connectivity, and storage health. Resources must write their vote to stable storage before responding to ensure they can survive a crash between the vote and the final commit decision. This step effectively pauses the transaction until every participant confirms its ability to participate, which introduces latency but guarantees that no partial updates will be applied.

func (c *Coordinator) prepareAll(ctx context.Context, resources []ResourceManager) int {
        responses := make(chan *prepareResponse, len(resources))
        for _, res := range resources {
            go func(r ResourceManager) {
                status, err := r.Prepare(ctx, c.xid)
                responses <- &prepareResponse{Status: status, Err: err}
            }(res)
        }
        for range resources {
            resp := <-responses
            if resp.Err != nil || resp.Status == Failed {
                return -1
            }
        }
        return 0
}

Step 3 — Aggregate the Vote

Once responses arrive, the coordinator aggregates the votes to determine the final decision. If any resource returns a negative status, the coordinator must abort the transaction immediately to preserve data integrity. This step ensures that a single point of failure in a resource doesn't corrupt the global state, preventing the "dirty read" scenario where data is half-updated. The coordinator acts as a quorum collector, ensuring that a majority of votes match the consensus required before proceeding. It prevents the scenario where one node commits while another fails, which would lead to data inconsistency in a distributed environment.

func (c *Coordinator) decide(ctx context.Context, votes map[string]int) error {
        if len(votes) < 1 || votes["ABORT"] > 0 {
                return errors.New("one resource voted to abort")
        }
        return nil
}

Step 4 — Finalize the Outcome

After voting, the coordinator broadcasts a COMMIT or ROLLBACK instruction based on the aggregate result. In the commit case, resources apply their changes and release locks. The coordinator must persist this decision locally to an atomic log before notifying resources to prevent a state where a resource has committed but the coordinator lost its record, leading to an inconsistent global state upon restart. Conversely, if the vote was negative, the rollback is mandatory. This ensures no data is left in an intermediate state. The protocol requires two stable storage writes (vote and decision) and ensures that no resource moves forward unless the coordinator has a record of the vote.

Performance and Trade-offs

Two-phase commit introduces performance overhead. The coordinator must wait for all participants before proceeding, which creates a bottleneck. In high-throughput e-commerce, this might delay order confirmation. However, the alternative is data inconsistency, which is unacceptable for financial systems. Modern implementations optimize 2PC by using asynchronous networking or batching commits, but the synchronous nature remains. It's crucial to monitor the latency introduced by the preparation phase, as resources holding long-held locks can starve other transactions.

Conclusion

Implementing a two-phase commit protocol is a complex but necessary strategy for systems requiring distributed atomicity. While it introduces latency and complexity, it provides the reliability needed for critical financial or inventory management applications. By carefully coordinating resources and handling failure states, we ensure that the ledger remains consistent across distributed nodes.

Part of the Architecture Patterns series.

Database Connection Pooling: What Every Backend Developer Should Know

Dylan Dumont — Sat, 11 Apr 2026 06:31:13 +0000

Without a pool, every request forces a TCP handshake, database authentication, and context switch, turning latency spikes into a denial-of-service scenario.

What We're Building

We are implementing a production-grade connection management pattern in Go. This scope focuses on configuring limits, handling lifecycle, and monitoring health to prevent resource exhaustion in a high-concurrency API service. We assume a standard PostgreSQL backend where drivers like pgx are used to expose pool metrics.

Step 1 — Initialize the Pool at Startup

Database connections should never be instantiated on-demand per request in a concurrent environment. Instead, the pool is initialized once when the application service starts, ensuring resources are pre-warmed. This approach avoids the latency of TCP handshakes for every incoming HTTP request.

db, err := sql.Open("postgres", "conn=...")
if err != nil {
    log.Fatalf("Failed to open database: %v", err)
}

This specific choice matters because sql.Open only returns a client handle; actual connection allocation happens lazily based on configured limits.

Step 2 — Configure Limits and Timeouts

You must define MaxOpenConns to limit the maximum number of connections the driver will open and SetConnMaxLifetime to ensure connections are recycled. Setting a timeout on idle connections prevents the pool from holding stale sockets indefinitely, which is critical for network instability.

db.SetMaxOpenConns(100)
db.SetMaxIdleConns(25)
db.SetConnMaxLifetime(5 * time.Minute)

This specific choice matters because the database server and the client process may share the same network subnet, increasing the risk of socket exhaustion if not strictly limited.

Step 3 — Manage Idle Connection Recycling

Idle connections are candidates for eviction if they sit too long without being used. The SetMaxIdleConns value combined with SetMaxIdleTime ensures that unused connections are closed automatically. This prevents the accumulation of "zombie" connections that consume file descriptors but provide no throughput.

db.SetConnMaxIdleTime(30 * time.Second)

This specific choice matters because idle connections often become invalid due to firewall timeout settings or intermediate proxy resets, causing sudden failures when accessed.

Step 4 — Monitor Metrics for Latency

Production systems require visibility into pool health, specifically the Stats() method which returns open and acquired connection counts. You should check these metrics to detect if the application is blocking on Acquire() due to saturation.

stats := db.Stats()
if stats.OpenConnections == stats.MaxOpenConnections {
    log.Printf("Pool is full: %d open, %d waiting", stats.OpenConnections, stats.WaitCount)
}

This specific choice matters because relying solely on HTTP response times often leads to reactive failure when the pool is actually empty and waiting for a handshake.

    +---------+      +---------+      +---------+
    |  Client | ---->|  Pool  | ---->|Database |
    +---------+      +---------+      +---------+
         |              |                 |
         | (Acquire)    | (Execute)       |
         v              v                 v
    [Wait Queue]  <- [Idle Connection] -> [Result]

Key Takeaways

Lazy Initialization: Connections should be pre-warmed at startup rather than created on demand to minimize latency during peak traffic.
Strict Limits: Enforcing MaxOpenConns prevents your application from consuming all available network sockets on the server, which could crash the database.
Recycle Logic: Recycling idle connections based on time is crucial to handling network instability where long-lived sockets might be dropped by proxies.
Visibility: Monitoring the pool stats allows you to react to saturation before it results in timeouts, rather than reacting to HTTP 503 errors after the fact.

What's Next?

To further optimize the service, consider using the pgx driver which provides pgxpool for finer control over connection recycling. You could also implement circuit breakers to stop hitting the database if the pool metrics indicate persistent saturation.

Building a TCP Server in Rust With tokio: From Accept Loop to Graceful Shutdown

Dylan Dumont — Fri, 10 Apr 2026 06:25:28 +0000

Handling concurrent network connections requires managing lifecycles without blocking the event loop, which dictates high-throughput and fault isolation.

What We're Building

We are constructing a production-grade TCP server using the Tokio runtime. The scope includes binding a socket to a local interface, establishing a non-blocking accept loop, spawning isolated tasks for each connection, and implementing a graceful shutdown mechanism that listens for SIGINT or SIGTERM. The resulting architecture ensures that a single failing client does not crash the entire server and allows the system to drain pending requests before exiting.

Step 1 — Initialize the Runtime and Listener

Before accepting connections, you must boot the asynchronous runtime and configure the TCP socket. You bind the listener to a specific address to restrict traffic to the intended interface.

use tokio::net::TcpListener;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let listener = TcpListener::bind("127.0.0.1:8080").await?;
    println!("Listening on port 8080...");
    loop {
        // Accept loop logic follows
    }
}

Using tokio::net::TcpListener ensures the accept operation does not block the current thread, allowing the runtime to handle I/O multiplexing efficiently.

Step 2 — The Non-Blocking Accept Loop

The core logic involves polling the listener for incoming connections. You iterate indefinitely, attempting to pull a stream from the queue, and handling errors like address in use gracefully.

use tokio::sync::mpsc;

// ... previous setup
while let Ok((stream, addr)) = listener.accept().await {
    println!("New connection from: {}", addr);
    // Spawn connection handler
    tokio::spawn(async move {
        // Handle stream...
    });
}

The while let pattern decouples the acceptance logic from the handling logic, preventing the thread from stalling while waiting for a socket to become available.

Step 3 — Isolate Connection Tasks

Each incoming connection must be handled by its own task. This prevents memory growth in a single heap space and allows individual connections to fail without affecting others.

use tokio::sync::oneshot;

let handle = tokio::spawn(async move {
    let (tx, mut rx) = oneshot::channel();
    // Process stream buffer
    // Drop `stream` when done to release resources
    drop(stream); 
});

Spawning connections ensures resource independence, which is critical for high-concurrency backends where a malicious or misbehaving client must not starve legitimate users.

Step 4 — Implement Graceful Shutdown

Stopping the server abruptly drops active sockets, potentially causing timeouts for clients. Graceful shutdown requires draining incoming requests and waiting for active handlers to finish.

use tokio::signal;
use std::time::Duration;

// Create a shutdown channel
let (shutdown_tx, mut shutdown_rx) = mpsc::channel::<()>(1);

tokio::spawn(async move {
    loop {
        tokio::select! {
            Ok((stream, addr)) = listener.accept() => {
                println!("New connection from: {}", addr);
                tokio::spawn(async move {
                    drop(stream); // Handle stream here
                });
            },
            _ = shutdown_rx.recv() => {
                println!("Shutting down...");
                break;
            }
        }
    }
});

// Wait for Ctrl-C, then signal shutdown
signal::ctrl_c().await.expect("Failed to listen for ctrl_c");
let _ = shutdown_tx.send(()).await;

This architecture uses tokio::select! to race between accepting connections and receiving shutdown signals, ensuring the runtime waits for cleanup before terminating.

Key Takeaways

Async I/O: Blocking the event loop inside the accept loop kills performance, so tokio::net::TcpListener is mandatory.
Task Isolation: Spawning connection handlers in the runtime prevents memory leaks and cascading failures across processes.
Signal Handling: Using tokio::signal allows the application to listen for OS-level interrupts and trigger cleanup logic instead of hard exiting.
Resource Cleanup: Dropping handles ensures the operating system reclaims file descriptors once the server stops accepting new connections.
Channel Synchronization: Using channels to signal shutdown allows the main loop to exit cleanly without relying on external processes.
Error Propagation: Handling Result types ensures that binding failures or listener accept errors do not silently ignore critical infrastructure states.

What's Next?

To harden this implementation, you should implement TLS encryption using tokio-rustls to secure data in transit. Adding metrics via Prometheus or OpenTelemetry allows you to monitor active connection counts and response times in real time. Next, consider migrating to a framework like Actix-web to integrate this handler into a full API surface with routing support. Finally, implement circuit breaker patterns to prevent cascading failures if upstream dependencies become unstable.

Stream Processing vs Batch Processing: When to Use Each

Dylan Dumont — Thu, 09 Apr 2026 17:17:02 +0000

Batch processing offers throughput and accuracy, while stream processing offers latency. Your architecture must decide which metric survives production requirements.

What We're Building

We are designing a financial transaction monitoring system. This system requires real-time fraud detection for immediate blocking but also daily reconciliation for regulatory compliance. We cannot choose one paradigm exclusively. We must structure our data pipelines to handle high-volume historical data efficiently while processing live events with low latency. The core trade-off involves computational resources, consistency models, and operational complexity.

Step 1 — Define Latency Requirements

The first step is distinguishing between micro-batch and real-time needs. Micro-batches are small chunks processed periodically. Real-time implies processing each record as it arrives. We define a latency threshold T. If user experience demands T < 100ms, batch is insufficient. If the system tolerates T < 5 minutes, batch becomes viable for cost savings.

# Latency configuration struct
from dataclasses import dataclass

@dataclass
class ProcessingConfig:
    batch_interval_seconds: int
    stream_latency_threshold_ms: int
    consistency_model: str

config = ProcessingConfig(
    batch_interval_seconds=60,
    stream_latency_threshold_ms=100,
    consistency_model="at_least_once"
)

Defining these constraints upfront prevents over-engineering the infrastructure with unnecessary state stores.

Step 2 — Implement Batch Layer

For the reconciliation task, we ingest terabytes of historical logs. Batch processing allows us to parallelize work across a cluster without worrying about message ordering for every single event. We use Apache Spark logic to aggregate data over fixed windows. This approach maximizes throughput by utilizing local disk storage and minimizing network overhead for heavy computation.

# Pseudo-code for Spark Batch Job
def process_daily_batch(input_path, output_path):
    df = spark.read.parquet(input_path)
    windowed_data = df.groupBy("date").sum("amount")
    windowed_data.write.parquet(output_path)
    spark.catalog.createTable("daily_report", windowed_data)

Batch processing is cost-effective for large datasets where immediate insight is not critical to business logic.

Step 3 — Implement Stream Layer

For fraud detection, we need a continuous consumer. We subscribe to a message broker like Apache Kafka. The stream processor maintains state in memory or in RocksDB to detect patterns within sliding windows. This requires handling backpressure and ensuring exactly-once semantics if financial integrity is non-negotiable.

# Pseudo-code for Stream Consumer
def process_stream_events(consumer_group):
    for event in kafka_consumer:
        risk_score = risk_model.predict(event)
        if risk_score > threshold:
            alert_service.notify(event.user_id)
            sink.write("alert_log", event)

Stream processing introduces overhead but enables immediate intervention, critical for high-stakes environments like finance.

Step 4 — Manage Stateful Processing

State management is the hidden cost of both paradigms. Batch layers often store state in Parquet files on object storage. Stream layers often require managed systems like Flink or Kafka Streams to retain checkpoints. Without managing state, you lose context required for windowing. A stateless stream consumer cannot aggregate totals or detect sequence anomalies across a session.

[BATCH] -> [SINK STORE] -> [LOAD ON DEMAND]
            |
[STREAM] <- [STATE STORE] <- [CHECKPOINT]

Separating state management from computation simplifies scaling and debugging during failures.

Step 5 — Operational Trade-offs

Finally, consider the operational reality. Batch jobs are easier to pause and restart but introduce a cold start latency. Stream jobs require constant monitoring of consumer lag and state store health. Hybrid architectures must orchestrate both via a service bus. Overlapping state stores can lead to data duplication. You must design APIs that unify these views.

# Hybrid orchestration logic
def orchestrate_system():
    if current_time % BATCH_INTERVAL == 0:
        trigger_batch_job()
    else:
        continue_stream_loop()

Balancing these requirements ensures robustness without sacrificing performance.

Key Takeaways

Latency vs. Throughput — Batch wins on volume and cost.
Consistency Models — Batch offers strong consistency. Stream needs eventual consistency.
State Management — Batch uses durable storage. Stream uses in-memory checkpoints.
Data Schemas — Batch handles batched schemas. Stream handles streaming schemas.
Failure Handling — Batch handles offline recovery. Stream handles replayable logs.

What's Next

Next, you might explore event-driven architectures using Apache Kafka and Apache Flink. You can look into optimizing micro-batch latency for near real-time needs. Consider whether a Lambda architecture or a Kappa architecture best fits your team's constraints.

Timeout Propagation: Why Your Deadlines Need to Flow Through the Entire Call Chain

Dylan Dumont — Wed, 08 Apr 2026 07:04:12 +0000

Timeout Propagation: Why Your Deadlines Need to Flow Through the Entire Call Chain

Ignoring a timeout on an API call doesn't isolate the failure; it poisons the shared thread pool and starves concurrent requests.

What We're Building

We are implementing a deadline-aware client in a Go microservice. Our goal is to ensure that when a service receives a request, it knows exactly how much time remains for the entire transaction, not just the current function. This involves calculating the time budget before entering the call chain and passing it explicitly to every downstream dependency.

Step 1 — Establish the Transaction Deadline

Every handler must declare the maximum time allowed for the whole request lifecycle. This prevents downstream services from running indefinitely if the parent service is overwhelmed.

const maxTransactionTime = 5 * time.Second

func handler(req interface{}) error {
    ctx, cancel := context.WithTimeout(context.Background(), maxTransactionTime)
    defer cancel()
    // Use ctx in subsequent logic
}

Defining a global constant prevents ad-hoc timeout logic from appearing scattered across the codebase. It sets a hard boundary for the entire request scope.

Step 2 — Calculate Remaining Window

Before making an external call, you must subtract the local processing time from the remaining transaction deadline. You cannot use the initial deadline for every nested call.

func processPayment(ctx context.Context) error {
    if deadline, ok := ctx.Deadline(); ok {
        remaining := time.Until(deadline)

        // Visualize the budget
        //     [================] -> 5s (Total)
        //        [======] -> 2s (Processing)
        //     [============] -> 3s (Remaining)
        externalCtx, cancel := context.WithTimeout(ctx, remaining)

        // Call external API with 3s remaining
        return api.Call(externalCtx)
    }
}

This ensures that if the first call takes too long, the downstream API is not given a window larger than the original client allowed. It protects your system from cascading delays.

Step 3 — Propagate Context Downstream

When calling out to external libraries, you must pass the new context rather than passing the original background context. The HTTP client should accept the context to handle timeouts.

func fetchUser(ctx context.Context) (*User, error) {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, userURL, nil)
    if err != nil {
        return nil, err
    }
    client := &http.Client{
        Timeout: 100 * time.Millisecond, // Fallback per-call limit
    }
    resp, err := client.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    var user User
    if err := json.NewDecoder(resp.Body).Decode(&user); err != nil {
        return nil, err
    }
    return &user, nil
}

Passing ctx ensures that if the transaction times out, the request is aborted immediately. Without this, the connection remains open, draining file descriptors and memory.

Step 4 — Listen for Deadline Exceeded

When a timeout occurs, the context will trigger a context.DeadlineExceeded error. You must handle this explicitly so the application does not crash on panic.

select {
case <-time.After(1 * time.Second):
    // Normal execution path
case <-ctx.Done():
    if ctx.Err() == context.DeadlineExceeded {
        return errors.New("deadline exceeded")
    }
}

This logic allows the code to exit gracefully or return a specific HTTP 504 Gateway Timeout status. It converts a system panic into a recoverable application error.

Step 5 — Isolate Parallelism Budgets

When launching goroutines or channels, you must ensure they do not consume the parent's entire budget. Split the remaining time among parallel tasks to prevent one slow child from starving others.

// If parent has 3s remaining, do not give all 3s to every child
childCtx, cancel := context.WithTimeout(ctx, 1.5 * time.Second)
go func() {
    // Process in background
}()

Allocating the full remaining budget to every parallel worker causes total starvation if one worker lags. Splitting the budget ensures that even a lagging task cannot block the whole request thread.

Key Takeaways

Deadline Budgeting: Calculate and subtract local processing time to prevent passing infinite time windows to downstream dependencies.
Cancellation Propagation: Pass the context explicitly so that a timeout at the entry point aborts all nested operations immediately.
Resource Starvation: Unbounded timeouts drain thread pools and file descriptors, leading to system-wide hangs rather than isolated errors.
Context Management: Every external call must receive a specific context derived from the original, shrinking deadline.
Graceful Failure: Catch context.DeadlineExceeded to return meaningful HTTP 504 errors instead of defaulting to panics.
Parallel Safety: Split time budgets for concurrent tasks to prevent a single slow worker from blocking the entire transaction.

What's Next?

Observe: Expose metrics for how often context.DeadlineExceeded fires. Use this to tune timeouts for different tiers.
Retry: Implement backoff logic only for retryable errors. Treat timeouts as terminal errors unless you have a specific transient failure pattern.
Document: Maintain a table of expected latency SLAs for your services. Use these numbers to set base timeouts in contracts.

Forem: Dylan Dumont

The RED Method: Request Rate, Errors, and Duration as Your Core SLIs

What We're Building

Step 1 — Instrument the Middleware

Step 2 — Aggregate Request Counts

Step 3 — Classify Error Labels

Step 4 — Measure Latency Histograms

Step 5 — Export Metrics via HTTP Endpoint

Key Takeaways

What's Next?

Further Reading

Building a Job Queue in Rust: Persistent Tasks With Retry Logic

What We're Building

Step 1 — Define the Job State Machine

Step 2 — Persist Job State in Storage

Step 3 — Implement Exponential Backoff Logic

Step 4 — Handle Permanent Failures in a DLQ

Takeaways

Next

Reading

Books

Further Reading

Log-Structured Merge Trees: The Data Structure That Powers Modern Databases

What We're Building

Step 1 — In-Memory Write Buffer

Step 2 — Flushing to SSTables

Step 3 — Indexing for Random Reads

Step 4 — Asynchronous Compaction

Key Takeaways

What's Next?

Further Reading

Exactly-Once Delivery Is a Lie: How Systems Approximate It Anyway

What We're Building

Step 1 — The Idempotency Key

Step 2 — The Outbox Pattern

Step 3 — State Machine Validation

Step 4 — Compensating Transactions

Key Takeaways

What's Next?

Further Reading

Architecture Patterns Series

Serde Deep Dive: Custom Serialization for Wire Protocols

What We're Building

Step 1 — Define the Internal Domain Model

Step 2 — Override Serialization Attributes

Step 3 — Manage Optional and Deprecated Fields

Step 4 — Implement Custom Deserialization Logic

Step 5 — Control Versioning and Protocols

Key Takeaways

What's Next?

Further Reading

Cooperative Cancellation: Propagating Shutdown Signals Through Async Trees

What We're Building

Step 1 — Establish a Broadcast Channel

Step 2 — Distribute Tokens to Children

Step 3 — Listen in a Select Statement

Step 4 — Implement Cleanup Closures

Step 5 — Trigger from the Root

Key Takeaways

What's Next?

Further Reading

Rate Limiting Deep Dive: Token Bucket vs Leaky Bucket vs Sliding Window

What We're Building

Step 1 — Define the Rate Limit Contract

Step 2 — Implement Token Bucket for Burst Tolerance

Step 3 — Implement Leaky Bucket for Traffic Smoothing

Step 4 — Implement Sliding Window for Precision

Step 5 — Persist State for Distributed Systems

Step 6 — Select the Right Algorithm

Takeaways

What's Next

Further Reading

Two-Phase Commit Demystified: When Distributed Transactions Are Unavoidable

What We're Building

Step 1 — Define the Transaction Coordinator

Step 2 — Request Prepared States

Step 3 — Aggregate the Vote

Step 4 — Finalize the Outcome

Performance and Trade-offs

Conclusion