Forem: Onur Cinar

Native Chaos Engineering: Testing Resilience with Fault & Latency Injection

Onur Cinar — Fri, 03 Apr 2026 14:51:48 +0000

You’ve implemented retries, circuit breakers, and timeouts. Your application is now "resilient." But how do you know these policies actually work? Waiting for a production meltdown to verify your configuration is a high-stakes gamble.

Native Chaos Engineering in Resile allows you to synthetically induce failure and latency directly into your application's execution path, ensuring your resilience policies are battle-tested before they're ever needed in production.

The Problem: "Dark Code" in Resilience Policies

Resilience policies—like retries and circuit breakers—are often "dark code." These are execution paths that are rarely traversed under normal operating conditions. Because they only trigger during failure, they are notoriously difficult to test and prone to:

Buggy Configurations: A retry limit that is too high, or a circuit breaker threshold that never trips.
Unintended Side Effects: A retry loop that accidentally consumes all available database connections.
Silent Failures: A fallback strategy that actually panics because it hasn't been executed in months.

Traditional chaos engineering tools often operate at the infrastructure layer (e.g., killing pods or dropping network packets). While powerful, these tools can be difficult to set up in local development or staging environments and often lack the granularity to test specific application-level logic.

The Solution: Fault & Latency Injection

Resile provides a Chaos Injector middleware that can be integrated directly into any execution policy. By injecting synthetic faults (errors) and latency (delays) with configurable probabilities, you can simulate various failure scenarios without touching your infrastructure.

Key Features:

Deterministic Randomness: Uses Go 1.22's math/rand/v2 for efficient and predictable random number generation.
Context-Aware: Latency injection strictly respects context.Context cancellation. If your request times out while Resile is injecting chaos latency, it exits immediately.
Zero Dependencies: Just like the rest of the Resile core, the chaos package depends only on the Go standard library.
Granular Control: Configure error and latency probabilities independently for fine-tuned simulation.

Practical Usage

Integrating chaos into your existing Resile policies is as simple as adding the WithChaos option.

1. Basic Chaos Configuration

You can define a chaos configuration that injects a 10% error rate and adds 100ms of latency to 20% of requests.

import (
    "github.com/cinar/resile"
    "github.com/cinar/resile/chaos"
)

// Configure chaos injection
cfg := chaos.Config{
    ErrorProbability:   0.1,                    // 10% chance of failure
    InjectedError:      errors.New("chaos!"),   // The error to return
    LatencyProbability: 0.2,                    // 20% chance of latency
    LatencyDuration:    100 * time.Millisecond, // Delay to inject
}

// Apply it to an execution
err := resile.DoErr(ctx, action, 
    resile.WithRetry(3),
    resile.WithChaos(cfg),
)

2. Testing Your Circuit Breaker

Chaos injection is exceptionally useful for verifying that your circuit breaker trips under pressure. By setting a high ErrorProbability, you can force the breaker to transition from Closed to Open in a controlled environment.

cb := circuit.New(circuit.Config{
    WindowSize:           10,
    FailureRateThreshold: 50.0,
})

// Force 80% error rate to trip the breaker quickly
cfg := chaos.Config{
    ErrorProbability: 0.8,
    InjectedError:    errors.New("synthetic failure"),
}

for i := 0; i < 20; i++ {
    resile.DoErr(ctx, action, 
        resile.WithCircuitBreaker(cb),
        resile.WithChaos(cfg),
    )
}

fmt.Printf("Circuit Breaker State: %v\n", cb.State()) // Should be Open

Configuration Reference

The chaos.Config struct provides the following options:

Field	Type	Description
`ErrorProbability`	`float64`	The probability of injecting an error (0.0 to 1.0).
`InjectedError`	`error`	The error to be returned when an error is injected.
`LatencyProbability`	`float64`	The probability of injecting latency (0.0 to 1.0).
`LatencyDuration`	`time.Duration`	The duration of the latency to be injected.

Best Practices

Environment Gating: Never enable chaos injection in production unless you are performing a planned game day. Use environment variables to gate the configuration:
```
if os.Getenv("ENABLE_CHAOS") == "true" {
    opts = append(opts, resile.WithChaos(loadChaosCfg()))
}
```
Observability: Ensure your Instrumenter (like slog or OTel) is active. This allows you to see the injected errors and latencies in your logs and traces, making it easier to verify how your application responds.
Start Small: Begin with low probabilities (e.g., 1-2%) to identify subtle race conditions or timeout issues before increasing the "blast radius."

Conclusion

Resilience is not a "set it and forget it" feature. It requires continuous verification. By bringing chaos engineering directly into your application's execution policies, Resile empowers you to build systems that aren't just theoretically resilient, but practically battle-hardened.

For more information and advanced usage, visit the github.com/cinar/resile project.

Beyond Static Limits: Adaptive Concurrency with TCP-Vegas in Go

Onur Cinar — Thu, 02 Apr 2026 19:11:51 +0000

Traditional concurrency limits (like bulkheads) are static. You pick a number—say, 10 concurrent requests— and hope for the best. But in the dynamic world of cloud infrastructure, "10" might be too conservative when the network is fast, or dangerously high when a downstream service starts to queue.

Static limits require manual tuning, which is often done after an outage has already happened. To build truly resilient systems, we need Adaptive Concurrency Control.

Here is how to implement dynamic concurrency limits in Go using Resile, inspired by the TCP-Vegas congestion control algorithm.

The Problem: The "Fixed-Limit" Trap

Imagine your service talks to a database. You've set a bulkhead limit of 50 concurrent connections.

Scenario A (Normal): Database latency is 10ms. 50 concurrent requests mean you're handling 5,000 RPS. Everything is fine.
Scenario B (Degraded): Database latency spikes to 500ms due to a background maintenance task. Your 50 "slots" are now filled with slow requests. Your throughput drops to 100 RPS, and new incoming requests start to pile up in your own service's memory, eventually leading to a cascade of failures.

In Scenario B, 50 is too many. You're holding onto resources that are essentially waiting on a bottleneck. You should have reduced your concurrency limit to prevent your own service from becoming part of the problem.

The Solution: Little's Law & TCP-Vegas

Adaptive Concurrency uses two core principles:

Little's Law (L = λW): The number of items in a system (L) is equal to the arrival rate (λ) multiplied by the average time an item spends in the system (W).
TCP-Vegas AIMD: An Additive Increase, Multiplicative Decrease (AIMD) logic based on Round-Trip Time (RTT).

How it works:

Baseline: The algorithm tracks the minimum RTT (the fastest the system can possibly go).
Additive Increase: If current latency is close to the baseline (no queuing detected), it cautiously increases the concurrency limit by 1.
Multiplicative Decrease: If latency spikes above a threshold (e.g., 1.5 x baseline), it assumes queuing is happening downstream and immediately slashes the concurrency limit by 20%.

This allows your service to automatically "breathe" with the network. It expands to use available capacity when things are fast and contracts instantly to protect itself when things slow down.

Implementing with Resile

Resile makes it trivial to add adaptive concurrency to your Go services.

// 1. Create a shared AdaptiveLimiter.
// This should be shared across multiple calls to the same resource.
al := resile.NewAdaptiveLimiter()

// 2. Use it in your policy.
p := resile.NewPolicy(
    resile.WithAdaptiveLimiterInstance(al),
)

// 3. Execute your action.
err := p.DoErr(ctx, func(ctx context.Context) error {
    return callDownstreamService()
})

if errors.Is(err, resile.ErrShedLoad) {
    // The limiter has dynamically reduced the limit and shed this request
    // to protect the system.
}

Why "TCP-Vegas"?

Unlike other congestion control algorithms (like TCP-Reno) that wait for packet loss to react, TCP-Vegas reacts to latency changes. This is perfect for microservices where "packet loss" usually means a timed-out request or a 503 error—both of which we want to avoid before they happen.

Zero-Configuration Resilience

One of the biggest benefits of Adaptive Concurrency is that it requires zero manual configuration. You don't need to know if your database can handle 50 or 500 connections. The AdaptiveLimiter will discover the optimal limit in real-time.

It even handles "Network Drift." Over time, the minimum baseline RTT is gradually decayed, allowing the system to recalibrate if you migrate your database to a faster region or if the network topology changes.

Conclusion

Resilience isn't just about surviving failures; it's about adapting to them. By moving from static bulkheads to adaptive concurrency, you're building a system that can intelligently protect itself from cascading failures while maximizing throughput during "peace time."

Check out the Adaptive Concurrency Example in the Resile repository to see it in action.

Respecting Boundaries: Precise Rate Limiting in Go

Onur Cinar — Tue, 24 Mar 2026 13:00:00 +0000

Traffic spikes are a double-edged sword. On one hand, you’re busy! On the other, those spikes can overwhelm your services or exceed your downstream quotas.

Whether you're protecting your own database from an unexpected burst or respecting a third-party API’s strict 100 requests-per-second (RPS) limit, you need a precise way to shape your traffic.

Enter the Token Bucket Rate Limiter in Resile.

The Problem: Unbounded Traffic

In a distributed environment, your clients don't know about each other. If 50 different microservice instances all decide to call a downstream API at the same time, the aggregate traffic can easily exceed the capacity of the target system.

When you exceed these limits, you'll often see:

HTTP 429 (Too Many Requests): Downstream services start rejecting you.
Cascading Latency: The target system slows down for everyone because it's processing too many requests at once.
Cost Overruns: Many cloud providers and SaaS APIs charge significant premiums for exceeding agreed-upon quotas.

The Solution: The Token Bucket Algorithm

The Token Bucket is a classic algorithm used for traffic shaping.

Imagine a bucket that refills with "tokens" at a constant rate (e.g., 100 tokens per second). Every request must consume a token from the bucket. If the bucket is empty, the request is rejected immediately. This allows for small "bursts" (filling the bucket) while maintaining a precise long-term average rate.

Implementing with Resile:

Resile makes adding rate limiting to your executions simple.

// Allow 100 requests per second.
// If the limit is exceeded, it fails fast with resile.ErrRateLimitExceeded.
err := resile.DoErr(ctx, action, 
    resile.WithRateLimiter(100, time.Second),
)

Rate Limiting vs. Adaptive Retries

Wait, doesn't Resile already have AdaptiveBucket? What's the difference?

AdaptiveBucket is success-based. It tracks how many requests are succeeding vs. failing and throttles retries accordingly. It's designed specifically to prevent "retry storms" when a service is failing.
RateLimiter is time-based. It enforces a strict, constant quota of requests over a time interval. It’s designed for general traffic shaping and quota management.

For maximum protection, you can even use them together!

Shared Rate Limiters

Often, you want to enforce a global rate limit across your entire service instance. You can create a shared RateLimiter and pass it to multiple executions:

// Shared rate limiter for a specific API key or downstream service
limiter := resile.NewRateLimiter(50, time.Second)

// Each call will consume tokens from the same shared bucket.
err := resile.DoErr(ctx, myAction, 
    resile.WithRateLimiterInstance(limiter),
)

Observability: Seeing the Shaping

Knowing when and why your traffic is being throttled is essential for operational visibility.

If you use Resile's telemetry integrations (like slog or OpenTelemetry), you'll get automatic visibility into these events. The OnRateLimitExceeded event is triggered whenever a request is rejected by the rate limiter, allowing you to monitor your quota utilization in real-time.

Conclusion

Rate limiting is not just about saying "no"; it's about being a good citizen in a distributed ecosystem. By respecting boundaries and shaping your traffic at the source, you protect both your own service and the systems you depend on.

Resile provides a production-grade rate limiter that integrates seamlessly into your resilience policies, giving you fine-grained control over your traffic flow.

Learn more about Resile: github.com/cinar/resile

Stop the Domino Effect: Bulkhead Isolation in Go

Onur Cinar — Sun, 22 Mar 2026 17:42:19 +0000

In a distributed system, failure is inevitable. But a failure in one part of your system shouldn't bring down everything else.

Imagine your Go service depends on three different downstream APIs: Payments, Inventory, and Recommendations. Suddenly, the Recommendations API starts taking 30 seconds to respond. If your service doesn't have isolation, your goroutines will start piling up waiting for Recommendations. Eventually, you'll hit your process limit, and even the critical Payments API calls will start failing because there are no resources left to handle them.

This is the Domino Effect, and the Bulkhead Pattern is how you stop it.

The Problem: Resource Exhaustion

When one dependency slows down, it consumes resources:

Goroutines: Blocked waiting for a response.
Memory: Each blocked goroutine carries a stack.
File Descriptors/Sockets: Open connections to the slow service.

Without a bulkhead, a single slow dependency can "starve" the rest of your application, leading to a total system collapse.

The Solution: The Bulkhead Pattern

Named after the partitioned sections of a ship's hull, a Bulkhead isolates failures. If one section of the ship is flooded, the others remain buoyant. In software, we achieve this by limiting the number of concurrent executions allowed for a specific resource or dependency.

Implementing with Resile:

Resile makes it trivial to add bulkhead isolation to any operation.

// Allow only 10 concurrent calls to this specific operation.
// If an 11th call comes in, it fails fast with resile.ErrBulkheadFull.
err := resile.DoErr(ctx, action, 
    resile.WithBulkhead(10),
)

Using a Shared Bulkhead

Often, you want to limit concurrency across multiple different call sites that hit the same downstream service. You can create a shared Bulkhead instance for this:

// Create a shared bulkhead for the "Inventory Service"
inventoryBulkhead := resile.NewBulkhead(20)

// Call Site A
resile.DoErr(ctx, fetchItem, resile.WithBulkheadInstance(inventoryBulkhead))

// Call Site B
resile.DoErr(ctx, updateStock, resile.WithBulkheadInstance(inventoryBulkhead))

By sharing the instance, you ensure that the total concurrency hitting the Inventory Service never exceeds 20, regardless of which part of your code is making the call.

Why "Fail-Fast" Matters

When a bulkhead is full, Resile immediately returns resile.ErrBulkheadFull.

This is much better than waiting for a timeout. By failing fast, you:

Preserve Resources: You don't spawn another goroutine or open another connection.
Provide Immediate Feedback: Your upstream callers get an error instantly and can decide how to handle it (e.g., show a cached result or a "service busy" message).

Observability: Monitoring the Walls

You need to know when your bulkheads are working. If a bulkhead is frequently full, it might mean your downstream service is struggling, or you need to re-evaluate your capacity limits.

If you use Resile's telemetry integrations (like slog or OpenTelemetry), you'll get automatic alerts when a bulkhead saturates. The OnBulkheadFull event is triggered every time a request is rejected due to capacity limits.

Conclusion

Bulkheads are a fundamental building block of resilient systems. By isolating your dependencies, you ensure that a local fire doesn't become a global conflagration.

Resile provides a clean, "Go-native" way to implement bulkheads without complex boilerplate, allowing you to focus on your business logic while keeping your system stable.

Explore Resile on GitHub: github.com/cinar/resile

Infinite Data Processing in Go: Building Resilient Data Pipes with Channels

Onur Cinar — Wed, 18 Mar 2026 16:00:00 +0000

When building data-intensive applications, we usually start with the most obvious approach: loading data into a slice or array, iterating over it to process the data, and returning the result. This batch-processing mindset works great—until the data never stops coming.

Whether you are dealing with live IoT telemetry, continuous log tailing, or real-time financial market feeds, you quickly run into the problem of "infinite" data. If you try to append an endless stream of stock ticks to a []float64, your application will inevitably consume all available memory and crash.

To handle infinite data gracefully, you need to shift your architecture from batch processing to stream processing. In Go, we have the perfect built-in primitive for this: Channels.

The Power of Channels as Data Pipes

Go channels are often taught primarily as a way to synchronize goroutines, but they are also incredibly powerful as sequential data pipes. By treating channels as standard inputs and outputs, you can build decoupled, memory-efficient pipelines where data flows through a series of transformations continuously.

When redesigning my open-source technical analysis library, cinar/indicator, for its v2 release, I faced exactly this challenge. In algorithmic trading, systems need to react instantly to live market feeds without accumulating massive memory overhead. Transitioning the library's core architecture from slice-based arrays to stream-based Go channels solved this elegantly.

Let's look at how to build a continuous data pipe, and some of the tricky edge cases you'll encounter along the way.

Building a Pipeline Stage

Imagine we want to calculate a Simple Moving Average (SMA) over a live stream of data. Instead of taking a slice, our function will accept a read-only channel as its input and return a read-only channel as its output.

package main

import (
    "fmt"
)

// SimpleMovingAverage acts as a pipe: it reads from 'input', processes, and writes to 'output'
func SimpleMovingAverage(input <-chan float64, period int) <-chan float64 {
    output := make(chan float64)

    go func() {
        // Ensure the output channel is closed when the input stream ends
        defer close(output) 

        window := make([]float64, 0, period)
        sum := 0.0

        for val := range input {
            window = append(window, val)
            sum += val

            // Keep the window size fixed
            if len(window) > period {
                sum -= window[0]
                window = window[1:]
            }

            // Only emit a value once we have enough data points
            if len(window) == period {
                output <- sum / float64(period)
            }
        }
    }()

    return output
}

Because the processing happens inside its own goroutine, the function returns the output channel immediately. The goroutine stays alive, eagerly waiting for new data to arrive on the input channel.

Handling Stream Complexities with Helpers

Once you start relying heavily on channels, you run into a few structural challenges. To make working with channels just as easy as working with slices, cinar/indicator includes a robust helper package.

If you are building your own stream-based application, you can leverage these helpers directly from the library rather than reinventing the wheel.

1. The Branching Problem

A major gotcha with Go channels: once a value is read from a channel, it's gone. What if you want to calculate an SMA and a Relative Strength Index (RSI) from the exact same live price ticker? You can't have two consumers read from one channel without them stealing data from each other.

To solve this, the library provides helper.Duplicate. This function takes one input channel and "fans it out" into multiple identical output channels. This allows you to safely branch your data stream to multiple independent technical indicators simultaneously without race conditions or data loss.

// Branching one price stream into three identical streams
priceStreams := helper.Duplicate(livePrices, 3)

smaStream := indicator.SMA(priceStreams[0], 14)
rsiStream := indicator.RSI(priceStreams[1], 14)
macdStream := indicator.MACD(priceStreams[2], 12, 26, 9)

2. Lookbacks and Sliding Windows

Many data processing algorithms require looking back at the last N periods. Instead of managing a sliding window manually inside every single function (like we did in the basic SMA example above), the library uses helper.Buffered. This provides a clean abstraction to maintain a rolling state over a continuous channel, vastly simplifying the development of complex logic.

3. Bridging the Gap: Slices vs. Streams

The rest of the world often still speaks in batches. You might be downloading historical CSV data for backtesting, or you might need to output an array for a charting UI. To bridge this gap, the helper package includes utilities to fluidly move between paradigms:

helper.SliceToChan: Converts a static historical array into a simulated live data stream. It spins up a goroutine, pushes every element from the slice into a channel, and closes it. It's perfect for feeding historical backtests into a live-stream architecture.
helper.ChanToSlice: The inverse operation. It drains a stream back into an array, which is incredibly useful for writing unit tests or rendering charts.

Chaining the Pipes Together

Because all the indicators and helpers in cinar/indicator take channels and return channels, they are highly composable. We can chain them together like Unix command-line pipes (|).

Here is what it looks like to wire up an application using these concepts:

package main

import (
    "fmt"
    "[github.com/cinar/indicator/v2/helper](https://github.com/cinar/indicator/v2/helper)"
    "[github.com/cinar/indicator/v2/trend](https://github.com/cinar/indicator/v2/trend)"
)

func main() {
    // 1. We start with a static slice of historical data
    historicalPrices := []float64{10.0, 12.0, 14.0, 13.0, 15.0, 18.0, 19.0, 17.0}

    // 2. Bridge the gap: convert the slice to a live stream
    marketTicks := helper.SliceToChan(historicalPrices)

    // 3. Pipe the ticks into a 3-period SMA processor from the library
    smaStream := trend.Sma(marketTicks, 3)

    // 4. Drain the output stream 
    for avg := range smaStream {
        fmt.Printf("New SMA tick processed: %.2f\n", avg)
    }
}

Why This Architecture Wins

Memory Efficiency: We only store the exact amount of data needed at any given moment. The Go garbage collector easily cleans up the rest, meaning we can process a continuous websocket stream for months without memory leaks.
Backpressure Handling: Go channels are blocking by nature. If a complex compound strategy at the end of the pipeline is too slow, the channels will naturally fill up, pausing the producers further up the chain until it catches up.
Decoupling: Each pipeline stage is completely isolated. The indicator doesn't know if the data is coming from a historical Tiingo repository, an Alpaca websocket, or a mock unit test. It just reads from <-chan T and writes to <-chan T.

Try It Out

By treating channels as native data streams and relying on robust helper utilities, you can build highly resilient, concurrent pipelines capable of processing truly infinite data sets.

If you are building financial tools, real-time dashboards, or are just looking to explore a Go codebase that relies heavily on generics and channel-based streaming, I highly recommend leveraging the cinar/indicator library. It comes batteries-included with all the helpers and technical indicators you need to get started with stream processing in Go.

How are you handling continuous data streams in your applications? Let me know in the comments!

Self-Healing State Machines: Resilient State Transitions in Go

Onur Cinar — Mon, 16 Mar 2026 13:00:00 +0000

Distributed systems are inherently stateful. Whether you're managing a database connection pool, a multi-step payment workflow, or a complex IoT device lifecycle, you need to transition between states reliably.

Standard state machines (FSMs) are great for logic, but they are often brittle. What happens if a transition involves a network call that fails? Most developers end up wrapping their machine.Transition() calls in manual retry loops, cluttering their business logic and losing visibility into why a transition failed.

Inspired by Erlang's gen_statem behavior, Resile introduces resile.StateMachine: a standardized, resilient state machine where every transition is inherently protected by resilience policies.

The Resile Way: One-Line Resilience for Transitions

With resile.StateMachine, you don't just define how to move from State A to State B. You define a Resilient Transition.

Here is how you implement a self-healing connection manager:

import "github.com/cinar/resile"

// 1. Define your State, Data, and Events
type State string
const (
    Disconnected State = "Disconnected"
    Connected    State = "Connected"
)

type Event string
const (
    Connect Event = "Connect"
)

// 2. Define the Transition Logic
transition := func(ctx context.Context, state State, data Data, event Event, rs resile.RetryState) (State, Data, error) {
    if state == Disconnected && event == Connect {
        // This transition involves a network call.
        // If it fails, Resile will automatically retry it 
        // using the configured backoff and jitter.
        err := apiClient.Connect(ctx, data.Endpoint)
        if err != nil {
            return "", data, err
        }
        return Connected, data, nil
    }
    return state, data, nil
}

// 3. Initialize the Resilient State Machine
sm := resile.NewStateMachine(
    Disconnected, 
    Data{Endpoint: "api.example.com"}, 
    transition,
    resile.WithMaxAttempts(3),
    resile.WithBaseDelay(100 * time.Millisecond),
)

// 4. Handle events safely
err := sm.Handle(ctx, Connect)

What happens under the hood?

When you call sm.Handle(ctx, event), Resile enters a execution envelope.
It executes your transition function.
If the transition returns an error, Resile applies your retry policy (e.g., Exponential Backoff with Jitter).
Only when the transition succeeds does the StateMachine update its internal state and data.
If the retries are exhausted, the StateMachine remains in its previous state, ensuring consistency.

Why "Self-Healing"?

Most state machine implementations are "fire and forget" or "fail and stop." A Self-Healing state machine assumes that transitions are risky and provides the infrastructure to recover from those risks automatically.

Automatic Retries: No more manual loops inside your state logic.
Circuit Breakers: If a specific transition (e.g., to a "Maintenance" state) is failing repeatedly, the circuit breaker can trip to prevent overwhelming the system.
Context Awareness: If the transition is part of a timed-out request, the state machine cancels the transition attempt immediately, preventing goroutine leaks.
Observability: Every transition attempt—including retries—is tracked by Resile's telemetry hooks. You can see exactly how many times your machine "struggled" to reach the Connected state.

Observability: Tracking State Success

By using Resile's OpenTelemetry or slog integrations, you get deep insights into your state machine's health:

Attempts per Transition: See which events are causing the most retries.
Transition Latency: Measure how long it takes to move from one state to another, including backoff time.
Failure Patterns: Identify if a specific state is a "dead end" due to persistent errors.

Conclusion

Resilience isn't just for simple API calls. By bringing resilience to the core of your stateful logic, you build systems that are not only more robust but also significantly easier to debug and monitor.

Stop writing manual retry loops around your state changes. Let resile.StateMachine handle the complexity of the "unreliable world" while you focus on the logic of your application.

Give Resile a star on GitHub: github.com/cinar/resile

How are you managing state transitions in your Go microservices? Let's discuss!

Preventing Microservice Meltdowns: Adaptive Retries and Circuit Breakers in Go

Onur Cinar — Sun, 15 Mar 2026 22:28:33 +0000

We’ve all been there. A downstream database has a momentary blip. Your service instances, being "resilient," immediately start retrying their failed requests.

Suddenly, the database isn't just "having a blip" anymore—it’s being hammered by a self-inflicted DDoS attack from its own clients. This is the Retry Storm (or Thundering Herd), and it’s one of the most common ways distributed systems experience total meltdowns.

Standard exponential backoff protects individual services, but it doesn't protect the cluster. To do that, you need a layered defense-in-depth approach.

Here is how to prevent microservice meltdowns in Go using Resile.

The Problem: Aggregate Load

Imagine you have 100 instances of your API. Each instance is configured to retry 3 times. If the database slows down, you suddenly have 300 extra requests hitting it exactly when it's struggling to recover.

Even with jitter, the aggregate load can be enough to keep the database in a "failed" state indefinitely. To solve this, we need two patterns working together: Adaptive Retries and Circuit Breakers.

1. Adaptive Retries (The Token Bucket)

Inspired by Google's SRE book and AWS SDKs, Adaptive Retries use a client-side token bucket to "fail fast" locally.

The logic is simple:

Every success adds a small amount of "credit" to your bucket.
Every retry consumes a significant amount of credit.
If the bucket is empty, Resile stops retrying immediately and fails fast locally.

This ensures that if a downstream service is fundamentally degraded, your fleet of clients will automatically throttle their retry pressure at the source, giving the service breathing room to recover.

Implementing with Resile:

// Share this bucket across multiple executions or even your entire service
bucket := resile.DefaultAdaptiveBucket()

err := resile.DoErr(ctx, action, 
    resile.WithAdaptiveBucket(bucket),
)

2. Circuit Breakers (The Kill Switch)

While retries assume "eventual success," a Circuit Breaker assumes "statistical failure."

If a service fails 5 times in a row, the breaker "trips" (opens). For the next 30 seconds, every call to that service will fail instantly without even trying to hit the network. This protects your downstream infrastructure from useless traffic and saves your local resources (threads, memory, sockets).

Layering it in Resile:

import "github.com/cinar/resile/circuit"

// Create a breaker: Trip after 5 failures, wait 30s to retry
cb := circuit.New(circuit.Config{
    FailureThreshold: 5,
    ResetTimeout:     30 * time.Second,
})

err := resile.DoErr(ctx, action, 
    resile.WithCircuitBreaker(cb),
)

The Ultimate Defense: Layered Resilience

The real power of Resile comes from combining these patterns. You can layer Retries, Circuit Breakers, and Adaptive Buckets into a single execution strategy.

err := resile.DoErr(ctx, action,
    resile.WithMaxAttempts(3),           // Layer 1: Handle random blips
    resile.WithCircuitBreaker(cb),      // Layer 2: Stop hitting a dead service
    resile.WithAdaptiveBucket(bucket),  // Layer 3: Prevent cluster-wide storms
)

In this setup:

Retries handle the "one-off" network glitches.
The Circuit Breaker stops you from wasting time on a service that is clearly down.
The Adaptive Bucket ensures that even if the breaker hasn't tripped yet, you won't overwhelm the system with aggregate retry load.

Observability: Seeing the Shield in Action

Protecting your system is great, but knowing you’re being protected is better.

If you use Resile's slog or OpenTelemetry integrations, you'll see exactly when these shields activate. Your logs will show retry.throlled=true when the adaptive bucket kicks in, or your traces will show a circuit.open error when the breaker prevents a call.

This visibility is crucial for SREs to understand why traffic is failing and how the system is self-healing.

Conclusion

Building resilient microservices isn't just about making individual calls "smarter." It's about ensuring that your entire architecture can survive a storm without collapsing under its own weight.

By combining opinionated retries, circuit breakers, and adaptive throttling, Resile gives you a production-grade resilience engine that scales with your infrastructure.

Try Resile today: github.com/cinar/resile

How do you prevent "retry storms" in your Go clusters? Let's discuss in the comments!

Beating Tail Latency: A Guide to Request Hedging in Go Microservices

Onur Cinar — Sat, 14 Mar 2026 03:03:35 +0000

In distributed systems, we often talk about "The Long Tail."

You might have a service where 95% of requests finish in under 100ms. But that last 1% (the P99 latency)? Those requests might take 2 seconds or more. In a microservice architecture where one user action triggers 10 different service calls, that one slow dependency will bottleneck the entire user experience.

Standard retries don't help here. Why? Because a "Tail Latency" request hasn't failed yet—it’s just slow.

Waiting for a 2-second timeout to trigger a retry is a waste of time. To beat the long tail, you need Request Hedging (also known as Speculative Retries).

Here is how to implement it safely in Go using Resile.

What is Request Hedging?

The concept is simple but powerful: If a request is taking longer than usual (say, longer than the P95 latency), don't kill it. Instead, start a second, identical request in parallel.

Whichever request finishes first, you take its result and cancel the other one.

This "speculative" approach drastically reduces P99 latency because the mathematical probability of two identical requests hitting the "long tail" simultaneously is extremely low.

The Complexity of Manual Hedging

Implementing hedging manually in Go is a nightmare of goroutine management:

You need a select block with a timer.
You need to coordinate between two (or more) goroutines.
You must ensure that once one succeeds, the others are cancelled immediately to save resources.
You have to handle race conditions where both might succeed at the exact same millisecond.

Most developers end up with hundreds of lines of brittle boilerplate code to handle just one hedged call.

The Resile Way: `DoHedged`

Resile makes request hedging as simple as a single function call. It handles the goroutine lifecycle, context cancellation, and race conditions for you.

Here is how you fetch data with a 100ms hedging delay:

import "github.com/cinar/resile"

// data is automatically inferred as *User
data, err := resile.DoHedged(ctx, func(ctx context.Context) (*User, error) {
    return apiClient.GetUser(ctx, userID)
}, 
    resile.WithMaxAttempts(3),
    resile.WithHedgingDelay(100 * time.Millisecond),
)

What happens under the hood?

Resile starts the first request.
It waits for 100ms (HedgingDelay).
If the first request hasn't finished, it starts a second request.
As soon as one returns a successful result, Resile cancels the context of the other request and returns the data to you.

Picking the Right Hedging Delay

The "magic" of hedging lies in the delay.

Too short: You double your traffic unnecessarily, putting extra load on your downstream services.
Too long: You don't gain much latency benefit.

Pro-Tip: A good rule of thumb is to set your HedgingDelay to your P95 or P99 latency. This ensures you only "hedge" the slowest 1-5% of requests, providing a massive latency win with minimal extra load.

Observability: Tracking the "Speculative" Wins

If you're using Resile's OpenTelemetry integration (telemetry/resileotel), you can actually see these wins in your distributed traces.

Each hedged attempt is recorded as a sub-span. When a hedged request wins, you'll see the first span get cancelled and the second one succeed—providing clear proof that hedging saved your user from a 2-second wait.

Conclusion

Request hedging used to be a technique reserved for companies with massive infrastructure teams. With Resile, it’s a tool that every Go developer can use to build snappier, more resilient microservices.

By moving from "Wait and Retry" to "Hedge and Win," you can turn your long-tail latency into a competitive advantage.

Give Resile a star on GitHub: github.com/cinar/resile

How are you handling tail latency in your Go services? Let's discuss in the comments!

Python's Stamina for Go: Bringing Ergonomic Resilience to Gophers

Onur Cinar — Tue, 10 Mar 2026 02:04:41 +0000

If you've ever worked in the Python ecosystem, you've likely encountered tenacity or its opinionated wrapper, stamina. They make retrying transient failures feel like magic: a single decorator, sensible production defaults (exponential backoff + jitter), and built-in observability.

The Go ecosystem has powerful tools, but they often require a lot of boilerplate, use reflection, or lack the "Correct by Default" philosophy that makes stamina so great.

That's why I built Resile. It’s a love letter to Python's ergonomics, written in idiomatic, type-safe Go 1.18+.

The Ergonomic Gap

In Python, retrying an API call with stamina looks like this:

import stamina

@stamina.retry(on=httpx.HTTPError, attempts=3)
def get_data():
    return httpx.get("https://api.example.com").json()

Before Go 1.18, achieving this level of simplicity was nearly impossible. You either had to write verbose for loops or use libraries that relied on interface{} and reflection—which meant losing type safety and slowing down your code.

With the arrival of Generics, the game changed. We can now have the best of both worlds: Python-level ergonomics with Go’s compile-time safety.

The Resile Way: One-Line Resilience

Resile uses Go 1.18+ Type Parameters to wrap your logic in a resilience envelope. Here is how you fetch data with Resile:

import "github.com/cinar/resile"

// data is automatically inferred as *User
data, err := resile.Do(ctx, func(ctx context.Context) (*User, error) {
    return apiClient.GetUser(ctx, userID)
}, resile.WithMaxAttempts(3))

It looks and feels like a simple function call, but under the hood, Resile is doing the heavy lifting:

AWS Full Jitter Backoff: Spreading out retries to protect your database.
Context-Awareness: Cancelling retries immediately if the request times out.
Memory Safety: Using managed timers to prevent goroutine leaks.

Why "Correct by Default" Matters

One of the best things about stamina is that it makes the right thing the easy thing. Resile follows this philosophy strictly:

Exponential Backoff is the Default: You don't have to configure it; it's there from attempt one.
Jitter is Not Optional: Resile forces randomization to prevent "thundering herd" outages.
Zero Dependencies: The core of Resile depends only on the Go standard library. No bloated dependency graphs.
No Reflection: Unlike many older Go retry libraries, Resile uses static type parameters. This means zero runtime overhead and zero chance of a "type mismatch" panic.

Batteries Included (But Removable)

Just like stamina integrates with structlog and Prometheus, Resile provides optional sub-packages for modern observability:

telemetry/resileslog: High-performance structured logging with Go 1.21’s slog.
telemetry/resileotel: Full OpenTelemetry tracing. See every retry attempt as a sub-span in your Jaeger or Honeycomb dashboard.
Adaptive Retries: A client-side token bucket (inspired by Google's SRE book) to prevent your fleet from killing a degraded service.

Fast CI/CD: The "Stamina" Secret

A hidden feature of stamina that I absolutely loved was the ability to globally disable wait times in unit tests.

Resile brings this to Go through Context Overrides. You can make your 30-second retry loop execute in 1 millisecond during tests without changing a single line of business logic:

func TestService(t *testing.T) {
    // This context tells Resile to skip all sleep timers
    ctx := resile.WithTestingBypass(context.Background())

    err := myService.PerformTask(ctx) // Retries instantly!
}

Conclusion

We don't have to choose between Go's performance and Python's developer experience. By leveraging modern Go features like Generics and slog, we can build tools that are both powerful and a joy to use.

If you’ve been missing stamina in your Go projects, give Resile a try.

Check it out on GitHub: github.com/cinar/resile

Stop Writing Manual Retry Loops in Go: Why Your Current Logic is Probably Dangerous

Onur Cinar — Mon, 09 Mar 2026 02:55:42 +0000

If you've been writing Go for more than a week, you've likely written a retry loop. It usually starts like this:

for i := 0; i < 3; i++ {
    err := doSomething()
    if err == nil {
        break
    }
    time.Sleep(1 * time.Second)
}

It's simple, idiomatic, and... a ticking time bomb in production.

In a distributed system, transient failures—network blips, database locks, rate limits—are mathematical certainties. While a simple for loop feels like enough, it often fails exactly when your system is under the most stress.

Here is why your manual retry logic is probably dangerous, and how to fix it using Resile.

The 3 Silent Killers of Manual Retries

1. The Thundering Herd (Missing Jitter)

If your service has 1,000 instances and the database goes down for a second, all 1,000 instances will fail at once. With a fixed time.Sleep(1 * time.Second), all 1,000 instances will then wake up at the exact same millisecond and hammer the database again.

This is a self-inflicted DDoS attack. Without Jitter (randomized delay), your retries are just synchronized waves of traffic that prevent your dependencies from ever recovering.

2. The Context Blindness

Does your retry loop respect context.Context? Most don't. If a user cancels their request or a global timeout is reached, a time.Sleep will block that goroutine until the timer expires.

In a high-concurrency environment, these "hanging" goroutines pile up, leading to memory exhaustion and silent failures.

3. The `time.After` Memory Leak

Even "advanced" developers trying to be context-aware often use this:

select {
case <-ctx.Done():
    return ctx.Err()
case <-time.After(delay): // DANGER!
    // proceed
}

According to the Go standard library, the timer created by time.After is not garbage collected until it fires, even if the ctx.Done() case is chosen. In a busy service with long retries, this creates a slow-motion memory leak that is incredibly hard to debug.

Introducing Resile: Ergonomic Resilience for Go

I built Resile because I missed the ergonomics of Python’s stamina and tenacity libraries, but I wanted the uncompromising type safety of Go 1.18+ Generics.

Resile is a zero-dependency, execution resilience library that makes the "Correct Way" to retry as easy as a single function call.

The "Hello, World" of Resile

Instead of a manual loop, you use DoErr for actions that only return an error:

err := resile.DoErr(ctx, func(ctx context.Context) error {
    return db.PingContext(ctx)
})

Or Do for value-yielding operations with full type safety:

// user is automatically inferred as *User
user, err := resile.Do(ctx, func(ctx context.Context) (*User, error) {
    return apiClient.GetUser(ctx, userID)
}, resile.WithMaxAttempts(3))

Why Resile?

1. AWS Full Jitter by Default

Resile implements the industry-standard Full Jitter algorithm. Instead of sleeping for a fixed time, it calculates an exponential backoff and then picks a random value between 0 and that maximum. This perfectly spreads out the load across your cluster.

2. Memory-Safe Timer Management

Resile doesn't use time.After. It uses a managed time.Timer with explicit cleanup. Whether your retry succeeds, fails, or the context is cancelled, Resile ensures all resources are returned to the runtime immediately.

3. Generic-First API

No interface{}, no reflection, and no type casting. Because it uses Go Generics, the compiler checks your types at build time. If your function returns a *User, Resile returns a *User.

4. Fast Unit Testing

One of the biggest pain points of retries is that they slow down your CI/CD pipelines. Who wants to wait 10 seconds for a test to finish because of backoff?

With Resile, you can use WithTestingBypass to make all retries execute instantly in your tests:

func TestMyService(t *testing.T) {
    ctx := resile.WithTestingBypass(context.Background())

    // This will retry 5 times INSTANTLY without sleeping.
    err := service.Handle(ctx)
}

Beyond Simple Retries

Resile isn't just a retry loop; it's a resilience toolkit. Out of the box, you get:

Request Hedging: Start a second request if the first one is taking too long (beating tail latency).
Adaptive Retries: A client-side token bucket to prevent "retry storms" across a cluster.
Circuit Breaker Integration: Stop retrying when a service is fundamentally down.
Panic Recovery: Convert unexpected panics into retryable errors (the Erlang "Let It Crash" way).

Conclusion

Retrying is a distributed systems problem, not just a loop problem. By moving away from manual loops to a dedicated resilience engine like Resile, you protect your downstream services, eliminate memory leaks, and keep your code clean and type-safe.

Check out Resile on GitHub: github.com/cinar/resile

How are you handling transient failures in your Go services? Let's discuss in the comments!

Forem: Onur Cinar

Native Chaos Engineering: Testing Resilience with Fault & Latency Injection

The Problem: "Dark Code" in Resilience Policies

The Solution: Fault & Latency Injection

Key Features:

Practical Usage

1. Basic Chaos Configuration

2. Testing Your Circuit Breaker

Configuration Reference

Best Practices

Conclusion

Beyond Static Limits: Adaptive Concurrency with TCP-Vegas in Go

The Problem: The "Fixed-Limit" Trap

The Solution: Little's Law & TCP-Vegas

How it works:

Implementing with Resile

Why "TCP-Vegas"?

Zero-Configuration Resilience

Conclusion

Respecting Boundaries: Precise Rate Limiting in Go

The Problem: Unbounded Traffic

The Solution: The Token Bucket Algorithm

Implementing with Resile:

Rate Limiting vs. Adaptive Retries

Shared Rate Limiters

Observability: Seeing the Shaping

Conclusion

Stop the Domino Effect: Bulkhead Isolation in Go

The Problem: Resource Exhaustion

The Solution: The Bulkhead Pattern

Implementing with Resile:

Using a Shared Bulkhead

Why "Fail-Fast" Matters

Observability: Monitoring the Walls

Conclusion

Infinite Data Processing in Go: Building Resilient Data Pipes with Channels

The Power of Channels as Data Pipes

Building a Pipeline Stage

Handling Stream Complexities with Helpers

1. The Branching Problem

2. Lookbacks and Sliding Windows

3. Bridging the Gap: Slices vs. Streams

Chaining the Pipes Together

Why This Architecture Wins

Try It Out

Self-Healing State Machines: Resilient State Transitions in Go

The Resile Way: One-Line Resilience for Transitions

What happens under the hood?

Why "Self-Healing"?

Observability: Tracking State Success

Conclusion

Preventing Microservice Meltdowns: Adaptive Retries and Circuit Breakers in Go

The Problem: Aggregate Load

1. Adaptive Retries (The Token Bucket)

Implementing with Resile:

2. Circuit Breakers (The Kill Switch)

Layering it in Resile:

The Ultimate Defense: Layered Resilience

Observability: Seeing the Shield in Action

Conclusion

Beating Tail Latency: A Guide to Request Hedging in Go Microservices

What is Request Hedging?

The Complexity of Manual Hedging

The Resile Way: DoHedged

What happens under the hood?

Picking the Right Hedging Delay

Observability: Tracking the "Speculative" Wins

Conclusion

Python's Stamina for Go: Bringing Ergonomic Resilience to Gophers

The Ergonomic Gap

The Resile Way: One-Line Resilience

Why "Correct by Default" Matters

Batteries Included (But Removable)

Fast CI/CD: The "Stamina" Secret

Conclusion

Stop Writing Manual Retry Loops in Go: Why Your Current Logic is Probably Dangerous

The 3 Silent Killers of Manual Retries

1. The Thundering Herd (Missing Jitter)

2. The Context Blindness

3. The time.After Memory Leak

The Resile Way: `DoHedged`

3. The `time.After` Memory Leak