Forem: siffiyan assauri

Retry and Backoff: Building Resilient Systems

siffiyan assauri — Mon, 24 Nov 2025 02:02:41 +0000

In network based applications, transient failures like network delays or temporarily unresponsive servers are common. Without a retry mechanism, these disruptions can cause immediate failures, affecting user experience and service reliability. Implementing retries is key to maintaining stability during such interruptions.

Case Study

Consider an application that connects to an external API to fetch weather data. If the API experiences a temporary disruption, our application will fail immediately without attempting to retry. This could result in failing to retrieve important data, even though the disruption might be temporary.

In the following example, the FetchWithoutRetry function attempts to make an HTTP request to the external API. If the network call fails, the application logs the error and returns nil, without retrying the request

func FetchWithoutRetry(apiURL string) (*http.Response, error) {
    response, err := http.Get(apiURL)
    if err != nil {
        return nil, fmt.Errorf("Request failed : %w", err)
    }
    return response, nil
}

This approach misses the opportunity to recover from a transient error that might be resolved with a simple retry.

Implementing a Retry Pattern

A retry mechanism attempts the same request again if it initially fails due to a transient issue. By implementing a retry pattern, you improve the resilience of your downstream services. Instead of immediately returning an error, your system attempts to retry the request a predefined number of times.

Here’s an example implementation in Go:

const (
    maxRetries = 3
    retryInterval = 2 * time.Second
)

func FetchWithRetry(apiURL string) (*http.Response, error) {
    for attempt := 1; attempt <= maxRetries; attempt++{
        if response, err := http.Get(apiURL); err == nil {
            return response, nil
        }

       fmt.Printf("Request failed, attempt %d/%d: %w\n", attempt, maxRetries, err)
        time.Sleep(retryInterval)
    }

    return nil, fmt.Errorf("Request failed after %d attempts", maxRetries)
}

In this example, the system attempts the request up to maxRetries times, waiting retryInterval between each attempt.

Exponential Backoff

While simple retries improve resilience, using fixed intervals can worsen transient failures, especially if a server is under heavy load.

Exponential backoff optimizes retries by increasing the delay between
each attempt. This reduces the load on the failing system and improves
the likelihood of a successful retry by giving the server more time to
recover with each successive attempt. The progressively longer intervals
also help prevent overwhelming a recovering server with repeated bursts
of traffic.

Exponential backoff gradually increases the wait time after each failure. For example, the wait time might start at one second, then increase to two seconds, four seconds, and so on. Here’s how you might implement exponential backoff in Go:

const (
    maxRetries = 3
    initialInterval = 2 * time.Second
    backoffFactor   = 2.0
)

func FetchWithExponentialBackoff(apiURL string) (*http.Response, error) {
    waitTime := initialInterval

   for attempt := 1; attempt <= maxRetries; attempt++{
        if response, err := http.Get(apiURL); err == nil {
            return response, nil
        } 

        fmt.Printf("Request failed, attempt %d/%d: %w\n", attempt, maxRetries, err)
        time.Sleep(waitTime)
        waitTime *= time.Duration(backoffFactor)        
    }

    return nil, fmt.Errorf("Request failed after %d attempts", maxRetries)
}

Adding Jitter to Backoff

While exponential backoff reduces server load by spacing out individual
client retries, it doesn't prevent the "thundering herd" problem when
multiple clients fail simultaneously. If 1,000 clients all experience
a failure at the same moment (for example, when a server crashes),
exponential backoff means they'll all retry together at 2 seconds,
then again at 4 seconds, then 8 seconds, and so on. Each wave of
synchronized requests can overwhelm the recovering server.

Jitter solves this by adding random variation to the retry intervals.
Instead of all 1,000 clients retrying at exactly 2 seconds, they might
retry anywhere between 2.0 and 2.2 seconds. This spreads the load over
a 200ms window instead of hitting the server with all 1,000 requests
in the same millisecond.

Here's how to implement jitter with exponential backoff:

const (
    maxRetries = 3
    initialInterval = 2 * time.Second
    backoffFactor   = 2.0
)

func FetchWithExponentialBackoffAndJitter(apiURL string) (*http.Response, error) {
    waitTime := initialInterval

    for attempt := 1; attempt <= maxRetries; attempt++{
        if response, err := http.Get(apiURL); err == nil {
            return response, nil
        } 

        fmt.Printf("Request failed, attempt %d/%d: %w\n", attempt, maxRetries, err)

         // Add jitter: up to 10% of the wait time
        jitter := time.Duration(rand.Int63n(int64(waitTime / 10)))
        actualWaitTime := waitTime + jitter

        time.Sleep(actualWaitTime)
        waitTime *= time.Duration(backoffFactor)        
    }

    return nil, fmt.Errorf("Request failed after %d attempts", maxRetries)
}

In this implementation, we add up to 10% random variation to the wait
time. This small adjustment can significantly reduce peak load during
recovery periods. The combination of exponential backoff (which gives
the server progressively more recovery time) and jitter (which prevents
synchronized retries) creates a robust retry strategy that helps both
client and server handle failures gracefully.

When to Retry Your Request

Although retry mechanisms enhance resilience, not every request to a downstream service should be retried. Consider the following factors:

Transient Errors : Focus on retrying errors that are likely to be temporary, such as network timeouts, server overload, or transient database issues
Type of Error : Not all errors should be retried. For example, retrying a 404 Not Found error is futile because the resource doesn’t exist.
Idempotency : Only retry idempotent operations like GET or DELETE requests. Retrying operations that modify state, such as POST, could cause unintended side effects.
Set Retry Limits : Set reasonable limits to prevent infinite loops and excessive resource usage.

Common Scenarios Where Retrying Might Not Help :

404 Not Found : The requested resource does not exist, so retrying won’t help.
401 Unauthorized : This error indicates invalid credentials. Retrying without correcting the credentials will fail.
403 Forbidden : The client does not have permission to access the resource. Retrying will not change the authorization status.
Permanent Errors : Errors caused by permanent issues, like configuration problems, should not be retried.

Conclusions

Implementing retry mechanisms with exponential backoff and jitter
significantly improves application resilience against transient failures
such as temporary network disruptions, server overloads, and brief
service outages. These techniques help applications recover automatically
from temporary issues without requiring manual intervention.

However, retry strategies come with important trade-offs to consider:

Benefits:

Automatic recovery from transient failures without user impact
Better tolerance for downstream service instabilities
Improved overall system reliability during peak loads

Trade-offs:

Increased end-to-end latency when retries are triggered
Additional resource consumption (network bandwidth, server capacity)
Risk of masking underlying problems that require investigation

Remember that retries are just one component of a comprehensive
resilience strategy. For production systems, consider combining retries
with complementary patterns such as:

Circuit breakers to prevent cascading failures when downstream services are consistently failing
Timeouts to avoid indefinitely waiting for responses
Monitoring and alerting to track retry rates and identify patterns that indicate deeper issues

Always evaluate the error type before retrying, not all failures benefit
from retries. Focus on idempotent operations and transient errors, and
avoid retrying authentication failures, permanent errors (4xx status
codes), or operations that modify state without proper idempotency
guarantees. By thoughtfully implementing these patterns, you build
systems that gracefully handle failures while remaining observable
and maintainable.

References

https://www.pullrequest.com/blog/retrying-and-exponential-backoff-smart-strategies-for-robust-software/

https://codecurated.com/blog/designing-a-retry-mechanism-for-reliable-systems/

Controlling Concurrency in Go with the Semaphore Pattern

siffiyan assauri — Fri, 10 Oct 2025 02:18:17 +0000

Concurrency is one of Go’s greatest strengths, but without proper control it can quickly overload your system and drain resources.

The Semaphore Pattern provides a solution by limiting how many goroutines can run concurrently, keeping applications stable and resource usage safe.

What Is a Semaphore?

A semaphore is a synchronization mechanism used to control concurrent access to a shared resource. A simple analogy is a traffic light: only a certain number of cars can pass at a time, while the rest must wait for their turn.

In the same way, a semaphore limits how many processes or goroutines can run concurrently, preventing the system from becoming overloaded.

Why Use a Semaphore?

In modern applications, it is often necessary to limit concurrency, for example :

Rate limiting API requests to avoid being throttled or blocked by an external service.
Controlling database connections so they don’t exceed the capacity of the connection pool.
Maintaining system stability under high load to prevent resource exhaustion.

Without concurrency control, applications risk spikes in CPU, memory, and network usage, which can lead to crashes or severe performance degradation.

Semaphores in Go

Go does not provide a built-in semaphore implementation. However, there are two common approaches :

1. Buffered Channel

Use a channel with a fixed capacity to limit the number of goroutines running concurrently. This approach is simple and works without extra dependencies.

2. golang.org/x/sync/semaphore

An external package that provides a more feature-rich semaphore, for example :

Supports weighted semaphores (a task can acquire more than one “slot”).
Tight integration with context.Context → enables cancellation and timeout handling.

Case Study: Limiting Bulk Email Sending

Imagine a marketplace needs to send 10,000 New Year promotional emails.

If all emails are sent at once :

The SMTP provider may throttle or block requests.
The application server will experience a resource spike.
There is a high risk of being blacklisted as spam.

The result: campaign failure and lost business opportunities.

Solution with Semaphore

By applying a semaphore, we can limit how many emails are sent concurrently, for example 50 emails per batch.

Other goroutines wait until slots are available.
The system remains stable and controlled.
The campaign runs smoothly without sacrificing performance.

Implementing a Semaphore in Go

1. Using a Buffered Channel

A straightforward idiomatic approach in Go. Each goroutine must acquire a slot from the channel before executing and release it when done.

package main

import (
    "fmt"
    "sync"
    "time"
)

func main() {
    totalEmails := 10000
    maxConcurrent := 50

    sem := make(chan struct{}, maxConcurrent) // max 50 emails concurrently
    var wg sync.WaitGroup

    for i := 1; i <= totalEmails; i++ {
        wg.Add(1)
        sem <- struct{}{}            // acquire slot
        go func(uid int) {
            defer wg.Done()
            defer func() { <-sem }() // release slot      
            SendPromoEmail(uid)      // simulate send promo email
        }(i)
    }

    wg.Wait() // wait for all goroutines to complete
}

func SendPromoEmail(uid int) {
    // Simulate email sending
    fmt.Printf("Sending email to user %d\n", uid)
    time.Sleep(100 * time.Millisecond)
}

2. Using sync/semaphore

For more advanced use cases, the external package provides better flexibility.

package main

import (
    "context"
    "fmt"
    "sync"
    "time"

    "golang.org/x/sync/semaphore"
)

func main() {
    totalEmails := 10000
    maxConcurrent := 50

    ctx := context.Background()
    sem := semaphore.NewWeighted(int64(maxConcurrent))
    var wg sync.WaitGroup

    for i := 1; i <= totalEmails; i++ {
        wg.Add(1)

        if err := sem.Acquire(ctx, 1); err != nil {
            fmt.Printf("Failed to acquire semaphore: %v\n", err)
            wg.Done()
            continue
        }

        go func(uid int) {
            defer wg.Done()
            defer sem.Release(1) // release slot

            SendPromoEmail(uid) // simulate send promo email
        }(i)
    }

    wg.Wait() // wait for all goroutines to complete
}

func SendPromoEmail(uid int) {
    // Simulate email sending
    fmt.Printf("Sending email to user %d\n", uid)
    time.Sleep(100 * time.Millisecond)
}

Other Use Cases

Besides bulk email sending, semaphores are useful in many scenarios, such as:

1. Rate Limiting API Calls
Restricting requests to an external API to avoid exceeding rate limits or being throttled.

2. Batch Job Processing
Controlling how many batch jobs (report generation, ETL, background tasks) run simultaneously to prevent server overload.

3. File I/O or Network Calls
Limiting the number of heavy I/O operations (file uploads/downloads, syncing data to cloud storage) so bandwidth and disk usage remain stable.

Best Practices

When using semaphores in Go, consider these best practices :

Choose the right limit : Match the semaphore size to your system’s capacity.
Use Context: Always handle cancellations with context.Context.
Avoid deadlocks : Always release the semaphore, preferably with defer.
Monitor performance: Profile your application to tune semaphore size.

Limitations and Alternatives

While semaphores are powerful, there are important limitations:

1. Memory Overhead with Large Workloads

If you spawn one goroutine per task and limit them with a semaphore, memory usage can grow significantly when the workload reaches hundreds of thousands of tasks.

Solution : use a worker pool to keep the number of goroutines constant, while tasks are queued and processed by workers in a controlled manner.

2. Distributed System Limitations

Semaphores only control concurrency within a single program instance.

In a distributed system with many instances (e.g., multiple email-sending services running in parallel in a cluster), a semaphore cannot enforce concurrency limits across nodes.

Solution :
Use a distributed lock (e.g., Redis) to coordinate concurrency across nodes. This ensures all instances share a global limit so that only a certain number of tasks run across the entire cluster at the same time.

3. System Reliability

Semaphores even with distributed locks like Redis only act as a concurrency control mechanism. They do not guarantee task reliability.

Things not handled by semaphores or Redis:

Retry & Failure Recovery → there is no built in retry mechanism.
Task Persistence → if the application crashes, in-progress tasks are lost.
Delivery Guarantee → no assurance that tasks are processed at-least-once or exactly-once.

For high reliability, use a Message Broker such as Kafka, RabbitMQ, or NATS.

Tasks are stored durably, surviving application crashes.
Built-in support for retries and delivery guarantees.
Workers can consume tasks at their own pace, keeping load stable.

Conclusion

Semaphores are a simple pattern for controlling concurrency at the application level. They are ideal for small to medium workloads where local resource stability is the main concern.
For very large workloads, worker pools are more efficient than spawning massive numbers of goroutines with semaphores.
In distributed systems, semaphores should be combined with distributed coordination (e.g., Redis) to enforce concurrency limits across nodes.
For high reliability, use a message broker (Kafka, RabbitMQ, NATS). Semaphores remain valuable for local concurrency control, while brokers ensure durability, retries, and stable task distribution.

Credits

Car icons created by Vectors Market - Flaticon
Electric car icons created by kosonicon - Flaticon
Traffic light icons created by xnimrodx - Flaticon
Promo icons created by Freepik - Flaticon
People icons created by Vitaly Gorbachev - Flaticon
Marketplace icons created by Dewi Sari - Flaticon