Forem: Gauri Katara

After Fixing Thread Pool Exhaustion, A Reader Asked: Did You Add a Circuit Breaker? Here's What I Did Next

Gauri Katara — Wed, 22 Apr 2026 03:38:10 +0000

This is a follow-up to my previous article —

Our Spring Boot API Froze Under Load

[https://hashnode.com/edit/cmny3jv0u00bg2dlndzza08zy]

A reader asked me a question I couldn't stop thinking about.

In my last article I wrote about how our Spring Boot API froze under load because of thread pool exhaustion. We fixed it with two things — adding timeouts on external calls and isolating slow dependencies into their own thread pool using @async. A reader left a comment that stopped me mid-scroll: "After the fix, did you keep monitoring thread pool metrics as a guardrail — or add any alerts around it? And did you consider a circuit breaker?" We had added monitoring. But the circuit breaker part? Honestly, we had talked about it and not implemented it yet. That comment pushed me to finally do it properly — and in this article I want to share what I learned about Resilience4j circuit breakers, why they are the missing layer in most Spring Boot microservices, and how to implement them step by step.

The problem with just timeouts

Why timeouts alone are not enough
After our fix, here is what our system looked like: — External service calls had a 5 second timeout — Slow calls ran on an isolated thread pool — CloudWatch alerted us when active threads crossed 70% of max This was much better than before. But there was still a problem. Imagine the external service we were calling started failing — not slowly, but completely. Every call we made was timing out after 5 seconds and returning an error. With just timeouts, here is what happens: every single request to our API still tries to call that external service, waits 5 seconds, times out, and then returns an error. We are calling a service we already know is down — over and over — wasting threads, wasting time, and making the recovery slower. What we needed was something that would say: "This service has been failing repeatedly — stop calling it for a while. Let it recover. Then try again carefully." That is exactly what a circuit breaker does.

What is a circuit breaker
Circuit breaker — the concept in plain English

The name comes from electrical engineering. A circuit breaker in your home trips when there is too much current — it breaks the circuit to prevent damage. You reset it manually when the problem is fixed. In software it works the same way — with three states: CLOSED — Everything is working normally. Requests go through. The circuit breaker monitors failure rate in the background. OPEN — Too many failures have happened. The circuit breaker trips. Instead of trying the actual call, it immediately returns a fallback response. No waiting, no timeouts — fast failure. HALF-OPEN — After a waiting period, the circuit breaker allows a few test requests through. If they succeed, it goes back to CLOSED. If they fail, it goes back to OPEN. The result: when a dependency is down, you fail fast instead of failing slow. Your threads are not stuck waiting. Your system stays responsive. And the failing service gets breathing room to recover.

Circuit breakers don't prevent failures. They contain them — stopping one failing dependency from bringing down your entire system.

Add Resilience4j
Step 1 — Add the dependency
Resilience4j is the most widely used resilience library for Java. It integrates cleanly with Spring Boot and is lightweight compared to alternatives like Hystrix (which is now in maintenance mode). Add this to your pom.xml:

<dependency>
    <groupId>io.github.resilience4j</groupId>
    <artifactId>resilience4j-spring-boot3</artifactId>
    <version>2.1.0</version>
</dependency>

<!-- Also needed for Actuator metrics -->
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-aop</artifactId>
</dependency>

Use resilience4j-spring-boot3 for Spring Boot 3.x. For Spring Boot 2.x use resilience4j-spring-boot2.

Configure it
Step 2 — Configure the circuit breaker
Add this to your application.properties or application.yml:

#application.properties
#Name your circuit breaker — use the same name in your code
resilience4j.circuitbreaker.instances.externalService.sliding-window-size=10

resilience4j.circuitbreaker.instances.externalService.failure-rate-threshold=50 

resilience4j.circuitbreaker.instances.externalService.wait-duration-in-open-state=10s 

resilience4j.circuitbreaker.instances.externalService.permitted-calls-in-half-open-state=3 

resilience4j.circuitbreaker.instances.externalService.automatic-transition-from-open-to-half-open-enabled=true 

resilience4j.circuitbreaker.instances.externalService.register-health-indicator=true

Let me explain what each setting means in plain English:

sliding-window-size=10
// Look at the last 10 calls to decide the failure rate
failure-rate-threshold=50
// If 50% or more of those 10 calls failed — trip the circuit (go OPEN)
wait-duration-in-open-state=10s
// Stay OPEN for 10 seconds before trying again (HALF-OPEN)
permitted-calls-in-half-open-state=3
// In HALF-OPEN, allow 3 test calls — if they pass, go back to CLOSED
automatic-transition-from-open-to-half-open-enabled=true
// Automatically move to HALF-OPEN after wait time — no manual reset needed.

Use it in code
Step 3 — Add the annotation and fallback

Now wrap the method that calls your external service with the @CircuitBreaker annotation:

@Service 
public class ExternalDataService {

// Circuit breaker name matches what you set in properties
@CircuitBreaker(name = "externalService", fallbackMethod = "getDataFallback")
public EnrichmentData getExternalData(String userId) {
    // This is the call that could fail or be slow
    return externalApiClient.fetchEnrichmentData(userId);
}

// Fallback — called automatically when circuit is OPEN
// Must have same parameters + a Throwable parameter
public EnrichmentData getDataFallback(String userId, Throwable ex) {
    // Return a safe default instead of failing the whole request
    // Options: cached data, empty object, default values
    log.warn("Circuit breaker open for externalService. " +
             "Returning fallback for userId: {}. Reason: {}",
             userId, ex.getMessage());

    return EnrichmentData.defaultData();
}
}

The fallback method signature must match the original method exactly — same parameters — plus one extra Throwable parameter at the end. If the signature doesn't match, Spring will throw an error at startup.

Combine with timeout
Step 4 — Combine circuit breaker with timeout

A circuit breaker works best when combined with a timeout. The timeout prevents threads from waiting too long. The circuit breaker prevents calling a service that is already known to be failing. Resilience4j has a built-in TimeLimiter for this:

#Add timeout configuration alongside circuit breaker
resilience4j.timelimiter.instances.externalService.timeout-duration=3s
resilience4j.timelimiter.instances.externalService.cancel-running-future=true

// Use both annotations together
@CircuitBreaker(name = "externalService", fallbackMethod = "getDataFallback")
@TimeLimiter(name = "externalService")
public CompletableFuture<EnrichmentData> getExternalData(String userId) {
    return CompletableFuture.supplyAsync(() ->
        externalApiClient.fetchEnrichmentData(userId)
    );
}

// Fallback for combined circuit breaker + time limiter
public CompletableFuture<EnrichmentData> getDataFallback(
        String userId, Throwable ex) {
    log.warn("Fallback triggered for userId: {}. Reason: {}",
             userId, ex.getMessage());
    return CompletableFuture.completedFuture(
        EnrichmentData.defaultData()
    );
}

Monitor it
Step 5 — Monitor circuit breaker state with Actuator

Once Resilience4j is set up with register-health-indicator=true, you can see the circuit breaker state through Spring Boot Actuator:

#Enable circuit breaker metrics in Actuator
management.endpoints.web.exposure.include=health,metrics,circuitbreakers 
management.health.circuitbreakers.enabled=true
#Check health endpoint
GET /actuator/health
#Response shows circuit breaker state:
#"externalService": { "status": "CLOSED", "details": {...} }
#Check detailed metrics
GET /actuator/metrics/resilience4j.circuitbreaker.state
GET /actuator/metrics/resilience4j.circuitbreaker.failure.rate

Add a CloudWatch alarm on the circuit breaker state metric — if it goes OPEN, you want to know immediately. A circuit breaker going OPEN is a signal that something in your system needs attention.

Full picture

The complete resilience stack — how it all fits together

After everything we implemented, here is the full picture of how an external call is protected in our service: 1. Request comes in to our API 2. Main Tomcat thread handles it normally 3. External call is handed off to isolated async thread pool (@async) 4. TimeLimiter ensures the call times out after 3 seconds maximum 5. Circuit breaker monitors failure rate across last 10 calls 6. If failure rate exceeds 50% — circuit opens, fallback returns immediately 7. After 10 seconds — 3 test calls go through in HALF-OPEN state 8. If tests pass — circuit closes, normal operation resumes 9. CloudWatch monitors active thread count AND circuit breaker state Each layer handles a different failure scenario. Together they mean one slow or failing external dependency cannot bring down our service.

This is called the Bulkhead + Circuit Breaker pattern. It is one of the core resilience patterns in microservices architecture — documented in the Microsoft Azure Architecture patterns and Netflix's engineering blog.

What I learned from this whole journey

The honest truth is that we should have had a circuit breaker from day one. Timeouts and thread isolation were necessary but they were always incomplete without it. What I found interesting about implementing Resilience4j is how much it forces you to think about failure modes upfront. What should the fallback return? What is an acceptable failure rate? How long should we wait before retrying? These are questions that reveal a lot about how well you understand your own system's dependencies. If I were starting a new microservice project today, these three things would go in before the first endpoint is written:
— Timeouts on every external call
— Isolated thread pools for external dependencies
— Circuit breaker with a sensible fallback Not after the first production incident.

Before. --- This article was directly inspired by a comment on my previous post about thread pool exhaustion. If you haven't read that one, it gives context for why we ended up here. If you found this useful or have questions about Resilience4j configuration

— drop a comment.

And if you are using a different resilience pattern in your Spring Boot services, I would genuinely love to hear how you approached it.

Follow me here for more practical Java backend content — no theory-only posts, just things that come from real production systems.

Our Spring Boot API Froze Under Load — Here's Exactly How We Fixed It

Gauri Katara — Thu, 16 Apr 2026 04:06:09 +0000

This article was originally published on
My Hashnode blog [https://gaurikatara.hashnode.dev]

The API didn't throw an error. It just... stopped responding or frozen.

No exceptions in the logs. No 500 errors. No obvious reason. Just requests piling up, response times climbing from milliseconds to 30 seconds, and then — timeouts. It was strange. Traffic had picked up slightly — nothing unusual for that time of day — and suddenly our backend service was barely responding. Users were seeing blank screens. This is the story from one of my typical workdays about how we diagnosed and fixed issue in our Spring Boot microservice — and what I learned from it that I hadn't found clearly explained anywhere online at single point.

I began investigating all the potential causes:

Network issues
Examine database metrics
Monitor CPU and memory usage
Inspect thread dumps
Review load balancer logs

What I discovered was surprising—it was related to the threads.

The issue stemmed from thread pool exhaustion in production.

WHAT IS THREAD POOL EXHAUSTION — and why is it so sneaky?

Spring Boot uses an embedded Tomcat server by default. Tomcat handles incoming HTTP requests using a fixed pool of threads. By default, this pool has a maximum of 200 threads. Here is how it works normally: a request comes in, Tomcat assigns it a thread, the thread does its work, returns a response, and the thread goes back to the pool. Fast, clean, repeatable.

Now here is the problem: what if a thread gets stuck waiting?

Maybe it's calling an external API that is slow.
Maybe it's waiting on a database query that is taking too long.
Maybe it's doing something blocking that it shouldn't be.

If enough threads get stuck waiting at the same time, the pool runs out. New requests come in but there are no threads available to handle them. So they wait in a queue. The queue fills up. New requests start getting rejected or timing out. Your API is not down. It is not throwing errors. It is just frozen. And the worst part — from the outside it looks exactly like a network issue or a deployment problem. Most developers waste hours looking in the wrong place.

Thread pool exhaustion does not always look like an error. It often looks like extreme slowness or total unresponsiveness — with perfectly clean logs.

HOW WE DIAGNOSED IT

Step 1 — Check the thread pool metrics

The first thing I did after ruling out obvious causes — no deployment had happened, no database was down — was check our application metrics on AWS CloudWatch. We had Spring Boot Actuator enabled, which exposes metrics including thread pool stats. I looked at the active thread count and it was sitting at exactly 200 — the Tomcat default maximum. That was the first real clue. If your team has Actuator set up, you can check this yourself at: /actuator/metrics/tomcat.threads.busy If that number is at or near your maximum thread count while the API is slow, thread pool exhaustion is almost certainly your problem.

Step 2 — Take a thread dump

To confirm, I took a thread dump while the service was under load. A thread dump shows you exactly what every thread in your JVM is doing at that moment. You can trigger one using:

Using jstack (get the PID first with: ps aux | grep java)

jstack > thread_dump.txt
Or via Spring Boot Actuator if enabled

Postman -- http://localhost:8080/actuator/threaddump
When I opened the thread dump, I saw the same pattern repeating across most of the 200 threads. They were all stuck in a WAITING or TIMED_WAITING state, blocked on an HTTP call to an external third-party service we were calling to enrich our response data. That external service had started responding slowly — averaging 25 seconds per call instead of the usual 200ms. Our code was calling it synchronously, blocking the thread the entire time. With enough concurrent requests, every thread in the pool was stuck waiting for that external service. New requests had nowhere to go.

THE FIX

Step 3 — The fix had two parts

We fixed it with two changes. The first was immediate — add a timeout to the external HTTP call so threads don't wait forever. The second was the proper fix — make the call non-blocking.

Part 1 — Add connection and read timeouts immediately

This was a quick fix we could deploy right away. We configured our RestTemplate with explicit timeouts:

@Configuration public class RestTemplateConfig {

@Bean
public RestTemplate restTemplate() {
    HttpComponentsClientHttpRequestFactory factory =
        new HttpComponentsClientHttpRequestFactory();

    // Don't wait more than 3 seconds to connect
    factory.setConnectTimeout(3000);

    // Don't wait more than 5 seconds for a response
    factory.setReadTimeout(5000);

    return new RestTemplate(factory);
}
}

Always set timeouts on any external HTTP call. No timeout means a thread can be stuck waiting forever. This is one of the most common mistakes in Java backend services.

Part 2 — Increase thread pool size as a short-term buffer

While we worked on the proper async solution, we also increased the Tomcat thread pool size to give us more breathing room under load. This goes in your application.properties:

Maximum number of threads to handle requests
server.tomcat.threads.max=400

Minimum threads always kept alive
server.tomcat.threads.min-spare=20

Max requests that can wait when all threads are busy
server.tomcat.accept-count=100 Increasing thread count is not a real fix — it just delays the problem. If your threads are blocking, adding more threads means more threads will block. Always fix the root cause.

Part 3 — The proper fix: make the external call async

The real solution was to stop blocking a thread while waiting for the external service. We used Spring's @async with a dedicated thread pool for external calls, isolating them from the main request-handling threads:

@Configuration 
@EnableAsync 
public class AsyncConfig {

@Bean(name = "externalCallExecutor")
public Executor externalCallExecutor() {
    ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
    executor.setCorePoolSize(10);
    executor.setMaxPoolSize(50);
    executor.setQueueCapacity(100);
    executor.setThreadNamePrefix("external-call-");
    executor.initialize();
    return executor;
}
}

Now the external call runs on its own isolated thread pool. Even if that external service slows down completely, our main Tomcat threads keep handling requests normally. The two concerns are fully separated.

What happened after the fix

After deploying the timeout fix first, the immediate crisis resolved. Threads were no longer getting stuck indefinitely — after 5 seconds they would time out, return an error or fallback response, and free up for the next request. After the async refactor, the problem disappeared entirely. Active thread count during peak traffic dropped from 200 (maxed out) to consistently under 40. Response times went back to normal. The external service could be slow or even temporarily down, and our API kept working.

What I learned from this

Always set timeouts on external calls. Every HTTP call your service makes to another service or API must have a connect timeout and a read timeout. No exceptions. This is the single most common cause of thread pool exhaustion.
Thread pool exhaustion looks like slowness, not errors. If your API is suddenly unresponsive with no exceptions in the logs, check your active thread count before you do anything else.
Increase thread pool size only as a temporary measure. It buys you time but does not fix the root cause. The real fix is to stop blocking threads in the first place.
Isolate slow dependencies. Any external call that could be slow or unreliable should run on its own thread pool, completely separate from your main request-handling threads. This is called bulkhead pattern — and it is one of the most important resilience patterns in microservices.
Actuator is your best friend in production. If you are running Spring Boot and you do not have Actuator enabled with metrics exposed, you are flying blind. Enable it. It will save you hours the next time something goes wrong. --- I hope this walkthrough helped.

Thread pool exhaustion may seem complex but is easily fixable once you know the signs. If you've faced this issue or are dealing with it now, share your approach in the comments. If you found this useful, follow me for insights on Java backend engineering, Spring Boot, Kafka, AWS, and real-world production challenges