Circuit Breaker & Back-Pressure

1. What Is It?

A circuit breaker is a fault-tolerance pattern that wraps outbound calls to a dependency (database, external API, microservice). When the dependency starts failing at a high rate, the circuit breaker "trips open" — it stops forwarding calls immediately, returning errors fast rather than waiting for timeouts. This prevents the caller from accumulating blocked threads waiting on a dead dependency, which would otherwise cascade the failure upstream.

Back-pressure is a flow-control mechanism where a downstream consumer signals to an upstream producer that it's reaching capacity and the producer should slow down or stop. Instead of the consumer being overwhelmed and dropping requests, back-pressure propagates the capacity constraint upstream so the system self-regulates.

These two patterns solve related but distinct problems:

• Circuit breaker: "The downstream service is broken — stop calling it."
• Back-pressure: "The downstream service is too slow right now — slow the upstream producer down."

Together with the bulkhead pattern (resource isolation) and aggressive timeouts, they form the four pillars of resilience in distributed systems.

2. How It Works

Circuit Breaker: State Machine

The circuit breaker is modeled as a state machine with three states:

                    Failure rate >= threshold
  ┌─────────────────────────────────────────┐
  │                                         ▼
┌─────────┐   Failures accumulate   ┌──────────┐
│ CLOSED  │ ─────────────────────► │   OPEN   │
│(normal) │                         │(fail fast)│
└─────────┘                         └──────────┘
     ▲                                    │
     │    Probe calls succeed             │ After sleep window
     │                                    ▼
     │                            ┌─────────────┐
     └──────────────────────────── │  HALF-OPEN  │
                                   │(test probes) │
                                   └─────────────┘
                                         │
                                         │ Probe calls fail
                                         ▼
                                   Back to OPEN

CLOSED (normal operation):

• All requests flow through to the dependency
• The breaker tracks a rolling window of calls and their outcomes (success/failure/timeout)
• When the failure rate reaches the configured threshold (default: 50% over 100 calls in Resilience4j), the breaker transitions to OPEN

OPEN (fail-fast):

• All calls to this dependency are immediately rejected — no network call is made
• Returns a CircuitBreakerOpenException or executes a fallback function instantly
• After a configurable sleep window (default: 60 seconds), transitions to HALF-OPEN
• This is the core protection: a thread that would have blocked for 10s waiting for a timeout instead fails in microseconds

HALF-OPEN (recovery testing):

• A limited number of probe requests (default: 10) are allowed through to the dependency
• If probe success rate ≥ threshold → breaker resets to CLOSED (dependency recovered)
• If probe failure rate ≥ threshold → breaker returns to OPEN (still broken, wait more)

Resilience4j Configuration Example

Copy
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)           // Trip at 50% failure rate
    .slidingWindowSize(100)             // Over last 100 calls
    .waitDurationInOpenState(Duration.ofSeconds(60))  // Stay open 60s
    .permittedNumberOfCallsInHalfOpenState(10)        // 10 probe calls
    .minimumNumberOfCalls(20)           // Need at least 20 calls to calculate rate
    .recordExceptions(IOException.class, TimeoutException.class)
    .build();

CircuitBreaker circuitBreaker = CircuitBreaker.of("paymentService", config);

// Wrap the call
Supplier<Response> decoratedCall = CircuitBreaker.decorateSupplier(
    circuitBreaker,
    () -> paymentService.charge(request)
);

Try<Response> result = Try.ofSupplier(decoratedCall)
    .recover(CircuitBreakerOpenException.class, ex -> fallbackResponse());

Mermaid: Circuit Breaker in Action

-+ResetFullscreen

Back-Pressure

Back-pressure is a feedback signal that flows upstream — the consumer tells the producer to slow down.

Without back-pressure:

Producer (fast: 10K msg/s) → Queue → Consumer (slow: 1K msg/s)
                                      Queue grows unboundedly
                                      → OOM → crash

With back-pressure:

Producer → asks: "can I send more?" → Consumer: "only request 100"
Producer sends 100 → Consumer processes 100 → signals: "ready for 100 more"
Rate matches consumer capacity — queue stays bounded

Reactive Streams protocol (request(n) pull model):

Copy
publisher.subscribe(new Subscriber<T>() {
    Subscription subscription;

    public void onSubscribe(Subscription s) {
        subscription = s;
        subscription.request(10); // "I can handle 10 items"
    }

    public void onNext(T item) {
        process(item);
        subscription.request(1); // "I've processed one, send me 1 more"
    }
});
// Publisher MUST NOT emit more items than requested

Back-pressure strategies for overflow:

Load Shedding vs Back-Pressure

Both address overload, but with different priorities:

Back-pressure:                      Load Shedding:
Producer ←[slow down]← Consumer    Producer → [DROP] ← Overloaded Consumer
Preserves data, increases latency   Preserves latency, loses data
First line of defense               Last resort safety valve

WHY: When a system is overwhelmed, there are exactly two choices: preserve data (accept higher latency) or preserve latency (drop data). Back-pressure chooses data; load shedding chooses latency/availability. A well-designed system uses back-pressure first and load shedding only when the queue would grow unboundedly.

Bulkhead Pattern

Isolate resource pools per downstream dependency so a failure in one doesn't exhaust shared resources:

WITHOUT Bulkhead:
Shared thread pool: [===========] 100 threads
  Payment service calls blocking: 90 threads
  User service calls:               8 threads
  Inventory service calls:          2 threads
  → Payment slow → ALL services starved

WITH Bulkhead:
  Payment pool:   [=====] 30 threads (only payment calls blocked here)
  User pool:      [=====] 30 threads (unaffected)
  Inventory pool: [=====] 30 threads (unaffected)
  → Payment slow → only payment calls fail; other services unaffected

Mermaid: Resilience Patterns Together

-+ResetFullscreen

Timeout Chains

Timeouts are the first line of defense against cascade failures:

Without timeouts:
  API Server waits 30s for DB
  → 100 concurrent requests × 30s wait = thread pool exhausted
  → New requests fail immediately
  → CASCADE

With aggressive timeouts (200ms):
  API Server waits 200ms, then returns error
  → Thread freed
  → Other requests unaffected
  → Circuit breaker eventually trips on high timeout rate
  → Cascade stopped

Timeout budgeting: Each layer in a call chain should have a shorter timeout than the layer above it:

Client timeout: 5000ms
  └─ API Gateway timeout: 3000ms
       └─ Service A timeout: 2000ms
            └─ Service B timeout: 1000ms
                 └─ Database timeout: 500ms

This ensures inner layers fail before outer layers, giving each layer a chance to handle the failure gracefully and allowing each service to provide a degraded-but-functional response.

3. Variants & Comparisons

Circuit Breaker Libraries

Envoy/Istio implement circuit breaking as a service mesh sidecar — language-agnostic, configured via YAML, no application code changes.

Hystrix vs Resilience4j

4. When to Use It (and When NOT To)

Use Circuit Breakers When:

• Calling external services: Payment processors, SMS gateways, third-party APIs that have their own availability SLAs and can go down independently.
• Microservices calling other microservices: Service A calling Service B — if B degrades, A should detect it and fail fast rather than blocking its own thread pool.
• Database connections: Wrap DB calls in a circuit breaker. DB slow → trip the breaker → return cached/degraded response rather than queuing 1000 threads waiting for DB connections.
• Any dependency with unpredictable latency: The circuit breaker is the safety net for when timeouts alone aren't enough.

Use Back-Pressure When:

• Producer-consumer pipelines with mismatched throughput: Kafka consumers falling behind producers; message processing queues growing unboundedly.
• Reactive/streaming systems: Reactive Streams / RxJava / Akka Streams — back-pressure is built into the protocol.
• Batch processing with downstream rate limits: ETL pipelines, rate-limited external API calls.
• WebSocket / SSE data feeds: Server pushing real-time data to clients — clients signal how fast they can consume.

Anti-Patterns:

• Circuit breaker with no fallback: A circuit breaker that trips open but just returns a 500 error without a fallback is better than nothing, but a cached response, default value, or "degraded mode" response is far better UX.
• Retry without exponential backoff: Retrying on failure is correct, but retrying immediately at full rate creates a "retry storm" that hammers an already-struggling service. Always use exponential backoff (wait 1s, then 2s, then 4s, then 8s between retries) + jitter (add randomness so all clients don't retry at the same instant).
• Shared thread pools across dependencies: Without bulkheads, one slow dependency consumes the entire thread pool. Always use separate pools or semaphores per dependency category.
• Not accounting for circuit breaker overhead in latency budgets: When the circuit breaker is open, calls fail in microseconds. When half-open, some probe calls have normal latency. Your SLO calculations should account for the state.

5. Real-World Usage

Netflix — Hystrix at Scale

Netflix built Hystrix to manage inter-service calls across 500+ microservices. Their canonical configuration settled on 50% failure rate over a 10-second rolling window. Netflix instrumented every downstream call with a circuit breaker and paired it with a real-time operational dashboard (Hystrix Dashboard + Turbine for aggregating metrics across instances). Their Chaos Monkey and Chaos Kong tools actively inject failures to validate that circuit breakers actually trip and the fallback path (degraded UI without recommendations, for example) works correctly. Netflix's design goal: losing the recommendation service should degrade Netflix, not take it down.

Google SRE — Back-Pressure and Load Shedding

Google's Site Reliability Engineering book documents their approach: services automatically shed non-critical load when approaching capacity limits. Their RESOURCE_EXHAUSTED gRPC status code is the explicit back-pressure signal in their service mesh. When a backend is overloaded, it returns RESOURCE_EXHAUSTED rather than accepting work it can't complete — callers are expected to back off, reducing overall system load and allowing recovery. Google explicitly designed their systems so that overload conditions are bounded rather than unbounded collapse.

AWS SQS — Back-Pressure via Queue Depth

Amazon SQS queues naturally implement back-pressure in event-driven architectures. When consumers fall behind producers, the queue depth grows. Auto-scaling groups can use SQS queue depth as a scaling metric (via CloudWatch) — as the queue grows, more consumer instances spin up, increasing processing rate until the queue stabilizes. This is back-pressure implemented at the infrastructure layer rather than in application code.

6. Interview Cheat Sheet

5 Sentences to Show Deep Understanding

1. "A circuit breaker prevents cascade failures by detecting a failing dependency and fast-failing subsequent calls without attempting the network round-trip — a call that would have blocked for 10s waiting for a timeout instead fails in microseconds, freeing the thread for other work and preventing the thread pool from filling up with blocked waiters."

2. "Back-pressure is the producer-side complement to the circuit breaker: instead of blocking calls to a broken downstream (circuit breaker), back-pressure slows the producer when the downstream is saturated — the consumer signals 'I can handle N more items' and the producer respects that limit rather than overwhelming the consumer."

3. "The three states of a circuit breaker form a control loop: Closed (normal operation, tracking failure rate) → Open (fail fast, stop all calls) → Half-Open (probe with limited calls to test recovery) → Closed again if recovery confirmed. The sleep window in Open state gives the dependency time to recover before probe calls restart."

4. "Bulkhead isolation is the reason a slow payment service shouldn't take down the user profile service — by allocating separate thread pools (or semaphores) per downstream dependency, the blast radius of one dependency's slowness is contained to that dependency's pool, leaving other service calls unaffected."

5. "Aggressive timeouts are the first line of defense: a 200ms timeout on a database call frees the thread if the DB is slow, whereas a 30s timeout holds the thread for 30 seconds — at 100 concurrent requests, 30s timeouts means thread pool exhaustion in seconds, while 200ms timeouts mean threads are recycled 150x faster."

Common Follow-Up Questions

Q: What is a retry storm and how do you prevent it? A: A retry storm occurs when a service becomes slow and multiple upstream services all start retrying simultaneously at full rate, multiplying the load on the already-struggling service. Prevention: (1) Exponential backoff with jitter — each retry waits longer, and the jitter spreads retries over time instead of synchronized waves; (2) Circuit breaker — after enough retries fail, the circuit trips open and stops retries entirely; (3) Max retry limit — don't retry forever, cap at 3-5 attempts.

Q: How do you implement a fallback when the circuit is open? A: Options in order of decreasing quality: (1) Cached response: return the last known good response (stale data is better than no data for many use cases); (2) Default/degraded response: return an empty list, a "feature unavailable" message, or a default value; (3) Alternative path: call a secondary service or a simpler backup implementation; (4) Fail gracefully: return a user-friendly error message rather than a stack trace. Which fallback to use depends on the business impact — for payments, fail loudly; for recommendations, show empty or default.

Q: What's the difference between timeout and circuit breaker? A: A timeout handles individual slow calls — it limits how long one call can block. A circuit breaker handles systemic failure patterns — it detects when many calls are timing out or erroring and proactively stops all calls rather than making each one pay the timeout penalty. Timeouts work call-by-call; circuit breakers work at the aggregate failure rate level. You need both: timeout detects the individual slow call; circuit breaker detects the pattern across many calls and stops trying.

Q: How would you tune circuit breaker thresholds in production? A: Start with Resilience4j defaults (50% failure rate, 100-call window, 60s sleep). Observe: (1) False positive rate — how often does the breaker trip when the dependency is actually healthy? Lower the threshold to reduce sensitivity. (2) Mean time to recovery — how long does the system stay degraded after a real failure? Reduce sleep window. (3) Use slow call thresholds as well (calls > 500ms count as failures) — a technically successful but very slow call is operationally a failure.

Connections to Other Building Blocks

• Message Queues (Kafka): Back-pressure is inherent in Kafka consumer groups — the lag metric tells you when consumers are behind. Kafka provides bounded queues that absorb bursts without blocking producers.
• Load Balancing: Load balancers and circuit breakers are complementary: the LB distributes to healthy instances (via health checks); the circuit breaker handles the case where an instance is technically healthy but its dependencies are failing.
• Rate Limiter: Rate limiter caps request rate to protect a service from too many callers. Circuit breaker stops calls to a failing dependency. Rate limiting is upstream protection; circuit breaking is downstream protection.
• API Gateway: API gateways (Kong, Envoy) implement circuit breaking as plugins — useful for protecting backend services from a failing shared dependency (e.g., all API routes share a database connection pool).
• Replication Strategies: Circuit breakers are especially valuable when one replica is degraded — the breaker detects high latency/errors from that replica and routes around it while it recovers.