Circuit Breaker Half-Open

The test state of the Circuit Breaker — the one preventing yo-yo between Open and Closed and making the pattern usable in production.

Problem

A basic Circuit Breaker has two states: Closed (traffic flows) or Open (traffic blocked). When the Walmart AS2 partner errors, the circuit opens after 5 failures in 10s, blocks for 30s, then retries. If Walmart still errors, the circuit closes — and 5 new requests try, fail, and the circuit reopens. Every 30s a burst of 5 requests crosses: we invented a system that DDoSes the partner the moment it returns. Worse: Open ↔ Closed transitions flip in a loop.

Forces

• Testing before reopening is necessary. Without a probe we do not know if the partner is back.
• The test must not be massive. If all traffic is sent at once and the partner is fragile, we bring it back down.
• The test must sample without blocking. The application service must not be blocked during the test.
• Multiple tests may be needed. One successful request is not proof of stability.

Solution

Add an intermediate Half-Open state to the Circuit Breaker. When the Open timer expires, go to Half-Open. In this state, allow at most N concurrent test requests (typically 1 to 3). Others continue receiving fail-fast 503. If the first M test requests (typically M=N=3) succeed, return to Closed: full traffic. If one fails, fall immediately back to Open and restart a timer (often longer, with exponential backoff). Resilience4j, Polly, Istio implement this state natively.

       [Closed]
       traffic OK
          │
   threshold exceeded
   (e.g. 5 failures / 10s)
          │
          ▼
       [Open]                  ◀───── failure on a test request
       traffic blocked                  │
       fail-fast 503                    │
          │                             │
   timer reset (e.g. 30s)               │
          │                             │
          ▼                             │
   [Half-Open]                          │
   N test requests  ──────────────────  │
   (e.g. 3 max concurrent)              │
          │                             │
   M consecutive successes              │
          │                             │
          ▼                             │
       [Closed] ──── full traffic ─────  │

EDI implementation

Concrete case: Java EDI hub with Resilience4j. Configuration circuitBreaker.walmart-as2: failureRateThreshold=50%, slidingWindowSize=20, waitDurationInOpenState=30s, permittedNumberOfCallsInHalfOpenState=3. At 11:00 Walmart starts replying HTTP 503. By 11:00:18 the circuit goes Open. Every AS2 to Walmart immediately replies CircuitBreakerOpenException, the application service routes to a retry queue. At 11:00:48 the circuit goes Half-Open. The first 3 AS2 messages are sent. Walmart replies 200 OK for all 3: circuit returns to Closed, full traffic resumes. If one had failed, Half-Open falls back to Open with timer doubled to 60s. On the ops console, the state diagram is live — Prometheus metrics resilience4j_circuitbreaker_state.

Anti-patterns

• No Half-Open. Circuit flapping between Closed and Open in a loop while the partner is unstable: retry bursts, partner collapsing the moment it reopens.
• Half-Open with too many test requests. If N=100, the DDoS is just delayed by 30s.
• Half-Open without cumulative backoff. If every return to Open always restarts at 30s, and the partner stays down 4h, 480 test bursts are generated. Double the timer on each failure.
• Read-only Half-Open. Testing with GET /health but automatically routing write traffic: the partner can be healthy and have a broken write pipeline. Test with a representative request.

Related patterns

• Circuit Breaker — the parent pattern.
• Retry + backoff — exponential backoff applies to the Open timer after each failure.
• Health Check — the test request in Half-Open is sometimes a health-check.
• Bulkhead — one Circuit Breaker per compartment.
• Timeout — each test request has its own timeout, distinct from the circuit.

Sources

• Nygard M. — Release It!, Pragmatic Bookshelf, 2nd ed. 2018. Chap. "Circuit Breaker" details the three states. pragprog.com — Release It! 2nd ed
• Fowler M. — CircuitBreaker (martinfowler.com, 2014). Pedagogical synthesis with the canonical state diagram. martinfowler.com — CircuitBreaker
• Resilience4j — CircuitBreaker. Java reference implementation with configurable Half-Open state. resilience4j.readme.io/docs/circuitbreaker
• Polly — Circuit Breaker. The .NET implementation. thepollyproject.org
• Istio — Circuit Breaking. The mesh's outlierDetection config implements the transition. istio.io — Circuit Breaking