Building Resilient Microservices with Circuit Breakers

When you’re running services that handle 500k+ daily transactions, transient failures stop being edge cases and start being the norm. Networks flap, downstream APIs rate-limit, and databases hiccup. The difference between a resilient system and a brittle one is almost never about avoiding failures — it’s about how gracefully you handle them.

Why fault tolerance matters

In a distributed system, a single slow dependency can cascade. If your service holds connections open while waiting on a downstream timeout, thread pools exhaust, and a minor blip becomes a full outage. This is the textbook definition of a cascading failure.

The fix is rarely more hardware. It’s patterns.

The core patterns

I rely on three, almost always in combination:

1. Timeouts — every outbound call needs one. No exceptions.
2. Exponential backoff with jitter — retry, but don’t retry naively.
3. Circuit breaker — stop calling a service that’s clearly broken.

// Simplified circuit breaker state machine
type State int

const (
    Closed State = iota
    Open
    HalfOpen
)

Backoff that actually works

Naive retry loops are a great way to turn a 100ms blip into a sustained thundering herd. Exponential backoff spreads load, but without jitter you still get synchronized retries. Always add randomized jitter — even ±20% makes a real difference under load.

Closing thoughts

Resilience is an architectural property, not a library. The patterns above are the vocabulary; the real work is reasoning about failure modes at design time and measuring them in production.