Leader Election

Designate a single node — re-designable in seconds — to do what several should not do simultaneously.

Problem

A service runs in N instances for HA, but some tasks must have only one executor at time T: cron poller, scheduler, sequential channel processor. If every instance runs in parallel, you get either duplication (sending the invoice 5 times) or corruption (two nodes reordering the same outbox).

Forces

• A crashed instance must hand off its role quickly (else the task is suspended).
• No node should believe it's the leader while another already is (split brain).
• Too-frequent elections create flapping and unnecessary latency.
• Heartbeat-based failure detection must be faster than the business tolerance window.

Solution

Use a coordination service (ZooKeeper, etcd, Consul, or Kubernetes Lease ConfigMap) where each candidate tries to take a lease with TTL. The lease holder is leader; it must renew before expiration. Others observe: if the lease expires (the leader is dead or unreachable), the first candidate to claim the lease becomes leader. Raft (etcd) and ZAB (ZooKeeper) algorithms handle registry consistency. For simple cases, a SELECT ... FOR UPDATE on a dedicated row in DB suffices (Postgres advisory lock).

Structure

Coordination service (etcd / ZooKeeper)
       lease key: "edi.outbox.poller.leader"
                  │
   ┌──────────────┼──────────────┐
   ▼              ▼              ▼
 Node A         Node B         Node C
 (current      (observer)    (observer)
  leader)
   │
   │ renew lease every 5s (TTL 15s)
   │
   X dies (network partition)
                  │
        lease TTL expires 15s later
                  │
        Node B acquires lease ──► becomes leader

EDI implementation

A 3-replica EDI hub needs to designate one node for: (1) polling AS2 MDNs — else two nodes would process the same MDN and mark the order "delivered" twice; (2) generate the end-of-day report sent to the manager; (3) purge the outbox table. Simple implementation: a Kubernetes Lease edi-mdn-poller with 30 s retention, renewed every 10 s by the leader. On node crash, another takes the lease in under a minute.

Anti-patterns

• TTL too long — a dead node blocks the task for minutes.
• No fencing token — a former leader woken after a partition keeps writing and corrupts state.
• Election on a single-instance coordination service — it becomes a SPOF.
• Leader that doesn't stop the task when it loses the lease — split brain guaranteed.
• Lease renewal interrupted by a Java GC pause — leader loses lease without knowing it's "dead".

Related patterns

• Consensus (Raft) — algorithm used by etcd and many others.
• Competing Consumers — alternative when concurrency is acceptable.
• Circuit Breaker — associate to handle coordination service failure.
• Health Check — basis of failure detection.

Sources

• Microsoft Azure Architecture — Leader Election. learn.microsoft.com/en-us/azure/architecture/patterns/leader-election
• Kubernetes — Coordinated Leader Election (Lease API). kubernetes.io/docs/concepts/architecture/leases
• Kleppmann M. — Designing Data-Intensive Applications, O'Reilly 2017, ch. 9 "Consistency and Consensus".