Exactly-Once Semantics

How to guarantee that an event is processed exactly once — not zero, not two — in a distributed streaming pipeline that may restart at any time.

Problem

In a distributed streaming pipeline, a worker may crash mid-processing, a Kafka broker may reassign the partition, a window may emit twice after recovery. The natural outcome is at-least-once: each event is processed at least once, possibly several times. For most cases (KPI dashboard, anomaly detection), this is enough because aggregates are idempotent (a count can be recomputed). But for cases where each event mutates non-idempotent external state (send a notification, write a bank settlement, call the tax administration API), double processing is a major application bug. How to eliminate this duplication?

Forces

• Latency cost: exactly-once requires additional coordination (checkpoint, two-phase commit).
• Throughput cost: Kafka EOS divides throughput by 1.5-3 vs at-least-once.
• Sink frontier: engine-internal EOS is simpler; end-to-end EOS requires sink idempotency or 2PC.
• FLP impossibility: perfect exactly-once is theoretically impossible in an asynchronous system; in practice multiple mechanisms are combined.
• Marketing confusion: Kafka 0.11 announced "exactly-once" with a precise definition (atomic Kafka write) that is not always the expected one.

Solution

Three combined mechanisms. (1) Idempotent producer: each Kafka producer sends a sequence number per partition; the broker auto-dedupes (Kafka 0.11+, configuration enable.idempotence=true). (2) Kafka transactions: a producer can wrap multiple writes in an atomic commit/abort transaction; consumers in read_committed mode only see committed transactions. (3) Chandy-Lamport snapshots: Flink uses a distributed checkpoint algorithm (1985 paper) — injects barriers into the stream, each operator snapshots its state at barrier traversal, and the globally consistent state is reconstructible. On recovery, Flink replays from the last consistent checkpoint. (4) Sink 2PC: for an external sink (PostgreSQL, S3), a 2PC connector guarantees that the Kafka commit and sink commit are atomically tied.

Structure

Kafka source ──► Flink job ────► Kafka sink ──► External sink (DB)
                    │
                    │ Every N seconds:
                    │ inject CHECKPOINT barrier into stream
                    │
Operator A ──► [state A]─[barrier]─► Operator B ──► [state B]
                    │                        │
                    │ snapshot state A       │ snapshot state B
                    │ to durable storage     │ to durable storage
                    ▼                        ▼
              Checkpoint metadata committed (atomic)

On failure:
  Flink restarts from last checkpoint
  Source replays from offset stored in checkpoint
  All operators restore state from snapshot
  Sink: if 2PC, only committed transactions visible
  → No double effect from work between barrier and crash

EDI implementation

Exactly-once is required for several critical EDI flows. (1) Fiscal DRR reports (ViDA, July 2030): each intra-EU B2B invoice must be declared exactly once to the administration; the Flink pipeline → fiscal API connector must eliminate any duplication even after recovery. (2) Automatic INVOIC generation from an ORDERS flow: double generation = double invoicing = customer dispute. (3) Reception notification to the partner: double notification = double accounting at the partner. Typical implementation: Kafka source in read_committed mode → Flink with EOS checkpointing every 30 seconds → transactional sink connector (TwoPhaseCommitSinkFunction for PostgreSQL fiscal store + Kafka sink for downstream propagation). Always combine with application-level idempotency at the sink — that is defence in depth: if EOS fails, idempotency catches it.

Anti-patterns

• Believing engine-internal EOS = end-to-end EOS — without a transactional sink, the guarantee is lost at commit.
• Disabling checkpoints to gain throughput — loss of all EOS guarantees.
• Non-idempotent HTTP sink without per-key dedup — double notification guaranteed on recovery.
• Checkpoint interval too long — large replay volume on recovery.
• Confusing Kafka EOS (atomic write batch) and application EOS (single business effect) — Kafka guarantees atomicity, not that the business effect occurs once.

Related patterns

• Idempotency — the ultimate defence at the sink.
• At-Least-Once — the default semantics without EOS.
• Two-Phase Commit — used for sink coordination.
• Transactional Outbox — producer-side alternative.

Sources

• Carbone P. et al. — State Management in Apache Flink: Consistent Stateful Distributed Stream Processing, VLDB 2017. vldb.org
• Confluent — Exactly-Once Semantics Are Possible: Here's How Kafka Does It, 2017. confluent.io
• Chandy K., Lamport L. — Distributed Snapshots: Determining Global States of Distributed Systems, ACM TOCS 1985.
• Kreps J. — Exactly Once Delivery and Transactional Messaging in Apache Kafka, KIP-98 design doc. cwiki.apache.org
• Akidau, Chernyak, Lax — Streaming Systems, O'Reilly 2018, ch. 5.