Transactional Outbox

A business command writes its business state and the event to publish in the same local DB transaction. An asynchronous relay drains the outbox table to the broker. Two writes become one local ACID guarantee, plus an eventually-consistent publication.

Problem

The classic dual-write scenario: an API ingests an EDIFACT ORDERS, must persist the order in PostgreSQL and publish an OrderReceived event to Kafka so downstreams (ERP, WMS, notifications) react. If you write to PostgreSQL first then publish to Kafka and the process crashes in between, the order exists in DB but no consumer is notified — lost business event. If you publish to Kafka first, you risk the inverse situation (consumer reacts to an order that was never committed). 2PC between PostgreSQL and Kafka exists (XA) but pays in latency and blocks on coordinator failure.

Forces

• Business atomicity: guarantee that no event is published without persisted state and vice-versa.
• No 2PC: avoid XA between DB and broker (see 2PC).
• Acceptable latency: publishing can be delayed from ms to seconds depending on the drain strategy.
• Consumer idempotency: the relay may publish multiple times (at-least-once); consumers must deduplicate.
• Order per aggregate: order of events for one order must be preserved.

Solution

Create an outbox table in the same database as the business tables. Any transaction modifying business state also writes the event-to-publish into outbox in one transaction. A relayer process drains this table periodically or via CDC, publishes to Kafka, marks the entry as processed (or removes it). Two drain strategies:

• Polling publisher: a worker reads every N seconds rows WHERE published_at IS NULL ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED, publishes, updates. Simple, but adds DB load.
• CDC publisher: Debezium reads the PostgreSQL WAL and publishes outbox inserts directly to Kafka. Zero lag, zero app load, but adds Debezium to the stack.

Structure

            ┌────── Application ──────┐
            │                          │
            │  BEGIN TX;               │
            │  INSERT INTO orders ...; │
            │  INSERT INTO outbox ...; │ ◄── same TX
            │  COMMIT;                 │
            │                          │
            └──────────┬───────────────┘
                       │
                       ▼
            ┌── PostgreSQL ───────────┐
            │  orders (business state) │
            │  outbox (events)         │
            └──────┬───────────────────┘
                   │
              ┌────┴────┐
              │         │
         polling     CDC (Debezium)
              │         │
              ▼         ▼
            ┌── Kafka ────────────────┐
            │  topic: edi.order.events │
            └──────────────────────────┘

EDI implementation

Typical schema for a 2026 EDI hub, with per-aggregate ordering preserved:

-- Outbox table with monthly partitioning for archiving
CREATE TABLE outbox (
  id              BIGSERIAL PRIMARY KEY,
  aggregate_id    VARCHAR(80) NOT NULL,      -- 'ORDER-12345'
  aggregate_type  VARCHAR(40) NOT NULL,      -- 'Order'
  event_type      VARCHAR(80) NOT NULL,      -- 'OrderReceived'
  payload         JSONB NOT NULL,
  headers         JSONB,                     -- correlation_id, source
  created_at      TIMESTAMPTZ DEFAULT now(),
  published_at    TIMESTAMPTZ,
  partition_key   VARCHAR(40) GENERATED ALWAYS AS (aggregate_id) STORED
);
CREATE INDEX outbox_unpublished
  ON outbox (created_at) WHERE published_at IS NULL;

-- Business EDIFACT ORDERS ingestion transaction
BEGIN;
  INSERT INTO orders (id, partner, total, status, raw_edifact)
  VALUES ('ORD-12345', 'WALMART', 1234.50, 'RECEIVED', $1);

  INSERT INTO outbox (aggregate_id, aggregate_type, event_type, payload)
  VALUES ('ORD-12345', 'Order', 'OrderReceived', $2);
COMMIT;

-- Polling drain worker (batch of 100)
BEGIN;
  SELECT id, aggregate_id, event_type, payload
  FROM outbox
  WHERE published_at IS NULL
  ORDER BY id
  LIMIT 100
  FOR UPDATE SKIP LOCKED;
  -- ... publish to Kafka ...
  UPDATE outbox SET published_at = now() WHERE id = ANY($ids);
COMMIT;

For Debezium, configure the PostgreSQL connector with outbox.table.name=outbox and use the Outbox Event Router SMT which transforms the insert into a structured key/value Kafka event automatically. Partition Kafka by aggregate_id to preserve order per order.

Anti-patterns

• Outbox in a different database than the business one — loses atomicity, back to dual-write problem.
• No purge — the outbox table grows indefinitely, performance degrades. Archive to cold storage or delete after publication confirmed.
• No index on WHERE published_at IS NULL — polling becomes a full table scan, latency explodes.
• Confusing outbox and event store — outbox is ephemeral (drained and purged), event store is permanent (see Event Sourcing).
• No Kafka partition key — order per aggregate is lost, consumer sees InvoiceAcked before InvoiceIssued.

Related patterns

• Outbox (architectural view) — the generic version.
• Transactional Inbox — the consumer-side mirror.
• Change Data Capture — alternative or complement (CDC on business tables instead of outbox).
• Idempotency Key (sender) — the relay must use an idempotent key to avoid creating duplicates downstream.

Sources

• Richardson C. — Pattern: Transactional Outbox, microservices.io. The canonical reference page. microservices.io
• Debezium Documentation — Outbox Event Router SMT. debezium.io
• Confluent — The Outbox Pattern in Practice, Gunnar Morling, 2019. The article that popularised the Debezium + Kafka implementation.
• Kleppmann M. — Designing Data-Intensive Applications, O'Reilly 2017, ch. 11 ("Stream Processing").
• Richardson C. — Microservices Patterns, Manning 2018, §3.2 ("Reliably publishing events").