Idempotent Sender

The sender that knows its retries will never create duplicates — the indispensable complement of idempotency receiver.

Problem

An EDI hub sends an INVOIC to Walmart via AS2. The send leaves, the connection times out after 60s. Did Walmart receive it? The hub retries. Walmart returns MDN for both sends: Walmart's ERP has two invoices, fires two identical payments. Without an idempotency receiver on Walmart's side, a hub timeout pays twice. But: even if Walmart makes the effort to dedupe, the hub still needs to send the same idempotency key twice. A hub that regenerates a random MessageId on each retry produces two "different" messages for Walmart.

Forces

• At-least-once is the realistic semantics. Every distributed network retries under doubt.
• The idempotency key must be stable across retries. If it changes, the receiver cannot dedupe.
• The key must be persisted before sending. If regenerated in memory at each attempt, a crash between two attempts destroys consistency.
• Retry can be triggered by another node. In cluster, node A sends, crashes, node B retries — must recover the same key.

Solution

Before any send attempt: (1) Generate a stable deterministic idempotency key from the domain: send-{invoiceNumber}-to-{partnerId}, or a ULID persisted with the invoice in DB. (2) Persist key + payload hash + status pending in a sender_log table in the same transaction as the business creation. (3) Send the message carrying the key in a stable header: AS2 Message-ID, EDIFACT RFF, HTTP Idempotency-Key header (IETF draft) or X12 BIG02. (4) On retry, read sender_log and reuse the same key, same hash, no regeneration. (5) The receiver, with its idempotency receiver, dedupes on the received key.

Idempotent Sender:

  1. Generate key    "send-INV-2026-0001234-attempt-1"
     (stable, deterministic, never regenerated on retry)
            │
            ▼
  2. Persist key + payload in DB     transaction T1
            │                          sender_log {
            │                            key, payload_hash,
            │                            status='pending',
            │                            attempt_count, created_at
            │                          }
            ▼
  3. Send AS2 / AS4 / REST API       message carries the key
                                     in Message-ID / RFF
            │
            ▼  ack received → status='delivered'
            ▼  timeout / 5xx → status='pending', attempt++
            ▼
  4. Retry → lookup key in DB
            if status='delivered' → do not resend (idempotent)
            if status='pending'   → resend same message, same key

EDI implementation

Concrete case: EDI hub generates an EDIFACT INVOIC D.96A for Walmart. The Outbox pattern stores in DB: {invoice_id, message_id: "EDI-INV-2026-0001234-v1", payload_hash: "sha256:abc...", status: "pending", attempt: 0}. The sender reads the Outbox queue, sends via AS2 with Message-Id: EDI-INV-2026-0001234-v1@hub.local. If MDN received: status='delivered'. If timeout: status stays pending, attempt++. The retry reuses exactly the same Message-Id. Walmart, which dedupes on Message-Id (see RFC 4130 §10), recognises the duplicate and only sends one INVOIC to its ERP. REST variant: HTTP header Idempotency-Key: EDI-INV-2026-0001234-v1 (see Stripe, Square IETF draft). On retry, the server returns the cached response. For EDIFACT: use BGM C002 1004 (Document Number) + UNB 0020 (Interchange Control Reference) — both must be retry-stable. AS4 / ebMS3 use eb:MessageId + eb:RefToMessageId for duplicate elimination.

Anti-patterns

• Random idempotency key. UUID regenerated on each attempt: two different UUIDs = two messages for the receiver.
• Key only in memory. If the service restarts, the key is lost, retry produces a duplicate.
• No payload hash. One might want to test that retry sends the same payload — if changed in between (bug), detect and alert.
• Key not transported in the message. If the receiver does not see the key, it cannot dedupe.
• Sender ignoring ACKs. Not reading MDN/CONTRL/200OK retries forever. Always couple acknowledgements and idempotent sender.

Related patterns

• Idempotency Key (sender) — close pattern, sometimes confused.
• Idempotency (receiver) — complementary pattern on the receiver side.
• Outbox — typically used to store the key atomically with the business.
• Retry + backoff — the sender retries reusing the key.
• At-Least-Once — the semantics that idempotent sender makes tolerable.
• Correlation Identifier — often the same key serves both.

Sources

• Kleppmann M. — Designing Data-Intensive Applications, O'Reilly 2017. Chap. 11 "Stream Processing" §"Producer Idempotency". oreilly.com — DDIA
• RFC 4130 §10 — AS2 Duplicate elimination. AS2 Message-ID handling for deduplication. rfc-editor.org/rfc/rfc4130
• IETF draft — The Idempotency-Key HTTP Header Field. Emerging standard for REST APIs. datatracker.ietf.org — Idempotency-Key
• Stripe — Idempotent Requests. The reference implementation in commercial REST API. stripe.com — Idempotent Requests
• OASIS ebMS 3.0 Core Specification — Duplicate detection. AS4/ebMS3 duplicate elimination via eb:MessageId. oasis-open.org — ebMS3
• Apache Kafka — Idempotent producer. enable.idempotence=true, PID + sequence number per partition. kafka.apache.org — Idempotent producer