Idempotent Receiver for EDI flows

An EDIFACT INVOIC received twice must produce one accounting invoice. Deduplication relies on the regulatory fields of each standard — not an application UUID — and takes contractual partner retransmission windows into account.

Problem

All EDI protocols explicitly expect retransmission: AS2 sends an MDN, otherwise the partner retransmits; AS4 has NRR + Retry-After; OFTP2 has EERP + retry timer; SFTP has its own application-level retry logic. The AS2 golden rule (RFC 4130 §10) even mandates that on missing MDN, the sender MUST retransmit. Consequence: every EDI receiver must treat each delivered message as potentially duplicated. The question is not "will this happen?" but "what happens when it does?". Not deduplicating = duplicate invoices, duplicate orders, lost stock, negative audit.

Forces

• Explicit standards: each EDI standard has a contractual uniqueness field (UNB 0020, ISA13, BT-1 in UBL, AS4 MessageId).
• Dedup window: must be at least as long as the maximum partner retry window (typically 7 days for AS2, 30 days for PEPPOL).
• Reference reuse: EDIFACT and X12 standards allow reusing UNB/ISA13 numbers after a period — dedup must be bounded by (sender, ref, date), not just ref.
• Auditability: each dedup trace must be stored to prove to the partner why their retry did not produce an invoice.
• Performance: one lookup per ingested message — must be O(log n) with proper DB index.

Solution

Identify the standard-specific uniqueness key. Persist this key before any business side effect (creating an order, invoice, internal ASN). If the key already exists in the dedup window, return a positive functional ACK (the partner thinks it arrived, which is true — it arrived on first transmission) without replaying the side effect. Keep the dedup history for audit.

Structure

Uniqueness keys per standard:

EDIFACT  → UNB segment, element 0020 (Interchange Control Reference)
           composed with 0004 (Sender), 0010 (Recipient) and S004 (date/time)
           e.g. "UNB+UNOC:3+SENDER:ZZ+RECVR:ZZ+260518:1023+REF12345'"

X12      → ISA segment, element 13 (Interchange Control Number)
           composed with ISA06 (Sender ID) and ISA08 (Receiver ID)
           e.g. "ISA*00*          *00*          *01*SENDER         *01*RECVR
                *260518*1023*U*00401*000012345*0*P*>~"

UBL      → cbc:UUID (BT-46) or cbc:ID (BT-1) per profile
           e.g. <cbc:UUID>3aa5d6dc-65d3-4f01-8a47-2e7c12a44b8b</cbc:UUID>

AS4      → eb:MessageId from WS-Addressing header
           e.g. <eb:MessageId>uuid-12345@sender.example.com</eb:MessageId>

PEPPOL   → SBDH/DocumentIdentification/InstanceIdentifier
           e.g. <InstanceIdentifier>peppol-1234-5678</InstanceIdentifier>

EDI implementation

Multi-standard dedup table schema, indexed by (partner_id, standard, interchange_ref, date):

CREATE TABLE edi_dedup (
  partner_id        VARCHAR(40) NOT NULL,
  standard          VARCHAR(20) NOT NULL,    -- 'EDIFACT' | 'X12' | 'UBL' | 'AS4'
  interchange_ref   VARCHAR(80) NOT NULL,    -- UNB 0020, ISA13, etc.
  message_type      VARCHAR(20),             -- 'INVOIC', '810', 'Invoice'
  payload_sha256    CHAR(64),                -- drift detection
  received_at       TIMESTAMPTZ DEFAULT now(),
  first_seen_at     TIMESTAMPTZ NOT NULL,
  retry_count       INT DEFAULT 0,
  PRIMARY KEY (partner_id, standard, interchange_ref, message_type)
);

-- Lookup before ingestion
WITH new_msg AS (
  INSERT INTO edi_dedup (partner_id, standard, interchange_ref, message_type, payload_sha256, first_seen_at)
  VALUES ('WALMART', 'EDIFACT', 'REF12345', 'INVOIC', $1, now())
  ON CONFLICT (partner_id, standard, interchange_ref, message_type)
  DO UPDATE SET retry_count = edi_dedup.retry_count + 1
  RETURNING (xmax = 0) AS is_new, payload_sha256
)
SELECT is_new, payload_sha256 FROM new_msg;

-- If is_new = true → process (create order, etc.)
-- If is_new = false → positive functional ACK without side effect
--   AND verify payload_sha256 vs new hash: if different, alert
--   (partner reuses ref with changed content = serious bug)

Purge window: for typical AS2/AS4 B2B, retaining 90 days is plenty. For PEPPOL, where the SBDH InstanceIdentifier must be unique forever, archive after 1 year into a frozen Elasticsearch index and keep the trace for 10-year fiscal audit.

Anti-patterns

• Using an application UUID at ingestion instead of the contractual reference — loses partner semantics, retry creates duplicates.
• Deduplicating only on interchange_ref without partner_id — Walmart and Carrefour may reuse the same ref.
• Not verifying payload hash — a partner re-submitting REF12345 with new content is silently ignored (silent corruption).
• Dedup window too short — a J+8 partner retry creates a duplicate because dedup has already purged J+7.
• Confusing business idempotency and broker idempotency (see Transactional Inbox) — both are needed, complementary.

Related patterns

• Idempotent Receiver (EIP view) — the generic Hohpe/Woolf version.
• Idempotency and deduplication — topic overview.
• Idempotency Key (sender) — sender-side counterpart.
• Message Deduplication Ledger — cross-flow generalisation.
• Acknowledgements — functional ACK must be positive even on detected duplicate.

Sources

• Hohpe G., Woolf B. — Enterprise Integration Patterns: Idempotent Receiver, Addison-Wesley 2003, §528. enterpriseintegrationpatterns.com
• RFC 4130 — MIME-Based Secure Peer-to-Peer Business Data Interchange Using HTTP, Applicability Statement 2 (AS2), §10 (Retransmission). datatracker.ietf.org/doc/rfc4130
• UN/EDIFACT Syntax Rules Version 4 (ISO 9735) — Annex C on UNB 0020 Interchange Control Reference semantics. iso.org
• X12 Foundations — TR3 Interchange Control Number Usage Guide. x12.org
• OpenPEPPOL — SBDH Implementation Guide, section 5.2 (InstanceIdentifier uniqueness rules).