Dead-Letter Queue: variants and runbooks

The Dead Letter Channel of EIP §119 is too monolithic for a 2026 EDI hub. Five operational variants — tiered retry, parking lot, per-cause DLQ, leak probe, replay runbook — separate what is recoverable from what is not.

Problem

A single DLQ rapidly becomes a graveyard: messages accumulate without triage, prioritisation, or context. The ops team opens the ticket "127 errors in DLQ" and must guess case by case whether it is a network outage (to retry), a broken schema (to fix), a corrupted payload (to delete), or an attack (to investigate). Without structure, mean time to resolution explodes. Rule of thumb: a single DLQ works for 100 messages/day, not 10 000.

Forces

• Error typology: transient (network, partner down), permanent (broken schema, corrupted payload), business (business validation), security (invalid signature).
• Leak probes: a sudden DLQ growth is the most reliable signal of a production problem — must trigger an alert.
• Asymmetric error costs: leaving a false positive (message wrongly rejected) costs 100x more than a false negative (wasteful retry).
• Compliance: messages in DLQ often contain business data — their retention is regulated (GDPR, fiscal archival).
• Idempotent replay: replaying a DLQ message must not create a duplicate if the partner has already retransmitted meanwhile.

Solution

Layer five variants:

• Tiered retry: intermediate queues retry-5s, retry-30s, retry-5min, retry-30min, retry-2h with exponential backoff. Failure after 5 retries → DLQ.
• Per-cause DLQ: dlq-schema-error, dlq-business-validation, dlq-partner-rejected, dlq-signature-invalid, dlq-unknown. Routing based on captured exception type.
• Parking lot queue: for messages ops decide to keep but do not yet know what to do with (waiting for business arbitration).
• Leak probe: alert if a DLQ depth grows by more than 10 messages in 5 min, or if the dlq/total rate exceeds 0.1%.
• Replay runbook: CLI/UI tool to filter, inspect, modify (with audit), retry or delete (with audit) DLQ messages.

Structure

Layered error flow:

   Producer ──► main-topic ──► consumer
                                  │
                                  ▼ fail (transient)
                              retry-5s ──► consumer (attempt 2)
                                  │
                                  ▼ fail
                              retry-30s ──► consumer (attempt 3)
                                  │
                                  ▼ fail
                              retry-5min ──► ... attempt 5
                                  │
                                  ▼ exhausted
                          ┌── Classify error ──┐
                          │                    │
                          ▼                    ▼
                   dlq-schema-error    dlq-partner-rejected
                          │                    │
                          ▼                    ▼
                    parking-lot      ops-runbook-required

Leak probe:
    Prometheus alert: dlq_depth_5min_diff > 10

EDI implementation

On Kafka, typical naming convention for an EDI hub:

# Kafka topology for a typical EDI flow
edi.ingest                  → raw ingestion
edi.parse                   → EDIFACT/X12/UBL parsing
edi.parse.retry.5s          → 5s retry after transient failure
edi.parse.retry.30s         → 30s retry
edi.parse.retry.5min        → 5min retry
edi.parse.dlq.schema        → invalid schema (action: fix mapping)
edi.parse.dlq.business      → business rule (action: review partner)
edi.parse.dlq.unknown       → unknown cause (action: investigate)
edi.parse.parking-lot       → awaiting business decision

# Enriched Kafka headers on DLQ message
x-original-topic: edi.parse
x-error-class: SchemaValidationException
x-error-msg: "Element 'UNH' missing"
x-retry-count: 5
x-first-failed-at: 2026-05-18T10:23:45Z
x-correlation-id: msg-abc-123
x-partner-id: WALMART
x-message-id: REF12345

# Spring Kafka error handler config (Java)
@Bean
public DefaultErrorHandler errorHandler(KafkaTemplate<String, Object> tpl) {
  DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(
      tpl,
      (record, ex) -> {
        if (ex instanceof SchemaValidationException) {
          return new TopicPartition("edi.parse.dlq.schema", -1);
        } else if (ex instanceof BusinessRuleException) {
          return new TopicPartition("edi.parse.dlq.business", -1);
        }
        return new TopicPartition("edi.parse.dlq.unknown", -1);
      });
  return new DefaultErrorHandler(recoverer,
      new ExponentialBackOffWithMaxRetries(5));
}

Replay runbook (minimal CLI): list by partner and cause (dlq inspect --topic dlq-schema --partner WALMART --since 2h), inspect a message (dlq show msg-abc-123), replay by injecting on main topic (dlq replay msg-abc-123 --to edi.parse), or delete with audit (dlq drop msg-abc-123 --reason "corrupted payload, partner notified ticket TKT-456"). Every action produces an immutable audit event.

Anti-patterns

• One DLQ for every hub flow — impossible to sort, runbook becomes unusable beyond 1 000 messages.
• No TTL or DLQ purge — old messages accumulate indefinitely, broker performance degrades, GDPR audit compromised.
• Auto-replay without ops intervention — silent crash re-injecting 100 000 corrupted messages in a loop.
• DLQ without correlation to original message (no correlation-id or source-topic) — context impossible to recover.
• No alert on DLQ growth — leak invisible until ops discover it browsing the console once a week.

Related patterns

• Dead Letter Channel (EIP) — the canonical version of the pattern.
• Invalid Message Channel — complementary pattern for well-formed but semantically invalid messages.
• Poison Message Handling — pattern for messages that systematically crash the consumer.
• Retry & Backoff — strategy that feeds the DLQ after exhaustion.
• Exception flow — overview of EDI error handling.

Sources

• Hohpe G., Woolf B. — Enterprise Integration Patterns: Dead Letter Channel, Addison-Wesley 2003, §119. enterpriseintegrationpatterns.com
• AWS — Amazon SQS Dead-Letter Queues. The reference doc for the multi-tier pattern (main → DLQ). docs.aws.amazon.com
• Confluent — Error Handling Patterns in Kafka Streams, 2019. confluent.io
• Spring Kafka — Dead Letter Topics and Error Handlers. docs.spring.io
• Nygard M. — Release It! Design and Deploy Production-Ready Software, 2nd ed. Pragmatic Bookshelf 2018, ch. 5 ("Stability Patterns").