Poison Message Handling

A message that systematically crashes the consumer (NullPointerException, OutOfMemoryError, infinite parsing) will never be consumed. In a FIFO ordered broker, it blocks the entire partition behind it. The poison message pattern detects and isolates it.

Problem

A Kafka consumer that crashes on the same message every attempt produces the worst operational scenario: the entire partition stops, subsequent messages pile up, lag explodes, partner SLOs are violated. Typical causes: a malformed EDIFACT triggers a regex catastrophic backtracking (CVE), an INVOIC with 100 000 lines triggers an OOM, a payload containing invalid UTF-8 breaks the parser. The message is neither legitimate nor cleanly rejectable — it kills the consumer. Without detection, the consumer restarts, attempts the same message, recrashes, restarts, ad infinitum.

Forces

• Ordering: if the broker guarantees per-partition order (Kafka, SQS FIFO), the poison message blocks everything behind it.
• Detection: the marker of a poison message is repetition of identical exception beyond a threshold, not first occurrence.
• Error cost: a false positive (skip a legitimate message) can be severe (lost invoice). A false negative (consumer in infinite loop) blocks everything.
• Diagnosis: poison message analysis requires an isolated environment (debug sandbox, beware malicious payloads).
• Audit: every poison message skip must be traceable and notifiable (ops alert + mandatory analysis ticket).

Solution

Combine three mechanisms:

• Per-message failure counter: track (message_offset, attempts) in process memory or a dedicated table. Beyond N attempts (typically 5), classify the message as poison.
• Quarantine queue: on detection, copy the message into a dedicated queue edi.poison (secured DLQ), commit the original offset to unblock the partition, alert ops.
• Analysis sandbox: an isolated, resource-bounded (CPU/RAM capped), instrumented consumer that attempts to re-parse the poison message to produce a precise error report without risking the production runtime.

Structure

Normal consumer:

  while true {
    msg = broker.poll(timeout: 30s);
    if not msg: continue;

    attempts = poisonCounter.get(msg.id);
    if attempts >= POISON_THRESHOLD {
      // ─── Poison detection
      quarantineQueue.publish(msg, headers={
        "x-poison-cause": lastException,
        "x-attempts": attempts,
        "x-original-topic": msg.topic,
        "x-detected-at": now()
      });
      alerter.fire("PoisonMessageDetected", {msg: msg.id, partition: msg.partition});
      broker.commit(msg.offset);  // unblock partition
      poisonCounter.reset(msg.id);
      continue;
    }

    try {
      process(msg);
      broker.commit(msg.offset);
      poisonCounter.reset(msg.id);
    } catch (Exception e) {
      poisonCounter.increment(msg.id);
      log.warn("Failed attempt {} for {}: {}", attempts + 1, msg.id, e);
      // NO commit → retry next poll
      sleep(exponentialBackoff(attempts));
    }
  }

EDI implementation

For an EDI hub, the typical Kafka strategy:

-- Poison counter table (PostgreSQL)
CREATE TABLE poison_counter (
  topic            VARCHAR(80) NOT NULL,
  partition_id     INT NOT NULL,
  offset           BIGINT NOT NULL,
  consumer_group   VARCHAR(80) NOT NULL,
  attempts         INT DEFAULT 0,
  last_exception   TEXT,
  last_attempt_at  TIMESTAMPTZ,
  PRIMARY KEY (topic, partition_id, offset, consumer_group)
);

-- TTL index for purge (daily cleanup)
CREATE INDEX poison_counter_old
  ON poison_counter (last_attempt_at)
  WHERE last_attempt_at < now() - INTERVAL '7 days';

-- Analysis sandbox (isolated consumer on quarantine queue)
@KafkaListener(
  topics = "edi.poison",
  containerFactory = "sandboxFactory"  -- 256MB heap, 30s CPU cap
)
public void analyzePoison(ConsumerRecord<String, byte[]> record) {
  try {
    Object parsed = parseAttempt(record.value());
    // If we got here, it's a real schema/version issue
    reportToOps(record, parsed, null);
  } catch (Throwable t) {
    // Capture full stack trace, dumps, etc.
    forensicReport(record, t);
    // Permanent quarantine with detailed report
    archiveBucket.upload("forensic/" + record.offset(), record.value(), t);
  }
}

EDIFACT-specific case: malformed messages may contain control characters (chars 0x00-0x1F outside UNA segment terminators) that crash some legacy Java parsers. Always pre-validate with a lightweight guard (UNB present, length < 10MB, valid charset) before invoking the heavy EDIFACT parser. This guard avoids 95% of partner-origin poison messages.

Anti-patterns

• Silent auto-skip after N retries — produces silent losses, audit explodes.
• No persisted counter — consumer restart resets counter and infinite loop resumes.
• Analyse poison message in the same process as main consumer — risk of re-killing the runtime.
• Confusing poison message and DLQ message — DLQ is a known catalogued failure; poison is an unexplained crash.
• No rate-limit on alerts — a partner sends 100 poison messages per minute, ops receive 100 SMS.

Related patterns

• Dead-Letter Queue: variants — parent pattern of which poison message is a variant.
• Dead Letter Channel — the canonical EIP version.
• Circuit Breaker — useful to protect the analysis sandbox.
• Bulkhead — isolate the sandbox from other hub consumers.
• Retry & Backoff — supplies the counter used by poison detection.

Sources

• Nygard M. — Release It! Design and Deploy Production-Ready Software, 2nd ed. Pragmatic Bookshelf 2018, ch. 4 ("Stability Antipatterns" — Cascading Failures).
• Microsoft — Handling Poison Messages, Azure Service Bus documentation. learn.microsoft.com
• Apache Kafka — Kafka Streams Error Handling Documentation. kafka.apache.org
• AWS — Amazon SQS Best Practices: Handling Poison Pills. docs.aws.amazon.com
• OWASP — ReDoS: Regular Expression Denial of Service. Frequent cause of poison messages via partner payload. owasp.org