KAFKA-EDI

Kafka (EDI) is the Apache Kafka distributed immutable log and streaming platform — foundation of large-scale (>10K msg/s) event-driven EDI architectures.

Definition

Kafka organises data in partitioned Topics replicated across Brokers. Consumers (Consumer Groups) progress independently via persistent offsets. Ideal for EDI event sourcing: every INVOIC, MDN, status flows into a dedicated Topic.

Origin

Created at LinkedIn in 2010 by Jay Kreps and Neha Narkhede. Apache top-level project in 2012. Confluent founded in 2014 for commercial support.

Use

A large retailer ingests 50M EDIFACT/day into a Kafka Topic 'edi.invoic'. A 'erp-loader' Consumer Group loads SAP HANA, an 'fraud-detector' applies ML rules, an 'audit-archive' writes to S3. Replay up to 30 days back via offset reset.

Related terms

• Event sourcing — central pattern.
• AWS SQS — queue alternative.
• OpenTelemetry — observability.
• Outbox pattern — outbound integration.