Pipes and Filters

The most cited architectural-structure pattern of the catalogue. An EDI integration pipeline is not a single block: it is a chain of atomic stages (parse, validate, enrich, translate, ship) linked by channels. Each stage is isolatable, testable, replaceable.

Problem

An incoming EDI message must go through a dozen transformations before landing in the ERP: detect format, parse, validate syntax, validate business rules, resolve identifiers, translate to canonical, translate to the ERP schema, deliver. Writing this journey as a single monolithic function produces unmovable code: every change ripples, unit testing becomes impossible, you cannot change one step without risking the others. The pipeline must be decomposed.

Forces

• Separation of concerns. Each stage solves one class of problem. The parser does not validate, the validator does not translate.
• Composability. Adding a step (e.g. GDPR anonymisation) = wiring a new filter between two existing ones, not rewriting.
• Unit testability. Each filter has an in/out message contract; it is tested in isolation against golden files.
• Native parallelism. Several messages can traverse the pipeline in parallel, each filter being an independent consumer on its input channel.
• Orchestration cost. Each filter = a process, a queue, a deployment. More moving parts to monitor.

Solution

EIP §70 (Hohpe & Woolf, 2003), borrowed from UNIX architecture (Doug McIlroy, 1964, formalised in Kernighan & Plauger's Software Tools, 1976). Decompose processing into filters — atomic units consuming a message on an input channel, transforming it, depositing it on an output channel — linked by pipes (the channels). Each filter obeys a contract: same message type in and out (the canonical envelope stays stable), only the content evolves. That lets us reorder or substitute a filter without breaking the chain.

┌───────┐   ┌───────┐   ┌─────────┐   ┌─────────┐   ┌──────┐
   │ parse │──▶│ valid │──▶│ enrich  │──▶│ trans-  │──▶│ ship │
   │ AS2   │   │ ate   │   │ (GLN→   │   │ late    │   │      │
   │ EDI   │   │ schema│   │ address)│   │ canon.  │   │ to   │
   └───────┘   └───────┘   └─────────┘   └─────────┘   │ part │
        │           │            │             │       │ ner  │
        ▼           ▼            ▼             ▼       └──────┘
     pipe        pipe         pipe          pipe
     (queue)     (queue)      (queue)       (queue)

   Each filter:
     - one input pipe, one (or many) output pipes
     - same message envelope contract
     - stateless if possible (state in canonical message)

EDI implementation

Typical inbound INVOIC pipeline:

# Inbound EDIFACT INVOIC pipeline
filters:
  - id: sniff-format
    in:   edi.inbound.raw
    out:  edi.detected.edifact     # another filter for x12 / ubl
    role: detect EDIFACT vs X12 vs UBL

  - id: parse-edifact
    in:   edi.detected.edifact
    out:  edi.canonical.invoic.raw
    role: parse UN/EDIFACT INVOIC D96A → JSON canonical

  - id: schema-validate
    in:   edi.canonical.invoic.raw
    out:  edi.canonical.invoic.validated
    fail: edi.deadletter.invalid
    role: JSON-schema + business rules

  - id: enrich-gln
    in:   edi.canonical.invoic.validated
    out:  edi.canonical.invoic.enriched
    role: GLN→party master, GTIN→product master

  - id: translate-erp
    in:   edi.canonical.invoic.enriched
    out:  erp.bookkeeping.invoice.in
    role: canonical → SAP IDoc INVOIC02

Each filter has its own input/output channel, its own DLQ (edi.deadletter.*), its own observability. Each can evolve independently: add anonymize-pii between schema-validate and enrich-gln, or replace parse-edifact with a new implementation without touching the rest. The golden rule is that every filter publishes on its output a message of the same canonical type as the one it consumed, enriched or pruned; never a different type. That distinguishes pipes-and-filters from a workflow where every stage speaks its own dialect.

Parallelism & order

Parallelism is native: each filter can run in N instances (competing consumers on its input queue). But that breaks order. If order matters (e.g. ORDERS followed by ORDCHG on the same PO), partition by key (Kafka with key = controlRef) or impose a serial in the affected filter. Other filters can stay parallel.

Anti-patterns

• God-object filter. A filter that parses, validates AND translates. You're back to coupled code.
• Filters sharing a database. Coupling through data: a schema change blocks two filters. Prefer the canonical passed in the message.
• No per-stage DLQ. If a message fails in enrich-gln, it should not bounce back to parse-edifact; it goes to the stage's DLQ with context.
• Message type changing at every filter. If the output of parse-edifact is one object, validate's is another, enrich's a third, the canonical contract is lost — substitution becomes impossible.

Related patterns

• Message Channel — each pipe is a named channel.
• Message Translator — a particular filter that changes the schema.
• Content Enricher — a particular filter that adds fields.
• Splitter and Aggregator — filters with non-1:1 cardinality.

Sources

• Hohpe G., Woolf B. — Enterprise Integration Patterns, Pipes and Filters (§70). enterpriseintegrationpatterns.com — Pipes and Filters
• Kernighan B., Plauger P. — Software Tools, Addison-Wesley, 1976. The educational formalisation of the UNIX pipe concept that inspires the pattern.
• Shaw M., Garlan D. — Software Architecture: Perspectives on an Emerging Discipline, Prentice Hall, 1996. Architectural-styles chapter, including pipes-and-filters.
• Apache Camel — the whole framework is built around the pattern. Camel routes are pipes-and-filters pipelines. camel.apache.org — Pipeline