Dark Launch

The new version receives real traffic in parallel with the old, but its results stay invisible to the partner. Real-load validation without user risk — the signature pattern of gradual refactoring.

Problem

One wishes to validate a new implementation (a new EDIFACT parser, a new EN 16931 validation engine) under real conditions, without any risk that its outputs diverge from the legacy behaviour expected by partners. Synthetic tests do not always capture real cases. A Canary exposes even 1% of traffic to risk. We want the realism of production without the consequence.

Forces

• Real-traffic realism is necessary: only real messages capture full diversity.
• User-facing impact must be nil: v2 output must never reach the partner.
• v1/v2 comparison must be quantified: divergence rate, deviation types, performance.
• Resource cost doubles for the involved flows: each message is processed twice.
• Audit must distinguish prod and shadow: shadow outputs must not pollute the fiscal archives.

Solution

Duplicate incoming traffic via a splitter or tee. The v1 pipeline (legacy) processes normally and returns the response to the partner. The v2 pipeline (dark) processes the same message in parallel, without returning any response. v2 outputs are persisted in a shadow store, compared to v1 outputs, and used to build a divergence dashboard. Once confidence is acquired (e.g., < 0.01% divergence over 1 week), switch to Canary then 100%. Github Scientist variant: a library that wraps this pattern at code level, statistically measuring conformance.

Duplicated traffic, compared output:

       Incoming INVOIC
            │
            ▼
   ┌────────────────┐
   │  Tee splitter  │  (copies the message)
   └─┬──────────┬───┘
     │          │
     ▼          ▼
  v1.0 prod   v2.0 dark
  (responds)  (computes, persists, does not respond)
     │          │
     ▼          ▼
  Response →   Compare-shadow storage
  to partner   • same results?
               • acceptable performance?
               • errors?
                       │
                       ▼
               v1 vs v2 divergence dashboard
               Investigate divergences

EDI implementation

Concrete case: refactoring a custom Java EDIFACT INVOIC parser that grew through 10 years of patches, replaced by a new Smooks-based parser. Without Dark Launch: synthetic tests validated but uncertainty on exotic cases (special characters, rare encodings, partner custom segments). With Dark Launch: each incoming INVOIC is processed by both parsers. The canonical JSON output is compared field by field. Over 100k messages in 1 week, we observe: 99.98% identical, 0.02% divergence (~20 messages). Investigation: 18 due to legacy undocumented behaviour (tolerance of a non-standard delimiter) that the new parser handles better. 2 due to a Smooks bug to fix. After fix, divergence at 0%, promotion to Canary 1% → 100%. Tools: Github Scientist (Ruby, Python/Java ports available), Apache Kafka MirrorMaker for asynchronous duplication, custom tee in the internal bus.

Anti-patterns

• Side effects in dark: if v2 dark actually sends an MDN or writes to a shared DB, it is no longer dark launch, it is a hidden deployment — production incidents.
• Naive comparison: if v1 and v2 produce semantically equivalent but syntactically different results (field order, whitespace), raw comparison explodes in divergence.
• Infinite dark launch: staying in dark 6 months durably doubles costs for no incremental benefit beyond the first weeks.
• No promotion criterion: moving to Canary "when it feels good" = no better than no dark launch.
• Blocking added latency: if the splitter blocks the v1 call waiting for v2, partner latency drifts — dark must be asynchronous and failure-tolerant.

Related patterns

• Canary Release — natural next step after validated dark launch.
• Feature Flag — drives dark launch activation.
• Wire Tap — neighbouring EIP pattern to observe a channel without disturbing it.
• Detour — neighbouring EIP pattern to activate an optional step.
• Blue-Green Deployment — complementary switching pattern after dark launch.

Sources

• Github Scientist: reference open-source library for code-level dark launch. Lets a new implementation be tested in parallel with the old and measures divergence. github.com/github/scientist
• Facebook Engineering — Building and testing at Facebook. The term "dark launch" was popularised by Facebook to describe progressive deployment of a new feature without making it visible to users before validation.
• Newman S. — Building Microservices, O'Reilly 2015 (2nd ed. 2021). Chapter on release and shadow testing in distributed systems.
• Humble J., Farley D. — Continuous Delivery, Addison-Wesley 2010. Chapter 10 on advanced release techniques.
• Beyer B. et al. — Site Reliability Engineering, O'Reilly 2016. Chapter 27 (Reliable Product Launches at Scale) on progressive-launch patterns. sre.google/sre-book
• LinkedIn Engineering — Dark Launching: public use cases of the pattern in LinkedIn engineering to test new features under real load.