Sliding Window

How to obtain a continuously refreshed view of a metric over a recent time window, without waiting for a complete bucket to close.

Problem

Tumbling Window answers "how many in this hour?" with a single emission at the end of the hour. But for operational cases ("what is the error rate over the past 5 minutes?", "what is the average AS4 latency over the past 60 seconds?"), we need a continuously refreshed answer, not waiting 5 minutes for the next point. This requires a window whose end always advances but which always covers the same duration — a sliding window.

Forces

• Storage cost: each event belongs to (size/step) windows simultaneously; proportional state multiplication.
• Emission cost: one emission per step (every 10 seconds, for example) instead of one per size.
• Visual granularity: the smaller the step, the smoother the curve, but the more the cost explodes.
• Late data: same problem as tumbling, but worsened by the multiplication of open windows.
• Approximation: for very large windows, sketches (HyperLogLog, Count-Min) are often used instead of an exact state.

Solution

Define a size W (duration covered by the window) and a step S (interval between two emissions). At time t, the active window covers [t-W, t]. If S < W (standard case), each event belongs to W/S windows. At each step S, the aggregate of events in the current window is emitted. Flink implementation: stream.window(SlidingEventTimeWindows.of(Time.minutes(5), Time.seconds(30))) computes a 5-minute window emitted every 30 seconds. For very large windows (hour, day) on very high-throughput streams, prefer an incremental sketch (HyperLogLog for distinct count, t-digest for percentiles) that maintains bounded state.

Structure

Time:     T-W ─────────────── T ─── T+S ─── T+2S ───►
              ◄─── Window 1 ──►
                    ◄─── Window 2 ──► (slid by S)
                          ◄─── Window 3 ──►

Event arriving at time t belongs to all windows W_i such that
  W_i.start ≤ t < W_i.end

Configuration "size 5min, slide 30s":
  Event at 10:00:15 belongs to windows ending at:
    10:00:30, 10:01:00, 10:01:30, ..., 10:05:00
  → 10 windows simultaneously

Emission cadence:
  Every 30 seconds, system emits aggregate over last 5 minutes.

EDI implementation

Three canonical EDI cases. (1) Real-time SLA: AS4 success rate over the past 5 minutes emitted every 30 seconds — feeds an SRE dashboard, triggers an alert if < 99.5 %. (2) Volume anomaly detection: if the number of invoices emitted over the past 10 minutes exceeds the historical mean for this time-of-day band by 3σ, alert (may signal a partner recovering from outage and replaying its backlog). (3) Operational trending: top-5 most frequent SBDH error codes over the past 15 minutes — emitted every minute to identify a degradation quickly. Spark implementation: df.groupBy(window(col("eventTime"), "5 minutes", "30 seconds"), col("partner")).count(). Mind allowed lateness and state TTL; without them, memory consumption explodes within a few hours.

Anti-patterns

• Step too fine (1 second on 1h window) — 3 600 simultaneous windows per key, unmanageable state.
• Sliding on very-high-cardinality data (by message ID) — explodes state immediately.
• No TTL state — RocksDB grows unbounded until OOM.
• Emitting on every event instead of every step — operator emits too many downstream events.
• Confusing sliding and hopping windows — Beam and Flink terminology diverges slightly; read the docs.

Related patterns

• Tumbling Window — the non-overlapping alternative.
• Session Window — activity-based window.
• Watermarks — close sliding windows.
• Circuit Breaker — often driven by a sliding-window metric.

Sources

• Akidau T. et al. — The Dataflow Model, VLDB 2015. research.google
• Apache Flink — Windows: Sliding Windows. nightlies.apache.org/flink
• Spark Structured Streaming — Window Operations. spark.apache.org
• Akidau T., Chernyak S., Lax R. — Streaming Systems, O'Reilly 2018, ch. 4.
• Apache Beam Programming Guide — Sliding time windows. beam.apache.org