Imported from Wbunker/skills-repo (streaming-systems-expert/SKILL.md). Install upstream with npx skills add Wbunker/skills-repo --skill streaming-systems-expert. Copyright stays with the author.

Streaming Systems Expert

Based on Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing by Tyler Akidau, Slava Chernyak, and Reuven Lax (O'Reilly, 2018).

Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│                    THE BEAM MODEL                                    │
│                                                                      │
│  WHAT are you computing?   → Transformations (grouping/nongrouping) │
│  WHERE in event time?      → Windowing (fixed/sliding/session)      │
│  WHEN in processing time?  → Triggers + Watermarks                  │
│  HOW do refinements relate?→ Accumulation (discard/accum/retract)   │
└──────────────────────────┬──────────────────────────────────────────┘
                           │
         ┌─────────────────▼──────────────────┐
         │         STREAMS AND TABLES          │
         │                                     │
         │  Stream ──[grouping]──► Table       │
         │  Table  ──[trigger]──►  Stream      │
         │                                     │
         │  Nongrouping ops: stream → stream   │
         │  Grouping ops:    stream → table    │
         └────────────────────────────────────┘
                           │
         ┌─────────────────▼──────────────────┐
         │             TIME DOMAINS            │
         │                                     │
         │  Event Time ────────────────────►  │
         │       │  ↑ skew/lag varies          │
         │  Processing Time ───────────────►  │
         │                                     │
         │  Watermark = oldest unprocessed     │
         │             event time (heuristic   │
         │             or perfect)             │
         └────────────────────────────────────┘

Quick Reference — Load the Right File

Reference Files

Core Decision Trees

What Windowing Strategy Do You Need?

What is the temporal structure of your computation?
│
├── No temporal grouping needed (pure element-wise transforms)
│   └── No windowing — nongrouping transformation
│       → stream-to-stream operation; no state accumulation
│
├── Align windows to fixed time boundaries (hourly reports, daily batches)
│   └── Fixed windows
│       ├── Aligned (default) — all keys share same window boundaries
│       │   → resource spikes at window boundaries; simpler reasoning
│       └── Unaligned — per-key offset boundaries
│           → smooths resource usage; trades simplicity for efficiency
│
├── Overlapping windows (rolling averages, sliding aggregates)
│   └── Sliding windows — window size + slide period
│       → each element appears in multiple windows; higher cost
│
├── Activity-based grouping (user sessions, event bursts)
│   └── Session windows — gap-based, data-driven
│       ├── Unbounded sessions (gap timeout only)
│       ├── Bounded by duration — cap session length
│       └── Bounded by element count — cap session size
│
└── Processing-time snapshots (legacy batch compatibility, no event-time skew)
    └── Processing-time windows
        ├── Via triggers — global event-time window + processing-time trigger
        └── Via ingress time — stamp events with arrival time at source
            → simpler but loses event-time ordering

When Should Results Be Materialized? (Trigger Choice)

When do you want output from a window?
│
├── As fast as possible (low latency, accept multiple updates)
│   └── Per-record trigger (repeated update)
│       → highest throughput of partial results; downstream must handle restatement
│
├── On a processing-time delay (throttle output rate)
│   └── Processing-time trigger (aligned or unaligned delay)
│       → reduces output volume; good for dashboards with refresh limits
│
├── When input is believed complete (correctness over latency)
│   └── Watermark trigger (on-time pane only)
│       → highest latency; loses data if watermark heuristic is wrong
│
└── Balance latency, completeness, and late data
    └── Compound early/on-time/late trigger
        ├── Early: per-record or per-delay (speculative results)
        ├── On-time: watermark passage (authoritative result)
        └── Late: per-record or per-delay (corrections for late arrivals)
            → pair with allowed lateness to bound late firings

Which Accumulation Mode?

How should multiple panes for the same window relate?
│
├── Each pane is independent; downstream overwrites previous
│   └── Discarding — cheapest; requires downstream overwrite semantics
│       → use when downstream stores only the latest value (key-value store)
│
├── Each pane includes all prior data (cumulative totals)
│   └── Accumulating — moderate cost; downstream must handle cumulative values
│       → use when downstream sums or aggregates across panes (e.g., counters)
│
└── Each pane corrects the previous via retraction
    └── Accumulating and Retracting — most expensive; most correct
        → required when downstream cannot distinguish cumulative from delta
           or when joins and multi-level aggregations need undo signals

Exactly-Once: Which Mechanism?

What is your exactly-once requirement?
│
├── Within the pipeline shuffle (internal correctness)
│   └── Shuffle-level deduplication
│       ├── Assign unique record IDs upstream
│       ├── Downstream catalogs seen IDs per key (with Bloom filter optimization)
│       └── GC catalog entries after watermark passes
│
├── At the source (replay/resume after failure)
│   └── Source checkpointing
│       → source API must support offset-based or token-based replay
│
└── At the sink (no duplicate external writes)
    ├── Idempotent sinks — writes are safe to repeat
    │   → files (overwrite), key-value stores (upsert), Pub/Sub (dedup by ID)
    └── Transactional sinks — atomic commit of a batch of records
        → BigQuery load jobs, database transactions
        → wrap in windowed commit; one transaction per window/bundle

Key Concepts Quick Reference