Data Engineering — Interactive Lab
A hands-on tour of the machinery underneath data systems — the hash functions, sketches, storage engines, joins, logs, and streams that move and shape data at scale. Three layers: Concept cards for the vocabulary, Math focus dives that derive the quantitative core step by step, and How it works dives that step you through system internals one operation at a time. Click, drag, and run each idea to see it work. A sibling to the ML, Deep-Learning, and Time-Series labs. Press Esc anytime for this menu.
Hash tables — buckets, collisions & resizing
A hash table turns a key into an array index in O(1): run the key through a hash function, take the result mod the number of buckets, and store it there. Two keys can land in the same bucket — a collision — so each bucket holds a small chain. As the table fills, the load factor α = n/B climbs and chains lengthen; cross a threshold and the table doubles and rehashes to keep lookups fast. This one structure underlies hash joins, group-bys, dedup, partitioning, and the sketches in the other dives.
The Bloom filter — probabilistic set membership
A Bloom filter answers "have I seen this before?" using a tiny bit array and k hash functions. To add an item, hash it k ways and set those k bits to 1. To test membership, hash again and check those bits: if any is 0 the item is definitely absent; if all are 1 it is probably present — but maybe a false positive, because other items may have set those same bits. No false negatives, ever. Used everywhere a cheap pre-check saves an expensive lookup: LSM-tree reads, join pruning, dedup, cache filtering.
The LSM-tree — memtable, flush & compaction
Disks hate random writes, so the log-structured merge-tree turns every write into an append. New writes land in a small sorted memtable in RAM; when it fills, it is flushed whole to an immutable sorted file (an SSTable) on disk. Files pile up, so a background compaction merge-sorts them into bigger sorted runs, dropping overwritten and deleted keys along the way. Reads check newest-to-oldest, using a per-file Bloom filter to skip files that can’t contain the key. Step the writes and watch the structure breathe.
The hash join — build then probe
Joining two tables by comparing every pair is O(|R|·|S|) — hopeless at scale. The hash join is O(|R|+|S|): in the build phase it scans the smaller table once and drops every row into a hash table keyed on the join column; in the probe phase it scans the larger table once, hashes each row’s key, and looks straight into the matching bucket to emit joined pairs. Two linear passes replace the quadratic blow-up. Step the build, then the probe, and watch joined rows fall out.
Row vs columnar storage
One logical table, two physical layouts on disk — and the query you run decides which is dramatically faster.
Same table, two physical orders
A table is logically a grid, but disk is one-dimensional. Row-major stores each row’s fields together; column-major stores each column together. The query you run decides which wins.
Storage layout
Query
Cells read
Partitioning & skew
Splitting data across machines only scales if the split is even — watch a single hot key melt one partition.
Spreading data across machines
To scale past one node, data is split into partitions. A good split keeps every partition the same size; a bad one creates a hotspot that bottlenecks the whole job.
Scheme
Partitions: 4
Hot-key skew: 0%
Balance
Quorum consistency: R + W > N
Replicate to N nodes, write to W, read from R. The overlap rule decides whether a read can ever go stale.
Reading your own writes
With data replicated to N nodes, a write waits for W acks and a read polls R nodes. If R + W > N the read set must overlap the write set, so a read always sees the newest value.
Replicas N: 5
Write quorum W: 3
Read quorum R: 3
Guarantee
Windowing a stream
You can’t aggregate infinity, so you bucket events by time — three ways, each with different overlap behaviour.
Cutting an endless stream into chunks
You can’t aggregate an infinite stream, so you bucket events by time. Tumbling windows are fixed and non-overlapping; sliding windows overlap; session windows grow until a gap of inactivity closes them.
Window type
Size: 4
Windows
Watermarks & late data
Out-of-order events need a rule for "done." The watermark advances, windows fire, and stragglers arrive late.
When is a window done?
Events arrive out of order, so the system tracks a watermark = newest event-time seen − a delay. A window fires once the watermark passes its end. Anything arriving with an event-time already behind the watermark is late.
Stream
Watermark delay: 2
State
HyperLogLog — cardinality from leading zeros
Counting distinct items exactly means remembering every item — impossible at scale. HyperLogLog cheats with a coin-flip insight: if the longest run of leading zeros you have ever seen in a hashed value is ρ, you have probably seen about 2^ρ distinct items. It splits the hash — the first bits pick one of m registers, the rest supply ρ — and each register keeps the largest ρ it has seen. Combining the registers with a bias-corrected harmonic mean turns those maxima into a count, using only a few bits per register no matter how many items stream by.
Consistent hashing — the ring
Plain hashing (server = hash(key) mod N) is a disaster when N changes: add one machine and almost every key moves. Consistent hashing puts both keys and servers on a circular hash ring; a key belongs to the first server clockwise from it. Now adding or removing a server only relocates the keys in one arc — about 1/N of them — instead of all of them. To stop any one server owning a huge arc by chance, each server is hashed to many virtual nodes scattered around the ring, which evens out the load.
Count-Min sketch — frequency estimation
How often has each item appeared, when there are too many items to keep a counter for each? The Count-Min sketch keeps a small d × w grid of counters and d hash functions. To record an item, increment one counter per row at the column its hash picks. To estimate its count, read those same d counters and take the minimum — collisions can only ever push a counter up, so the smallest of the d readings is the closest to the truth. It never undercounts, and the overcount shrinks as the grid widens.
Reservoir sampling — k fair picks from a stream
You want k random items from a stream, but you don’t know how long it is and can’t hold it in memory. Reservoir sampling keeps exactly k slots: the first k items fill them, then for the i-th item you flip a biased coin — with probability k/i it replaces a random slot, otherwise it’s discarded. That single rule keeps the invariant that, at every moment, the k slots are a uniform random sample of everything seen so far. One pass, constant memory, provably fair.
MapReduce — map, shuffle, reduce
Almost every big-data batch job is the same three moves. Map runs a function over each input record independently and emits key-value pairs — here, splitting lines into (word, 1). Shuffle moves those pairs across the network so that every pair with the same key lands on the same reducer, sorted by key. Reduce then folds each key’s values into a result — summing the 1s into a count. Map is embarrassingly parallel, reduce is parallel across keys, and the shuffle in the middle is the expensive part that moves data between machines.
The Kafka log — partitions, offsets & lag
A Kafka topic is just an append-only log, split into partitions for parallelism. Producers append records to the end of a partition; each record gets a monotonically increasing offset. Consumers don’t delete anything — they simply remember an offset per partition and read forward, so many independent consumers can replay the same log at their own pace. Within a consumer group, each partition is owned by exactly one consumer, and adding consumers triggers a rebalance. The gap between the newest offset and a consumer’s position is its lag.
Write-ahead log — durability & crash recovery
A database can’t safely change its data files in place — a crash mid-write would corrupt them. The fix is the write-ahead log: before touching any data page, append a record describing the change to a sequential log and flush it to disk. Only then is the change applied to the (in-memory) pages. The rule is "log first, apply second." If the machine crashes, the in-memory pages vanish — but every committed change is safe in the log, so recovery simply replays the log from the start and rebuilds the exact state. Crash it and recover it and watch the data come back.
The B-tree — the index behind every database
A B-tree keeps sorted keys in wide, shallow nodes so that finding any key takes only a few steps — the structure behind almost every database index and file system. Each node holds several keys and sits between its children’s ranges; a search walks down comparing against a handful of keys per level. Inserts go into a leaf, and when a node overflows its capacity it splits: the median key rises into the parent and the node becomes two. Splits ripple upward, and when the root splits the whole tree gains a level — which is the only way a B-tree ever grows taller.
ETL vs ELT
Same three letters, opposite order — and the order decides where your transforms run and what they cost.
Where does the transform run?
ETL transforms data in a separate engine before loading the clean result into the warehouse. ELT loads raw data first and transforms it inside the warehouse using its compute. Cheap warehouse compute made ELT the modern default.
Pipeline
Trade-off
Change data capture (CDC)
Read the database’s own commit log and turn every row change into an ordered event — sync without polling.
Turning a table into a stream
Instead of repeatedly querying a database for what changed, change data capture reads the database’s own commit log and emits one event per row change — insert, update, or delete — in order. Downstream systems stay in sync with near-zero load on the source.
Make a change
Stream
Idempotency & retries
Networks retry, so messages arrive twice. The fix isn’t fewer retries — it’s making the sink not care.
Why pipelines double-count
Networks retry. A message sent "at least once" can arrive twice. A non-idempotent sink counts both; an idempotent sink keys each event by a unique id and ignores repeats — so retries become harmless and the result is correct.
Sink
Result
Batch vs streaming
The size of the chunk you process at once is a dial between fast reactions and efficient bulk work.
Latency vs throughput
Batch waits to accumulate many records, then processes them together — high throughput, high latency. Streaming processes each event as it arrives — low latency, more per-event overhead. Most modern stacks blend both (micro-batching).
Mode
Batch size: 1
Result
Schema evolution
Data outlives the code that wrote it, so the shape has to change without breaking everyone downstream.
Changing the shape over time
Data outlives code, so schemas must evolve without breaking old readers or writers. Adding an optional field is safe both ways; removing or renaming can break readers that still expect the old shape. Formats like Avro/Protobuf encode rules for this.
Apply a change
Compatibility
Compression & encoding
A column is one type, highly repetitive — which is exactly what run-length and dictionary encoding feast on.
Columns compress beautifully
A column holds one type with lots of repetition, so it shrinks far better than a row. Run-length encoding stores (value × count) for repeated runs; dictionary encoding replaces values with tiny integer codes into a lookup table.
Encoding
Size
File layout & pushdown
Per-block min/max stats let a query skip most of a file before reading a single value of it.
Read less by knowing more
Columnar files keep per-block min/max statistics. A query’s filter lets the engine prune whole blocks that can’t match (predicate pushdown) and read only the needed columns (projection pushdown) — often skipping 90%+ of the file.
Query: value > 50
Columns read
I/O saved
Vectorized execution
Pushing a batch of values through each operator at once amortizes overhead and unlocks the CPU.
Process many, not one
Row-at-a-time execution pays interpreter overhead per tuple. Vectorized engines push a batch of values through each operator at once, amortizing overhead and letting the CPU use tight loops and SIMD — often 10×+ faster for analytics.
Batch size: 1
Throughput
The CAP theorem
When the network splits, you get consistency or availability — not both. The rest of the time, CAP is free.
Partition forces a choice
When the network splits two replicas (a partition), you can stay consistent (refuse writes on the cut-off side) or stay available (accept writes and reconcile later) — not both. With no partition you keep both; CAP only bites during the split.
Network
Priority
Behaviour
Replication & lag
Followers scale out reads but trail the leader — so a follower read can hand you slightly stale data.
Followers trail the leader
Writes go to the leader and replicate to followers asynchronously. Reads can be served by followers to scale out — but a follower may be behind by some replication lag, so it can return slightly stale data.
Actions
Lag
Event vs processing time
When it happened and when you saw it are different clocks — and the gap is where streaming bugs live.
Two clocks
Event time is when something happened; processing time is when your system saw it. Network delays and retries mean events arrive late and out of order, so correct analytics must reason in event time, not arrival order.
Network delay: 3
Skew
Delivery semantics & exactly-once
Failures force a choice between losing messages, duplicating them, or doing the work to get it exactly right.
Three delivery guarantees
If a step can fail and retry, you choose: at-most-once (never retry — may lose), at-least-once (retry — may duplicate), or exactly-once (retry + dedup by id + atomic commit — correct).
Semantics
Outcome
Backpressure
If arrivals outpace service, a queue grows without bound — unless the system pushes back. Watch λ vs μ.
When producers outrun consumers
If arrival rate λ exceeds service rate μ, an unbounded queue grows forever. A bounded queue forces backpressure — the system slows the producer (or drops) to stay stable. Little’s law: queue length L = λ × wait W.
Producer λ: 8/s
Consumer μ: 10/s
Queue
OLTP vs OLAP
Transactions and analytics want opposite things from storage — which is why one database rarely does both well.
Opposite access patterns
OLTP (transactions) touches a few rows across all columns — point reads and writes. OLAP (analytics) scans a few columns across millions of rows — aggregates. They want opposite storage: row-stores vs column-stores.
Workload
Best fit
The star schema
A fact table of measurements surrounded by dimension tables of context — join them to reassemble a readable row.
Facts in the middle, dimensions around
A fact table holds the measurable events (a sale, with foreign keys and numbers); dimension tables describe the context (which product, store, date). Joining the fact to its dimensions reassembles a full, human-readable row.
Join dimensions
Result row
Normalization vs denormalization
Store each fact once and join, or pre-join into wide rows — trading write-safety for read-speed.
Duplicate data, or join it?
Normalized keeps each fact once in its own table — clean writes, but reads must join. Denormalized pre-joins into wide rows — fast reads with no joins, but the same value is copied everywhere, so updates risk anomalies.
Model
Trade-off
Slowly changing dimensions
When a dimension attribute changes, do you overwrite the past or keep every version? That choice is SCD.
Keeping history when attributes change
A customer moves city. Type 1 overwrites the old value — simple, but history is lost. Type 2 closes the old row and inserts a new versioned row with effective dates, preserving the full timeline for point-in-time analysis.
Strategy
Dimension table
Data contracts & validation
A producer promises a shape; validating it on ingest stops bad rows before they corrupt everything downstream.
Reject bad data at the door
A data contract declares the shape a producer promises: required fields, types, value ranges. Validating on ingest catches violations early and quarantines them, instead of letting bad rows silently corrupt everything downstream.
Contract
Result
Deduplication
Retries make the same record arrive twice; a windowed dedup keeps the first and drops the echoes.
Same event, more than once
Retries and replays mean the same record arrives multiple times. A dedup operator keeps the first occurrence of each key seen within a time window and drops the rest — so downstream counts and joins stay correct.
Dedup
Window: 5
In / out
Data lineage
The dependency graph of your datasets answers two questions: where did this come from, and what breaks if it fails?
What feeds what
Lineage is the dependency graph of datasets — which tables a model is built from, and which dashboards depend on it. It answers two questions: where did this number come from (upstream), and what breaks if this job fails (downstream impact).
Select a dataset
Impact
Consensus & quorum
A write is only committed once a majority of replicas acknowledge it. That majority rule is what lets the system survive a minority of failures without splitting in two.
Majority must agree
With N replicas, a quorum is ⌊N/2⌋+1. The leader replicates an entry and waits for a quorum of acks before committing. A minority that loses contact (a partition) can’t reach quorum, so it can’t commit — preventing split-brain.
Cluster state
Can it commit?
Two-phase commit vs saga
Atomic distributed writes have two styles: 2PC locks everyone until all agree (and blocks on coordinator failure); sagas commit locally and undo with compensating steps.
Lock-and-agree vs commit-and-compensate
2PC: a coordinator asks all participants to prepare, then commit only if all vote yes — strong atomicity, but participants hold locks and block if the coordinator dies. Saga: each step commits independently; on failure, run compensating actions to unwind.
Protocol
On failure
Vector clocks & causality
Without a shared clock, a vector clock per node tracks what each has seen — letting the system tell whether one event caused another or they happened concurrently.
Order without a global clock
Each node keeps a counter vector. A local event bumps its own entry; a message carries the sender’s vector and the receiver takes the element-wise max. If neither vector ≤ the other, the events are concurrent (a conflict).
Step
Relationship
Join strategies
The optimizer picks how to join from the inputs’ sizes and sort order: broadcast a tiny side, hash-shuffle two big ones, or merge two already-sorted streams.
Three ways to join
Broadcast: ship a small table to every node — no shuffle of the big side. Shuffle (hash) join: repartition both sides by key, then hash-join locally. Sort-merge: sort both by key and merge — ideal when inputs are already sorted.
Strategy
Chosen when
Cost-based optimization
A cost-based optimizer uses table statistics to pick the cheapest plan — above all, a join order that keeps intermediate results small.
Order matters
Joining the same three tables in different orders produces wildly different intermediate sizes. Using row-count and selectivity stats, the optimizer estimates cost and chooses the order that minimizes rows flowing between operators.
Plan
Estimated cost
Shuffle & spilling
Wide operations exchange data across the network by key; if a partition is too big for memory, the engine spills it to disk to avoid running out of RAM.
The expensive part
groupBy, join, and sort are wide: rows must be repartitioned by key across the cluster (a shuffle). When a partition exceeds executor memory, it spills to local disk — correct, but slow. Skew makes one partition spill while others don’t.
Memory pressure
Effect
B-tree vs LSM: amplification
B-trees update pages in place — great reads, costly random writes. LSM-trees append sequentially then compact — great writes, but reads check multiple levels and compaction rewrites data.
Pick your amplification
Every storage engine trades three costs. B-tree: low read amp, higher write amp (in-place updates, page splits). LSM: low write amp (sequential appends), higher read amp (merge levels, bloom filters) and space amp (old data until compaction).
Engine
Profile
Primary vs secondary index
A primary (clustered) index stores rows in key order, so the index is the table. A secondary index is a side structure mapping other columns to row locations.
Where the rows live
With a primary index the data is physically sorted by the key — one lookup lands on the row. A secondary index on another column points back to the primary key or row id, so a lookup is index → pointer → row (an extra hop).
Index
Lookup path
Buffer pool / page cache
Databases keep hot pages in an in-memory buffer pool. A hit is fast; a miss reads from disk and evicts a cold page (usually LRU) to make room.
Memory in front of disk
Reads and writes go through fixed-size pages cached in RAM. The hit rate dominates performance. On a miss the engine fetches from disk and evicts the least-recently-used page; dirty pages are flushed (often via the WAL) before eviction.
Access a page
Cache
Stateful stream processing
Aggregations and joins on a stream keep per-key state in a checkpointed store. Watermarks let old state be evicted so it doesn’t grow forever.
Memory that survives restarts
A running count or windowed join must remember prior data — that’s state, kept per key in a state store and checkpointed for fault tolerance. Without bounds it grows unboundedly; a watermark lets the engine drop state for windows that can no longer receive data.
Operation
State store
Surrogate vs natural key
A natural key carries business meaning but can change or be reused; a surrogate key is a stable, meaningless ID that keeps references valid forever.
Stability vs meaning
A natural key (email, SKU, SSN) is intuitive but mutable — when it changes, every foreign-key reference must change too. A surrogate key (an auto ID) never changes, so facts and history stay correctly linked; the natural key becomes a regular attribute.
Primary key
Event
References
Orchestration DAG
A scheduler runs tasks as a dependency graph: independent tasks run in parallel waves, a task starts only after its parents finish, and failures retry then block what’s downstream.
Dependencies decide order
Model the pipeline as a DAG. The scheduler computes a topological order and runs each wave of ready tasks in parallel. On failure it retries with backoff; if a task ultimately fails, its descendants are skipped and an alert fires.
View
Behavior
Backfill & reprocessing
When logic changes or data arrives late, you re-run historical partitions — idempotently overwriting each day so the fix applies to the past, not just new data.
Fixing the past
A normal run processes today. A backfill re-executes a range of historical partitions (e.g., after a bug fix or a new column). Because each partition is overwritten idempotently, reprocessing is safe to repeat and parallelize across days.
Run
Partitions recomputed
Dead-letter queue
One malformed record shouldn’t crash the whole pipeline. A dead-letter queue diverts records that fail parsing or validation so good data keeps flowing and bad data is inspected later.
Isolate the poison pill
Without a DLQ, a single unparseable message can stall or kill a consumer. With one, failures are routed to a separate queue/table with the error and original payload — preserving throughput and giving you a place to debug and replay.
Inject
Routing
Data quality dimensions
Quality is multidimensional — completeness, accuracy, consistency, timeliness, and uniqueness each fail in their own way and need their own checks.
Five lenses on “good data”
A pipeline asserts these as rules and an observability layer tracks them over time. Pick a dimension to see a concrete violation; the right fix differs for each.
Dimension
Violation
PII & anonymization
Protecting personal data isn’t one technique: masking hides a field, hashing makes it non-reversible, and k-anonymity generalizes quasi-identifiers so no row is unique.
Three levels of protection
Mask a direct identifier in output. Hash (with a salt) to join without exposing the raw value. k-anonymity generalizes quasi-identifiers (age→band, zip→region) so every row is indistinguishable from at least k−1 others — defeating re-identification.
Technique
Result
Sort-Merge Join
Sort both inputs on the join key, then merge them with two pointers in a single pass — emitting matches and advancing the side with the smaller key.
External Merge Sort
Data too big for memory is sorted in two phases: create sorted runs from memory-sized chunks, then k-way merge the runs by repeatedly taking the smallest head.
Raft Consensus
Step through a full cycle: election by majority vote, log replication, commit on majority acknowledgement, leader failure, and re-election in a new term.
Skip List
Search a value by starting in the top express lane and dropping down whenever the next node would overshoot — skipping most of the list instead of scanning it.
Merkle Tree
Two replicas differ in exactly one block. Compare hashes top-down: prune any subtree whose hashes match, and descend only the mismatching side to locate the bad block.
Token Bucket
Tokens drip in at a fixed rate up to the bucket’s capacity. Each request consumes one; if the bucket is empty the request is rejected. Bursts are allowed up to capacity, average rate stays bounded.
Gossip & anti-entropy
Nodes spread state by telling a few random peers each round. Like an epidemic, information reaches everyone in about log(n) rounds — no central coordinator needed.
Epidemic propagation
Each round, every node that knows a fact forwards it to a random peer. The infected set roughly doubles per round, so all n nodes converge in ≈ log₂(n) rounds — robust to failures and requiring no membership leader. Used for membership, failure detection, and replica anti-entropy.
Spread
Convergence
Snowflake IDs
A 64-bit ID packs a timestamp, a machine id, and a per-millisecond counter — so every node mints unique, roughly time-sortable IDs with zero coordination.
Coordination-free unique IDs
Bits = 1 sign + 41 timestamp (ms since an epoch) + 10 machine id + 12 sequence. The timestamp prefix makes IDs sort by creation time; the machine bits keep nodes from colliding; the sequence allows many IDs per millisecond per node.
Field
Role
Lambda vs Kappa
Two ways to serve both historical and real-time views: Lambda runs parallel batch and speed layers; Kappa runs one streaming pipeline and reprocesses by replaying the log.
Two layers vs one
Lambda: a batch layer recomputes accurate views from all history while a speed layer covers the recent gap; the serving layer merges them — powerful but two codebases to keep in sync. Kappa: a single stream pipeline; to “recompute,” replay the retained log through a new version — one codebase, simpler ops.
Architecture
Trade-off
Data freshness & SLAs
Freshness is how old the data is: now minus the last successful load. An SLA sets the maximum allowed lag; crossing it should page someone, not silently serve stale data.
Lag against a promise
Freshness lag = current time − last successful update. A freshness SLA (e.g., “≤ 1 hour”) turns staleness into an alertable condition. Dashboards should surface lag, and pipelines should fail loudly when the window is breached rather than serve quietly outdated numbers.
State
Status
Trie (Prefix Tree)
Words sharing a prefix share a path. Search a query character by character; the node reached roots all completions of that prefix — that’s how autocomplete works.
Cuckoo Hashing
Each key has two candidate buckets. Insert into a free one; if both are taken, evict an occupant and re-home it in its alternate bucket — chaining kicks until everything settles.
Leaderless replication
With no leader, clients write to and read from many replicas at once. If the write and read quorums overlap (W+R>N), every read is guaranteed to see the latest write.
Quorum overlap
Dynamo-style stores replicate to N nodes and require W acks to write, R responses to read. When W+R>N the two sets must share a node, so a read always catches the newest value; read repair then updates the stale replicas it noticed. Lower quorums trade consistency for latency/availability.
Quorum (N=3)
Guarantee
Hash vs range partitioning
Hash partitioning spreads keys evenly but scatters ranges; range partitioning keeps order for fast scans but risks hot partitions. The choice follows your access pattern.
Even spread vs ordered scans
Hash: key → hash → partition. Load is balanced and point lookups are easy, but a range query must hit every partition. Range: contiguous key ranges per partition. Range scans touch one partition, but sequential keys (timestamps, autoincrement ids) create hotspots.
Scheme
Behavior
CRDTs
Conflict-free replicated data types let replicas update independently and merge deterministically — so concurrent edits converge to the same state without coordination or a leader.
Merge without conflict
A CRDT defines a commutative, associative merge so that applying updates in any order yields the same result. Two replicas can accept concurrent writes offline and reconcile later — no locks, no last-write-wins data loss. Counters, sets (OR-Set), and text sequences all have CRDT forms.
Conflict handling
After reconciling
Materialized views
A materialized view caches a query’s result. The win is incremental maintenance — applying just the delta from a base-table change instead of recomputing the whole thing.
Cache the answer, update the delta
A regular view re-runs on every read; a materialized view stores the result for fast reads. Keeping it current can mean a full recompute (scan everything) or incremental maintenance (apply only the rows that changed) — the latter is what makes pre-aggregated tables and streaming materializations cheap.
Refresh strategy
Work to refresh
Bitmap Index
Each distinct value gets a bit-vector over the rows. A multi-predicate filter becomes a bitwise AND/OR of those vectors — the set bits of the result are the matching rows.
Binary Heap (min-heap)
A complete tree stored in an array. Insert appends a leaf and sifts it up; extract-min lifts the smallest from the root, moves the last leaf up, and sifts it down — each O(log n).
Columnar encodings
Columnar storage compresses each column with the scheme that fits its shape: run-length for repeats, dictionary for low cardinality, delta for sorted numbers, bit-packing for small ranges.
One column, many encodings
Because a column holds one data type with similar values, encodings get huge wins: RLE collapses runs, dictionary maps distinct values to tiny codes, delta stores differences of sorted/monotonic data, and bit-packing uses only as many bits as the value range needs.
Encoding
Best for
Fencing tokens
A paused old leader can wake up and corrupt state. A monotonically increasing fencing token, checked by storage, makes any write from a stale leader get rejected.
Stop the zombie leader
If a leader stalls (a long GC pause) the cluster may elect a new one. When the old leader resumes it still thinks it’s in charge. Each lock grant carries an increasing token; the storage layer remembers the highest token it has seen and rejects any write with a smaller one — fencing out the zombie.
Protection
Stale write
Count-distinct: exact vs HLL
Counting unique values exactly means remembering every value — memory grows with cardinality. HyperLogLog estimates the count in a few kilobytes, flat, with ~2% error.
Memory vs accuracy
An exact distinct count keeps a set of all seen values, so memory scales with the number of uniques — fine for thousands, ruinous for billions. HyperLogLog hashes values and tracks only the maximum leading-zero run per bucket, giving a cardinality estimate in fixed, tiny space at the cost of a small error.
Method
At 1 billion uniques
Checkpointing & exactly-once
A streaming job survives failures by periodically snapshotting operator state and the input offset. On restart it restores the snapshot and replays from that offset — with an idempotent sink, exactly-once.
Snapshot + replay
The engine writes a consistent checkpoint of all operator state plus the source offset it had consumed to. After a crash it restores the latest checkpoint and replays the log from that offset — no data lost. Pairing replay with an idempotent or transactional sink yields exactly-once output, not just at-least-once.
Recovery
After a crash
Roaring Bitmap
Within one 65,536-key chunk, Roaring stores a sorted array while sparse and switches to a dense bitmap once the array would exceed the bitmap’s fixed 8 KB — always the smaller of the two.
Segment Tree
Each node aggregates a range; a query selects the few canonical nodes that exactly cover the asked interval (O(log n)), and a point update repairs only one root-to-leaf path.
Adaptive query execution
A plan chosen from estimates is often wrong. Adaptive execution re-optimizes mid-query using real runtime stats — coalescing tiny partitions, switching joins, and splitting skewed ones.
Re-plan with real numbers
Static optimizers guess from stale statistics. After a shuffle, the engine knows actual partition sizes, so it can coalesce many small partitions into fewer tasks, switch a planned shuffle join to a broadcast when a side turns out small, and split a skewed partition across tasks.
Plan
At runtime
Zone maps & min/max pruning
Each data block stores the min and max of its columns. A query skips any block whose range can’t contain the value — reading a fraction of the file without an index.
Skip what can’t match
Columnar files keep per-block (row-group) statistics. For WHERE x = 57, the engine checks each block’s [min,max]; blocks that can’t contain 57 are pruned unread. Data clustered/sorted on the predicate column makes zone maps dramatically more effective.
Layout
Blocks scanned for x=57
Tombstones & compaction
In append-only stores a delete is just a tombstone marker. Reads hide the deleted key, and a later compaction physically drops the key and its tombstone.
Delete = write a marker
LSM-trees and log-structured tables never modify data in place, so a delete appends a tombstone. Queries skip any key shadowed by a newer tombstone; until compaction runs, the old value still occupies space. Compaction merges files, drops shadowed values, and removes expired tombstones.
Stage
Storage
Hinted handoff
When a target replica is down, a coordinator temporarily stores a hint and replays the write once the replica returns — keeping writes available during transient failures.
Hold the write, deliver later
In a leaderless system, if one of the N replicas is unreachable, the write still succeeds on the others; a healthy node keeps a hint (the write plus its intended destination). When the down node recovers, the hint is handed off to it, restoring full replication without blocking the original writer.
State
Replication
Late-arriving dimensions
A fact can reference a dimension row that hasn’t loaded yet. Rather than drop it, insert an inferred placeholder member now and backfill its attributes when the dimension arrives.
Fact before dimension
Events often beat their reference data. Dropping the fact loses revenue; blocking the load stalls the pipeline. The standard fix: create an inferred member (a dimension row with the key and unknown attributes), point the fact at it, and update the attributes later when the real dimension record lands — the fact’s key never breaks.
Strategy
Outcome
Schema registry & compatibility
A schema registry versions the schemas producers and consumers share. Compatibility rules decide which changes are safe, so a new producer can’t silently break old consumers.
Govern the contract
Producers register each message schema and get a version id. The registry enforces a compatibility mode before accepting a new version: backward (new schema reads old data — safe to add optional fields), forward (old schema reads new data), or full (both). Removing a required field or changing a type breaks compatibility and is rejected.
Change
Registry verdict
Data skew & salting
One hot key can send most rows to a single reducer, stalling the whole job. Salting splits the hot key with a random suffix, spreads the load, then re-aggregates.
Break up the hot key
If 80% of rows share one key, the partition for that key overwhelms one task while others idle. Salting appends a small random number to the key so its rows fan out across many partitions; a second aggregation step combines the partial results back into the true total.
Approach
Load balance
Grouping sets & rollup
One query can compute many aggregation granularities at once — totals, subtotals, and grand totals — instead of running and unioning several separate GROUP BYs.
Subtotals in one pass
GROUP BY ROLLUP(region, product) produces the detail rows plus per-region subtotals and a grand total in a single scan. GROUPING SETS generalize this to any chosen combinations, and CUBE computes every combination — far cheaper than separate queries.
Query
Rows produced
Data drift taxonomy
Drift is when incoming data stops matching expectations. The five kinds fail differently — schema, type, value-distribution, null-rate, and cardinality — and each needs its own detector.
Five ways data drifts
Pipelines break quietly when the data changes, not the code. Catch each kind with a specific check: structural (schema/type) vs statistical (distribution/null/cardinality). Pick one to see a concrete example and its detector.
Drift type
Example & detector
Table grain & grain mismatch
Grain is the meaning of one row. Joining two tables at different grains without aligning them multiplies measures — the classic silent reporting bug.
One row means what?
Declare the grain explicitly (one row per transaction, per customer-day, per month). A grain mismatch — e.g., joining daily sales to a monthly target — fans the coarse side out across the fine side, double-counting the target. Fix by aggregating to a common grain before joining.
Scenario
Result
Fan-out, fan-in & join traps
A fan-out turns one row into many; a fan trap is a join that secretly fans out and double-counts a measure; a chasm trap joins two facts through a shared dimension and explodes both.
When joins multiply rows
Fan-out (one order → many items) and fan-in (many transactions → one daily total) are intended. A fan trap is an accidental one-to-many that inflates a sum; a chasm trap joins two fact tables via one dimension, producing a cross-product. Fix by aggregating before joining or splitting into separate queries.
Pattern
Effect
Referential integrity & orphans
Every foreign key should point to a real parent. Rows whose key has no matching dimension are orphans — they silently drop from inner joins and corrupt totals.
Keep facts anchored to dimensions
Referential integrity means sales.customer_id always exists in customer. An orphan fact (product_id with no product row) disappears in an inner join or shows null attributes in a left join. Enforce with a referential check, and handle the orphan via an inferred dimension member or a quarantine table.
State
Join behavior
Keys & hash collisions
Beyond surrogate vs natural: composite keys combine columns, hash keys fingerprint them into one value, and a hash collision (two rows, same hash) silently merges distinct records.
Identify a row, safely
A composite key is several columns together (customer_id + order_date + product_id). A hash key hashes those into one stable id (sha2(...)) — compact and great for joins/CDC. The risk is a collision: two different inputs mapping to the same hash, merging unrelated rows. Wide hashes (SHA-256) and including all identifying columns make collisions negligible.
Key style
Note
Soft vs hard delete & tombstones
A soft delete flags a row deleted; a hard delete physically removes it; a tombstone is the delete event CDC emits so downstream tables can propagate the removal.
Three ways to delete
Soft delete sets is_deleted=true — reversible, keeps history, but readers must filter it. Hard delete removes the row — needed for right-to-be-forgotten. A tombstone is a delete-flagged CDC record that tells downstream pipelines to delete the matching row too (soft/hard delete propagation).
Method
Behavior
Control, audit & sidecar tables
Operational metadata lives beside the data: a control table tracks pipeline state, an audit table logs every run, and a sidecar table carries per-load metadata you reconcile against the main table.
Metadata that runs the pipeline
Control table: the state the next run reads — last watermark, last successful load. Audit table: one row per run (job_id, start, end, status, row_count) for history and SLA tracking. Sidecar: per-batch metadata (checksum, source_file, batch_id); sidecar correctness means its row counts/checksums match what actually landed.
Table
Holds
Reconciliation & change detection
Trust comes from comparing source to target: row counts, null counts, duplicate counts, and hash diffs prove the load is complete and detect exactly which rows changed.
Prove the load is correct
After a load, reconcile: do source and target row counts match? Did null or duplicate counts move? For incremental work, a hash diff (a checksum of the important columns) tells you precisely which rows changed without comparing every field. These checks turn “looks fine” into a verifiable guarantee.
Check
Verdict
Master data & golden records
MDM resolves many source records of the same real-world entity into one trusted golden record — via matching rules, then survivorship rules that pick the winning value per field.
One trusted version of the truth
The same customer appears across CRM, POS, and web. Matching (exact + fuzzy) and entity resolution link them; survivorship rules decide which source wins per attribute (CRM email beats POS email); the result is a golden record — the basis of Customer 360 and householding. Standardization/canonicalization first makes matching reliable.
Stage
Step
Determinism & reproducibility
A deterministic pipeline yields the same output from the same input and code every time. Hidden non-determinism (now(), random, unordered reads) breaks reproducibility and backfills.
Same input → same output
Deterministic logic is reproducible: re-running on the same data and code gives identical results — essential for backfills, testing, and audits. Common non-determinism: current_timestamp(), random IDs, unordered aggregations, or reading “latest” at runtime. Pin them (pass the run date in, sort before first(), seed randomness) to restore reproducibility.
Logic
Re-run result
Pipeline reliability patterns
Production pipelines must survive retries and restarts: idempotency, reentrancy, atomic writes, and restartability together make a run safe to repeat after any failure.
Safe to run again
Idempotent: re-running gives the same result (no duplicates). Reentrant: a new run can start while/after a failed one without corruption. Atomic write: a step fully succeeds or fully fails — no partial state. Restartable / fault-tolerant: resume from the last checkpoint. A no-op run (no new data) should change nothing.
Property
On retry
Poison pills, retries & circuit breakers
One bad message shouldn’t take down a consumer. Route poison pills to a DLQ, retry transient failures with backoff, and trip a circuit breaker when a dependency is failing.
Contain the failure
A poison pill (unparseable record) goes to the dead-letter queue so good data keeps flowing. Transient errors (a brief timeout) go to a retry queue with backoff. When a downstream dependency is consistently failing, a circuit breaker trips — fail fast and stop hammering it — then half-opens to test recovery.
Failure
Handling
Table types: append, current, history, snapshot
The same data supports different table shapes: append-only event logs, current-state tables (latest only), history tables (every version), and snapshot tables (state as of a date).
Mutable vs immutable shapes
An append-only (immutable) table only adds rows — an event/state log. A current-state table keeps only the latest row per key. A history table keeps every version (SCD2-style). A snapshot table stores the full state as of a point in time. Each answers a different question; you often derive the others from the append-only log.
Shape
Holds
Fact table types
Facts come in flavors: transaction (one row per event), periodic snapshot (state each period), accumulating snapshot (a process with milestone dates), and factless (just the occurrence).
Four fact patterns
Transaction: one row per event, additive measures. Periodic snapshot: one row per entity per period (daily balance). Accumulating snapshot: one row per process instance with milestone date columns that fill in over time (order → ship → deliver). Factless: records that an event happened (attendance, eligibility) with no measure.
Type
Grain
Dimension patterns
Dimensional modeling has named patterns: conformed (shared across marts), role-playing (one dim, many roles), junk (grab-bag of flags), degenerate (key with no dimension), and bridge (many-to-many).
The dimension toolbox
Conformed: the same dimension reused across fact tables so metrics align. Role-playing: one date dimension joined as order_date, ship_date, etc. Junk: low-cardinality flags collapsed into one small dimension. Degenerate: a transaction id kept on the fact with no separate dimension. Bridge: resolves a many-to-many (account ↔ customer).
Pattern
Use
Nested & semi-structured data
JSON/Avro carry arrays and structs. Explode turns an array into one row per element; flatten lifts struct fields into top-level columns — the two moves that tabularize semi-structured data.
From documents to tables
Semi-structured formats (JSON, Avro, XML) nest arrays and structs. Explode expands an array column into one row per element (an order with 3 items → 3 rows). Flatten promotes nested struct fields (address.city → city) into columns. Together they turn a document into a query-friendly relational shape.
Operation
Result
Window types & stream joins
Streaming aggregations bucket events by time: tumbling (fixed, non-overlapping), sliding (overlapping), and session (gap-based) windows — plus stream-stream and stream-batch joins.
How events get grouped
Tumbling: fixed back-to-back buckets (every 5 min). Sliding: fixed-size but overlapping (5-min window every 1 min). Session: dynamic windows that close after an inactivity gap. Stream-stream join matches two live streams within a time bound (needs watermarks for state cleanup); stream-batch join enriches a stream with a static/slowly-changing table.
Window / join
Behavior
CDC methods & lag
Change data capture comes in three flavors — log-based, trigger-based, and snapshot — trading source impact for completeness. CDC lag is how far behind the target is from the source.
How changes are captured
Log-based reads the database’s transaction log — low source impact, captures every change including deletes (tombstones). Trigger-based fires DB triggers into a change table — captures all DML but adds write overhead. Snapshot/incremental snapshot diffs periodic copies — simple, but misses intermediate changes. CDC lag = source commit time − target apply time.
Method
Trade-off
Event sourcing & ordering
Event sourcing keeps the log of events as the source of truth and derives state by replaying it. Ordering and out-of-order handling decide whether that replay is correct.
State is a fold over events
Instead of storing current state, store every event; current state = replaying them in order. This gives free audit history and replay (rebuild any past state, or reprocess with new logic). The catch is ordering: events must be applied in the right sequence, and out-of-order arrivals (network delays) need sequence numbers or event-time + watermarks to reorder.
Aspect
Behavior
Scaling & resource patterns
Scale out (more nodes) or up (bigger nodes); autoscale to load. The classic failure modes are the small-file problem, stragglers, and resource contention from skew.
More machines vs bigger machines
Horizontal scaling adds nodes (near-linear for partitionable work); vertical adds CPU/RAM to one node (simpler, capped). Autoscaling matches cluster size to load. Watch for the small-file problem (too many tiny files → metadata overhead), stragglers (one slow task holds up a stage), and resource contention when a hot partition monopolizes an executor.
Topic
Note
Observability, impact & blast radius
Observability is metrics + logs + traces that let you detect, diagnose (root-cause), and scope failures. Lineage turns a broken job into a precise blast-radius and impact analysis.
Detect, diagnose, scope
Pipelines emit metrics (freshness, row counts, durations), logs, and traces so you can alert and run root-cause analysis. When something breaks, lineage answers the scoping questions: which downstream tables, dashboards, and models are affected (impact analysis) and how far the damage spreads (blast radius).
Capability
What it gives
Testing the data pipeline
Data pipelines need a test pyramid: fast unit tests on transforms, integration tests across steps, end-to-end on real data shapes, plus data-quality, contract, and chaos tests.
More than code tests
Unit: a single transform with fixed input. Integration: several steps wired together. End-to-end: the full pipeline on representative data. Beyond code: data-quality tests (assertions on the output), contract tests (source matches the agreed schema), regression/snapshot tests (output didn’t change unexpectedly), and chaos tests (inject failures to prove resilience).
Level
Tests
NFRs & reliability metrics
Architecture reviews hinge on non-functional requirements: throughput, latency, availability, durability — measured by SLAs and the recovery metrics RPO, RTO, MTTR, and MTBF.
The “-ilities” and their numbers
Throughput (work/sec) and latency (time per request) size the system; availability (uptime %) and durability (data not lost) bound reliability. Recovery is quantified: RPO (max acceptable data loss, in time), RTO (max time to restore), MTTR (mean time to recover), MTBF (mean time between failures).
Metric
Means
Governance roles & policies
Governance assigns accountability — owner, steward, custodian — and enforces policy: classification and sensitivity, retention and purging, right-to-be-forgotten, residency, and least privilege.
Who is accountable, and the rules
Owner is accountable for a data asset; steward defines quality and meaning; custodian runs the systems that store it. Policies: classification + sensitivity labels drive access; retention/purging and right-to-be-forgotten govern lifecycle; residency/sovereignty constrain location; least privilege and segregation of duties limit access.
Area
Means
More dimension patterns
Beyond the basics: mini dimensions split off fast-changing attributes, outriggers are dimensions of dimensions, hierarchy dimensions nest levels, helper tables resolve multi-valued links, and rapidly-changing dimensions need special care.
The rest of the toolbox
Mini dimension: pull volatile/banded attributes (age band, income band) into a small separate dim so the main dim stays stable. Outrigger: a dimension referenced by another dimension. Hierarchy: parent→child levels in one dim. Helper: like a bridge, resolves a multi-valued attribute. Rapidly-changing dimension: changes so often that SCD2 explodes — solved with a mini dimension.
Pattern
Use
Customer 360 & reference data
Identity resolution links a person’s many identifiers into one entity; householding groups people into households; Customer/Product 360 is the unified view; reference data is the managed lookup vocabulary everything joins to.
One entity, one view
Identity resolution links email, phone, and device ids to a single person (deterministic + probabilistic). Householding rolls individuals up into a household. A Customer 360 / Product 360 is the resulting unified profile drawing on every source. Reference data management curates the controlled vocabularies (country codes, currencies, product categories) that all of these join against.
Concept
Means
Data trust & quality SLOs
Trust is made measurable: a reliability/trust score rolls quality checks into one number, certification marks a dataset as vetted, and quality SLAs/SLOs set the targets the data must meet.
Make trust a number
A reliability/trust score combines freshness, completeness, validity, and uniqueness into a single 0–100 signal stakeholders can rely on. Certification formally blesses a dataset (“certified gold”) after review. A quality SLA is a committed target (e.g., 99.9% valid rows, refreshed by 8am); a quality SLO is the internal objective — usually stricter — that you manage to so you never breach the SLA.
Mechanism
Means
FinOps: chargeback & showback
Cloud cost is engineering’s problem: tag resources to attribute spend, then either showback (report each team’s cost) or chargeback (actually bill it), with workload isolation and capacity planning to keep it predictable.
Make spend visible and owned
Resource tagging + cost attribution map every dollar to a team/project. Showback reports that cost back for awareness; chargeback goes further and bills the team’s budget, driving accountability. Workload isolation (separate clusters/warehouses) keeps costs clean and stops one job starving another; capacity planning forecasts the compute you’ll need so you neither overspend nor throttle.
Lever
Effect
Data mesh & data products
Data mesh shifts ownership to domains that publish data as products on a self-service platform, governed by federated standards — decentralizing data the way microservices decentralized apps.
Decentralize, but standardize
Domain ownership: the team that knows the data owns its pipelines end-to-end. Each publishes a data product — discoverable, documented, with an owner and a data product SLA/SLO. A central self-service platform provides the paved-road tooling so domains don’t reinvent it. Federated governance sets global rules (security, quality, interoperability) enforced locally — often computationally.
Pillar
Means
Producer & consumer contracts
A data contract has two sides: the producer guarantees a schema and semantics, the consumer states what it depends on. Consumer-driven contracts turn those expectations into tests the producer runs before shipping a change.
Who promises what
A producer contract is the publisher’s guarantee — schema, types, semantics, freshness. A consumer contract declares exactly which fields and rules a consumer relies on (often a subset). A consumer-driven contract collects consumers’ expectations and runs them as tests in the producer’s pipeline, so a breaking change is caught before release. Compatibility rules (backward/forward/full) define what evolution is safe.
Side
Means
Parallelism, locality & contention
Throughput comes from task slots (executors × cores) chewing through partitions in waves. Data locality avoids shipping data over the network; resource contention is what happens when too many tasks fight for the same CPU, memory, or I/O.
How the work actually runs
Parallelism = task slots = executors × cores; a stage’s partitions run in waves of that many tasks. Data locality schedules a task where its data already lives (same node/rack) to avoid network transfer — the “move compute to data” principle. Resource contention appears when slots, memory, or disk I/O are oversubscribed: tasks queue, GC thrashes, and the stage slows even though “CPU is busy”.
Aspect
Means
Load, stress & smoke tests
Correctness tests aren’t enough for pipelines: a smoke test proves it runs at all, a load test proves it meets SLAs at expected volume, a stress test finds the breaking point, and a soak test catches slow degradation over time.
Non-functional testing
Smoke: a fast end-to-end sanity check — does it even run? Load: at expected volume, does it hold latency/throughput SLAs? Stress: push past expected load to find where and how it breaks (graceful vs cliff). Soak: sustained load for hours/days to surface memory leaks, state growth, and small-file accumulation that short runs miss.
Test
Finds
Encryption at rest & in transit
Two complementary protections: encryption at rest secures stored bytes on disk/object storage, encryption in transit secures bytes moving over the network — both anchored by managed keys.
Protect data still and moving
At rest: files in object storage and disks are encrypted with keys, so a stolen disk reveals nothing. In transit: TLS encrypts data moving between client, services, and storage, defeating eavesdropping. Key management (a KMS) controls the keys — rotation, access policies, and customer-managed keys (CMK) so you, not just the provider, hold the root of trust. Strong systems require both, always.
Protection
Means
Incident response & on-call
When a pipeline breaks, a defined flow contains the damage: detect via alerts, triage and assign a severity, mitigate and communicate, then run a blameless postmortem. Distributed tracing and error/quarantine tables speed diagnosis.
From alert to postmortem
Detect: monitoring fires an alert (freshness/quality/failure). Triage: assign a severity (SEV1 customer-facing → SEV3 minor) and an owner. Mitigate & communicate: stop the bleeding (pause downstream, roll back) and post status. Postmortem: a blameless write-up of timeline, root cause, and action items. Distributed tracing across services and an error table of failed records make the root cause findable fast.
Phase
Step
Data profiling
Before asserting quality you must know the data. Profiling computes per-column statistics — null rate, distinct count, min/max range, value distribution, and format conformity — that seed quality checks, drift baselines, and master/golden-record building.
Measure the data first
A profile summarizes each column: completeness (null rate), cardinality (distinct count), range (min/max), distribution (histogram), and conformity (share matching the expected format/pattern). These numbers become the baselines for drift detection, the thresholds for validity/range checks, and the inputs to entity matching when building a master record. You profile on first load and re-profile to catch change.
Measure
Finds
Privacy: GDPR, PDPA & subject rights
Privacy regimes — the EU’s GDPR and Singapore’s PDPA among them — grant individuals rights over their data and obligate you to honor them: lawful basis to process, access and erasure (right to be forgotten), breach notification, and residency limits.
Engineering for privacy law
GDPR (EU) and PDPA (Singapore) share a shape: you need a lawful basis (consent or legitimate interest) to process personal data; individuals have data subject rights — access, correction, portability, and erasure (the right to be forgotten); breaches trigger notification within set windows; and data may be bound by residency/sovereignty. Practically, you must be able to locate, export, and delete one person’s data across every table on request.
Obligation
Means
The pipeline, one stage at a time
Everything a batch pipeline does, wired into a single loop. Press Next to advance one stage and watch a micro-batch flow from the source, through ingest, parsing, validation, dedup, transform and shuffle, into an atomic commit — while bad rows peel off to quarantine, a failed check lands in the error table, and a duplicate is dropped. The watermark advances, downstream reads the new version, then it loops for the next batch.
The stream, one stage at a time
Everything a streaming job does each trigger, wired into a loop. Press Next to advance one stage and watch a micro-batch read from the source offset, get an event-time, push the watermark forward, update keyed state, drop a late record, persist to the checkpoint, and commit exactly-once to the sink. State survives across triggers — that’s what the checkpoint protects.
The warehouse, one stage at a time
A dimensional load, end to end. Press Next to extract source orders, stage and standardize them, conform the dimensions (an SCD2 version opens when an attribute changes), swap natural keys for surrogate keys, assemble a fact row at its declared grain, load the star, then run a BI query that joins fact to dimensions and aggregates. Watch the star assemble and the join paths light up.
The shuffle, one stage at a time
What happens when data has to cross the cluster. Press Next to watch map tasks emit keyed records, partition them by hash, spill sorted runs to disk under memory pressure, exchange them over the network (every reducer fetches from every mapper), merge, and reduce. A hot key skews one partition into a straggler — exactly what AQE skew-splitting targets.
The join, one stage at a time
How two tables actually meet. Press Next to plan a strategy from the table sizes, distribute the data (broadcast the small side, or shuffle both by key), build a hash table from the build side, probe it by streaming the other side, spill a partition that outgrows memory, and emit matches. Use strategy to flip between broadcast and shuffle and compare the cost.
The orchestrator, one stage at a time
How a pipeline of tasks actually runs. Press Next to schedule the DAG, run tasks whose dependencies are met, hit a failure, retry it with backoff, run the rest behind their dependency gates, check the SLA, and backfill a past date partition through the same graph. Tasks light up by status and downstream waits until upstream succeeds.
The deploy, one stage at a time
How a pipeline change reaches production. Press Next to open a PR, run CI (unit + contract + data-quality tests), build a versioned artifact, validate in staging, release to a canary and compare to baseline, promote blue/green, then monitor and finalize. Use outcome to flip between a healthy deploy and a regression that triggers a rollback.
The contract, one change at a time
How a producer and consumer agree on shared data. Press Next to define a versioned contract, have the producer emit data validated at the boundary, let a consumer bind to it, ship a compatible change (an optional field → minor bump), attempt a breaking change (drop a required field → blocked in CI), version to a major with dual-publish, and migrate consumers before retiring the old one.
The cost model, one step at a time
How the optimizer chooses. Press Next to gather statistics, estimate filter selectivity and join cardinality, enumerate candidate plans, cost each by IO + CPU + network, choose the cheapest, and then meet reality — the actual row counts — to see why stale stats mislead the planner.
The ingest, one batch at a time
How new data lands safely. Press Next to discover files arrived since the last offset, read them, infer + evolve the schema when a new column appears, rescue malformed values, write to the table, and commit the checkpoint so a restart resumes exactly once.
The lineage, one hop at a time
How a column connects end to end. Press Next to pick a target column, trace it upstream through transforms, see the full graph, introduce an upstream change, run impact analysis on everything downstream, and notify the affected owners before it ships.
The partitioning, one step at a time
How layout drives scan cost. Press Next to choose a partition key, write the layout into per-value directories, run a query with a predicate, prune the partitions that cannot match, hit a skew from a hot value, and fix it with bucketing or salting.
The backfill, one partition at a time
How history is rebuilt safely. Press Next to choose a range, isolate the work from the live pipeline, reprocess each day idempotently, reconcile late data, atomically publish the corrected partitions, and verify the counts — all while the live stream keeps running.
The replication, one step at a time
How data survives a failover. Press Next to write to the primary, replicate to a replica, observe lag, hit a primary failure, fail over by promoting the replica, and reconcile when the primary returns. Use mode to flip between synchronous and asynchronous replication and watch RPO/RTO change.