Databricks Data Engineering — Interactive Lab
A hands-on tour of the machinery underneath Databricks — the Delta transaction log, file compaction, Z-ordering & data skipping, MERGE, VACUUM, Auto Loader, and the Spark query engine (Catalyst, shuffle, AQE, broadcast joins, jobs & stages, Structured Streaming). Two layers of interactive dives — Lakehouse internals and the Spark engine — plus concept cards for the vocabulary. Click, drag, and run each idea to see it work. Press Esc anytime for this menu.
The Delta transaction log
A Delta table is just Parquet data files plus an ordered _delta_log of JSON commits. Every write is a new version containing actions — add file and remove file. The table’s state at any version is simply every file added up to that point minus every file removed. Nothing is mutated in place, which is what gives Delta its ACID guarantees and lets you time-travel to any past version just by replaying the log up to it.
OPTIMIZE — compacting small files
Streaming and frequent appends create a flood of small files. Every file a query touches carries fixed overhead — open it, read its footer, plan around it — so a thousand tiny files is far slower to scan than a few large ones, even at the same total size. OPTIMIZE bin-packs small files together into new files near a target size (1 GB by default), writing them as a single commit and tombstoning the originals. Ingest a pile of small files, then optimize and watch the file count collapse.
Z-ordering & data skipping
Delta stores per-file min/max statistics for each column. When a query filters on a column, any file whose min/max range can’t contain a match is skipped without being read. The catch: skipping only works if each file’s values are tightly ranged. Z-ordering sorts rows along a space-filling curve that interleaves multiple columns, so each file becomes a compact cluster in several dimensions at once — shrinking its min/max box and letting far more files be skipped for multi-column filters.
MERGE — upserts & SCD Type 2
MERGE is the workhorse of Delta pipelines: MERGE INTO target USING source ON key, then declare what to do when matched and when not matched. As a plain upsert it updates rows that exist and inserts those that don’t. For slowly changing dimensions Type 2 it does more: when a matched row’s attributes changed, it expires the old version (sets current = false) and inserts a new current row — preserving the full history instead of overwriting it. Step a source batch through and watch each decision.
VACUUM, tombstones & retention
When Delta removes a file it only writes a remove action — the file stays on disk so older versions remain readable. Those tombstoned files accumulate, so VACUUM physically deletes data files that are no longer referenced by the current version and older than the retention window (7 days by default). The cost is that time travel to a version whose files were vacuumed no longer works. Overwrite a few times to pile up tombstones, set retention, then VACUUM and watch which past versions stop being readable.
Auto Loader — incremental file ingestion
Pointing a batch job at a growing cloud folder re-reads everything every time. Auto Loader instead remembers which files it has already processed in a durable checkpoint, so each run ingests only the new files — and because that bookmark survives restarts, a crash never causes a file to be processed twice or skipped. Drop files into the folder, run Auto Loader to ingest the new ones, then restart the stream and confirm nothing is re-ingested.
Catalyst — logical plan optimization
You write what you want; Spark’s Catalyst optimizer decides how. A query becomes a tree of operators, and a sequence of rewrite rules transforms it into a cheaper-but-equivalent plan before any data moves. Two of the highest-impact rules: column pruning reads only the columns the query actually needs, and predicate pushdown shoves filters down into the scan so the data source skips non-matching data instead of Spark reading and discarding it. Step through the rules on SELECT name, amount WHERE region='US' AND amount>100 and watch the bytes read collapse.
Broadcast vs shuffle (sort-merge) join
Joining two tables means getting matching keys onto the same executor. There are two ways. If one side is small enough, Spark broadcasts a full copy to every executor and joins locally — the big side never moves. If both sides are large, Spark falls back to a sort-merge join: it shuffles both tables across the network by join key so equal keys meet, which is correct but expensive. The deciding factor is whether the smaller side fits under the broadcast threshold. Slide the right-table size across the threshold and watch the strategy — and the network cost — flip.
Shuffle partitions & Adaptive Query Execution
Every shuffle splits data into a fixed number of partitions (200 by default). Too many for the data and you get hundreds of tiny partitions, each with scheduling overhead; a skewed key and one partition balloons into a straggler that holds up the whole stage. Adaptive Query Execution waits until the shuffle has actually run, looks at the real partition sizes, then coalesces small partitions into target-sized ones and splits oversized skewed partitions. Switch between uniform and skewed data and toggle AQE to watch it rebalance.
Structured Streaming — micro-batches & watermarks
A stream is processed as a series of small micro-batches, each appended to the result like a row to a table. The hard part is event time: records arrive late and out of order, so a windowed count can’t know when a window is “done”. The watermark solves it — it’s the max event time seen so far minus an allowed-lateness delay. Windows entirely behind the watermark are finalized and dropped from state; events arriving behind it are too late and discarded. Advance the stream and tune the delay to trade completeness against state size.
Jobs, stages & tasks
Transformations are lazy — nothing runs until an action (write, count, collect) triggers a job. Spark splits that job into stages at every wide transformation: narrow ops (filter, map, select) need only the local partition and pipeline together inside one stage, while wide ops (groupBy, join, repartition) require a shuffle and force a stage boundary. Each stage runs as a set of tasks — one per partition — in parallel across executor cores. Run the pipeline stage by stage and watch the tasks fan out.
Unity Catalog — privilege resolution
Unity Catalog governs every table through a three-level namespace — catalog.schema.table — and privileges inherit downward: a grant on a catalog applies to all its schemas and tables, a grant on a schema applies to all its tables. To decide whether a principal may run SELECT, Unity Catalog checks the object, then its schema, then its catalog, against the groups the user belongs to — the first matching grant wins. Pick a user and a table and watch the resolver walk the hierarchy.
SELECT on a catalog to a group once instead of touching thousands of tables, and new tables are covered automatically. Because everything resolves through groups rather than individuals, access reviews stay tractable. The same model underpins finer controls — row filters and column masks attach to a table but evaluate per-querying-user — and every resolved access is logged for lineage and audit. One permission model spans tables, views, volumes, and files.Schema enforcement
A Delta table rejects any write that doesn’t match its schema — bad data is stopped at the door instead of silently corrupting the table.
Write on schema mismatch
Delta validates every incoming batch against the table schema. Columns of the wrong type or unexpected extra columns cause the whole write to fail — protecting downstream readers.
Incoming batch
Result
Schema evolution
When the source legitimately gains a column, mergeSchema lets the table evolve to absorb it — controlled change instead of a hard failure.
Source added a column
Enforcement blocks unexpected columns, but real sources change. With mergeSchema, Delta safely adds the new column (old rows get null); without it, the write is rejected.
mergeSchema
Resulting table
Change Data Feed
CDF turns a Delta table into a stream of row-level changes — every insert, update (before & after), and delete between two versions.
Row-level changes
With Change Data Feed enabled, Delta records what each commit did to every row. Downstream pipelines read just the changes between versions instead of recomputing from the full table.
Operation at v→v+1
CDF output
ACID & optimistic concurrency
Two writers can commit at once without locks — Delta lets them race, and only forces a retry when they actually touch the same files.
Concurrent commits
Both writers read the same snapshot and prepare changes. At commit, Delta checks the log: if their file sets don’t overlap, both succeed; if they do, the later writer retries against the new version.
Scenario
Outcome
Medallion architecture
Raw data earns trust in stages — Bronze keeps it as-is, Silver cleans and conforms it, Gold shapes it for the business.
Bronze → Silver → Gold
Each layer has one job. Bronze lands raw source data unchanged (an auditable record). Silver deduplicates, validates, and conforms it. Gold aggregates into business-ready tables for BI and ML.
Inspect layer
What happens here
Lakehouse vs warehouse vs lake
The lakehouse aims to end the tradeoff: the open, cheap storage of a data lake with the transactions and governance of a warehouse.
One table for everything
Classic warehouses give ACID and fast SQL but lock data in and struggle with ML. Data lakes are cheap and open but lack transactions and governance. The lakehouse puts a transactional layer (Delta) over open files to get both.
Highlight capability
Reading
Full vs incremental load
Reprocessing the whole source every run is simple but quadratically wasteful — incremental loads touch only what changed.
Reload all, or just the new?
A full load rereads the entire source each run — easy, but cost grows with total size. An incremental load reads only rows added since the last run, so cost tracks the change, not the size.
Strategy
Source
Rows processed
Streaming trigger modes
The same streaming query can run as periodic micro-batches, a one-shot catch-up job, or a continuous low-latency loop — pick the latency you’re paying for.
How often to fire a batch
A trigger controls when Structured Streaming processes available data. The mode trades latency against cost: tighter intervals mean fresher results but a cluster running more of the time.
Trigger
Trade-off
Star vs snowflake schema
A star keeps dimensions flat and fast to join; snowflaking normalizes them into sub-tables — less redundancy, more joins.
How to shape dimensions
A star puts a central fact table around denormalized dimension tables — one join per dimension. Snowflaking normalizes a dimension into a chain of sub-dimensions, saving storage but adding joins to every query.
Schema
Trade-off
Slowly changing dimensions
When a dimension attribute changes, how you record it — overwrite, version, or keep-previous — is the difference between SCD Type 1, 2, and 3.
A customer moves city
c2 changes from NYC to KL. Type 1 overwrites (no history). Type 2 expires the old row and inserts a new current one (full history). Type 3 keeps a “previous” column (one step of history).
SCD type
Resulting row(s)
Photon & vectorization
Photon replaces row-at-a-time execution with a vectorized engine that processes columns in batches — far more work per CPU cycle.
Per-row vs per-batch
The classic engine interprets one row at a time, paying overhead on each. Photon processes a column batch at once with tight, SIMD-friendly loops — fewer virtual calls, better cache and CPU use, big speedups on scans and aggregations.
Engine
Throughput
Caching hot data
The first read pulls remote files over the network; the cache keeps them on local SSD so repeat queries skip the slow hop entirely.
First read vs repeat read
Cloud object storage is durable but far away. The Delta/disk cache stores recently read files on the executor’s local SSD, so the next query on the same data is served locally — often an order of magnitude faster.
Cache
Run
Latency
Row & column security
The same table returns different data to different people — row filters hide records and column masks redact sensitive fields, evaluated per querying user.
One table, per-user views
A row filter restricts which rows a principal can see (e.g. only their region); a column mask redacts sensitive columns (e.g. email) unless the user is privileged. Both attach to the table and run for whoever queries it.
Querying user
What they see
DLT expectations
Declarative data-quality rules that ride with the pipeline — choose whether a violating row is kept-and-counted, dropped, or fails the run.
Quality as code
An expectation is a constraint like amount > 0 attached to a Delta Live Tables step. The action on violation is your choice: warn (keep, just track), drop (quarantine bad rows), or fail (abort the update).
On violation
Outcome
Fact & dimension tables
A fact table records measurable events; dimensions describe the context those events point to — and the shape of each is very different.
Two kinds of table
A fact table is tall and narrow: many rows, each a business event, holding numeric measures plus foreign keys. A dimension is short and wide: few rows of descriptive attributes the facts reference.
Show
Shape
Data Vault modeling
Data Vault splits the model into Hubs, Links, and Satellites so the warehouse can absorb source change and load in parallel without redesign.
Hubs · Links · Satellites
Hubs hold immutable business keys, Links capture relationships between hubs, and Satellites store the descriptive, time-stamped attributes. New sources attach as new satellites — the core never changes.
Highlight
Holds
Kimball vs Inmon
Two classic warehouse philosophies: Kimball builds dimensional marts bottom-up; Inmon builds a normalized enterprise warehouse top-down first.
Bottom-up vs top-down
Kimball: build conformed star-schema marts per business process, integrated by shared dimensions — fast to deliver. Inmon: build a normalized (3NF) enterprise warehouse as the single truth, then derive dependent marts.
Methodology
Flow
Surrogate vs business key
A business key comes from the source and can change; a surrogate key is a stable system ID that keeps joins and history intact when it does.
What changes when the source key changes?
Use the business key as primary key and a source-side change (re-keying, merge) ripples through every fact and history row. A surrogate key stays fixed; the business key becomes just another attribute.
Primary key
Event
Result
SCD Types 0, 4 & 6
Beyond the common Types 1–3, SCD Type 0 freezes a value, Type 4 splits history into its own table, and Type 6 combines 1+2+3.
The other SCD types
Type 0: never update (retain original). Type 4: keep only current in the dimension and push history to a separate table. Type 6: 1+2+3 together — versioned rows plus a current-value column.
Type
Structure
ETL vs ELT
The only real difference is when and where you transform — before loading in a separate engine (ETL), or after loading inside the lakehouse (ELT).
Order of operations
ETL transforms data in a dedicated engine before loading the clean result. ELT loads raw into the lakehouse first and transforms in place with its elastic compute — the modern default, since storage is cheap and you keep the raw copy.
Approach
Pipeline
Batch vs streaming
Batch processes a bounded chunk on a schedule for throughput; streaming processes an unbounded feed continuously for low latency.
Bounded vs unbounded
Batch runs periodically over a finite dataset — simple and high-throughput, but results lag the schedule. Streaming processes events as they arrive (as micro-batches in Spark) for fresh, low-latency results at higher operational cost.
Mode
Trade-off
Change Data Capture (CDC)
CDC reads a source database’s change log and replays just the inserts, updates, and deletes downstream — no full reload.
Capture changes at the source
Instead of re-extracting whole tables, CDC taps the source’s transaction log (or Delta CDF) and emits a stream of row changes. A MERGE then applies them to the target, keeping it in sync cheaply.
Change type
Applied to target
Deduplication
Sources resend the same record, so pipelines keep the latest row per key and drop the repeats — usually with a window + row_number.
Same event, more than once
At-least-once delivery and retries mean duplicates are normal. Deduplication partitions by the natural key, orders by event time, and keeps row number 1 — discarding older copies.
Run
Result
Validation & reconciliation
Two trust checks: validation tests rows against rules before they land, and reconciliation confirms source and target counts agree afterward.
Before and after
Validation rejects or quarantines rows that fail rules (types, ranges, not-null, referential). Reconciliation compares aggregate counts/sums between source and target to prove the load was complete and correct.
Check
Outcome
Processing operations
The verbs of a pipeline: cleanse, transform, enrich, aggregate, standardize, and profile — each reshaping data toward the target model.
What “processing” means
Real transformation is a sequence of operations. Pick one to see what it does to a sample record on its way from raw to gold.
Operation
Effect
Normalization vs denormalization
Normalizing splits data into related tables to kill redundancy; denormalizing flattens it back for read speed.
Split apart or flatten
Normalization (toward 3NF) stores each fact once across related tables — great for write integrity, more joins to read. Denormalization pre-joins into wide tables for fast analytical reads, accepting duplication.
Form
Trade-off
Delta Sharing
An open protocol that shares live Delta data with any client — the recipient reads the real files in place, with nothing copied or moved.
Share without copying
The old way emails extracts or builds bespoke pipelines, duplicating data and going stale instantly. Delta Sharing grants a recipient governed, read access to the actual table — they always see current data, across clouds and tools.
Approach
Result
Data products
A data product is a dataset treated like a product — owned, documented, quality-guaranteed, and built for known consumers, not a one-off table.
Table vs product
A raw table is just bytes. A data product adds an owner, a contract/SLA, quality checks, documentation, and discoverability — so consumers can trust and reuse it. Central to Data Mesh.
View as
Has
Exactly-once vs at-least-once
After a failure and retry, at-least-once can process a record twice; exactly-once uses checkpoints and idempotent writes to ensure it counts once.
What a retry does
At-least-once guarantees nothing is lost but may reprocess after a crash — duplicates. Exactly-once pairs offset checkpoints with idempotent/transactional sinks so each record affects the result a single time.
Guarantee
Simulate
Count
Event time vs processing time
Windowing by event time puts a late record in the window it belongs to; windowing by processing time puts it in whenever it happened to arrive.
Which clock counts?
A network blip delays a 12:00 event until 12:07. Event-time windowing (with a watermark) still counts it in the 12:00 window — correct. Processing-time windowing counts it at 12:07 — wrong bucket, but simpler and watermark-free.
Clock
Where the late event lands
Checkpointing
A checkpoint durably records stream offsets and aggregation state, so after a crash the query resumes exactly where it stopped — no loss, no replay gaps.
Recover without re-reading
Each micro-batch commits its source offsets and state to the checkpoint location. On restart the stream reads the checkpoint and continues from the next offset — the foundation of exactly-once and fault tolerance.
Simulate
State
Unity Catalog objects
Everything Unity Catalog governs lives in one hierarchy — catalogs and schemas group the securables: tables, views, volumes, and external locations.
One namespace for all assets
UC organizes data as catalog.schema.object. Objects can be tables and views (tabular), volumes (files), and access to outside storage is defined by external locations built on storage credentials.
Object
What it is
Clusters · serverless · SQL warehouses
Three ways to get compute: classic clusters you size and manage, instant serverless, and SQL warehouses tuned for BI — different startup, ops, and use.
Pick your compute
A cluster is a driver plus executors you configure (most control, slower start, you manage idle). Serverless starts in seconds with nothing to manage. A SQL warehouse is Photon-powered compute purpose-built for dashboards and SQL.
Type
Profile
File & table formats
Row vs columnar vs table format decides how data is read. Pick a format to see its layout and whether it’s typed, splittable, and transactional.
How bytes are laid out
Text formats (CSV/JSON/XML) are simple but slow and untyped. Binary columnar formats (Parquet/ORC) are fast for analytics. Table formats (Delta/Iceberg/Hudi) add a transaction log on top for ACID and time travel.
Format
Properties
Data quality dimensions
Quality isn’t one thing — completeness, accuracy, consistency, freshness, and uniqueness each fail differently. Pick one to see a failing example.
Five ways data goes wrong
Monitoring and contracts track each dimension separately because the fixes differ. Validation rules and profiling surface them; observability watches them over time.
Dimension
Failure
Data lineage
Lineage is the graph of what feeds what — trace a column upstream to its sources and downstream to every dashboard and model that breaks if it does.
Where did this come from?
Unity Catalog captures lineage automatically. It answers two questions: upstream — which tables produced this number; and downstream — what is impacted if this job fails or a column changes.
Trace from
Impact
Partition · repartition · coalesce
Three ways data gets divided: partitioning by a column writes directories; repartition reshuffles into N even parts; coalesce merges down without a shuffle.
Dividing the data
Partitioning physically splits a table by a column into folders so queries prune them. Repartition(N) does a full shuffle into N balanced partitions. Coalesce(N) reduces partitions by merging neighbors — cheap, no shuffle, but can leave them uneven.
Operation
Effect
Transformations vs actions
Spark is lazy: transformations only build a plan and run nothing — until an action triggers the whole optimized plan at once.
Lazy evaluation
filter, select, join, withColumn are transformations — they return a new DataFrame and execute nothing. count, collect, write, show are actions — they force Spark to optimize and run the accumulated plan.
Build the plan
State
Window functions
Window functions compute across a set of related rows — rank, lag, running totals — without collapsing them into one row like a groupBy would.
Compute over a window
OVER (PARTITION BY … ORDER BY …) defines a window per row. Unlike aggregation, every input row survives and gets an extra computed column — ranks, previous values, cumulative sums.
Function
Output column
UDF vs pandas UDF
A plain Python UDF runs one row at a time with serialization overhead; a pandas (vectorized) UDF processes whole columns via Arrow — far faster.
Row-at-a-time vs vectorized
A scalar UDF serializes each row to Python and back — flexible but slow, and a black box to Catalyst. A pandas UDF ships an Arrow batch to vectorized pandas code, amortizing the overhead across many rows.
Type
Throughput
Governance pillars
Enterprise governance is several disciplines at once — a catalog, lineage, master & reference data, contracts, and observability. Pick one to see its job.
The pieces of governance
Unity Catalog anchors most of these on Databricks. Each pillar answers a different question about trust, ownership, and control of data.
Pillar
Role
Data mesh vs data fabric
Two answers to scaling data: mesh is organizational (domains own data as products); fabric is technical (a unified metadata/access layer over everything).
People vs plumbing
Data Mesh decentralizes ownership — each domain publishes governed data products under federated rules. Data Fabric is an architecture that stitches sources together through a smart metadata and access layer. They’re complementary, not rivals.
Model
Idea
RBAC & access control
Access is granted to roles, not people. A query is allowed only if the user’s role chain holds the privilege on the object — and every check is audited.
User → role → privilege
Role-based access control assigns users to groups/roles, and privileges (SELECT, MODIFY) to roles on securables. Switch the role to see whether the same query is allowed, and note the audit log entry it produces.
Acting role
Decision
PII protection
Three different protections for sensitive data: masking hides it at read time, tokenization swaps it for a reversible token, encryption scrambles it with a key.
Mask, tokenize, or encrypt
Masking redacts in query results while the stored value stays (good for least-privilege views). Tokenization replaces the value with a meaningless token mapped in a vault. Encryption transforms it with a key so only key-holders can read it.
Method
Stored / shown as
Workflow orchestration
Orchestration runs tasks in dependency order on a schedule, retries failures automatically, and alerts you when something stays broken.
A DAG that runs itself
Databricks Workflows model tasks as a DAG: each runs only after its parents succeed. On failure a task retries with backoff; if it still fails the run is marked failed and alerts fire. Press run to step through.
Run
Status
CI/CD for data pipelines
Pipeline code moves through dev → test → prod behind gates: tests and quality checks must pass before a deploy is promoted to the next stage.
Promote only what passes
Version pipelines in git; a CI run executes unit tests and data-quality checks on every change. A failing gate blocks promotion, so broken logic never reaches production. Toggle a failing test to see the gate hold.
Pipeline
Stage
Declarative pipelines (DLT / Lakeflow)
You declare the tables and the quality rules; the engine figures out dependencies, incremental vs full refresh, and recovery. Lakeflow is the current branding.
Declare, don’t orchestrate
In a declarative pipeline you define each dataset as a query plus expectations. A streaming table ingests new data incrementally; a materialized view recomputes its result when inputs change. The framework manages the DAG, retries, and lineage.
Dataset type
Refresh behavior
Event-driven architecture
Producers emit events to a log; many consumers react independently and in real time. Adding a consumer doesn’t touch the producer.
React, don’t poll
Instead of batch jobs polling for changes, services publish events to a stream (Kafka, Event Hubs). Consumers subscribe and act as events arrive — enabling real-time analytics and loose coupling. Add a consumer to see fan-out.
Consumers
Fan-out
Object storage & the data lake
Cloud object storage (S3/ADLS/GCS) holds files as objects in buckets — cheap, durable, and effectively unlimited. It’s the foundation the lakehouse sits on.
Why the lakehouse rests on it
Object storage decouples storage from compute: store any volume of files cheaply and spin compute up/down against them. A data lake is raw files in this storage; Delta adds the transaction log that turns those files into reliable tables.
Show
Note
High availability vs disaster recovery
HA keeps you running through small failures within a region; DR brings you back after a region-wide disaster. They’re measured by RTO and RPO.
Stay up vs come back
High availability uses redundancy across availability zones so a node/AZ loss is invisible — near-zero downtime. Disaster recovery replicates to another region and restores after a major outage, accepting some recovery time (RTO) and possible data loss window (RPO).
Scenario
Outcome
DataFrame · SQL · RDD
The three Spark APIs aren’t three engines — DataFrames and Spark SQL compile through Catalyst to the same plan; only the low-level RDD API skips the optimizer.
Same engine underneath
Write it as a DataFrame, as Spark SQL, or with the legacy RDD API — DataFrame and SQL both go through Catalyst and produce identical optimized plans. RDDs run exactly what you wrote, with no optimization, which is why they’re rarely used now.
API
Path to execution
Cost optimization
Compute is the big bill, and idle time is the waste. Always-on clusters pay for gaps; autoscaling trims them; serverless bills only while work runs.
You pay for idle, not just work
A workload runs in bursts across the day. An always-on cluster is billed the whole time. Autoscaling sheds workers between bursts. Serverless spins to zero when idle — least waste, at a per-second rate.
Compute
Daily cost
Cloud networking
Where does your data traffic actually travel? A locked-down deployment keeps workspace-to-storage traffic on the cloud’s private backbone instead of the public internet.
Public path vs private path
By default, compute can reach storage and control-plane services over the public internet. A secure setup puts the workspace in your own VPC/VNet, uses private endpoints to storage, and enables secure cluster connectivity (no public IPs) — shrinking the attack surface.
Network mode
Posture
Self-service analytics
Self-service lets business users answer their own questions against governed data — instantly, but still bounded by Unity Catalog permissions — instead of filing a ticket.
Ticket queue vs governed self-service
The old model routes every question through an engineering backlog. Self-service publishes trusted data products in the catalog and gives users SQL, dashboards, and notebooks over them — answers in minutes, with row/column security and lineage still enforced automatically.
Model
Time to answer
Deletion Vectors
A delete marks row positions in a bitmap beside the data file instead of rewriting it. Reads apply the vector to hide those rows; OPTIMIZE later rewrites the file and clears the vector.
Liquid Clustering
Rigid partitioning by a high-cardinality column explodes into tiny files; liquid clustering instead groups rows by clustering keys incrementally as data arrives, keeping files well-sized and skip-friendly.
Delta Log Checkpoint
Each commit appends a JSON file. Reading means replaying them — so every 10 commits Delta writes a Parquet checkpoint of the whole state; readers start from it and replay only the tail.
Vectorized Execution (Photon)
A row engine interprets one value per operator call; a vectorized engine applies each operation to a whole column batch at once — amortizing dispatch and enabling SIMD.
Whole-Stage Code Generation
Volcano executes a row by calling next() down a chain of operator iterators — a virtual call per operator per row. Codegen compiles the chain into one fused loop with the operators inlined.
Shuffle Spill & Executor Memory
After a shuffle, each reduce partition accumulates in a bounded memory region. A partition that exceeds the limit spills sorted runs to disk; skew makes one partition spill far more than the rest.
Delta clone: deep vs shallow
A deep clone copies the data files into an independent table; a shallow clone copies only the metadata and points at the source files — instant and cheap until you write.
Copy data, or just pointers?
A deep clone physically copies all data files plus the log — a fully independent table, good for backups and migration. A shallow clone copies only the transaction-log metadata and references the source’s existing files — near-instant and storage-free, ideal for dev/test or short-lived experiments. Writes to a shallow clone copy-on-write only the touched files.
Clone type
Cost & independence
Predictive Optimization
Predictive Optimization lets Databricks automatically run table maintenance — OPTIMIZE, compaction, and VACUUM — when it predicts they will help, instead of you scheduling them by hand.
Maintenance on autopilot
Well-run Delta tables need periodic OPTIMIZE (compaction), clustering, statistics, and VACUUM. Doing this manually means guessing schedules and over- or under-running jobs. Predictive Optimization watches access patterns and table state and runs the right maintenance at the right time, charging only for work that pays off — fewer small files and faster queries without operational toil.
Mode
Outcome
Lakehouse Federation
Lakehouse Federation queries external systems — Postgres, MySQL, Snowflake, Redshift — in place through Unity Catalog, without first ingesting the data into the lakehouse.
Query where the data lives
Sometimes copying data in is wasteful or not allowed. Federation registers an external source as a catalog in Unity Catalog; queries are pushed down to the remote engine and joined with lakehouse tables — unified governance and lineage, no ingestion pipeline. Great for exploration and joining a few external tables; for heavy repeated workloads, ingesting still wins on performance.
Approach
Trade-off
Snapshot isolation & MVCC
Every reader sees a consistent snapshot — one specific table version — while writers create new versions without blocking anyone. This multi-version concurrency (MVCC) is how Delta gives readers and writers isolation without locks.
Readers and writers never block
Delta is multi-version: a commit produces a new set of files and a new log version, never mutating existing files. A query pins the latest version at start (snapshot isolation) and reads it consistently even as new commits land — the basis of time travel too. Writers use optimistic concurrency: they assume no conflict, validate at commit, and the loser retries against the new snapshot.
Scenario
Behavior
RBAC vs ABAC & dynamic masking
Role-based access grants through roles you assign; attribute-based (ABAC) grants through tags/attributes on the data and the user, so one policy covers everything tagged. Dynamic masking redacts sensitive columns at query time based on who is asking.
How access scales
RBAC grants privileges to roles, then roles to users — simple, but policy count explodes as data grows. ABAC (attribute-based) writes one rule against tags/attributes (e.g., “anyone without the PII-clearance attribute cannot read columns tagged PII”) that applies everywhere automatically. Dynamic masking (via row filters / column masks) returns the same query with sensitive values redacted for unauthorized users instead of denying the whole query.
Model
Means
The lakehouse engine, one stage at a time
Two journeys in one loop. Press Next to advance a write through Delta — read snapshot, write files + deletion vector, optimistic-concurrency commit, append a new log version — then a query through Spark: Catalyst plan, data-skipping file pruning, Photon vectorized scan, AQE shuffle, result. Watch the version grow, the files prune, and the answer return.
The CDC journey, one stage at a time
A single row’s change, from source to served. Press Next to advance one stage and watch it commit in the source, get captured from the log (a DELETE becomes a tombstone), land in bronze, get ordered + deduped, MERGE into the current-state silver table, update the SCD2 history, report CDC lag, and serve. The key cycles INSERT → UPDATE → DELETE across triggers.
The query optimizer, one step at a time
One SQL statement, walked through Catalyst. Press Next to advance a stage and watch the plan tree change: parse to an unresolved tree, analyze to resolve columns, optimize with rule-based rewrites (the filter slides down to the scan, columns are pruned), cost-based join selection (the small table broadcasts), then a physical plan of real operators, and finally execute.
The medallion, one layer at a time
One dataset, promoted up the layers. Press Next to watch it land raw and messy, get ingested as-is into bronze (full fidelity + metadata), be cleaned and conformed in silver (schema, dedup, validation, key resolution), aggregated into gold business marts, and finally served. The record changes shape each layer while the row count settles and data quality climbs.
The state store, one stage at a time
How a streaming aggregation remembers. Press Next to read a micro-batch, look up each key’s state, update the running aggregate, advance the watermark, evict keys that have fallen behind it (so state stays bounded), checkpoint incrementally to durable storage, and emit. A key that stops sending events goes stale and is evicted — watch it disappear.
The adaptive optimizer, one step at a time
A plan that learns from reality. Press Next to build a static plan from estimates, run stage 1 and collect the actual statistics, then let AQE re-plan: switch a sort-merge join to a broadcast when a side is actually tiny, coalesce the 200 default shuffle partitions to a handful, and split a skewed partition. Then run stage 2 on the adapted plan.
The cluster, one stage at a time
How a job becomes work on machines. Press Next to submit a job, let the driver plan stages and tasks, autoscale the executor pool up to fit, schedule tasks into slots, run stage 1 in waves, shuffle between stages, run stage 2, then autoscale down idle executors. Watch executors appear, slots fill with tasks, and the pool shrink when the work drains.
The time machine, one commit at a time
How time travel works and where it ends. Press Next to make three commits (each a new version, old files untouched), time-travel to version 1, run OPTIMIZE to compact small files (originals become unreferenced), VACUUM to delete unreferenced files past the retention window, and then watch a time-travel read of version 1 fail — its files are gone.
The recovery, one step at a time
How a streaming job survives a crash. Press Next through a job running, a crash mid-batch, the restart, reading the checkpoint, restoring state, replaying the uncommitted batch, dedup via idempotent commit, and resuming — exactly-once.
The MERGE, one step at a time
How an upsert really runs. Press Next to line up inputs (source changes + target), join on the key, classify rows as update / insert / delete, plan which files hold matched rows, rewrite just those files (copy-on-write), and commit a new version with an OCC check.
The Photon engine, one step at a time
Why columnar + SIMD wins. Press Next to plan the operators and check Photon eligibility, read data as column batches, vectorize filter/project over whole vectors, run a vectorized aggregate, compare throughput to row-at-a-time, and hit a fallback when an expression isn’t supported.
The liquid clustering, one step at a time
How clustering enables data skipping. Press Next to choose clustering keys, ingest data clustered by them, collect per-file min/max stats, run a query with a predicate, skip files whose range cannot match, and recluster new data incrementally.
The catalog, one check at a time
How governance runs on every query. Press Next to resolve the three-level namespace, receive the request, authenticate the identity, check grants down the hierarchy, apply column/row policy, capture lineage + audit, and serve a masked result.
The Auto Loader, one file at a time
How new files are ingested at scale. Press Next as a file notification arrives, lands on a queue, the loader reads it, infers + evolves the schema (rescuing surprises), loads to Delta, and records a checkpoint so the file is processed exactly once.