Databricks Mastery Path¶
Databricks is Apache Spark and Delta Lake plus a lot of platform: a governance layer (Unity Catalog), compute you do or don't manage,
pipelines and jobs, SQL warehouses, ML tooling and AI features. It's easy to click through and hard to know what is actually
happening underneath. A one-row DELETE that rewrites 49,999 other rows. A partition column that turns 2 MB into 86,665 files and a
seven-minute write. A deleted checkpoint that duplicates every record. An INSERT * merge that silently drops a new column. An ALTER TABLE ... CLUSTER BY that rewrites nothing. A window function whose result changes with tied sort keys. None of these are
platform bugs. They follow from how Delta and Spark work, and this course shows each one with something you can run.
It assumes you know SQL and some Python. It doesn't assume any Databricks experience, and most of Levels 1, 2 and 4 runs on your own laptop with open-source Spark and Delta Lake.
Level 1 is the foundation: the lakehouse and its control and compute planes, compute types and notebooks, Spark DataFrames and SQL,
reading and writing files, Delta tables, the Delta log, time travel and VACUUM, tables, views and volumes, ending with a local
sales lakehouse. Level 2 is Delta and data engineering: MERGE and the change feed, schema enforcement, OPTIMIZE and clustering,
partitioning, how Spark executes and why joins skew, Structured Streaming, Auto Loader and the medallion architecture, ending with an
incremental bronze/silver/gold pipeline. Level 3 is the platform: Unity Catalog's object model, privileges, storage governance, row
filters and masks, lineage and system tables, declarative pipelines, jobs, OpenSharing and SQL warehouses, ending with a governance-as-code
project. Level 4 is building and operating: bundles and CI/CD, testing, MLflow and model serving, AI features, cost, performance,
table maintenance and migration, ending with a platform plan and checker.
Every lesson has a How It Actually Works section: why Delta commits are atomic, why a file can be vacuumed out from under a rollback, why Z-order can cut files read from 16 to 2 on one column and not another, why AQE can fix partition counts but didn't fix a hot key on a small test, and why a deletion vector hides a row without erasing it.
Versions and what was actually run
This course was written without access to a Databricks workspace, and no output is presented as coming from Databricks.
What was run, and is labelled in each lesson: Apache Spark 4.2.0 and Delta Lake 4.4.1 (open source) on a laptop for the Spark
and Delta behaviour and the local pipeline project; the open-source Spark Declarative Pipelines runtime that ships with PySpark;
MLflow 3.17.2, scikit-learn, pytest, sqlglot 30.24, and the delta-sharing and databricks-sql-connector clients (API checks
only). Unity Catalog, jobs, SQL warehouses, Auto Loader, OpenSharing, Model Serving, AI Functions, Genie, Photon, serverless compute and
Predictive Optimization could not be run; those lessons are built from the Databricks documentation as of October 2026 and say
so, and where a small Python or Spark program models documented behaviour, it is labelled as a model and not as Databricks code.
The cost lesson uses invented mock billing rows. Where a local experiment failed or showed nothing (a caching micro-benchmark, a hung
Python UDF timing), the lesson reports that instead of inventing results.
Databricks renames and changes things often: Delta Live Tables became Lakeflow Spark Declarative Pipelines, Asset Bundles became Declarative Automation Bundles, Delta Sharing is now documented as OpenSharing, Genie Spaces became Genie Agents and Vector Search became AI Search. The lessons say which name is current. Re-check anything marked "as of October 2026" against your own workspace.
How the program is organized¶
| Level | Focus | Modules | What you can do after |
|---|---|---|---|
| 1 · Lakehouse & Spark Foundations | Architecture, compute, Spark, files, Delta, the log, time travel, tables and volumes | 10 | Build, change and recover a small Delta lakehouse |
| 2 · Delta Lake & Data Engineering | MERGE, schema, clustering, partitioning, execution and skew, streaming, Auto Loader, medallion | 10 | Build incremental, idempotent pipelines and make them fast |
| 3 · Unity Catalog, Pipelines & Sharing | Governance, privileges, storage, masks, lineage, declarative pipelines, jobs, sharing, SQL | 10 | Design and operate a governed, orchestrated platform |
| 4 · Development, ML & Operations | Bundles, testing, MLflow, serving, AI features, cost, performance, maintenance, migration | 10 | Ship, test, monitor and afford it, and plan the whole thing |
Each level has 9 teaching modules plus a hands-on project (module 10), for 40 lessons total.
How to use this site¶
- Run the local labs. You need Python 3.12 and Java 17 or newer.
pip install pyspark delta-sparkis enough for Levels 1 and 2. SetPYSPARK_PYTHONto the same interpreter, or Spark will stop with a version-mismatch error (one of the things that happened while writing this course). - Get a Free Edition workspace for the platform-only lessons. It's a serverless, quota-limited, no-cost workspace for learning; check its current limitations before relying on a feature.
- Read the "what was run" note at the top of each lesson. It tells you whether the output you see is observed, modelled or documented.
- Do the exercises, especially those that say "predict, then run". The surprises are the point.
- Do the projects. Each combines its level into something you could adapt, and each states what it doesn't cover.
More from the Mastery Path series¶
Free, structured, module-wise training across 97 other languages, platforms and disciplines — expand a category to browse: