Skip to content

Level 1 · Lakehouse & Spark Foundations

Level 1 takes you from "Databricks is a Spark platform" to building and recovering a small Delta lakehouse. Most of it runs on your own laptop with open-source Spark and Delta Lake, which means the code is real and the outputs in these lessons are observed, not paraphrased. Along the way it shows the quiet failures that cost people days: inferSchema turning a numeric column into strings, a CASE expression labelling missing amounts "small", a window function that returns different running totals for tied rows, and a VACUUM that makes a rollback impossible.

What was run, and what wasn't

This course was written without access to a Databricks workspace. Spark 4.2.0 and Delta Lake 4.4.1 (open source, from PyPI) were run locally for everything that doesn't need the platform; those outputs are labelled "local". Platform-only features (Unity Catalog, serverless compute, SQL warehouses, Auto Loader, Lakeflow, MLflow serving) are described from the Databricks documentation as of October 2026 and are marked as not run. A Databricks Free Edition workspace is enough to try the platform-only exercises.

Prerequisites: basic Python and SQL. Java 17+ and Python 3.12 to run the local labs. If SQL is new, start with the SQL Mastery Path; for Spark in more depth see the PySpark Mastery Path.

Modules

  1. The Lakehouse: Control Plane, Compute Plane & Storage: what a lakehouse is, the two planes, accounts, workspaces and metastores.
  2. Workspaces, Compute Types & Notebooks: serverless vs classic vs SQL warehouses, notebooks, dbutils, the CLI.
  3. Spark DataFrames: Transformations & Actions: lazy plans, explain, null semantics, joins.
  4. Spark SQL, Built-in Functions & Window Functions: GROUP BY ALL, windows and ties, QUALIFY, ANSI mode.
  5. Reading & Writing Files: CSV/JSON/Parquet, parse modes, partition pruning, save modes.
  6. Delta Lake Tables: create, update, delete, inspect, managed vs external.
  7. Inside the Delta Log: actions, statistics, checkpoints, a real write conflict.
  8. Time Travel, RESTORE & VACUUM: what you can recover and when it's gone.
  9. Tables, Views & Volumes: the object model and where files belong.
  10. Project: A Sales Lakehouse You Can Run Locally: bronze/silver/gold with a quarantine table and a rollback.

After Level 1

Level 2 makes tables fast and pipelines continuous: MERGE, schema enforcement, OPTIMIZE and clustering, how Spark executes queries, joins and skew, Structured Streaming, Auto Loader and the medallion architecture.