Databricks & Snowflake · Chapter 5 of 10
Databricks in practice
Workspaces, clusters, jobs, Delta Live Tables and Unity Catalog — what each is and when it bites.
Databricks is a workspace over Spark, plus a set of managed capabilities around it. Knowing what each piece is for prevents the most common mistake, which is using notebooks for everything.
Compute: the thing that costs you money
| Type | What it is | Use it for |
|---|---|---|
| All-purpose cluster | Interactive, shared, stays up | Development and exploration. Expensive if left running — set auto-termination |
| Job cluster | Created for one job, destroyed after | Every scheduled workload. Cheaper and isolated |
| SQL warehouse | Optimised for BI-style SQL, serverless option | Dashboards and analyst queries |
| Serverless | No cluster to manage, starts in seconds | Removes idle cost and cold starts; less control |
The single biggest cost mistake is running production jobs on all-purpose clusters and leaving them up. The second is oversizing: a bigger cluster does not help a job bounded by a shuffle or by skew, and you pay for the idle cores.
Delta Live Tables and the declarative model
DLT lets you declare tables and the transformations between them, and the platform works out dependencies, orchestration, retries and data-quality expectations. It removes a lot of orchestration code. The trade is less control and a proprietary layer — a reasonable choice when the pipeline is genuinely declarative, a poor one when you need custom behaviour.
Unity Catalog, which is the governance answer
- Three-level namespace — catalog, schema, table — consistent across workspaces.
- Central grants, including row filters and column masks, instead of per-workspace access control.
- Lineage automatically, table and column level, which is worth more than it sounds when an incident happens.
- It is where the governance story lives, so ‘how do you control access?’ on Databricks is a Unity Catalog answer.
What people get wrong
Notebooks as production. Notebooks are for exploration. Production belongs in version-controlled code with tests, run as jobs. A pipeline that only runs when someone opens a notebook is not a pipeline.
collect() on a large DataFrame. Pulls everything to the driver and kills it. The driver is a single machine.
Ignoring skew until it is a production incident. Check the Spark UI for one task far outlasting its stage — that is the signature, and salting or a broadcast join is usually the fix.
Leaving auto-termination off. The most common line item on a surprising bill.