Databricks & Snowflake · Chapter 5 of 16
Databricks in practice
Workspaces, clusters, jobs, Delta Live Tables and Unity Catalog — what each is and when it bites.
Databricks is a workspace over Spark, plus a set of managed capabilities around it. Knowing what each piece is for prevents the most common mistake, which is using notebooks for everything.
Compute: the thing that costs you money
| Type | What it is | Use it for |
|---|---|---|
| All-purpose cluster | Interactive, shared, stays up | Development and exploration. Expensive if left running — set auto-termination |
| Job cluster | Created for one job, destroyed after | Every scheduled workload. Cheaper and isolated |
| SQL warehouse | Optimised for BI-style SQL, serverless option | Dashboards and analyst queries |
| Serverless | No cluster to manage, starts in seconds | Removes idle cost and cold starts; less control |
The single biggest cost mistake is running production jobs on all-purpose clusters and leaving them up. The second is oversizing: a bigger cluster does not help a job bounded by a shuffle or by skew, and you pay for the idle cores.
Delta Live Tables and the declarative model
DLT lets you declare tables and the transformations between them, and the platform works out dependencies, orchestration, retries and data-quality expectations. It removes a lot of orchestration code. The trade is less control and a proprietary layer — a reasonable choice when the pipeline is genuinely declarative, a poor one when you need custom behaviour.
Unity Catalog, which is the governance answer
- Three-level namespace — catalog, schema, table — consistent across workspaces.
- Central grants, including row filters and column masks, instead of per-workspace access control.
- Lineage automatically, table and column level, which is worth more than it sounds when an incident happens.
- It is where the governance story lives, so ‘how do you control access?’ on Databricks is a Unity Catalog answer.
What people get wrong
Notebooks as production. Notebooks are for exploration. Production belongs in version-controlled code with tests, run as jobs. A pipeline that only runs when someone opens a notebook is not a pipeline.
collect() on a large DataFrame. Pulls everything to the driver and kills it. The driver is a single machine.
Ignoring skew until it is a production incident. Check the Spark UI for one task far outlasting its stage — that is the signature, and salting or a broadcast join is usually the fix.
Leaving auto-termination off. The most common line item on a surprising bill.
The pieces, and where each one bites
| Piece | What it is | Where it bites |
|---|---|---|
| Workspace | The collaborative surface — notebooks, jobs, repos | Notebooks make production code easy to write and hard to review. Anything scheduled should live in a repo, not a notebook someone can edit while it runs |
| All-purpose cluster | Interactive compute you attach a notebook to | Left running. This is the single largest avoidable cost on the platform. Set aggressive auto-termination and mean it |
| Job cluster | Created for a run, destroyed after | Cheaper per unit and adds startup latency. For short frequent jobs that latency can dominate — which is what pools exist for |
| Declarative pipelines | Declare tables and dependencies; the platform manages the flow | Genuinely good for straightforward ingest-and-transform. Awkward the moment you need control the abstraction does not expose, and you cannot partially opt out |
| The catalog | Three-level namespace, permissions, lineage, credential vending | The real reason to be on the platform. Migration onto it is a project, not a setting |
| The vectorised engine | A native execution engine under the same API | Silent fallback on an unsupported function — you keep paying the faster meter for the slower engine, and nothing errors |
The two cost mistakes that account for most surprise bills.
A cluster sized for the peak, running at the average. Somebody sized it for the worst hour of the month and it runs that way permanently. Autoscaling helps and does not fix the floor — the minimum worker count is the number that actually costs you.
An interactive cluster nobody terminated. An analyst attached a notebook on Friday afternoon and the cluster idled through the weekend. Auto-termination is the whole fix and it is routinely set to a value long enough to be useless.
The diagnostic: plot cluster uptime against actual job runtime. The gap is the bill you are paying for nothing, and it is usually far larger than anyone expects.
And the tuning order that works: check for skew first — uneven task durations within a stage — then file sizes, then partition pruning, and only then cluster shape. Resizing to fix skew adds workers that have nothing to do, which is the most common way a performance problem becomes a cost problem as well.
Check yourself