Databricks & Snowflake · Chapter 8 of 16
Cost control, which is the skill people get hired for
Half of data-platform postings name cost. This chapter is the practical part.
Both platforms are consumption-priced, which means a single badly-written query or a forgotten cluster can cost more than an engineer's week. Cost control is not finance work here — it is engineering work, and it is one of the most reliably valued things you can bring.
Where the money actually goes
| Cause | Databricks | Snowflake |
|---|---|---|
| Idle compute | All-purpose clusters left running | Warehouses with long auto-suspend |
| Oversizing | Big clusters on shuffle-bound jobs | XL warehouses for small queries |
| Full recompute | Full refresh where incremental would do | Same — full table rebuilds nightly |
| Unpruned scans | No partitioning or clustering | No clustering key on a filtered column |
| Dashboard refreshes | Scheduled jobs nobody reads | Hourly refreshes on weekly dashboards |
| Retained history | Old Delta versions never vacuumed | Long Time Travel plus Fail-safe |
| Dev environments | Copies of production data | Copies rather than zero-copy clones |
The sequence that works
- Attribution before optimisation. Tag clusters and warehouses by team. Until spend is attributable nobody owns it and every conversation is about the total. This is the step people skip and it is the one that unlocks the rest.
- Find the top ten consumers. Spend is always concentrated — a handful of jobs or queries account for most of it. Both platforms expose usage views for this.
- Kill the free wins first. Auto-suspend, auto-termination, dropping unused dashboards' schedules, and vacuuming or expiring history. No engineering effort and often a double-digit percentage.
- Then incremental. Converting the biggest full refresh to incremental is usually the single largest saving available.
- Then pruning. Clustering the most-scanned table on the most-filtered column.
- Then commitments — and deliberately last, because buying a multi-year commitment on an architecture you are about to change locks in waste.
The number to report
Cost per unit of business value — per active user, per pipeline run, per resolved query — and its trend. Aggregate spend rising while cost per unit falls is a healthy, growing platform. Aggregate flat while cost per unit rises is a problem, and the total hides it in both directions.
Then publish it per team, monthly, and let them see their own number. They know which pipelines are safe to turn off and you do not.
Where the money actually goes, and the meters people miss
- Idle compute is the largest avoidable line in most accounts. A warehouse or cluster that is running and doing nothing bills at the same rate as one doing work. Auto-suspend is the fix, and it has a real counter-argument: suspending too aggressively drops the local cache, so the next query pays to re-read data it would otherwise have had. Tune the suspend interval to the query arrival pattern, not to a round number.
- A faster engine can cost more per query while being cheaper per workload, and the confusion is worth resolving explicitly: when the per-unit price is unchanged but the faster runtime emits more units per second, the bill can rise even though wall-clock time falls. Whether it is worth it depends entirely on your workload, and it cannot be settled by looking at the rate card.
- Silent fallback is the version of that which actually hurts: a query using an unsupported function drops back to the slower engine while still billing at the faster meter. Nothing errors. The only symptom is cost.
- Billing minimums are the surprise for short queries. Different resource types carry different minimum charges — some measured in a minute, some in an hour. A workload of many short queries against a resource with a long minimum is the worst possible match, and it is invisible until the invoice.
- Storage is rarely the problem; movement usually is. Cross-region reads and egress appear on a different line from the one anyone looks at.
Attribute before you optimise
An unattributed bill is a shared problem, and shared problems are nobody's.
The same bill split by team and published monthly produces reductions that no optimisation project achieves — because the person who can delete the unused hourly-refreshing dashboard is the only one who knows it is unused, and they currently have no reason to look.
The mechanism is tags or separate warehouses per team, and the work is organisational: agreeing what a team is and getting the tag applied consistently. That is genuinely most of the effort, and it is worth more than any query rewrite.
Then make cost a review criterion in the same sentence as correctness. A query costing a hundred times what it should is a defect, and it is never treated as one because nobody sees the number at review time.
And measure the right thing: total spend grows with the business and tells you nothing. Cost per unit of work — per active user, per pipeline run, per report served — tells you whether the platform is getting more or less efficient, which is what anyone is actually asking when they ask about cost.
Check yourself