Databricks & Snowflake · Chapter 8 of 10
Cost control, which is the skill people get hired for
Half of data-platform postings name cost. This chapter is the practical part.
Both platforms are consumption-priced, which means a single badly-written query or a forgotten cluster can cost more than an engineer's week. Cost control is not finance work here — it is engineering work, and it is one of the most reliably valued things you can bring.
Where the money actually goes
| Cause | Databricks | Snowflake |
|---|---|---|
| Idle compute | All-purpose clusters left running | Warehouses with long auto-suspend |
| Oversizing | Big clusters on shuffle-bound jobs | XL warehouses for small queries |
| Full recompute | Full refresh where incremental would do | Same — full table rebuilds nightly |
| Unpruned scans | No partitioning or clustering | No clustering key on a filtered column |
| Dashboard refreshes | Scheduled jobs nobody reads | Hourly refreshes on weekly dashboards |
| Retained history | Old Delta versions never vacuumed | Long Time Travel plus Fail-safe |
| Dev environments | Copies of production data | Copies rather than zero-copy clones |
The sequence that works
- Attribution before optimisation. Tag clusters and warehouses by team. Until spend is attributable nobody owns it and every conversation is about the total. This is the step people skip and it is the one that unlocks the rest.
- Find the top ten consumers. Spend is always concentrated — a handful of jobs or queries account for most of it. Both platforms expose usage views for this.
- Kill the free wins first. Auto-suspend, auto-termination, dropping unused dashboards' schedules, and vacuuming or expiring history. No engineering effort and often a double-digit percentage.
- Then incremental. Converting the biggest full refresh to incremental is usually the single largest saving available.
- Then pruning. Clustering the most-scanned table on the most-filtered column.
- Then commitments — and deliberately last, because buying a multi-year commitment on an architecture you are about to change locks in waste.
The number to report
Cost per unit of business value — per active user, per pipeline run, per resolved query — and its trend. Aggregate spend rising while cost per unit falls is a healthy, growing platform. Aggregate flat while cost per unit rises is a problem, and the total hides it in both directions.
Then publish it per team, monthly, and let them see their own number. They know which pipelines are safe to turn off and you do not.