Data Platform & Analytics · Chapter 7 of 16
Cost, because half the postings name it
The warehouse bill is usually the second-largest line in a data organisation, and it is yours.
Cost appears in 50% of these postings. Consumption pricing means a single badly-written query can cost more than an engineer's week, and nobody notices until the invoice.
Where the money actually goes
- Full scans on unpartitioned tables. The most common and most fixable. Partition and cluster on what people actually filter by, which you can read from query history rather than guess.
- Recomputing what has not changed. Full refreshes where an incremental model would do. This is usually the single largest saving available.
- Dashboards on a schedule nobody needs. Hourly refreshes on a report read weekly. Check last-viewed timestamps — a meaningful share of scheduled refreshes serve dashboards nobody has opened in months.
- Idle warehouses with generous auto-suspend, and oversized ones chosen once and never revisited.
- Storage nobody owns — old snapshots, abandoned dev schemas, unpartitioned raw retained forever.
Attribution before optimisation
The blocker is almost never knowing what to optimise; it is that nobody can attribute spend to a team, so nobody owns it and every conversation is about the total.
Tag queries and warehouses by team, publish cost per team monthly, and let them see their own number. Visibility with ownership moves more spend than any central optimisation programme, because they know which pipelines are safe to turn off and you do not.
Where the money actually goes
Data platform bills are dominated by a small number of patterns, and they are rarely the ones teams look at first. Almost every team starts by trying to make queries faster; almost every large saving comes from making queries not run.
| Pattern | Why it costs | The fix |
|---|---|---|
| Full-table scans on a partitioned table | The partition column is not in the filter, or is wrapped in a function that defeats pruning | Filter on the raw partition column. WHERE date_col >= x, never WHERE cast(date_col ...) = x |
| Dashboards refreshing on a schedule nobody set | Every panel is a query, running hourly, viewed weekly | Check refresh frequency against actual view counts. This is routinely the largest single saving and takes an afternoon |
| SELECT * in a columnar warehouse | You pay per column scanned; the star is every column | Name the columns. In a wide table this is a 10× difference on the same query |
| An idle warehouse that never suspends | Billed for being available | Auto-suspend — but not too aggressively, because a cold start drops the local cache and the next query pays for it twice |
| Rebuilding everything nightly | You reprocess history that did not change | Incremental models. The most common large structural saving, and the one that takes real work |
| Cross-region or cross-zone transfer | Egress is priced separately and invisibly | Co-locate compute with storage. This is often the single biggest surprise line |
The FinOps habit that beats every optimisation
Attribute cost to a team or a product before you try to reduce it.
An unattributed bill is a shared problem, and shared problems are nobody's. The same bill split by team, published monthly, produces reductions that no optimisation project achieves — because the people who can delete the unused dashboard are the only ones who know it is unused.
The mechanism is tagging or per-warehouse separation, and the work is mostly organisational: agreeing what a team is, and getting the tags applied consistently.
Then set the expectation that cost is a review criterion, in the same sentence as correctness. A query that costs a hundred times what it should is a defect, and it never gets treated as one because nobody sees the number at review time.
And measure the right thing: total spend is a bad metric because it grows with the business. Cost per unit of work — per active user, per pipeline run, per report served — tells you whether the platform is getting more or less efficient, which is the question anyone is actually asking.
Check yourself