Databricks & Snowflake · Chapter 2 of 16
Files, columns and compression
Why analytics runs on Parquet. The single idea that makes everything above it possible.
Everything in this book sits on top of one decision: store data by column rather than by row.
What Parquet actually is
- Columnar, on disk. Values for one column are stored contiguously, so reading one column is one sequential read.
- Chunked into row groups. A file is split into row groups — typically 128MB — and within each, into column chunks. This is what allows parallel reading.
- Self-describing. The schema is in the file footer, so a reader needs no external catalogue to parse it.
- Statistics per chunk. Min, max and null count for each column in each row group. This is the important one: a query filtering
WHERE date = '2026-08-01'can skip an entire row group whose max date is earlier, without reading it. That is called predicate pushdown and it is where most of the speed comes from. - Encoded before compressed. Dictionary encoding replaces repeated values with small integers; run-length encoding collapses repeats. Then general compression on top. A region column with twelve distinct values across a billion rows becomes almost nothing.
The practical consequences you will meet
| Symptom | Cause | Fix |
|---|---|---|
| Queries slow despite small result | Reading every row group because statistics cannot help | Sort or cluster the data by the column you filter on |
| Thousands of tiny files | Frequent small writes, each producing files | Compaction — rewrite into fewer, larger files |
| One file, no parallelism | A single large file or a non-splittable compression codec | Target ~128MB–1GB files; use Snappy or ZSTD rather than plain gzip on a whole file |
| Wide table, slow scans | Selecting * when you need four columns | Select the columns you need — with columnar this is a real saving, unlike in a row store |
The habit that makes you faster than most people
Filter on a column the data is physically ordered by. Partitioning and clustering exist so that statistics can eliminate files before reading them. A query that filters on an unordered column has to read everything, no matter how good the engine is.
That single principle explains most of the performance advice in chapters 7 and 8, on both platforms.
Why columnar storage changes the arithmetic
A row-oriented file keeps each record's fields together, so reading two columns of a forty-column table still reads every field of every row. A columnar file groups each column separately. The same query now reads two fortieths of the bytes, and every other property of an analytics engine follows from that one change.
- Compression gets dramatically better, because a column holds values of one type that are often similar. A column of country codes compresses to almost nothing; the same values scattered through rows do not.
- Encoding beats generic compression. Dictionary encoding replaces repeated strings with small integers; run-length encoding collapses repeated runs; delta encoding stores differences rather than values. These are chosen per column, per chunk, based on what the data actually looks like.
- Statistics per chunk enable skipping. Each row group stores the minimum and maximum of each column, so a query filtering on a date can discard whole chunks without decompressing them — predicate pushdown. This is where most of the practical speed comes from, and it is why sort order matters so much.
- Which is the point most people miss: min/max skipping only works if the values are clustered. On a column sorted by date, a date filter skips almost everything. On the same column shuffled randomly, every chunk's range covers the whole span, nothing can be skipped, and the statistics are useless. Sorting on your common filter column is often a larger win than any amount of extra compute.
The small-file problem, and why it is the most common performance bug
Every file carries fixed overhead: an open, a metadata read, a footer parse, a scheduling decision. On one large file that is nothing. On fifty thousand small ones it is the entire query.
The symptom is unmistakable once you know it: planning time exceeds execution time. The engine spends longer working out what to read than reading it.
The cause is almost always streaming or frequent writes — a job committing every few minutes produces a file every few minutes, forever, and nothing cleans up after it.
The fix is compaction, and the target is a file size in the hundreds of megabytes rather than kilobytes. Both major platforms can do it automatically and it is frequently not switched on.
And the thing that makes this bite: no catalog will tell you a table has this problem. The report that reaches you is ‘queries got slower and nobody changed anything’ — which is true, and the table degraded on its own.
One more property worth knowing, because it separates a format from a file: a columnar file is immutable. You cannot update a row in place. That is why table formats exist on top — to give you updates, deletes and transactions over a pile of immutable files, by tracking which files and which rows are current. Understanding that the format underneath cannot change explains most of the design above it.
Check yourself