Lakshya

Databricks & Snowflake · Chapter 2 of 10

Files, columns and compression

Why analytics runs on Parquet. The single idea that makes everything above it possible.

3 min read1 diagramAll 10 chapters

Everything in this book sits on top of one decision: store data by column rather than by row.

SELECT region, amount FROM orders — A TABLE WITH 40 COLUMNS ROW STORAGE — records stored whole row 1row 2row 3row 4row 5row 6row 7row 8 every row read in full — 40 columns fetched to answer a 2-column question COLUMNAR — each column stored together regionamountcust_iddateskuqtytax… only two column files opened — the other 38 are never touched This is why analytics runs on columnar files and transactions do not. Columnar also compresses far better, because a column holds one type and often repeats — a region column of a billion rows might hold twelve distinct values, which encodes to almost nothing.
A query that needs two columns reads two column files, not every row. On a 40-column table that is a twentyfold reduction in bytes read before any other optimisation. Columnar also compresses far better, because a column holds one type and often repeats.

What Parquet actually is

  • Columnar, on disk. Values for one column are stored contiguously, so reading one column is one sequential read.
  • Chunked into row groups. A file is split into row groups — typically 128MB — and within each, into column chunks. This is what allows parallel reading.
  • Self-describing. The schema is in the file footer, so a reader needs no external catalogue to parse it.
  • Statistics per chunk. Min, max and null count for each column in each row group. This is the important one: a query filtering WHERE date = '2026-08-01' can skip an entire row group whose max date is earlier, without reading it. That is called predicate pushdown and it is where most of the speed comes from.
  • Encoded before compressed. Dictionary encoding replaces repeated values with small integers; run-length encoding collapses repeats. Then general compression on top. A region column with twelve distinct values across a billion rows becomes almost nothing.

The practical consequences you will meet

SymptomCauseFix
Queries slow despite small resultReading every row group because statistics cannot helpSort or cluster the data by the column you filter on
Thousands of tiny filesFrequent small writes, each producing filesCompaction — rewrite into fewer, larger files
One file, no parallelismA single large file or a non-splittable compression codecTarget ~128MB–1GB files; use Snappy or ZSTD rather than plain gzip on a whole file
Wide table, slow scansSelecting * when you need four columnsSelect the columns you need — with columnar this is a real saving, unlike in a row store

The habit that makes you faster than most people

Filter on a column the data is physically ordered by. Partitioning and clustering exist so that statistics can eliminate files before reading them. A query that filters on an unordered column has to read everything, no matter how good the engine is.

That single principle explains most of the performance advice in chapters 7 and 8, on both platforms.

← The problem both are solvingDelta and Iceberg — how files become a table →