Lakshya

Databricks & Snowflake · Chapter 4 of 10

How a query actually executes

Spark's model and Snowflake's, side by side. The mental model that makes performance work intelligible.

3 min read0 diagramsAll 10 chapters

Both engines do the same broad thing — read columnar files in parallel and combine results — but the unit you control differs, and that is what makes them feel different.

Spark, which is what Databricks runs

  • You write a transformation, not a query plan. Operations are lazy: nothing runs until an action (write, count, collect) forces it.
  • The plan splits into stages at shuffle boundaries. A shuffle is any operation that must move data between machines — a join on a non-collocated key, a groupBy, a repartition. Shuffles are where time goes.
  • Each stage runs as tasks, one per partition. Parallelism is bounded by partitions and by executor cores, so too few partitions leaves cores idle and too many creates scheduling overhead.
  • Skew is the classic failure. If one key holds most of the rows, one task does most of the work and the stage takes as long as that task. You see it as 199 tasks finishing in seconds and one running for an hour.
  • Broadcast joins avoid the shuffle entirely by sending a small table to every executor. Getting a join broadcast rather than shuffled is often a ten-times difference.

Snowflake, which hides more

  • You write SQL and the optimiser plans it. Far less to tune, and less to control when you need to.
  • Data lives in micro-partitions — immutable compressed chunks of roughly 50–500MB uncompressed, created automatically, each with metadata about the values it holds.
  • Pruning is the whole game. The optimiser uses that metadata to skip micro-partitions entirely. A well-clustered table on a filtered column can skip 99% of them; a badly clustered one skips none and scans everything.
  • A virtual warehouse is just compute, sized T-shirt style, and doubling the size roughly doubles both throughput and cost per second. Multi-cluster warehouses add more clusters for concurrency rather than for single-query speed — a distinction people routinely get wrong.
  • Result and metadata caching are aggressive. An identical query with unchanged data can return from cache in milliseconds and cost nothing, which makes naive benchmarks meaningless.

The one-sentence contrast worth carrying

On Databricks you tune the job; on Snowflake you tune the data.

Databricks gives you cluster sizing, partitioning, join strategy and caching to control — more power and more ways to be slow. Snowflake gives you clustering, warehouse size and query shape — less to get wrong and less to reach for when the optimiser chooses badly.

← Delta and Iceberg — how files become a tableDatabricks in practice →