Lakshya

Databricks & Snowflake · Chapter 3 of 10

Delta and Iceberg — how files become a table

The mechanism that makes a lakehouse possible. Understand this and both products stop being magic.

3 min read1 diagramAll 10 chapters

Object storage has no transactions. You cannot edit a file in place, you cannot update two files atomically, and two writers can overwrite each other. A table format solves all of that with one idea: the data files are immutable, and a separate log records which files constitute the table right now.

WHAT A “TABLE” ON OBJECT STORAGE ACTUALLY IS OBJECT STORAGE — S3 / ADLS / GCS part-0.parquetpart-1.parquetpart-2.parquetpart-3.parquet part-4.parquetpart-5.parquetpart-6.parquetpart-7.parquet immutable files. never edited — only added or superseded THE METADATA LOG v1: files 0-3 · schema Av2: + files 4-5v3: - file 1, + file 6v4: + file 7 · schema B an ordered list of which files ARE the table, now what you get atomic commitstime travelschema change Delta and Iceberg are both this idea. The files are ordinary Parquet; the table is the log. A write appends new files and then appends one entry to the log. Until that entry lands, readers see the old version — which is how you get atomicity on storage that has no transactions. Deleting a row does not edit a file. It writes a new file without that row and supersedes the old one, which is why small frequent writes create thousands of tiny files and why compaction exists.
The files are ordinary Parquet; the table is the log. A write adds new files and then appends one entry saying which files are now current. Until that entry lands, readers see the previous version — which is how you get atomicity on storage that has none.

What that buys you, and why each matters

  • Atomic commits. A job that fails halfway leaves orphaned files that nobody is pointed at, rather than a half-written table. This alone fixes the worst data-lake failure.
  • Time travel. Older log entries still describe a valid set of files, so you can query the table as of last Tuesday, or restore it. Invaluable when a bad pipeline run corrupts something.
  • Schema evolution. The log records schema per version, so adding a column is a metadata operation rather than a rewrite.
  • Updates and deletes. Deleting a row writes a new file without it and supersedes the old one. This is what makes GDPR erasure possible on a lake at all.
  • Concurrency. Two writers can proceed and the log resolves conflicts — usually optimistically, so one may need to retry.

Delta and Iceberg, compared honestly

Delta LakeApache Iceberg
OriginDatabricksNetflix, then Apache
Log designAn ordered set of JSON commit files plus periodic checkpointsA tree of metadata files with manifest lists
StrengthDeep Databricks integration; very mature toolingEngine-neutral by design; strong partition evolution
Partition changesRequires careHidden partitioning — you can change the scheme without rewriting or changing queries
Who reads itSpark, and increasingly othersSpark, Trino, Flink, Snowflake, BigQuery, DuckDB

Both are now widely interoperable and the industry has largely stopped fighting about it. For a candidate the useful position is: they solve the same problem the same way, Iceberg was designed to be engine-neutral, and interoperability is what makes running two engines over one storage layer realistic.

The operational realities nobody mentions

  • Small files are the recurring problem. Streaming writes every few seconds create thousands of tiny files, and query planning slows down. Compaction is a scheduled job you will end up owning.
  • Old versions cost money. Time travel means superseded files are retained. Vacuum or expire-snapshots reclaims them, and forgetting to run it is a common cost surprise.
  • Deletes can be logical. Merge-on-read writes a delete file rather than rewriting data, which makes the write fast and every subsequent read slower until compaction.
  • Concurrent writers conflict. Optimistic concurrency means one writer retries — fine at low rates, painful if many jobs write the same table.
← Files, columns and compressionHow a query actually executes →