Before either product makes sense you need the shape of the problem they were built for. This chapter assumes nothing beyond knowing what a table is.

An operational database — the one behind an application — is optimised for reading and writing single rows quickly and safely. Analytics wants the opposite: scan a billion rows, touch two columns, and never mind if it takes a minute. Running both on one system means analytics queries lock tables and slow the application, which is why the first move in every organisation is to copy data somewhere else.

A warehouse is a database purpose-built for that scan-heavy workload — columnar storage, compression, and a query planner that assumes big aggregations. Teradata, Netezza, then Redshift and BigQuery. It works well and it has two structural problems. It is schema-on-write: you must model the data before you can load it, so anything unstructured or fast-changing is awkward. And storage and compute are coupled, so to store more you buy more machines whether or not you need the compute.

Object storage — S3 and its equivalents — made storage effectively unlimited and very cheap. So: put every file in a bucket, schema-on-read, figure it out later. That solved cost and flexibility and created a new problem, which is that a directory of files is not a table. No transactions, so a failed job leaves half-written data. No schema enforcement, so a producer changes a field and every reader breaks. No way to update or delete a single row, which matters enormously once privacy regulation requires erasure. The phrase ‘data swamp’ came from this era and it was fair.

The lakehouse is one specific idea: keep the cheap object storage, and add a metadata layer that makes a set of files behave like a table — atomic writes, schema enforcement, updates and deletes, and history. That layer is a table format, and Delta Lake and Apache Iceberg are the two that matter. Chapter 3 is entirely about how that works, because it is the single most important mechanism in this book.