Data Platform & Analytics · Chapter 16 of 16
Table formats, catalogs and the credential fight
The format war ended, a different war replaced it, and the thing no catalog will tell you is whether your table is actually healthy.
The open table format debate is over in a way that is genuinely unusual in this industry: one format won, and its main competitor's vendor adopted it. What replaced the argument is a fight one layer up, and it is about something more specific than it appears.
What the winning format actually gives you
A table format turns a directory of files in object storage into something with database semantics: atomic commits, snapshot isolation, time travel, schema evolution without rewriting data, and partition layouts that can change without breaking existing queries.
Its latest specification version reached general availability in the first half of 2026 and the three additions are all worth knowing:
- Deletion vectors — a much more efficient representation of merge-on-read deletes than the previous positional-delete files, which reduces the read amplification that made merge-on-read painful.
- Row lineage — row-level tracking that makes genuine incremental consumption possible rather than approximated with timestamps.
- A variant type for semi-structured data, so JSON-shaped columns get proper typed treatment rather than being stored as strings and parsed on every read.
A note on the competing format, because the honest picture is not the one either camp presents: it appears in a low single-digit percentage of job postings, which makes it look marginal as a hiring keyword. But it is the storage layer underneath one of the two dominant lakehouse platforms, so it is architecturally central for an enormous number of teams who simply never name it. Low keyword frequency is not low importance, and conflating the two is a common analysis error when reading job-posting data — including the job-posting data this site is built from.
The catalog war, and what it is actually about
The catalog fight is a fight over who holds the storage credentials. That one sentence explains the whole thing.
A modern catalog does not just list tables. It vends short-lived, scoped credentials to query engines — so the catalog, not the storage bucket policy, is where access control is genuinely enforced.
Which means whoever owns the catalog owns authorisation for the entire lakehouse, regardless of which engine runs the query. That is a far stronger position than owning a metadata listing, and it is why every vendor is fighting for this layer specifically.
One open catalog implementation graduated to top-level status at its foundation in early 2026 — genuinely vendor-neutral governance, credential vending, and a specification-first design.
The competing implementation from the other major vendor is open-sourced, but be precise: the open-source build trails the managed service in capability. Both sentences are true at once — ‘it is open source’ and ‘the open version is not the product people actually buy’ — and saying only the first is the most common piece of vendor-shaped teaching in this area.
There is also a genuinely interesting third design worth mentioning because it reframes the problem: put the table metadata in an ordinary SQL database rather than in JSON and Avro manifest files on object storage. Manifest-based metadata means many small reads and a commit-conflict story that gets awkward under concurrency; a transactional database handles both natively. It reached a stable version in 2026 and is not yet a mainstream choice, but the critique is sound and worth being able to articulate.
The thing no catalog tells you
Table maintenance is the unowned problem
No catalog tells you whether a table is healthy. Not one.
An open-format table degrades in specific, predictable ways: too many small files from frequent streaming commits, accumulated delete files under merge-on-read that make reads progressively slower, stale snapshots and orphaned files quietly paying storage rent, and metadata bloat that makes planning slow before scanning even starts.
The operations that fix these — compaction, delete-file rewriting, snapshot expiry, orphan cleanup — all exist and are well documented. What is missing is anything that tells you when to run them.
So it lands on the platform team as unglamorous, unscheduled, invisible work, and the symptom presented to you is never ‘the table needs compaction’. It is ‘queries got slower and nobody changed anything’.
Being the person who recognises that symptom is a genuinely differentiating piece of knowledge in a platform interview, because it is operational experience that cannot be acquired from documentation.
What the consolidation means for how you choose
The composable-stack idea — assemble best-of-breed tools, keep every layer swappable — was the organising principle of data platform work for about six years. Between 2024 and 2026 the vendor landscape underneath it collapsed into a handful of acquirers.
But separate the architecture from the vendors, because they moved in opposite directions. Separating storage, compute, transformation and serving is still correct and is more achievable now than it was, precisely because the formats and protocols got more open in the same window: a permissive table format won, a catalog specification became a neutral standard, and a major engine was relicensed toward Apache 2.0 rather than away from it.
What died is the vendor-neutral composable middle. The independent tool for each layer was mostly acquired. So the practical posture is: choose vendors knowing consolidation is the base case, and buy portability at the format and specification layer rather than at the vendor layer. An open table format with a specification-governed catalog survives your vendor being acquired. A proprietary storage layer does not, no matter how good the current relationship is.
Check yourself