Data Platform & Analytics · Chapter 14 of 16
Ingestion, transformation, and who owns your stack now
Fivetran bought both sides of the dbt-versus-SQLMesh debate, one licence went more open while another went less, and the change-data-capture failure that fills a disk.
The ingestion and transformation layer consolidated faster than any other part of the data stack, and the consolidation changes what you are actually choosing between.
The consolidation, in fourteen months
Fivetran acquired Census (reverse ETL) in May 2025, then Tobiko Data — the company behind SQLMesh — in September 2025, then completed a merger with dbt Labs on 1 June 2026, and then took stewardship of Great Expectations when its commercial arm was sold to an analytics firm.
The practical consequence: almost every ‘dbt versus SQLMesh, pick a side’ article written before mid-2026 is now describing two products with the same owner. Both continue independently, and the bet has changed from ‘which project survives’ to ‘which execution model do I prefer’.
The licence reversal, and the pairing that teaches it
dbt went more open, not less.
dbt Labs had publicly committed to releasing the Fusion engine's runtime under a restrictive source-available licence. When it shipped, it was released under Apache 2.0 instead, and merged into the open-source repository. The stated reason: preventing competitive deployments was not a priority.
And in the same window Soda Core went the other way — from Apache 2.0 to the Elastic License, with the usual restrictions on offering it as a hosted service.
Teach the pairing rather than either case. Licence direction is not a one-way ratchet and it is not predictable from vendor size or funding stage. What you can do is read the actual terms, check whether prior versions stay permissively licensed, and ask whether a fork exists — which is exactly the analysis from the infrastructure-as-code chapter, applied here.
The technical substance of that new engine is worth knowing separately: a rewrite in a systems language with a parser that understands SQL semantically, giving column-level lineage, compile-time error detection before the warehouse ever runs anything, and much faster parsing on large projects.
The two transformation models
| dbt | SQLMesh | |
|---|---|---|
| Model | Stateless between runs by design | State-aware — tracks what has been computed for which intervals |
| Environments | Build a copy of the project in a separate schema | Virtual environments — logical pointers to existing physical tables; promotion swaps pointers rather than rebuilding |
| Lineage | Column-level, via the Fusion semantic parser | Column-level, via SQL parsing from the start |
| Incremental correctness | You express the logic | The engine knows which intervals are missing |
| Ecosystem | Enormous — packages, integrations, hiring pool | Much smaller |
The virtual-environment idea is the genuinely distinctive one, and it is a cost argument as much as an ergonomics one: a development environment that points at production tables rather than copying them removes a large amount of warehouse spend that most teams treat as unavoidable.
On the benchmark claiming a ninefold advantage: it was published by Tobiko Data itself, run on Databricks. Treat it exactly as the streaming benchmarks in the earlier chapter — the mechanism (skipping unnecessary recomputation) is real and checkable; the multiple is marketing.
The practical verdict: most teams stay with dbt because migration cost is real and the ecosystem is enormous; sophisticated new deployments increasingly pick SQLMesh. With one owner, that is now a preference rather than a bet.
Ingestion, and the pricing model that shapes your schema
Fivetran prices on monthly active rows — inserts and updates count, unchanged rows and the initial sync do not.
That model has a specific and under-appreciated consequence: it punishes wide, frequently-updated tables and is nearly free for append-only ones. A single `updated_at` touch on a large dimension table can dominate the bill. If you are on this pricing and your costs are surprising, look at update patterns before you look at volume — and note that this makes the ingestion vendor's pricing an input to your schema design, which is an unusual and slightly uncomfortable coupling.
The alternatives split by licensing posture: Airbyte uses a restrictive licence for the platform and permissive licences for its connectors; Meltano is fully open with connectors under version control and a decentralised, uneven quality distribution. And both warehouse vendors now ship their own: Databricks LakeFlow and Snowflake Openflow, the latter built on Apache NiFi and billed on runtime node size and active time rather than data volume — so an idle runtime still costs money.
Change data capture, and the failure that fills a disk
Change data capture reads the database's write-ahead log rather than querying tables, so it is low-impact on the source and near-real-time. Debezium supports three deployment shapes, and one common belief about it is false:
- As a Kafka Connect connector — the classic deployment.
- Debezium Server, writing directly to a cloud queue, a message bus or another sink. No Kafka required — the claim that Debezium needs Kafka is repeated constantly by competitors and is not true.
- As an embedded library inside your own application.
The classic change-data-capture outage, and it is always the same one.
On a write-ahead-log-based source, the capture process holds a replication slot. The database cannot discard log segments the slot has not confirmed.
So if the consumer stops — a crashed connector, a paused pipeline, a full downstream queue — the source database's write-ahead log grows until the disk fills. And the thing that fails is not your pipeline. It is your production database.
Monitor replication slot lag as a first-class alert on the source, not just pipeline health on your side. And know what your policy is for an abandoned slot, because the safe action — dropping it — means a full re-snapshot.
The other two places teams get hurt: the handoff from initial snapshot to streaming, and schema changes arriving mid-stream. Both are solved problems in the tooling and both need you to have chosen a behaviour deliberately rather than accepted a default.
Check yourself