Data Platform & Analytics · Chapter 13 of 16
Orchestration, and the consolidation nobody predicted
What actually changed in the dominant scheduler, why three-quarters of installs have not upgraded, and the acquisition that ended the alternatives debate.
Orchestration is the least glamorous and most load-bearing part of a data platform, and the landscape changed more in 2025–26 than most practitioners have noticed.
The major version, and what it fixed
Apache Airflow — the dominant open-source scheduler — shipped Airflow 3.0 in April 2025, addressing several long-standing complaints:
- Versioned pipeline definitions. Structural changes are tracked over time, the interface shows which version a past run used, and editing a pipeline no longer mutates runs already in flight. That last part was a real correctness problem for years.
- A task execution API. Tasks no longer connect to the metadata database directly — every state transition and value exchange goes through a hardened API layer. That is what makes remote and edge execution possible, and it means task code now imports from a dedicated software development kit rather than from internal modules.
- Remote execution reached general availability, so tasks can run outside the cluster.
- A rebuilt interface, with version-aware views and first-class backfill management.
- And later: partitioned assets, so a downstream task can trigger on a slice rather than a whole dataset, plus expanded human-in-the-loop approval and multi-team isolation within one deployment.
Three removals that silently break older Airflow pipelines and older tutorials.
Nested sub-pipelines are gone — replaced by task groups and data-aware scheduling.
Service-level agreements are gone, replaced by a deadline-alert mechanism in a later release. Anyone relying on the old feature loses it silently on upgrade.
execution_date was removed from the task context in favour of logical_date. This is the one that errors on first run, and it appears in essentially every tutorial written before 2025.
Ruff's AIR30 rules auto-fix much of it, and the migration also has a schema change during which in-flight tasks can hang and need manual clearing. Plan a maintenance window rather than a rolling upgrade.
The adoption number that tells you the real story
Astronomer's community survey — nearly six thousand respondents, so skewed toward engaged Airflow users — found roughly a quarter on Airflow 3 a year after general availability, with the great majority of the rest intending to upgrade.
And adoption is lower at large enterprises, not higher — around a fifth at the biggest organisations. The pattern is the opposite of what vendors imply.
What that actually tells you: the upgrade is real work, and ‘this scheduler is legacy’ commentary is wrong about the market even where it is right about the ergonomics. Around eighty thousand organisations run it and downloads have grown more than thirtyfold in five years.
One more figure from the same survey worth carrying into any conversation about AI-generated pipelines: only about one in eleven respondents was satisfied with pipelines produced by generic AI tools, citing hallucinated operators and outdated syntax. The syntax point is mechanical — the models learned from years of pre-major-version tutorials.
The consolidation that ended the alternatives debate
For several years the interesting question was whether Prefect or Dagster would displace Airflow. On 13 July 2026 Prefect acquired Dagster.
Dagster continues under its own name and open-source licence, its commercial offering remains supported, existing contracts are unchanged, and no migration is required. The combined company operates under the acquirer's name.
So the two leading Airflow ‘successors’ are now one company — and the reported rationale is agent orchestration rather than data pipelines, which tells you something about where these products believe their growth is.
Asset-based versus task-based: does it matter?
The distinction was the central argument of the last five years, and the honest answer is that it was absorbed rather than won.
The claim was that you should declare the data assets that must exist and let the orchestrator derive the execution, giving you lineage, freshness policies and partition-aware backfills for free. Airflow answered with datasets, then assets, then asset partitioning. Prefect added an asset layer with checks, and the declarative pipeline frameworks arrived from the other direction. And then Dagster was acquired — by Prefect.
Asset thinking won the argument; Dagster, which argued it, did not win the market. In practice the distinction matters most when you backfill often across partitioned data, when you want lineage without a separate catalog, and when your team thinks in tables rather than jobs. It matters least when your pipelines are mostly external triggers and API calls, where task semantics are the honest model anyway.
Durable execution is a different thing entirely
A category conflation worth clearing up, because it appears in interviews: Temporal — and durable execution generally — is not a data orchestrator. It runs long-lived workflows written in ordinary code, persisting and replaying event history so state survives worker crashes, with signals, timers, human approvals and compensating transactions.
Use durable execution when the unit of work is a business transaction with durable state — an order, an onboarding, a payment saga. Use a data orchestrator when the unit of work is a scheduled or data-triggered batch. They overlap far less than the marketing suggests, and picking the wrong one produces a system that fights you constantly.
Check yourself