SDLC, DevOps & Developer Experience · Chapter 15 of 15
Developer portals and measurement, with the evidence
What an internal platform reliably delivers, what it cannot yet evidence, and the research finding that reversed the most-quoted claim in the field.
This is the chapter where the honest answer is most different from the marketing, and where there is now enough published research to say so with precision rather than as scepticism.
Developer portals: what they deliver and what they promise
Backstage is the open-source portal that defines the category. It moved to foundation governance, expanded its maintainer base well beyond its originating company, and now claims adoption in the low thousands of companies. Port and Cortex are the commercial alternatives, priced per seat, and one portal bundled into a suite most enterprises already buy is three to five times cheaper per seat than the specialists — a serious commercial threat to the category regardless of feature depth. Roadie sells managed Backstage, which exists because of the next paragraph.
Two of those names collide with unrelated products, and both appear elsewhere on this site
Cortex here is the developer portal. It is not Palo Alto's Cortex Cloud security platform (in the security tooling book), and it is not Cortex the long-term storage for Prometheus metrics (in infrastructure). Three unrelated products, one word.
Port is likewise a product name here, competing with a networking port, and with the ‘ports and adapters’ sense used in architecture.
Worth knowing before a meeting rather than during one — ‘we should look at Cortex’ means three different projects depending on who said it.
The build-versus-buy line, stated in the unit that actually decides it
Backstage is a framework, not a product. Adopting it means you are running and extending a React application, and the honest running cost is two to four engineers, ongoing — not a one-off integration.
That number is the whole decision. If you have those engineers and want the plugin ecosystem and no licence, build. If you want a catalog and scorecards in weeks and can absorb a per-developer licence, buy. If you want the plugins without the operational load, managed Backstage exists precisely for that gap.
Teams routinely compare the licence cost of the commercial options against zero, because the open-source one has no invoice. The two-to-four engineers do not appear on that comparison, and they are usually the larger number.
What a portal actually has to do
Five capabilities, and the reason the list matters is that a portal delivering only the first is the failure mode described further down:
| Capability | The developer question it answers |
|---|---|
| Service catalog | What exists, and who owns it? Fed from CI and from a descriptor file in each repository — not hand-curated, which is where catalogs die. |
| Scaffolder | How do I start something that is already compliant? One action creates repo, pipeline, scans, catalog entry and on-call rota. |
| TechDocs | Where are the docs, and are they current? Markdown beside the source, built in CI. Docs kept anywhere else go stale. |
| Self-service infrastructure | How do I get a database or a cluster without a three-week ticket? Request → policy → Terraform → pipeline. |
| Scorecards | Am I meeting the standard? Has an owner, has a runbook, scans passing, SLOs defined. And see the evidence problem below. |
Almost every portal that dies does so the same way.
It becomes a catalog that duplicates information developers can already get faster elsewhere — from the repository, from the cloud console, from a colleague.
The portal has to be the fastest path to at least one thing developers genuinely need — usually ‘create a new service’ or ‘get me a database’. Win that one, and the catalog becomes worth keeping accurate as a side effect. Lead with the catalog and you are asking people to maintain a directory for someone else's benefit.
A 2026 peer-reviewed literature review is the most useful thing published in this field, and its findings are uncomfortable.
It surveyed 88 sources across six years, academic and industry. Two findings matter most.
Only about 2% of the sources were peer-reviewed work with platform engineering as its primary focus. Academic research lags practitioner knowledge by two to three years, and essentially all the authoritative frameworks and adoption data come from vendors and foundations.
And there is no peer-reviewed empirical evidence of effectiveness for scorecards — the governance mechanism at the centre of every commercial portal — despite universal commercial adoption. Nobody has rigorously evaluated whether scorecard-driven governance improves quality or creates perverse incentives.
More broadly: large surveys consistently show that mature platform practice correlates with better delivery performance, and longitudinal and interventional evidence is absent. There are no before-and-after studies tracking organisations through adoption, so the correlation cannot distinguish ‘platforms cause performance’ from ‘high performers adopt platforms’. The hypothesised productivity dip before recovery has never been quantified.
A specific number worth handling carefully: the widely-repeated claim that the open-source portal holds around 89% share among organisations using such tools comes from a vendor-run survey of about a hundred engineering leaders. It is repeated everywhere without that context. Cite it with the caveat or not at all — and the same applies to the various ‘60–70% of portal initiatives fail’ figures, which trace to marketing content with no primary attribution.
So what should you actually buy one for?
Two things it reliably delivers, and they are worth real money:
Discoverability. A catalogue that answers ‘who owns this and where does it run’ during an incident. This alone justifies it in most organisations above a certain size.
Paved-path provisioning. A template that produces a correctly-wired service in minutes rather than days.
What it promises and cannot evidence is the productivity story. Buy a portal because your engineers cannot find things and cannot provision things safely. Do not buy one because a vendor's deck shows a delivery-metric uplift — that chart has no independent support, and the reviewers looked.
And the one consistent finding across the survey literature: organisations treating the platform as a user-centric product rather than as mandated infrastructure report better outcomes. Which is the same conclusion as the platform-as-product chapter, arrived at independently.
The measurement frameworks, and how they fit together
| Framework | What it measures | Use it for |
|---|---|---|
| The four delivery metrics | The system's output — deployment frequency, lead time, change failure rate, time to restore | Whether delivery is improving. Never in isolation |
| The five-dimension model | Insists you measure more than one thing — satisfaction, performance, activity, communication, efficiency | A defence against optimising a single number into meaninglessness |
| The three-dimension developer model | Why the output is what it is — feedback loops, cognitive load, flow state | The most actionable of the three: each dimension names something you can shorten, reduce or protect |
| The four-quadrant commercial framework | Speed, effectiveness, quality and impact | Its real contribution is impact — the share of engineering time going to new capability, which is the number a finance director understands |
A team using only the delivery metrics will optimise deployment frequency into a number that means nothing. The frameworks are complementary rather than competing: one measures output, one insists on breadth, one explains cause, and one translates to money.
The finding that reversed the field's most-quoted claim
For a decade the defining result of delivery research was that throughput and stability rise together — the speed-versus-safety tradeoff was false, because the same practices produced both. It was the single most quoted finding in the field.
The 2025 research programme, across roughly five thousand professionals, found that AI adoption has a positive relationship with delivery throughput and a negative relationship with delivery stability. The pairing broke.
The proposed mechanism is more valuable than the finding. It is not that generated code is bad. It is that removing the coding bottleneck saturates every bottleneck downstream of it — review, testing, quality assurance, release. If review capacity was your real constraint, AI makes your system worse.
The framing the researchers use is that AI is an amplifier: it magnifies an organisation's existing strengths and its existing weaknesses. Which makes the practical question not “should we adopt AI” but “what is our next bottleneck, and have we widened it first?” — and that is a very good thing to be able to say in an interview in 2026.
- Other findings from the same work worth carrying: about 90% of respondents report using AI at work and over 80% believe it has increased their productivity — belief, self-reported, not measured.
- 70% report trusting AI-generated code; 30% report little or no trust. That split is itself a finding.
- A follow-up modelling exercise puts gains at 35–40% on simple greenfield work and 10% or less on complex legacy code. That spread is the most useful single number in the literature, because it explains why two teams report wildly different experiences of the same tool.
- And it describes a productivity dip before the gain — caused by workflow adaptation and the cost of reviewing generated output — which leaders should expect rather than read as failure. Note that the financial modelling in that follow-up is a vendor publication modelling returns on its own product category; the greenfield-versus-legacy split is the defensible part.
The measurement tooling, and what it is really sold for
Several vendors sell delivery-metric platforms. The differentiator is not the metrics — any of them can produce the four delivery numbers — it is who they are sold to:
- One's distinguishing feature is research-and-development cost capitalisation reporting. That is not a productivity feature; it is a finance feature, and it is why the product gets bought.
- Another is positioned almost entirely at the investment-allocation layer for executives.
- The one that is methodologically strongest combines survey-based and telemetry-based measurement, which is what the developer-experience researchers actually advocate — perception and system data together, because neither alone is sufficient.
- And the gap none of them closes: any of these will produce metrics; almost none will tell you whether the metrics mean anything for your organisation. Which is exactly the gap the literature review identified.
What to say about all this in an interview
The position that reads as senior is neither enthusiasm nor cynicism. It is: “These frameworks are useful for starting a conversation and dangerous as targets. The delivery metrics measure output, the developer-experience dimensions explain it, and the honest state of the evidence is that platform investment correlates with performance and has never been shown to cause it. So I would measure, and I would be careful about what I claimed.”
That answer is defensible, it is accurate, and it distinguishes you from every candidate who recites four metric names.
Check yourself