LLM Systems Engineering · Chapter 8 of 20
Managed agent runtimes — what you are actually buying
AWS, Azure and Google all shipped one inside twelve months, and they solve narrower problems than the marketing implies.
Between May 2025 and October 2025 all three hyperscalers shipped a managed place to run an agent. That is a genuinely new product category — which means anyone asked “which agent runtime would you use” in an interview is being asked about something roughly a year old, where the honest answer includes what is still unsettled.
What you are buying
Strip the marketing and a managed agent runtime sells four things, each of which you would otherwise build:
- Session isolation. A hostile or merely buggy tool call cannot reach another user's session. AWS gives a microVM per session; Azure a VM-isolated sandbox with a persistent home directory that is restored on resume. Building that yourself is not hard so much as it is never finished.
- Identity propagation. A per-agent credential that can call models, tools and downstream services — and, where a user is present, act on that user's behalf with scoped consent rather than inheriting broad access.
- Memory as a service, so you are not operating a vector store and a summarisation job as a side effect of shipping an agent.
- An execution ceiling with somebody else's on-call attached to it.
What you are giving up, which is less discussed
Portability, and more of it than the “framework agnostic” claim suggests. All three genuinely will run LangGraph, CrewAI or your own loop — but the agnosticism sits at the transport layer, not the orchestration layer. AWS wants a container answering specific paths on specific ports. Azure wants a different protocol shape on a different port. Google wants a Python class with particular method names. “Deploy anywhere” code does not exist here; you write one thin adapter per target, not zero.
Debuggability. You cannot attach a shell to a microVM. During an incident you have exactly as much visibility as the provider's tracing gives you, which is a real cost that only shows up on the worst day.
And cost predictability. All three bill by active resource-second rather than by request, so an agent that thrashes CPU in a retry loop costs differently from one that mostly waits on a model. None of the three publishes a “cost per conversation” figure you could put in a budget line without instrumenting first. AWS bills memory on the peak footprint within each second, which rewards flat allocation over spiky.
The question that separates a real answer from a fluent one
“An agent that runs for twenty minutes and can crash halfway” is not a new problem. It is workflow durability, and there is a decade of prior art — Temporal shipped that primitive in 2019. The useful question about these runtimes is therefore whether they solve it or repackage it.
They do not solve it, and Microsoft says so in writing
Microsoft Foundry's resilient execution provides lease-based crash detection and handler reentry. Its own documentation states that “Recovery reenters the handler from its beginning. It isn't deterministic replay” and directs you to “Use a workflow engine for fan-out, fan-in, durable timers, or child workflows.”
It generalises. AWS gives an eight-hour ceiling, a health-check keep-alive and optional persistent storage — none of which is event-sourced replay. If the process dies mid-task, resuming is your application's problem, usually solved by your framework's own checkpointer writing outside the session. Google is blunter still: a ten-minute streaming ceiling and documentation telling you to keep state in Sessions or Memory for anything longer.
So these are session-hosting and identity services, not durability services. Buy the runtime for isolation, identity and memory; keep a real orchestrator for anything that must survive a crash and resume where it stopped.
And AWS agrees by action. In July 2026 it shipped native Step Functions integration with Harness — which is a tacit admission that the harness alone does not cover this.
“Harness” means three unrelated things, and two of them are a sentence apart above
Harness capitalised, in that last line, is AWS’s own AgentCore feature — config-only, three API calls, no container.
“the harness” in lower case is the general sense: the scaffolding around a model — the loop, the tools, the context management — covered in harness engineering.
And Harness (harness.io) is an unrelated continuous delivery platform that sits in a great many enterprise pipelines. If someone raises ‘Harness’ in a deployment or developer-platform conversation, that is almost certainly the one they mean.
Three products, one word. Worth checking which before you answer.
When self-hosting still wins
- You already run Kubernetes, or Amazon’s container service ECS, at a scale where another platform's abstractions cost more in workarounds than they save.
- Your language is not supported. Azure hosted agents are Python and C# only; Google Agent Engine is Python only; AWS is the most permissive, taking any container.
- You need a sub-second cold start — and note that none of the three publishes a cold-start figure at all, which is a genuine gap for anyone planning against a user-facing latency budget.
- A regulator wants your inference in a specific facility that is not on any of their region lists.
Check yourself