This is the eleventh part in a series on how AI is crossing out of software and into the physical world. This time, Jan Bosch looks at when software stops answering and starts acting.
The promise is easy to state. Software that doesn’t merely answer questions but executes workflows: reading the ticket, checking the inventory, issuing the credit, updating the record, sending the mail. Not a tool a person uses, but a colleague-shaped process that runs.
Gartner’s assessment of how that’s going is bracing. The firm predicts that more than 40 percent of agentic-AI projects will be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. It also estimates that of the thousands of vendors now marketing agentic AI, only around 130 are actually doing it – the rest being chatbots and RPA with a new label, a practice Gartner calls agent washing.
I could write a whole piece from that paragraph, and it would be the wrong piece. The cancellations are a symptom. What I want to spend this one on is the arithmetic underneath them, because once you see it, almost everything about agent deployment stops being mysterious.
Here’s the arithmetic. Reliability doesn’t add; it multiplies. An agent step that succeeds 95 percent of the time is, in isolation, impressive. Chain five of them and you’re at 77 percent. Chain twenty and you’re at 36 percent. Chain thirty, which is a perfectly ordinary number of steps for a real business process with lookups, validations, approvals and writes, and you’re at 21 percent. Push the per-step reliability to 99 percent and thirty steps still only gets you to 74 percent: a process that fails one time in four. To reach 97 percent on a thirty-step task you need 99.9 percent per step, which is a figure almost nothing in software achieves against unstructured inputs.
This is why demos are so misleading, and the mechanism is worth naming precisely. A demo is three steps. Real work is thirty. The gap between them isn’t polish, effort or model maturity; it’s exponentiation.
The benchmark data says exactly this, once you look at the right metric. Sierra’s τ-bench was built to test agents on multi-turn tasks with real tool use and written policies, and its most useful contribution isn’t a leaderboard but a measure: pass^k, the probability that an agent succeeds on the same task across k repeated attempts. The original results had GPT-4-class agents succeeding on fewer than half of tasks on a single attempt and on roughly a quarter when the same task was run eight times. Frontier models have since pushed past 80 percent on pass^1 in the retail domain, which is real progress. But pass^1 is the metric that flatters, and the pass^k curves are, in Sierra’s own words, far from ideal.
An agent that works four times out of five is a demo. An agent that works four times out of five every time you run it is a product. Enterprises buy the second thing and are shown the first. Which brings me to the design consequence, and to the idea I’ve been circling since the summer.
You can’t test your way to confidence in a thirty-step agent by sampling end-to-end outcomes. The state space is too large, the failures aren’t independent and a passing run tells you almost nothing about the next one. So, the architecture that actually survives contact with production isn’t a long autonomous chain; it’s a short sequence of narrow steps, each with a stated contract: what must be true going in, what must be true coming out. And each input and output is checked.
This is the fullest form of the argument I’ve been making throughout this series, and in agent systems, it stops being a philosophical position and becomes the only testable unit available. The durable artifact isn’t the prompt, the model or the orchestration graph; it’s the contract at each boundary, plus the running evidence that it held. You can’t verify the whole; you can verify the joints.
There’s a second design axis, and it’s the one I see neglected most often. The question isn’t what the agent is capable of doing; it’s what it’s capable of doing that can’t be undone. Reversibility changes the reliability you need by orders of magnitude. An agent that drafts a reply, proposes a reconciliation or stages a configuration change can be 80 percent reliable and still enormously useful, because the 20 percent is caught and costs a few seconds. An agent that issues a payment, cancels an order, files with a regulator or changes a physical setpoint needs a reliability that chained language models don’t currently reach. And more importantly, needs its authority written down. Blast radius belongs in the architecture diagram, as a first-class parameter, with the irreversible actions few, explicit and gated by something other than the model’s own confidence.
For startups, the agent-native company is real, and I want to be precise about why, because it’s usually explained wrongly. The advantage isn’t better model access; everyone rents the same models. The advantage is that a company founded last year owns its process definitions. Its workflows were designed once, recently, by people still employed there, with the automation in mind. That’s an enormous structural edge over an organization whose processes accreted across thirty years, four acquisitions and two ERP migrations. Small headcount at significant revenue is downstream of that, not of the models.
The corollary is a warning. If your product requires the customer to have a documented process, you’re not selling software; you’re selling a transformation program, and you should price and staff accordingly.
For established enterprises, the obstacle has a name: process debt. You can’t automate a process that nobody has written down. And most enterprise processes exist as a documented happy path plus an enormous, undocumented body of exception handling that lives in experienced people’s heads: what to do when the customer is in two systems, when the invoice predates the migration, when the field is blank because of a 2019 decision nobody recorded. Agents expose this immediately and unforgivingly. The pilot works on the happy path, which is 60 percent of volume, and then meets the other 40 percent, which is where all the institutional knowledge is and none of the documentation.
Which means the work is specification. Interestingly, the same conclusion the previous part in this series reached from the opposite direction. There, it was objective functions for design search; here, it’s process contracts for execution. Two different technologies, two different industries, the same scarce skill: the ability to state precisely what should happen, including in the cases nobody wanted to think about. If I had understood at the start of this series how often that answer was going to come back, I wouldn’t have needed fourteen parts.
Note also that every reason Gartner gives for cancellation – cost, unclear value, inadequate risk controls – is a governance failure rather than a capability failure. Nobody is cancelling these projects because the models can’t write the email. For society, the agent question is where this series touches knowledge work directly, and the honest framing is uncomfortable in an unexpected way.
Gartner expects at least 15 percent of day-to-day work decisions to be made autonomously through agentic AI by 2028, from essentially none in 2024. Even allowing for the usual optimism, the direction is clear. But the exposure isn’t distributed the way people assume. It isn’t the hardest thinking that’s most automatable; it’s the best-documented work. A role with clean inputs, written procedures and measurable outputs is a role an agent can attempt; a role whose value lies in undocumented judgment is, for now, safe. This creates the perverse result that the most professionalized, best-proceduralized functions are the most exposed and the ones that resisted documentation are protected by their own disorder.
And underneath that sits an accountability question most organizations haven’t answered. When an agent issues the credit, who’s answerable for it? Not in principle. On the org chart, by name, in the way that a bank knows which officer approved a loan. The agent can’t hold the accountability; it has to land on a person or a role, and in most deployments I’ve seen, it simply hasn’t been assigned.
Which brings me to where I want to end, and it isn’t a warning about robots taking jobs. In 1998, Richard Cook, then at the University of Chicago’s Cognitive Technologies Laboratory, wrote a short treatise called “How complex systems fail,” which has become required reading in safety engineering and, belatedly, in software operations. Two of his propositions have been following me around while I wrote this piece. Proposition five: Complex systems run in degraded mode. As Cook puts it, “complex systems run as broken systems,” functioning not because everything works but because enough redundancy and compensation exists to cover what doesn’t. And proposition twelve: “Human practitioners are the adaptable element of complex systems.” Practitioners and first-line management, he writes, “actively adapt the system to maximize production and minimize accidents,” and these adaptations “often occur on a moment-by-moment basis.”
Read those together and you have the real risk in enterprise agents, which has nothing to do with capability. The people whose work you’re automating weren’t merely executing the process; they were the layer that continuously, invisibly, unremarkably compensated for the fact that the process has never once worked as written. They noticed the thing that looked wrong. They made the call. They just didn’t log it.
Replace them with a system that executes the documented process faithfully and you don’t get the same organization more cheaply; you get an organization with nothing left in it that adapts. Running the written procedure precisely, into the exceptions, at machine speed.
So, the question I’d put to any executive midway through an agent program isn’t whether the technology works. It’s this: When the agents are running the workflow, who’s doing the moment-by-moment adapting that your people were doing without being asked, without being paid for it and without anyone writing it down? If the answer is nobody, you haven’t automated the process; you’ve removed its shock absorbers.
