Your Production Agent Is a Distributed System

The demo was flawless. The agent read the ticket, pulled the customer record, checked the refund policy, drafted the reply, updated the CRM, and posted to the finance queue, all on its own, in front of the whole room. Two weeks later the same agent was running the same workflow against real tickets when the model provider throttled the account mid-run. The process died at step fourteen of twenty. Nothing had recorded which steps were already done, so the retry started over from step one, and the three refunds it had already issued went out a second time. The model had performed every individual step correctly, both times. The run was what failed.

That gap, between a model that can take each step and a system that can finish the whole sequence, is where most agent pilots quietly stall. It is not a prompt problem and it is rarely a model problem. It is the oldest problem in distributed systems, wearing a new coat.

The capability-completion gap

There are two different things people mean when they say an agent “works,” and conflating them is how a dazzling demo turns into a stalled pilot. The first is capability: can the model, presented with the current state, choose and execute the right next step? The second is completion: will a run of twenty of those steps, spread over minutes or hours and crossing several tools, actually reach the end and stay correct if something goes wrong partway? Capability is a property of the model. Completion is a property of the system wrapped around it, and almost nobody is measuring the second one until production forces the issue.

The most careful public measurement of the first property is METR’s time-horizon study from March 2025. They asked how long a task, measured by how long a human professional would take on it, a frontier agent can complete autonomously, and they found the answer has been climbing fast: the length of task a model can finish with fifty percent reliability has a doubling time of around seven months, and on a real-world software benchmark the doubling was under three months. Capability is not the constraint, and it is getting less constraining by the quarter.

Look at the same study from the completion side, though, and the picture inverts. Current models, METR reports, “have almost 100% success rate on tasks taking humans less than 4 minutes, but succeed <10% of the time on tasks taking more than around 4 hours.” A four-minute task is one step. A four-hour task is a run. The model that is essentially perfect on the step falls off a cliff on the sequence, and the authors are explicit that what drives the horizon upward is “greater reliability and ability to adapt to mistakes,” not raw intelligence. Reliability over a long horizon is the scarce resource, and it is exactly the resource a single model call cannot supply, because a single call has no memory of the fourteen calls before it and no say over what happens after it returns.

A model that can take any single step is not the same as a system that can finish the whole sequence. The first is bought from a model provider. The second you have to build.

A better model will not save you

The reflexive fix, when an agent flakes in production, is to wait for the next model. It reasons better, it holds context longer, surely it will just get through the run. That bet misreads the failure. When the provider throttled the account in the story above, no amount of model intelligence was on the critical path; the process was dead. When step fourteen writes to a CRM and step fifteen crashes, a smarter model does not un-write the CRM. The failures that kill long-horizon runs are crashes, timeouts, rate limits, partial writes, and lost state, and those live in the space between calls, where the model does not operate at all.

This is why the industry’s disappointment numbers keep pointing away from capability. Gartner predicted in June 2025 that over forty percent of agentic AI projects will be canceled by the end of 2027, and the reasons it named were escalating costs, unclear business value, and inadequate risk controls. Its analysts framed the maturity problem as models not yet able to “follow nuanced instructions over time,” which sounds like a model critique but is really a systems one: following instructions over time is what a run does, not what a call does. A year on, the prediction reads less like a forecast and more like a description of pilots that shipped a capable model inside a fragile run.

There is a second reflex worth heading off, because it is currently fashionable: treat the reliability gap as a governance problem and convene a committee, a policy, an approval board. Governance matters, and we have argued elsewhere that agent authority needs real boundaries. But a policy document does not checkpoint a run or make a refund call safe to retry. You cannot govern your way out of a missing spine any more than you can prompt your way out of it. The gap is engineering, and the engineering already exists.

You have solved this before

Step back from the word “agent” and describe what is actually running: a long-lived process that calls multiple remote services, holds state between those calls, must survive the failure of any one of them, and must not corrupt the world if it dies halfway. That is not a novel AI artifact. That is a distributed system, and it is the exact shape of problem that order-processing pipelines, payment flows, and provisioning workflows have been solving for decades. The uncomfortable, liberating truth is that the patterns are sitting on the shelf, mature and boring, and agent teams keep reinventing the fragile version instead of reaching for them.

Four properties turn a fragile run into a durable one, and each one has a named, battle-tested pattern behind it. A run should be durable, meaning its state is persisted after every step so a crash resumes from the last good point rather than the beginning; this is durable execution, the model behind workflow engines like Temporal, AWS Step Functions, and Azure Durable Functions. It should be idempotent, meaning any step can be safely repeated without doubling its effect, which is what idempotency keys give you. It should be compensable, meaning completed steps can be undone when a later step fails unrecoverably, which is the saga pattern from the 1980s. And it should be supervised, meaning some component outside the model owns the run, bounds it, and can resume or abandon it, the way a job scheduler owns a batch job. None of this is exotic, and none of it depends on which model you call.

The failure modes have names too, and naming them is how you catch yourself building them. Map each anti-pattern to the pattern that retires it, and the table below is the artifact to bring to your next agent design review and argue over row by row.

The shape of the table is the argument. Every row moves the fix out of the model and into deterministic machinery the model’s output cannot talk its way past. The model still chooses the steps. The system decides what happens when a step fails, repeats, or never returns.

A durable run has a lifecycle, and drawing it makes the difference concrete. The states below are the ones a fragile agent collapses into a single “hope it finishes,” and the transitions, especially the ones back into a running state, are the whole point.

stateDiagram-v2
    direction LR
    [*] --> Running
    Running --> Checkpointed: step done
    Checkpointed --> Running: next step
    Checkpointed --> Done: last step
    Running --> Interrupted: crash / timeout / rate limit
    Interrupted --> Running: resume from checkpoint
    Running --> Failed: unrecoverable step
    Failed --> Compensating: undo completed steps
    Compensating --> [*]: clean rollback
    Done --> [*]

When this is over-engineering

The patterns are not free, and pretending every agent needs all four is its own failure mode. The deciding questions are simple: is the run long-horizon, and does any step have side effects the world would notice if repeated or half-done? If both are true, you need durability. If either is false, you may not.

Where you do need it, the next choice is build versus buy. For a single important agent, hand-rolling the essentials, a state row per run, an idempotency key on each side-effecting tool, an explicit undo for the reversible steps, is often enough and keeps you close to the metal. Once you are running many such agents, a workflow engine earns its keep, because durable execution, retries, and observability come built in and you stop reimplementing them per team. Choose the engine when the count of durable agents, not the ambition of any one of them, crosses the line where shared infrastructure beats bespoke plumbing.

The constraints nobody puts on the slide

Two honest frictions decide whether this actually lands. The first is that idempotency is easy to demand and hard to retrofit. Plenty of internal tools and third-party endpoints were built assuming a single trusted caller who would never retry, and they offer no key, no dedup, no safe replay. You often cannot change them, so the durable layer has to carry the weight: a dedup table keyed on the run and step in front of the offending call, so the agent’s retry is absorbed before it reaches a tool that would happily charge the card twice. Budget for that shim work; it is where the real hours go.

The second is organizational, and it is the one most likely to be skipped. Durable execution slows the thing that sold agents in the first place, the promise that anyone could stand one up in an afternoon. Adding checkpoints, keys, and compensation is real engineering with a learning curve, and it will feel like friction to the team that just wants the demo in production by Friday. This is also why a named owner matters so much: a run that belongs to someone gets checkpointed, monitored, and resumed, while a fire-and-forget agent with no owner is the one still silently dying at step fourteen a quarter later. Someone has to own the run as a system, not just the prompt as a feature.

The agent demo that dazzles the room is a claim about capability, and capability, as the data keeps showing, is the part getting cheaper and more abundant every quarter. Production is a claim about completion, and completion has never come from the model. It comes from the durable, idempotent, compensable, supervised machinery around it, the same machinery that has carried payments and provisioning through partial failure for decades. The teams pulling agents across the line to production are not the ones with the best prompt or the newest model. They are the ones who recognized what they had actually built, a distributed system, and reached for the patterns that already solve it.

If your own agent pilots are stalling somewhere between a great demo and a dependable production service, the useful first question is whether the thing failing is the model or the run. We help teams draw that line: retrofitting idempotency onto tools that never expected a retry, and moving fragile in-memory agents onto durable execution without giving up the speed that made them attractive. If that is where you are, we are glad to compare notes.