Written for engineers who are about to give an agent write access to something they cannot undo.
For about ten weeks I have run an LLM agent unattended, 24/7, on a Linux box, taking irreversible actions with real money — placing real orders against three separate venues. Not a demo, not a chatbot with a sandbox. A production agent with a wallet and no human in the loop.
It has not lost me much. It has not made me much either. That is not what this is about.
What I actually got out of it is an inventory of the ways an autonomous agent fails in production — and almost none of them are the ways the literature says it will. The published threat models for agentic AI are overwhelmingly adversarial: prompt injection, memory poisoning, tool abuse, identity spoofing. Those are real. They are also not what happened to me. I ran a text search across OWASP's Agentic AI – Threats and Mitigations v1.1, which defines seventeen threats across six playbooks. Occurrences of fail-closed: zero. watchdog: zero. dead-man: zero. heartbeat: zero. idempotency: zero. circuit breaker: zero. kill switch: zero. staleness: zero. outage: zero. NIST AI 600-1 has twelve risk categories and not one of them is operational failure.
Nobody attacked my agent. My agent was blinded by a billing lapse, fed by a data pipe that had been silently dead for weeks, and reported its own costs three times too high for months. None of that is in a threat model. All of it is in a production incident.
Here are eleven, with the control that now catches each. Every one is from my own logs.
Three layers, and the ordering is the whole point.
This is not a novel architecture. It is Layers of Protection Analysis, and process-safety engineering has used it for twenty-plus years under IEC 61511. Layer 2 is a Safety Instrumented Function acting as an Independent Protection Layer — independent of the basic process control system, which in our world is the LLM. The reason to borrow the vocabulary is that the discipline already knows the failure everyone else is about to rediscover: a protection layer that shares a dependency with the thing it protects against is not a protection layer.
Which brings me to the thing that made all of this concrete.
My model provider access was interrupted. Every rung of the chain — primary model, fallback model, degraded-effort retry — failed identically within about six seconds, because they all share one transport and one identity. Three consecutive decision cycles produced no decision at all, with real positions open.
The generalisable form: your model provider is a single point of failure, and your fallbacks probably aren't. Fallback across models is not fallback if they share an auth path, a quota pool, or a vendor. Mine looked like three layers of redundancy and was one.
The control: the deterministic enforcer makes no model calls at all, so it kept running through the entire outage and kept every position guarded. That is the whole argument for layer 2 in one incident. It is also why "add a retry" is not an answer — a retry shares the dependency.
When the subscription path died, a metered API rung was supposed to take over. It had a 420-second wall-clock limit. Measured across four firings on identical work: one success at 337s, and three timeouts at 427s, 466s and 472s.
Nothing hung. The job takes 340–470 seconds and I had given it 420. I had drawn the timeout through the middle of the work's own variance distribution, so it failed roughly half the time by construction.
The control: measure the distribution before you set the bound. A timeout is a claim about a latency distribution you have probably never plotted.
Worse than the last one: an aborted stream still bills for every token generated before the kill. Those three timeouts each cost roughly what a success costs and produced nothing. My books showed only the one call that finished.
The abort path logged no usage record — so failed calls were free in the accounting and invisible to the daily spend cap that was supposed to bound a runaway night. The cap could be exceeded indefinitely by failing.
The control: log usage on the failure path, and count failures against the cap. If your telemetry only fires on success, your cost ceiling is decorative. This is the single most transferable finding here: every metering system I have looked at since has the same hole.
The rate table was a generation out of date — it priced my model at $15/$75 per million tokens against a real $5/$25. Every cost figure the system ever reported was overstated by exactly 3×.
It never broke anything, which is precisely why it survived. Overstating spend is the safe direction: no alert fires, no budget is blown, nothing crashes. It was simply wrong, in the one place the system reported money.
The control: pin published rates in a test that fails when they drift, and date-stamp the table with the day it was verified. Treat "our own cost number" as a measurement with an error bar, not a constant.
Of the 299 decision cycles up to a fixed date, 196 produced no decision at all — 66%. Cause: quota exhaustion plus a decide-step timeout that a grown prompt had quietly started brushing against. After the fixes, the same measurement over the following 555 cycles: 79 silent, 14%.
The reason it went unnoticed is the ugly part: a cycle where the agent could not think looks exactly like a cycle where it thought and chose to do nothing. Both emit no action. From the outside, an agent that is broken two-thirds of the time is indistinguishable from an agent that is being selective — and the second reading is the flattering one, so it is the one you will reach for.
A second trap sits inside the first, and I hit it while re-deriving this number: the journal has a decided field, and it reads False on 96% of all cycles — because it means "decided to act," not "produced a decision." Had I trusted the obviously-named field I would have reported a catastrophe. The signal that actually meant "the mind produced nothing" was an empty thesis string. Check what your health field counts before you build an alarm on it.
The control: the ledger must distinguish declined from could not decide. They are different events and only one of them is your system working. If you cannot query "how many times did the agent fail to produce an answer today," you do not know whether your agent is running.
One of my inputs was reading from a path that no longer existed. It returned an empty result, cleanly, with no error. Empty is a completely legitimate answer for that feed — most days genuinely have nothing — so it read as normal for weeks.
The control: a feed that returns empty must prove it was reachable. Separate "queried successfully, no results" from "could not query." Assert freshness explicitly — a timestamp on the last successful contact, not the last non-empty payload. Silence is not data.
An API key got flagged into sandbox mode. The system fell back to a free public data source and kept running, with materially worse data feeding real decisions, for about a day. Everything downstream reported healthy — data was arriving.
The control: assert the provider identity, not just the presence of data. "I got numbers" is not a health check. Health is "I got numbers from the source I intended." Any degradation to a fallback must be a loud, visible state, not a quiet one.
A whole class of actions could never succeed, because the venue enforced a minimum balance my account sat below. Orders were submitted and simply never filled. No error, no refusal, no log line — requests went out and nothing came back.
The control: encode venue and platform preconditions as explicit local refusals with a stated reason, checked before the request goes out. A precondition you have not encoded is one you will discover as a silence. "It stopped working and nothing complained" is the signature of an unencoded precondition.
I decommissioned two subsystems by stopping their processes. Their open positions stayed open. The snapshot taken the next morning shows five open positions across the two accounts worth $1,579.23 — unmanaged, for four days. And because those positions kept moving, they continued rewriting the very performance records the decommissioning was meant to freeze.
The remediation script I later wrote states the lesson better than I can, in its own header:
"Stopping a box does not stop its positions: four days later both accounts still held open trades, quietly rewriting the final records those boxes are judged on."
It also records the second-order failure — the first cleanup attempt ran outside market hours and closed only five of six legs, because equity orders queue but options are rejected outright. A cleanup that can only run in a window is not a cleanup you can leave to someone remembering. It had to become a timer.
The control: teardown must include state, not just process. For any agent that creates external side effects, "stop the service" is step one of a procedure, and the procedure is not complete until the side effects are reconciled and closed. Write the teardown runbook before you write the launch runbook.
Protective orders resting against positions that no longer existed. Harmless-looking, until one fires against a position you have since re-entered in the other direction — and now your protection is an attack.
The control: an orphan sweep that reconciles protections against actual current state on a timer, and an invariant that exposure and protection are created and destroyed together. The general rule: every control needs a reconciler, because controls drift out of sync with the world exactly as readily as data does.
I moved money into the account. The system counted the transfer as a gain. The number was wrong in the most seductive possible direction, and I had no reason to look at it because it agreed with what I wanted to believe.
The control: deposit-neutral accounting — detect flows in and out and exclude them from performance, permanently, with no way to switch it off. Generalised: the metric that tells you whether the agent is working must be constructed so that it cannot be improved by anything other than the agent working. If money moving in makes the number go up, the number is not measuring the agent.
The agent is roughly flat. Down about 7% on a small account over ten weeks — which on a sub-$1,000 book is a loss of about fifty dollars, over six and a half weeks of head-to-head running.
That is not a track record. It is statistically indistinguishable from noise, and it licenses no conclusion in either direction: anyone claiming it proves LLM agents can't trade is doing bad statistics, and so is anyone — including me at 2am — claiming the edge is there and just needs time.
I am telling you this because the eleven items above are worth something and the P&L is not, and because a document like this that quietly omitted the returns would deserve your suspicion. The value here is the failure inventory, not the strategy. Every one of these eleven is domain-agnostic: replace "position" with "provisioned instance," "order" with "deploy," "stop" with "spend cap," and every single one still lands.
If your agent has write access to something you cannot undo — cloud provisioning, deploys, a production database, refunds — these are the questions I would want answered, in this order:
proceed, "so that a slow, erroring, or unreachable Guardian does not halt production." Its own text names the cost: "Fail-open trades enforcement for availability under disruption: an adversary who can disrupt the channel converts control into audit." To be fair to the spec, it also offers on_decision_failure: deny and requires every bypassed step to be audited. But defaults are what ships, and a control you have to remember to switch on is not a control. Go and read which setting your deployment is actually running. Then ask the second question: the Guardian is itself an agent — what happens when it is the thing that's rate-limited?I got all six of those wrong at least once, on a system I built specifically to be careful. The questions are cheap. The incidents are not.
I do this as fixed-scope work: an inventory of your agent's irreversible actions with a number on each, and the deterministic controls to bound them. Deliberately outside financial markets — that domain has its own regulator, and this is an engineering practice, not an advisory one.