Recovery Tools Help Agents Respond to Corrupted Tool Results
The tool call did not fail. It returned valid JSON, in the expected shape, with a value that could not be true.
The agent accepted it anyway.
It used the bad value in its next decision, called another tool, and built a confident answer on top of the mistake. By the time the failure became visible, the observation that caused it was several steps behind. Nothing crashed, and no exception pointed back to the tool call that caused it.
The difficult case is a call that succeeds at the transport layer but returns plausible-looking bad data. A cached error page arrives as content. A price is negative. An order status falls outside every state the system knows. Because the call itself succeeded, the agent has no explicit error telling it to stop.
In one study of a production agent runtime, humans reading the final output were the first to catch 70% of its silent failures. The system had logs, but nothing flagged the bad result while the agent was still acting on it.
A corrupted result leaves detection and recovery to the agent
When a tool times out, the failure is already labeled. The agent can retry or choose another route. A silent failure asks it to solve two harder problems at once:
- Notice that the observation is inconsistent.
- Find another action that can still finish the task.
Most agent loops leave both jobs to the language model. The raw tool result enters the context, the model consumes it, and the loop continues. Even a capable model that can explain the inconsistency afterward may not stop when the result first appears.
The intervention I tested inserts one deterministic check at that boundary. I call the full system an outcome monitor. Its checks are outcome contracts: properties that a valid result should satisfy.
A simplified receipt looks like this:
{
"tool_result": {
"item": "wireless keyboard",
"price": -79.00
},
"outcome_monitor": {
"violated": "price must be positive",
"observed": -79.00,
"recovery_tools": [
"retry_catalog_lookup",
"search_inventory"
]
}
}
The original result is preserved. The monitor does not call a tool, delete an action, choose a repair, read the evaluator, or reveal a benchmark fault label. It says what property broke and which public alternatives remain. The agent still decides what to do.
A hard guard could improve a score by blocking the wrong action itself. This runtime only changes what the agent can see.
A deterministic program learns per-tool rules from clean executions
There is no universal list of impossible tool results. A negative price is suspicious in a catalog and ordinary in a ledger. The monitor therefore learns or derives contracts for each tool.
In the main experiment, a deterministic program scanned clean tool executions while keeping every evaluated workflow out of the data used to build its checker. For each tool, it tested candidate rules about required fields, stable types, positive quantities, small categorical domains, echoed arguments, date ordering, and exact affine relationships between fields. It kept a rule only if the rule held in every qualifying training example and had enough supporting examples. This is what “mined” means here: the program proposed and filtered rules from observed executions. A researcher chose the candidate rule families and support threshold; the program applied those choices without a model call.
In other environments, the same interface used contracts derived from public API schemas or a public retail database. Contract construction was deterministic, local, and required no model calls. The workflows were split into five folds, and each fold's checker was built only from the other four, so no evaluated workflow fed its own checker. The five resulting rule sets contained 15,041 accepted invariants mined from 400 nominal workflows and built in under two seconds.
Recovery tools came from the public tool interface; the monitor had no access to solutions or hidden recovery paths. At 443 of 484 receipt events in the primary study, or 91.5%, the agent still had at least two advertised recovery actions. So the receipt usually left the agent a choice between routes.
Receipts raised ToolMaze completion from 10.9% to 28.1%
The primary evaluation used 80 difficult ToolMaze workflows with implicit tool failures. Baseline and monitored episodes were paired, randomized, and interleaved. The same workflows were run across four models from DeepSeek and Qwen, with extended reasoning disabled.
Without the monitor, the models completed 35 of 320 model-workflow pairs: 10.9%. With the receipt, they completed 90: 28.1%. That is a 17.2 percentage-point gain, with a task-clustered 95% bootstrap interval from 11.25 to 23.44 points and a sign-flip test below .00001.
Even with receipts, 230 of 320 episodes failed. Most tasks still failed with the receipt in place.
Every primary model improved. DeepSeek V4 Flash moved from 14/80 to 27/80; V4 Pro from 13/80 to 23/80; Qwen 3.7 Plus from 3/80 to 15/80; and Qwen 3.7 Max from 5/80 to 25/80. A separately frozen MiniMax M3 replication moved from 5/80 to 20/80, extending the result to a third model family without pooling it into the primary analysis.
The models did not all react the same way after a receipt. Across 484 receipt events:
- 48.6% switched to one of the listed recovery tools;
- 33.5% called some other tool;
- 14.0% retried the same tool;
- 3.9% made no further tool call.
So the receipt led to different next steps in different models.
The recovery-tool list produced the detectable gain
I began with a story about localization: tell the model exactly which property is inconsistent, and it can reason its way back. The controls tell a more specific story.
The baseline prompt already warned every model that tools could return unexpected data and instructed it to inspect fields, values, and types. In a separate follow-up, I turned extended reasoning back on for two DeepSeek tiers with an 8,192-token budget, and the monitor's effect remained. A generic warning triggered at the same moment performed about as well as a detailed, localized witness. Moving the warning or changing its salience produced no detectable advantage either.
Then I removed the recovery-tool list.
The stripped receipt fell back to baseline. Restoring the list added 11.4 percentage points on the same paired workflows. The diagnostic detail described what looked wrong; the gain came from listing where the agent could go next.
The null contrasts are power-bounded, so this does not prove that diagnostic wording never matters. It does identify the component that carried the detectable gain in these experiments: make the viable alternatives legible at the moment of failure.
Receipts helped when the corrupted result blocked completion
A result on one fault-heavy benchmark could be an artifact of that benchmark. I therefore carried the same nonbinding interface into a stateful retail environment, where a simulated user and database state determine whether the task is actually complete.
On 50 retail tasks, each run under two persistent fault types, DeepSeek V4 Flash improved by 14 points overall and V4 Pro by 12. But the aggregate hides the useful part.
When an order status was replaced with a value outside the valid set, the monitor helped sharply: Flash rose from 6/50 to 20/50, a 28-point gain with 14 paired wins and no losses. When a payment-conservation relationship was corrupted, Flash completed 35/50 tasks with or without the monitor. There were nine rescues and nine harms, producing no net effect.
The same pattern appeared elsewhere. In held-out AppWorld, baseline and monitored agents both completed 25 of 32 tasks. The checker fired in every monitored fault episode, so detector silence was not the explanation. Twenty-three of the 32 baselines completed despite consuming the fault. There was little left for a receipt to rescue.
Across the frozen studies, gains were largest where the injected fault usually blocked baseline completion. Where agents often completed despite the corrupted observation, rescues were offset by harms. That relationship is post hoc and descriptive, and it bounds where the headline result applies.
False receipts can redirect an agent even when they block nothing
On clean ToolMaze workflows, baseline and monitored agents both completed 74 of 114 model-workflow pairs. The aggregate difference was zero. Underneath it were five paired rescues and five paired harms.
Every action remains available in a nonbinding interface. A false alarm can still redirect attention, add context, and make an agent overreact. Leaving every action available does not make the monitor safe on its own.
A noisier schema-only detector makes the tradeoff visible. On a fault-enriched sample it recovered much of the completion gain: +10 points versus +15 for the learned monitor, with no significant head-to-head difference. But it emitted 166 receipts to the learned monitor's 111. Only 12% of its receipts exposed multiple recovery actions, compared with 91.9% for learned contracts. The learned system's clearest advantage was firing less often and listing more recovery options; its completion gain was not significantly larger.
Deployment therefore depends on the traffic mixture. A monitor that helps on faulted calls can still lose value if faults are rare and false receipts are common. Any real deployment needs to measure both.
Detection falls to 46% outside the learned rule classes
The original experiments align the injected failures with properties the contracts can represent. That is useful for asking whether a detected signal helps recovery, but it does not show that the detector can recognize arbitrary failures.
To test that boundary, I used eight fault types transcribed from a published taxonomy of production incidents. The person defining the faults saw the taxonomy and data shapes, but not the monitor's contract vocabulary.
Detection fell to roughly 46%.
The split explains why. The monitor caught 25 of 30 violations expressible over structured values, or 83%, but only 6 of 27 corruptions hidden inside plausible-looking strings, or 22%. A contract can reject a negative price or an impossible enum. It is much harder to recognize a polished paragraph whose content is wrong.
Outcome monitoring works when a tool's promises can be stated as checkable properties. Wrong results that stay well-formed pass through, and no receipt is sent for them.
Treat tool results as claims that may need a recovery path
The usual agent loop treats a successful tool response as an observation of the world. A safer interface treats it as a claim that can sometimes be checked before the agent builds on it.
Outcome monitors improved recovery across five models from three families and transferred to a second positive environment. They helped most when silent failure actually blocked the task. They did not help everywhere, they sometimes hurt individual clean runs, and their reach ended at the edge of their contract vocabulary.
The interface result is specific: agents often already have another route, but the runtime must identify the inconsistent result and name the available recovery tools at the moment the failure appears.
Research context
- ToolMaze: When Tools Fail provides the implicit-failure workflows used in the primary evaluation.
- τ-bench provides the stateful retail environment.
- AppWorld provides state-based tests that recognize alternative valid solutions.
- When Errors Become Narratives provides the production silent-failure taxonomy used for the incident-derived boundary study.
- Reinforced Agent is the inference-time reviewer method used for a separate retrospective comparison.
This article reports the results from the paper, now public: Outcome Monitors: Recovery Affordances for Silent Tool Failures (Panthi and Abdelfattah, arXiv 2026).
The poster is at spanthi.com/poster/outcome-monitors.
Cite this work
@misc{panthi2026outcomemonitors,
title = {Outcome Monitors: Recovery Affordances for Silent Tool Failures},
author = {Panthi, Sugam and Abdelfattah, Rabab},
year = {2026},
eprint = {2608.19303},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
doi = {10.48550/arXiv.2608.19303},
url = {https://arxiv.org/abs/2608.19303}
}