Outcome Monitors: Recovery Affordances for Silent Tool Failures
The problem
A cached error page looks valid, so the agent uses it as fact
A timeout labels the failure, so the agent can choose another route. A cached error page or a negative price arrives in the expected format. Nothing tells the agent to doubt it, and the wrong value enters the next decision as fact.
Method
Detect and advise: an in-band receipt, not a controller
A deterministic program learns rules from clean tool executions: required fields, stable types, positive values, ordered dates. It keeps a rule only if it holds in every qualifying example, and five-fold splitting keeps each tested workflow out of its own checker.
On a violation the monitor keeps the raw result and appends the violated rule, a witness, and callable recovery tools. It makes no model call and chooses no repair.
Core result: frozen paired ToolMaze, Table 1
Completion rises from 10.9% to 28.1%: every model, every stratum nonnegative
The recovery-tool list is the active content
What agents do after a receipt, across 484 detector events:
Generalization: frozen studies + dev rows
Gains concentrate where the fault blocks completion
Validity check: incident-derived faults, authored blind
The contract vocabulary bounds detection
46%
of injected incident faults flagged (46.1 / 45.3 Flash / Pro)
1 in 5
trackable episodes propagate a corrupted value; ⅔ still pass
0.80%
fire rate on 53,078 organic responses
Suite transcribed from published production incidents; detector unchanged. Rules learned from clean traces catch structured violations, not plausible-string corruption. Data: paper §6.
Takeaway
Attach recovery affordances at the point of failure
- Silent failures need an in-band signal: fault-aware prompts and extended reasoning budgets do not substitute for the receipt.
- Put recovery tools in the receipt: diagnostics without them perform at baseline in our controls.
- Keep it nonbinding and cheap: +$0.13 (+7%) per episode with no extra model call beats a five-call reviewer port (+$9.24, +498%) by +11.25 points.
- Expect gains where the fault blocks completion (baseline ≤ 16%); elsewhere rescues and harms roughly cancel.
- Detection stops at the learned rule classes: corruption inside plausible strings mostly slips through. Extending it is open.
TL;DR
What we did
Outcome Monitors check tool results against rules learned from clean traces or public schemas. On a violation they keep the result and append a nonbinding receipt: violated property, witness, and recovery tools. Frozen, prespecified evaluations with injected failures.
What we found
ToolMaze completion rises 10.9% → 28.1% across four models in two families and replicates in a third; τ-bench retail improves +14.0 / +12.0 points. Ablations identify the recovery-tool list, not diagnostic detail or timing, as the active content.
What you should do
When a tool result can be silently wrong, deliver recovery options with the flag: a one-line in-band receipt outperforms a 5-call reviewer at 1/70th the added cost. And treat the contract vocabulary as the boundary of trust: outside it, detection drops to 46%.
This poster online