arXiv:2608.19303, cs.AI, cs.CL, cs.SE

Outcome Monitors: Recovery Affordances for Silent Tool Failures

Sugam Panthi and Rabab Abdelfattah

Detect and advise: a nonbinding in-band receipt naming the violated property and public recovery tools

Scan for paper

The problem

A cached error page looks valid, so the agent uses it as fact

A timeout labels the failure, so the agent can choose another route. A cached error page or a negative price arrives in the expected format. Nothing tells the agent to doubt it, and the wrong value enters the next decision as fact.

10.9% of ToolMaze episodes complete under injected silent faults (35 of 320 across four models), with a fault-aware system prompt already in place.

Method

Detect and advise: an in-band receipt, not a controller

For ToolMaze, a deterministic program scans clean tool executions for required fields, stable types, positive values, small categorical domains, echoed arguments, ordered dates, and exact relationships between fields. It keeps a rule only if it holds in every qualifying example and meets a class-specific support threshold.

Five-fold task splitting keeps each tested workflow out of the traces used to build its checker. AppWorld rules come from its released API schemas; τ-bench rules come from its public retail database. On a violation, the monitor keeps the raw result and appends the violated rule, a witness, and callable recovery tools. It makes no model call and chooses no repair.

Core result: frozen paired ToolMaze, Table 1

Completion rises from 10.9% to 28.1%: every model, every stratum nonnegative

Open circles are baseline, filled green the monitor arm; Δ in points, W/L discordant pairs. Aggregate effect +17.2 points (95% bootstrap [11.25, 23.44], p < .00001); MiniMax M3 is a frozen third-family replication (p = .00073). Data: paper Table 1.

Ablations: 114 paired workflows

The recovery-tool list is the active content

A receipt stripped of its recovery-tool list performs at baseline; restoring the list adds +11.4 points (p = .028). Witness detail and delivery timing show no detectable difference (≈18-point MDE bounds the nulls). Data: paper Figure 3.

What agents do after a receipt, across 484 detector events:

91.5% of detector events retain at least two admissible recovery actions.

Generalization: frozen studies + dev rows

Gains concentrate where the fault blocks completion

τ-bench retail replicates the effect in a stateful second environment: +14.0 points on V4 Flash (p = .028) and +12.0 on V4 Pro (p = .023), driven by categorical status violations. Where baselines already complete despite the fault (≥ 65%), rescues are offset by harms. Clean traffic stays quiet: one spurious receipt in 100 τ-bench episodes, a 0-point estimate on 114 clean ToolMaze pairs. Data: paper Figure 2.

Validity check: incident-derived faults, authored blind

The contract vocabulary bounds detection

46%

of injected incident faults flagged (46.1 / 45.3 Flash / Pro)

1 in 5

trackable episodes propagate a corrupted value; ⅔ still pass

0.80%

fire rate on 53,078 organic responses

Suite transcribed from published production incidents; detector unchanged. Rules learned from clean traces catch structured violations, not plausible-string corruption. Data: paper §6.

Takeaway

Attach recovery affordances at the point of failure

Silent failures need an in-band signal: fault-aware prompts and extended reasoning budgets do not substitute for the receipt.
Put recovery tools in the receipt: diagnostics without them perform at baseline in our controls.
Keep it nonbinding and cheap: +$0.13 (+7%) per episode with no extra model call beats a five-call reviewer port (+$9.24, +498%) by +11.25 points.
Expect gains where the fault blocks completion (baseline ≤ 16%); elsewhere rescues and harms roughly cancel.
Detection stops at the learned rule classes: corruption inside plausible strings mostly slips through. Extending it is open.

TL;DR

What we did

Outcome Monitors check tool results against rules learned from clean traces from other tasks or public schemas. On a violation they keep the result and append a nonbinding receipt: violated property, witness, and public recovery tools. Frozen, prespecified evaluations with injected failures.

What we found

ToolMaze completion rises 10.9% → 28.1% across four models in two families and replicates in a third; τ-bench retail improves +14.0 / +12.0 points. Ablations identify the recovery-tool list, not diagnostic detail or timing, as the active content.

What you should do

When a tool result can be silently wrong, deliver recovery options with the flag: a one-line in-band receipt outperforms a 5-call reviewer at 1/70th the added cost. And treat the contract vocabulary as the boundary of trust: outside it, detection drops to 46%.