ACORN GLOBAL · ENGINEERING FIELD GUIDE · AGENT RELIABILITY
Diagnose AI agent retry loops
A repeated tool invocation can look like progress while delivering no useful outcome. Treat retries, failures and user impact as separate measurements.
Diagnostiquer les reprises en boucle — mesures vérifiables, aucune garantie ni résultat client inventé.
Look for these observable signals
- Repeated identical tool calls with unchanged inputs
- Retries without a bounded maximum attempt count
- Agent outputs that claim success after a failed tool result
A REPRODUCIBLE METHOD
Four steps to a useful baseline
Record one reproducible prompt, permitted tool call and observed outcome.
Count attempts, failures and retries over a declared sample interval.
Check timeouts, idempotency, stopping conditions and tool result handling.
Run a bounded before/after evaluation; keep the failure cases visible.
Evidence worth retaining
Timestamped tool outcomes, reproducible fixture, failure share, attempt count and accepted safety constraints.
Limit: An isolated successful run is not a reliability benchmark. Avoid automatically replaying external effects.
If required measurements are unavailable, record them as not measured. Never replace an unknown value with zero or claim a successful repair without a verified readback.