AI Agents

Agent Failure Analysis Prompt

Turn a failed case into a fix — diagnose where in the agent's flow it went wrong, categorize the failure, and point at the prompt, tool, or context that caused it.

Overview

An agent that fails a case isn't useful feedback until you know WHY it failed — bad instruction, wrong retrieval, a tool error, or a reasoning slip. This prompt analyzes a failure: it walks the agent's trace to find where it went off the rails, categorizes the failure type, identifies the likely root cause (prompt, context, tool, or model), and recommends where the fix belongs — so evaluation feeds improvement instead of just a red mark.

Why This Works

Finding the first failure point stops you fixing a downstream symptom
Attributing to a layer tells you where the fix actually belongs
Spotting a pattern turns one failure into a permanent test

Best for

Debugging agent failures during evaluation
Multi-step agents where failures cascade
Turning eval red marks into actionable fixes

Not for

Detecting that a failure occurred — use the Scorecard or Scenario prompts
Code-level debugging — use a debugging prompt

Use cases

Diagnosing why an agent failed a test case
Attributing a failure to prompt, retrieval, tool, or model
Deciding where a fix should go after a bad output

Customize This Resource

Opens this resource in the related tool with the fields pre-filled. Customize role, tone, and behavior rules — then generate your prompt.

Prompt Template

ROLE
You are diagnosing why an AI agent failed a specific case, to find where the fix belongs.

INPUT
CASE & EXPECTED BEHAVIOR:
[The input and what correct looks like]
AGENT TRACE:
[The agent's full trace — instructions, retrieved context, tool calls, reasoning, output]

ANALYZE
1. WHERE: pinpoint the step where the agent first went wrong (instruction-following, retrieval, tool call, reasoning, or final generation).
2. FAILURE TYPE: categorize it (e.g. ignored instruction, bad retrieval, wrong tool, hallucination, reasoning error, format break, over/under-refusal).
3. ROOT CAUSE: the most likely cause — and which layer owns it: the PROMPT/instructions, the CONTEXT/retrieval, a TOOL, or the MODEL itself.
4. NOT THE CAUSE: rule out the layers that worked, so the fix isn't misdirected.
5. FIX LOCATION: where the change should go, and a specific suggestion.
6. PATTERN: whether this looks like a one-off or a class of failures to add to the eval set.

RULES
- Identify the FIRST point of failure; later steps may just inherit it.
- Attribute to a layer (prompt/context/tool/model), not just 'the AI was wrong'.

OUTPUT
The failure point, type, root cause and owning layer, the fix location with a suggestion, and whether it's a pattern.

Related Resources

Explore more resources that work well with this one.

Related Tools

AI Output Validator

Tip: Save time by exploring related resources and tools that integrate with this workflow.

Explore all resources