AI Agents

Agent Failure Analysis Prompt

Turn a failed case into a fix — diagnose where in the agent's flow it went wrong, categorize the failure, and point at the prompt, tool, or context that caused it.

Open in AI Output Validator

Overview

An agent that fails a case isn't useful feedback until you know WHY it failed — bad instruction, wrong retrieval, a tool error, or a reasoning slip. This prompt analyzes a failure: it walks the agent's trace to find where it went off the rails, categorizes the failure type, identifies the likely root cause (prompt, context, tool, or model), and recommends where the fix belongs — so evaluation feeds improvement instead of just a red mark.

How to use this resource

Assemble the failed case

Gather the agent trace for the case that failed, the task it was given, and what the correct behavior would have been. The analysis needs the full path to find where it went wrong.
Open this resource in AI Output Validator

Load the prompt into AI Output Validator and paste in the failed run. The tool runs the analysis so you get the failure category and likely root cause without tracing by hand.
Review the root-cause findings

Read where the failure occurred, its category, and whether the cause points at the prompt, the context, a tool, or the model.
Send the fix to the right layer

Apply the recommendation to whichever layer it names - instruction, retrieval, or tool - then re-run the case to confirm the failure is gone.

Why This Works

Finding the first failure point stops you fixing a downstream symptom
Attributing to a layer tells you where the fix actually belongs
Spotting a pattern turns one failure into a permanent test

Best for

Debugging agent failures during evaluation
Multi-step agents where failures cascade
Turning eval red marks into actionable fixes

Not for

Detecting that a failure occurred — use the Scorecard or Scenario prompts
Code-level debugging — use a debugging prompt

Use cases

Diagnosing why an agent failed a test case
Attributing a failure to prompt, retrieval, tool, or model
Deciding where a fix should go after a bad output

FAQ

How does this prompt decide whether an agent failure belongs to the prompt, the context, a tool, or the model?

Step 3 (ROOT CAUSE) forces the blame onto exactly one owning layer - PROMPT/instructions, CONTEXT/retrieval, a TOOL, or the MODEL itself - rather than settling for 'the AI was wrong'. Step 4 (NOT THE CAUSE) rules out the layers that worked so the fix isn't misdirected. Run over the trace you paste in the assistant you choose, the named layer is a diagnosis; you apply the fix.

Why does the analysis look for the FIRST point of failure instead of the step where the wrong output appeared?

Its RULES pin the analysis to the FIRST point where the agent went wrong, because later steps often just inherit that error - a bad retrieval quietly poisons the reasoning and the final generation. Step 1 (WHERE) pinpoints that first break across instruction-following, retrieval, tool call, reasoning, or final generation, so you fix the source rather than a downstream symptom that would reappear on re-run.

After AI Output Validator names the failure category, what do I do with the PATTERN finding?

Step 6 (PATTERN) labels the case a one-off or a recurring class of failures, and step 5 (FIX LOCATION) names where the change belongs with a concrete suggestion. Both are suggestions, not actions: you decide whether to write the new eval case, apply the FIX LOCATION change to that layer, then re-run the failed case to confirm it now passes.

Customize This Resource

Opens this setup in AI Output Validator. Validate to see the score, every issue found, and the repair prompt to send back to the model.

Open in AI Output Validator

Prompt Template

Copy it as-is, or use Open in AI Output Validator to load it pre-filled and customize it with your own context.

ROLE
You are diagnosing why an AI agent failed a specific case, to find where the fix belongs.

INPUT
CASE & EXPECTED BEHAVIOR:
[The input and what correct looks like]
AGENT TRACE:
[The agent's full trace — instructions, retrieved context, tool calls, reasoning, output]

ANALYZE
1. WHERE: pinpoint the step where the agent first went wrong (instruction-following, retrieval, tool call, reasoning, or final generation).
2. FAILURE TYPE: categorize it (e.g. ignored instruction, bad retrieval, wrong tool, hallucination, reasoning error, format break, over/under-refusal).
3. ROOT CAUSE: the most likely cause — and which layer owns it: the PROMPT/instructions, the CONTEXT/retrieval, a TOOL, or the MODEL itself.
4. NOT THE CAUSE: rule out the layers that worked, so the fix isn't misdirected.
5. FIX LOCATION: where the change should go, and a specific suggestion.
6. PATTERN: whether this looks like a one-off or a class of failures to add to the eval set.

RULES
- Identify the FIRST point of failure; later steps may just inherit it.
- Attribute to a layer (prompt/context/tool/model), not just 'the AI was wrong'.

OUTPUT
The failure point, type, root cause and owning layer, the fix location with a suggestion, and whether it's a pattern.

More resources from AI Output Validator

Resource

Fix Invalid JSON from AI

The JSON won't parse and you can't see why. Deterministic cause-sniffing — trailing commas, single quotes, unclosed brackets — and the repair prompt that fixes it.

Engineering

Resource

Validate AI Output — Catch Format Violations

Paste the response, get the verdict: real JSON parsing, missing-field detection, and a repair prompt for everything found.

Prompt Engineering

Resource

Agent Evaluation Scorecard Prompt

Grade agent output the same way every time — a rubric scoring correctness, grounding, safety, tone, and completeness, with a pass threshold instead of a gut call.

AI Agents

Resources that pair well

Resource

Force JSON Output from AI

Stop getting 'Sure, here is the JSON…' — the output-contract pattern that forces models to return only parseable JSON: schema, example, and a strict rule block.

Prompt Engineering

Resource

Extract Data From Text with AI

Free text in, named fields out. The extraction prompt pattern that turns any unstructured text into consistent, parseable records.

Prompt Engineering

Resource

Classify Support Tickets with AI

Billing, Technical, Account, How-To, Feature Request — ticket triage with definitions that decide the borderline cases for the model.

Support

Related tools

Tool

AI Output Validator

Paste an AI's output and validate it against the expected format — with a repair prompt for every failure found.

Structured Output

Tip: Save time by exploring related resources and tools that integrate with this resource.