AI Agents

Conversation Quality Evaluation Prompt

Judge the whole conversation, not one reply — evaluate a multi-turn exchange for context retention, coherence, goal completion, and recovery from misunderstanding.

Overview

A chat agent can give good individual replies and still fail the conversation — losing context across turns, contradicting itself, or never actually resolving the user's goal. This prompt evaluates the multi-turn exchange as a whole: does it hold context, stay coherent turn to turn, recover when it misunderstands, and reach the user's goal — the qualities single-reply scoring can't see.

Why This Works

Session-level qualities (retention, recovery) are invisible to single-reply scoring
Goal completion measures what users actually care about, not reply polish
Counting turns-to-resolution catches the agent that gets there inefficiently

Best for

Conversational agents and chatbots
Support and assistant agents judged on whole sessions
Teams scoring only single replies and missing session-level failures

Not for

Single-turn output scoring — use the Agent Evaluation Scorecard
Generating conversation test cases — use the Agent Test Scenario Prompt

Use cases

Evaluating a multi-turn chatbot or support agent
Catching context loss and contradictions across turns
Measuring whether conversations actually resolve the goal

Customize This Resource

Opens this resource in the related tool with the fields pre-filled. Customize role, tone, and behavior rules — then generate your prompt.

Prompt Template

ROLE
You are evaluating a full multi-turn conversation with an AI agent — the whole exchange, not one reply.

INPUT
GOAL:
[What the user was trying to accomplish]
CONVERSATION:
[The full multi-turn transcript]

EVALUATE (0–5 each, with reason)
1. CONTEXT RETENTION: does it remember and use earlier turns, or forget/re-ask?
2. COHERENCE: is it consistent turn to turn — no contradictions or drift?
3. GOAL COMPLETION: did the conversation actually accomplish the user's goal, and how efficiently (turns to resolution)?
4. RECOVERY: when it misunderstood or the user corrected it, did it recover gracefully?
5. CLARIFICATION: did it ask when genuinely ambiguous, rather than guessing or over-asking?
6. TONE & CONSISTENCY: appropriate and stable persona across the conversation?

AGGREGATE
- Overall conversation quality with the deciding factor.
- The single turn where it most went wrong (if any), and why.
- Did the user leave with their goal met? (yes / partially / no)

RULES
- Judge the trajectory, not isolated replies — a good reply that ignores prior context still fails retention.
- Efficiency counts: reaching the goal in ten turns it could do in three is a partial failure.

OUTPUT
The per-dimension scores with reasons, the overall judgment, the worst turn, and whether the goal was met.

Related Resources

Explore more resources that work well with this one.

Related Tools

AI Output Validator

Part of these Blueprints

End-to-end build journeys that use this resource.

Build a Customer Support System with AI Blueprint · 7 stages

Tip: Save time by exploring related resources and tools that integrate with this workflow.

Explore all resources