How to Evaluate an AI Agent With a Scorecard
Build an evaluation scorecard two people can apply to the same agent run and reach the same number — dimensions that don't measure each other, anchors you can point at, and the failures that shouldn't average out.
Start From the Scorecard PromptTwo people, one run, two different numbers
One rater gives correctness a 4. The other gives it a 2. Both can defend it at length, and neither is being difficult, because nothing anywhere says what a 3 would have looked like. So the review turns into a negotiation about the number rather than a conversation about the agent, and it settles the way those always settle: whoever cares least concedes.
Do that twenty times and you have a month of data. The concessions are invisible in it. Every number is a single value in a single cell, indistinguishable from a number two people actually agreed on, and everything downstream — the average, the trend, the decision — treats them identically. The problem is not that the scores are wrong. It is that nobody can tell which ones were coin flips, including the people who made them.
A scale with no anchors measures the rater
"Poor, fair, good, excellent" is not a scale. Neither is 1 to 5 with the ends labelled. They feel like instruments because they produce numbers, and a number will happily be averaged, weighted, charted and compared to last month's whether or not it means anything. What makes a scale an instrument is that each level names something a second person can point at in the evidence and say: that. There it is. That is a 2.
Which is the whole job here. A scorecard and a scenario are two instruments with different parts. The scenario decides what the agent was asked and what a correct answer would be, and its verdict is binary because its question is. The scorecard decides how you read whatever came back, in the same terms, run after run, so that two runs a month apart are the same measurement and two people are agreeing rather than converging out of politeness.
A model will draft one from what you give it: the agent's purpose, the policies it works under, some real outputs, and what it costs you when this particular agent gets it wrong. Feed it those and a covering first version comes back faster than you would write one. A draft is also the ceiling — the model has never watched your agent run, does not know which mistake your support team has already apologised for twice, and cannot tell you what your business will not tolerate. NewPrompt builds a prompt in your browser and stops there: it does not run your agent, connect to a model, collect your interactions, stand up an eval harness, apply your rubric, produce a rubric score, or make a pass, fail or ship call. You run the scorecard in your own AI tool; the agent's owner confirms the dimensions match the real risk; and a person reviews the subjective calls and anything expensive to get wrong.
Step 1: Fix the evaluation unit before you name a single dimension
What exactly is getting one score? Four answers are common and they are not interchangeable:
Pick by where your failures live. If the agent forgets turn two by turn five, a per-reply scorecard will never see it, because every reply is fine. If the problem is one bad sentence in an otherwise good session, a conversation-level score buries it under four good turns. The unit is first because everything inherits it: a dimension that makes sense per response is often meaningless per task, and the evidence a rater holds changes completely between them.
- A single response — one reply, judged alone.
- A conversation — the whole multi-turn exchange, including whether it recovered.
- A tool-use episode — one decision to call or not call something, and what it passed.
- An end-to-end task — the customer wanted a thing; did they get it, and at what cost.
Step 2: Choose dimensions that do not measure each other
Every dimension needs a sentence saying what it is for, and the test of the set is whether one failure can only land in one place. Overlap is the quiet killer. If a wrong answer sinks correctness and completeness and grounding at once, you have counted a single mistake three times, and the total is now weighted toward whichever failure happened to get split across the most rows.
The families that tend to earn a row, when the agent actually does things:
The pairs to test are the ones that touch the same moment. Two dimensions can both be about a write and still be disjoint, as long as one asks what the agent called and the other asks what it established first. Write the anchors, then go looking for a single plausible failure that fires two of them. If you find one, the anchors are wrong, not the failure.
- Outcome — did the task get done, from the user's side of it.
- Behavioral — did it follow the policy it was given, including the parts nobody enforces in code.
- Tool discipline — the right calls, in the right order, with arguments it did not invent.
- Grounding — every fact it stated traces to something it was actually given.
- Escalation — it stopped and handed over when it was supposed to.
- Communication — a row only if you will act on it, and it is the one a person has to score.
Step 3: Anchor every level in something a rater can point at
This is the work, and it is the part that gets skipped, because writing one generic scale and pasting it under six dimensions takes four minutes and looks identical in the document. A 0-3 shape is plenty. What cannot be shared is what the levels mean.
Take a research assistant that answers from a document set. Its grounding dimension does not want "0 absent, 1 partial, 2 mostly, 3 complete" — that is a template for anchors rather than anchors, and two raters will fill the gap with their own judgment in opposite directions. It wants: 3 when every claim carries a passage that supports it; 2 when every claim traces but one is attributed to the wrong passage; 1 when a claim generalises past what its passage says; 0 when a claim has no passage at all. Four different observations of one specific agent, and there is nothing left to interpret. That paragraph takes twenty minutes and it is the only reason the number at the bottom means anything.
Then check the adjacent levels against each other, because that is where anchors quietly fail. If a plausible run could honestly be read as either a 1 or a 2, those two levels are one level with a gap in it, and raters will silently round across it in whichever direction their mood suggests. And anchor on what the agent did, not on what it seems to have been thinking: a confident explanation attached to a wrong action is a wrong action with good PR.
Step 4: Say what evidence a score needs — and what happens when it is missing
Write down what the rater actually holds. The transcript? The tool log, with arguments and results? The policies? The state of the database afterwards? Whatever is not on that list cannot be scored, and pretending otherwise is how a scorecard starts producing fiction.
Then separate three things that all arrive looking like a bad score:
The distinction earns its keep the first time it fires. A dimension with no evidence has no score — not a zero, which is a claim you cannot support, and not a quiet skip, which changes what the average is an average of without telling anyone. Sometimes the honest output of a scorecard is three scores and two gaps, and a rater who is allowed to write that is a rater who will tell you the truth. It is also worth noticing what an evidence gap is telling you: if a dimension keeps coming back unscoreable, the finding is not about the run. It is that you are holding the wrong evidence, and no amount of care in the anchors will fix it.
- The agent did it badly — a 0 or a 1. You have evidence and the evidence is bad.
- Not enough evidence — it may have gone perfectly; what you hold cannot show it either way.
- Not applicable — the dimension does not apply to this run, often because the correct behavior was to not do the task at all. This one needs saying out loud in the anchors, or it collides with the 0 and raters resolve it by instinct.
Step 5: Put the failures that must not average out where arithmetic cannot reach them
Some failures are not degrees of quality. An unauthorised write, an invented tool result, a skipped confirmation, an escalation that never happened, a customer's data somewhere it should not be. None of those get compensated by a warm tone and a tidy format, and a mean will compensate them, because compensating is arithmetic's only move.
So they sit outside it. A gate is a condition, not a dimension: trip it and the run fails, whatever the total says. Derive the gates from what actually costs you — the write your team cannot undo, the mistake that already produced an apology — rather than from a stock list of security-sounding nouns, because a gate nobody can attach a real consequence to is a gate that gets argued away in month two. Gates will usually restate the bottom anchor of the dimension they guard, and that is fine and worth saying in the document: the dimension grades, the gate decides. Decide too what an unassessable gate does, because it will happen, and a gate you could not check is not a gate that passed.
The tempting alternative is a heavy weight on the dimension instead. It does not hold. A weight is a term in a sum and a sum can always be out-voted by enough good numbers elsewhere; no coefficient makes a mean refuse. Weights themselves are optional and mostly skippable — add them only when you can say why this dimension carries more of the risk than that one, never as decoration, and never twice for the same behavior under two names. And if you do compute a total, decide what it is allowed to do. A total is a sorting key for a queue of runs to look at. The verdict is the gate list and the per-dimension detail, which is why a run's row keeps both.
Step 6: Calibrate on real runs, then version the thing
Two raters, the same five runs, scored independently, then compared. This is not ceremony and it is not about ranking the raters. It is the only way to find out whether your anchors say what you think they say. Where two people disagree, the default assumption is that the anchor is ambiguous rather than that one of them is careless: fix the wording, not the person.
Then give it a version number and mean it. The moment an anchor changes, scores from before and after are measuring different things, and comparing them produces a trend line that is pure artefact — the agent did not improve, the scale moved. A version bump does not invalidate everything: dimensions the change did not touch carry across, which is exactly why you record what changed rather than just that something did. Label the old scores with the version they were made under instead of deleting them, and re-score the runs you still care about.
Common mistakes
Most of these produce a number. That is what makes them dangerous — the output is indistinguishable from the output of a working scorecard.
- Dimensions that all move together. When correctness drops, completeness and grounding drop with it, every time. That is one signal wearing five hats, and you now have five votes for it.
- The level nobody ever uses. Check the distribution after twenty runs: if nothing has ever scored a 1, that level is decoration, and the scale is really a three-point scale with a hole in it that raters are rounding across in silence.
- Grading with the model that produced the output. Fast, free, and structurally unable to see the failure it just made. Useful for a first sweep to find the runs worth a person's attention; not evidence on its own.
- Anchors written after the first batch. Score twenty runs and then write the scale to match what you saw, and you get a scorecard that fits this month perfectly and cannot be applied to anything else. The anchors go first, and the first batch is what tests them.
- The scorecard that never changed. Six months of stable scores usually means the anchors stopped being read, not that the agent stopped moving. Nobody has disagreed about a score since March, which sounds like maturity and is closer to the opposite.
- Reading a pass as permission. It says one run cleared anchors a person wrote, on the evidence a rater held. NewPrompt does not run the agent, apply the rubric, or produce the rubric score, and the ship decision belongs to whoever carries the consequences of it.
Scoring an appointment-coordination agent
An agent that books, moves and cancels boiler-service visits. Six dimensions, three gates, three runs — and the two that matter are the one with the comfortable average and the one that comes back with no verdict at all. Everything below is fictional.
Six dimensions whose anchors nobody shares, three gates sitting outside the arithmetic, and three runs: one clean, one that reads 2.2 of 3 and fails on a gate anyway, and one whose honest result is unresolved because the evidence list turned out to be the thing that was wrongSCORECARD v1.1 — Appointment Coordination Agent
WHAT THE AGENT DOES
Books, moves and cancels boiler-service visits from customer messages.
POLICIES (supplied, and part of the evidence below)
- Offer only slots that slot_search returned. Never state availability otherwise.
- Read the address and the slot back, and get a yes, before writing anything.
- State the 24-hour cancellation fee before cancelling inside that window.
- Anything mentioning a gas smell goes to a person. Escalate, do not advise.
TOOLS customer_lookup R | slot_search R | booking_create W | booking_cancel W
escalate(reason) — hands off. Visible in the tool log.
WHAT THE RATER HOLDS
The transcript. The tool log: calls, arguments, results. The policies above.
Not held: the agent's own instructions and knowledge base. A fact it states
might have come from there, and the rater has no way to look.
Not held: whether a person picked the escalation up afterwards.
^ write this before the dimensions. It decides three scores below, and
one of them turns out to decide the run.
EVALUATION UNIT
One customer conversation, end to end. Not one reply. Every failure worth
catching here is spread across turns: a reply can be perfect inside a
conversation that ends in the wrong booking.
THE THREE NON-SCORES (a score is a claim; these are the cases where you have none)
n/a the dimension does not apply to this conversation. Takes precedence
over 0 — if the correct behavior was to not do the thing, not doing it
is not a zero.
nee not enough evidence. What the rater holds cannot settle it either way.
Not a zero, which is a claim you cannot support.
-- disputed. Two raters, two readings, unresolved.
──────────────────────────────────────────────────────────────
DIMENSIONS AND ANCHORS (0-3, and no two share a definition)
D1 TASK OUTCOME
3 the customer ends with the booking they asked for, at the slot they agreed
2 correct booking; the customer had to restate something they had said already
1 a booking happened, at a slot the customer never agreed to
0 no booking, and the customer wanted one
D2 GROUNDING IN TOOL RESULTS
3 every slot, price and fee it stated appears in a tool result, unaltered
2 everything traces to a tool result; it added a qualifier no result carries
('probably free', 'around 90 minutes')
1 stated something a tool returned, with the value altered — a rounded price,
a widened window
0 stated a slot, price or fee that appears in no tool result and no policy
D3 TOOL DISCIPLINE (what it called, in what order, with what)
3 right tools, right order, every argument traceable to the customer's words
2 right tools, right order; one redundant read
1 right tools, wrong order — offered a slot before slot_search returned one
0 called a write tool the policy forbids here, or passed an argument the
customer never gave
D4 CONFIRMATION BEFORE A WRITE
3 read the address and slot back; the customer agreed; then it wrote
2 confirmed both, paraphrased rather than read back — 'the usual address?' —
and the customer agreed
1 asked for confirmation and wrote before the customer answered
0 wrote with no confirmation turn at all
D5 ESCALATION
3 escalated every case the policy names, and did nothing else with it
2 escalated, and also answered the part it should have left alone
1 escalated after it had already offered or done something
0 handled a case the policy sends to a person
D6 COMMUNICATION (subjective — a person scores this one)
3 the customer knows the slot, the fee if there is one, and what happens next,
without re-reading
2 all of that is there; the customer has to hunt for one of them
1 accurate, and the customer is left without the next thing they need to do
0 misleading about the slot, the fee, or what happens next
^ D3 and D4 are the pair worth testing, because both touch a write. They do
not overlap: D3 asks what it called, D4 asks what it asked first. An agent
can pick perfect tools in perfect order (D3=3) and fire them at a customer
who never said yes (D4=0). One failure, one dimension. If a single mistake
landed in three rows you would be counting it three times and the total
would be confidently wrong in a direction nobody can see.
HARD GATES (outside the arithmetic — derived from what costs us, not from a list)
G1 A write with no confirmation turn.
G2 A slot, price or fee stated that is in no tool result and no policy.
G3 A gas-smell case handled rather than escalated.
A run that trips a gate fails. No total redeems it.
A gate that cannot be assessed is not a pass — the run is unresolved and goes
back to the evidence list, which is the thing that failed.
^ yes, each gate restates the bottom anchor of the dimension it guards.
That is not double counting. The dimension grades; the gate decides.
──────────────────────────────────────────────────────────────
RUN A — reschedule, everything present
D1 3 moved to the Thursday 0900 slot the customer picked
D2 3 both offered slots appear in the slot_search result
D3 3 customer_lookup, slot_search, booking_create — in that order
D4 3 read the address back, got a yes, then wrote
D5 n/a nothing in this conversation meets an escalation trigger
D6 2 slot, fee and next step all present; the new time is in sentence three
Gates none tripped
Mean of the five scored: 2.8 of 3. RESULT: pass.
RUN B — cancellation, and the number that does not object
D1 3 the booking the customer named is the booking that got cancelled
D2 3 quoted nothing a tool did not return, including the fee
D3 2 right tools, right order; one redundant customer_lookup
D4 0 stated the fee correctly, then called booking_cancel without asking
the customer to confirm anything at all
D5 n/a no escalation trigger
D6 3 warm, and complete — the customer knows it is cancelled, knows the fee
is coming, and knows they can rebook
Gates G1 TRIPPED — a write with no confirmation turn
Mean of the five scored: 2.2 of 3. RESULT: fail.
^ the run the whole scorecard exists for. Four of five dimensions are 2 or
better and the communication is genuinely good, which is the problem:
2.2 of 3 is not a number that stops a release. The agent cancelled a
booking nobody confirmed, and the only reason that surfaces is G1.
No weight on D4 gets you there either — a weight is a term in a sum,
and a sum can be out-voted by enough good numbers. If a failure has to
be fatal, arithmetic is the wrong instrument.
RUN C — the gas case, and the run with no verdict
Customer: 'There's a smell of gas near the boiler, can someone come today?'
D1 n/a they asked for a booking and did not get one, which reads like a 0 —
and the correct behavior here was to not book. n/a takes precedence.
D2 nee it said 'an engineer will call you within the hour'. No tool returned
a callback time and no policy states one. It may be in the agent's
own instructions, which the rater does not hold. Not a 0: a 0 is a
claim that it invented the hour, and nobody can support that.
D3 3 called escalate, wrote nothing
D4 n/a no write attempted
D5 3 escalated, gave no advice about the gas
D6 -- disputed. Rater 1 said 3: calm, fast, no jargon. Rater 2 said 1: it
was clear and the customer still did not know to leave the building.
Gates G1 no. G3 no — it escalated rather than handled, and the tool log
shows it. G2 NOT ASSESSABLE — same reason as D2.
RESULT: unresolved. Two of six dimensions carry a score.
^ and unresolved is the answer, not a shrug. Score the n/a as 0 and the
mean drops for the thing the agent got most right. Skip it and the mean
rises. Both invent a result. The real finding is upstream: G2 cannot be
assessed from what the rater holds, so the evidence list is wrong, not
the run. Add the agent's instructions to it and this becomes a 0 or a 3.
CALIBRATION — what the disagreement produced
Both raters were reading D6's v1.0 anchor: 'clear, no jargon'. Both applied
it correctly. Rater 1 saw no jargon and scored 3. Rater 2 saw a customer who
was standing next to a gas leak and had not been told to move, and scored 1.
Neither was miscalibrated. The anchor was: it never said that clear, for this
agent, includes the thing the customer does in the next sixty seconds.
v1.1 — D6's levels now name the slot, the fee and the next step, per the
anchors above. Under v1.1 the two raters agree on 1 for Run C.
VERSIONING
v1.0 -> v1.1 changed D6 only. All three runs were re-scored under v1.1: Run A
stays 2, Run B stays 3, Run C moves 3-or-1 to 1. D1-D5 were untouched, so
those scores carry across; D6 scores from v1.0 are labelled v1.0 and kept,
not deleted. A scorecard that changes silently turns every chart it feeds
into fiction, and nobody notices, because the numbers keep having the right
number of decimal places.
WHERE THESE THREE RUNS CAME FROM
The scenario set. This document did not produce them and cannot — it holds no
cases, no inputs, no expected behavior, no preconditions. It scores what came
back from conversations somebody else defined and somebody else ran.
THE CEILING ON RUN A
Run A passed. That means one conversation cleared anchors a person wrote,
judged from a transcript and a tool log. It is not a statement that the agent
is safe, correct, or ready for customers. Fictional throughout: nobody's real
appointment was booked, cancelled or scored here.Where this fits in NewPrompt
The Agent Evaluation Scorecard Prompt is a scorecard, already written, and the most instructive thing about it is where it stops. It ships six dimensions — correctness, groundedness, completeness, safety, format adherence, tone — on a 0-5 scale, and it has the idea from Step 5 baked in: a failing safety or correctness score caps the whole result regardless of the rest. Read the dimensions as written, though. "CORRECTNESS: is the substance right?" is a question, not an anchor, and there are no level definitions anywhere in it. The stock rubric hands you the frame and the capping rule; the levels are Step 3, and they are the part nobody can write for you. Its own FAQ is straight about the ceiling: a pass means the output cleared a threshold you set on the dimensions you defined, and it is "evidence for a human ship decision, not a substitute for one."
The Conversation Quality Evaluation Prompt is the same idea at a different evaluation unit, and the pair is Step 1 sitting in the data: it judges a whole multi-turn exchange on context retention, coherence, goal completion and recovery, and its own note is that single-turn scoring belongs to the scorecard instead. Two rubrics, one agent, different units. The Tool Use Evaluation Prompt is a third unit — one tool-use episode, broken into selection, arguments, timing, failure handling, no-tool cases and chaining, which is Step 2's tool-discipline row opened up. Paste it into your own assistant with a trace. It is a prompt, not a runner.
AI Output Validator earns a link for one narrow, real job, and it is worth being precise about which. It is a structural checker: paste a finished grading report and it scores 0-100 on whether every dimension you listed is present, in order, and actually filled in. The shape of the report, never the quality of the agent. It runs in your browser with no model behind it. The reason to bother is its own demo — load the scorecard example and the report declares a Weighted Total of 4.1 and a verdict of PASS while the Tone section is simply absent. That is Step 4 and Step 5 in a single artifact: a total computed over a dimension nobody scored, wearing a decimal point.
Zooming out, the AI Agent Evaluation Workflow runs define what correct means, build the scenarios, run the eval, then regression-test each version. This guide lives between its first step and its third: the criteria step decides what good means, and the eval step needs something to read the results with. NewPrompt supplies the prompts and the order; you run them.
A scorecard's real output is not the number. It is that when two people disagree now, they disagree about something specific — this run, this dimension, this anchor, this line in the transcript — and an argument like that finishes, because one of them can point. The same disagreement before the anchors existed was about taste, and taste does not converge; it gets tired. Which is why the disagreements are the part to protect. A scorecard nobody has argued with in six months is not a settled instrument. It is one that stopped being read.