Resource
Fix Invalid JSON from AI
The JSON won't parse and you can't see why. Deterministic cause-sniffing — trailing commas, single quotes, unclosed brackets — and the repair prompt that fixes it.
Agent Evaluation Scorecard Prompt
Grade agent output the same way every time — a rubric scoring correctness, grounding, safety, tone, and completeness, with a pass threshold instead of a gut call.
Overview
Evaluating agents by reading a few outputs and nodding doesn't scale and isn't consistent. This prompt builds a scorecard: the dimensions that matter (correctness, groundedness, safety, format adherence, tone, completeness), a concrete scale for each, and a weighted pass threshold — so two reviewers grade the same output the same way and 'good enough to ship' is a number.
How to use this resource
-
Assemble the evaluation inputs
Gather the agent output you want to grade, the task it was given, and what a correct response looks like. The rubric scores against that expected result, so the clearer it is, the more consistent the grade.
-
Open this resource in AI Output Validator
Load the scorecard into AI Output Validator and paste in the output alongside the expected result. The tool runs the prompt so you get the per-dimension scores without grading by hand.
-
Review the scored dimensions
Read the score and one-line reason for each dimension, then check the weighted total against the pass threshold and note the single issue holding the score down.
-
Feed the failures back into the agent
Use the dimensions that scored low to revise the agent's prompt or instructions, then re-score the next output against the same rubric to confirm it improved.
Why This Works
- A rubric makes two reviewers grade the same output the same way
- Weighting correctness and safety highest reflects what actually matters
- A capping rule stops a polished but unsafe answer from scoring well
Best for
- Teams evaluating agents by ad-hoc reading
- Eval pipelines needing a structured score
- Agents judged on more than just correctness
Not for
- Generating the test inputs — use the Agent Test Scenario Prompt
- Comparing two versions for drift — use the Agent Regression Test Prompt
Use cases
- Scoring agent outputs consistently across reviewers
- Setting a numeric 'ready to ship' threshold
- Comparing prompt versions on the same rubric
FAQ
What pass threshold should I set for the agent evaluation scorecard?
The prompt leaves the threshold for you to set — it asks the model to "state it" rather than assuming a number. Pick a weighted-total cut-off that matches your risk tolerance (a customer-facing agent needs a higher bar than an internal draft), then apply it consistently so every output is judged against the same line. You run the scorecard in your own AI tool.
How are the weights set on this agent scorecard, and how is the weighted total calculated?
You choose the weights and the prompt makes them explicit — it instructs the model to "state the weights," with correctness and safety weighted highest by design. The weighted total combines each dimension's 0–5 score by its weight into one aggregate number, which is then compared to your threshold for the PASS or FAIL. Nothing is auto-graded; the model applies the rubric you supply.
Why did my agent fail the scorecard even though most dimensions scored high?
A high average can still FAIL because of the capping rule: a failing CORRECTNESS or SAFETY score caps the overall result regardless of how strong the other four dimensions are. So an output that is well-formatted, complete, and on-tone but wrong or unsafe won't pass. The one-line justification on each score shows exactly which dimension capped it.
Does a passing score on this scorecard mean the agent output is safe to ship?
No — a PASS means the output cleared the weighted threshold you set on this rubric, not that it's guaranteed safe or correct. The score is a structured signal that makes grading consistent and comparable, not a shipping sign-off; a rubric only checks the dimensions you defined against the expected result you supplied. Treat a PASS as evidence for a human ship decision, not a substitute for one.
More resources from AI Output Validator
Resource
Validate AI Output — Catch Format Violations
Paste the response, get the verdict: real JSON parsing, missing-field detection, and a repair prompt for everything found.
Resource
Agent Failure Analysis Prompt
Turn a failed case into a fix — diagnose where in the agent's flow it went wrong, categorize the failure, and point at the prompt, tool, or context that caused it.
Resources that pair well
Resource
Force JSON Output from AI
Stop getting 'Sure, here is the JSON…' — the output-contract pattern that forces models to return only parseable JSON: schema, example, and a strict rule block.
Resource
Extract Data From Text with AI
Free text in, named fields out. The extraction prompt pattern that turns any unstructured text into consistent, parseable records.
Resource
Classify Support Tickets with AI
Billing, Technical, Account, How-To, Feature Request — ticket triage with definitions that decide the borderline cases for the model.
Related tools
Tool
AI Output Validator
Paste an AI's output and validate it against the expected format — with a repair prompt for every failure found.
Projects that use this resource
Project
Build an AI Support Agent with AI
The full path to a support agent you can put in front of customers — write its instructions, ground it in your docs, route and handle tickets, then evaluate and cost-control it before it goes live.
Project
Build a Customer Support System with AI
The full path to a support operation, not just a bot — stand up the knowledge base, route the tickets, add the AI agent, integrate your stack, close the feedback loop, evaluate, and deploy.
Guides for this resource
Guide
How to Make AI Follow Acceptance Criteria
The output reads well and answers the question, so you almost ship it — then you notice it broke a rule you cared about and skipped a section you needed. "Looks good" was never the test. Here's how to give AI the acceptance criteria up front and check the output against them before you rely on it.
Guide
How to Create an AI QA Checklist Before Using Generated Content
AI hands you a polished email, landing page, or support reply that looks ready to send — and only after you send it do you find the claim you can't back or the audience you got wrong. Here's how to build a reusable QA checklist and review generated content against it first.
Guide
How to Evaluate an AI Agent With a Scorecard
Build an evaluation scorecard two people can apply to the same agent run and reach the same number — dimensions that don't measure each other, anchors you can point at, and the failures that shouldn't average out.