Resource
Playwright Test Prompt
getByRole over CSS chains, auto-wait over sleep, web-first assertions — Playwright tests written the way Playwright wants.
Agent Safety & Refusal Evaluation Prompt
Test the two failure directions — does the agent refuse what it must, and does it stay helpful on the benign requests it shouldn't over-refuse?
Overview
Agent safety has two failure modes, and most evals only test one. This prompt evaluates both: does the agent refuse the genuinely out-of-bounds requests (harmful, policy-violating, out-of-scope, injection), and does it NOT over-refuse the benign ones that merely pattern-match as risky? It scores refusal correctness in both directions, because an agent that refuses everything is as broken as one that refuses nothing.
How to use this resource
-
Assemble both kinds of request
Gather the genuinely out-of-bounds requests the agent must refuse and the benign ones it must not over-refuse. Safety is scored in both directions, so you need both.
-
Open this resource in Test Case Prompt Generator
Load the prompt into Test Case Prompt Generator and fill in your policy and scope. It builds the refusal and over-refusal cases so the eval covers both failure modes.
-
Review the refusal scoring
Read how the agent scored on refusing what it should and staying helpful on what it should not, noting any case it got backwards.
-
Tighten the boundary and re-run
Use the failures to adjust the agent refusal instructions, then re-run the set so it neither under-refuses nor over-refuses.
Why This Works
- Testing both directions catches the over-refusal most safety evals ignore
- Separating false-refusals from missed-refusals shows which way to tune
- Judging refusal quality avoids the blunt-wall refusals that frustrate users
Best for
- User-facing agents with safety boundaries
- Agents exposed to adversarial input
- Teams tuning the refuse/help balance
Not for
- Pure correctness evaluation — use the Agent Evaluation Scorecard
- Generating the adversarial cases — use the Agent Test Scenario Prompt
Use cases
- Testing agent refusals on harmful and injection inputs
- Catching over-refusal that hurts benign users
- Evaluating refusal quality, not just refusal presence
FAQ
How is this different from a normal agent evaluation — does it only test refusals?
It tests refusal behavior in both directions, which most safety evals miss: does the agent refuse the genuinely out-of-bounds requests (harmful, out-of-scope, policy-violating, injection), and does it stay helpful on benign requests that only look risky? Over-refusing a safe request is scored as a real failure, not a safe default. Pure correctness — is the answer right — is a separate scorecard.
What do I give the agent safety and refusal evaluation prompt?
Two inputs: the POLICY — the agent's scope and safety rules — and a set of cases paired with the agent's actual responses. It classifies each case SHOULD-REFUSE or SHOULD-HELP and judges the response against that, so you need both harmful and benign cases. Generating those adversarial cases is a separate step, not something this eval produces.
What does the agent safety evaluation output look like?
For each case it classifies the request, judges whether the agent handled it correctly, and rates refusal quality — graceful and pointing somewhere useful, or a blunt wall. It counts correct versus incorrect in both directions and lists false-refusals (over-cautious) and missed-refusals (under-cautious) separately, flagging any jailbreak that produced restricted output as CRITICAL.
Does passing this eval mean my agent is safe?
No — it scores only the cases you supply, in both directions, so a clean run means the agent handled your test set, not that it's safe against everything. Coverage is only as strong as the adversarial and benign cases you feed it. Run it in your own AI tool, treat the CRITICAL flags as blockers, and keep the safety sign-off with a human.
More resources from Test Case Prompt Generator
Resource
Unit Test Prompt — Isolation Done Right
Mock the dependencies, test the business logic, one behavior per test — the unit testing contract that bans plumbing tests.
Resource
Agent Test Scenario Prompt
Build the test set an agent has to pass — scenarios across the happy path, edges, and adversarial inputs, each paired with the expected behavior to grade against.
Resources that pair well
Resource
Code Review Prompt — the Review Contract
"Review this code" gets shallow comments. The review contract gets findings with severities, a checklist, and a verdict.
Resource
Debugging Prompt — the Investigation Contract
"Fix this error" gets guesses. The investigation contract gets a ten-stage diagnosis: facts separated from assumptions, alternatives weighed, fixes justified.
Resource
Fix Invalid JSON from AI
The JSON won't parse and you can't see why. Deterministic cause-sniffing — trailing commas, single quotes, unclosed brackets — and the repair prompt that fixes it.
Related tools
Tool
Test Case Prompt Generator
Build test generation prompts — unit, integration, or E2E — with framework modes and edge-case coverage rules.