Classify and Tag Data With AI Consistently
AI tags one batch pricing, billing, bug — then tags the same items cost, plan confusion, error next week, so every count measures a moving target. Here's how to classify and tag data consistently: fix a taxonomy, define every label, and hold the rules steady across runs.
Build a Data Classification PromptWhen the same feedback gets a different label every run
You hand the AI a stack of customer feedback and say "tag these by topic." The first batch comes back clean: pricing, billing, bug, UX, performance. Next week you run the same prompt on the same kind of items and it comes back cost, plan confusion, error, onboarding, slow. Nothing has gone obviously wrong — every label is reasonable — but "pricing" and "cost" are now two labels for one thing, "bug" and "error" a second pair, and your dashboard is counting them separately. The taxonomy quietly split, and every number built on it is measuring a moving target.
That is what this guide fixes. To classify and tag data with AI consistently, you set the taxonomy before the text arrives instead of letting the model invent one per run: a fixed, closed label set, a one-sentence definition for each label, rules for the borderline cases, and a lane for the items that fit nothing. NewPrompt builds that classification prompt for you — the Data Classification Prompt assembles the label set, definitions, and rules into one prompt — and it is honest about the boundary. It doesn't import your spreadsheet, upload a CSV, run the model, label your rows, store your data, or measure how accurate the labels are. You paste your items into your own AI assistant or API; the model returns candidate labels; you spot-check them. A fixed taxonomy makes the model apply the same rules every run, which lowers the drift — it doesn't guarantee identical output.
Why AI tagging drifts between runs
A model told to "tag by topic" is inventing the taxonomy on the fly, every single time. With no label set to choose from, it generates labels from whatever wording this particular batch happens to use — so the taxonomy is a side effect of the input rather than a fixed thing you control. Here is where the consistency leaks out:
- There's no label set to choose from. "Tag by topic" lets the model mint a fresh label per item, so the same complaint becomes "pricing" one run and "cost" the next.
- The labels have no definitions behind them. "Billing" means whatever the model assumes this run — invoices today, plan questions tomorrow — so identical items land in different buckets.
- Synonyms run free. Nothing tells it that "refund" belongs under billing, or that "cost" should map to the canonical "pricing," so near-identical meanings scatter across near-identical labels.
- Ambiguous items get forced. An item that fits nothing gets crammed into the nearest label instead of flagged, so wrong labels get made silently and "other" fills with things that should have been reviewed.
- The taxonomy isn't carried between batches. Classify 200 rows in four calls and each call re-derives the label set unless you pin it — batch two doesn't know what batch one decided.
- Examples get copied as patterns. Show a few labeled samples with no rules and the model pattern-matches the surface features of your examples instead of applying a definition.
Step 1: Fix the job and the label set before any data
Decide two things before a single row goes in: what the classification is for, and the exact, closed set of labels it may use. The purpose narrows the set — "route to a support queue" wants a handful of routing categories; "measure feature demand" wants request types — and a closed set is the whole difference between labeling and inventing. Write the labels down as the only allowed answers, and say so in the prompt: use only these labels, and never create a new one.
The Data Classification Prompt builds this scaffold for you — you supply the labels and it assembles a prompt with the label set, the classification rules, the edge-case rules, and an output contract wired together. The Text Classification Prompt is the anatomy laid out to study: the five blocks a reliable classifier needs — defined labels, classification rules, edge-case rules, an ambiguity policy, and a confidence contract — built around a worked taxonomy you can adapt. Both hand you a prompt you run yourself; NewPrompt writes the structure, and your own AI does the labeling.
Step 2: Define every label, and say what each one excludes
A label name is a guess; a definition is a rule. "Billing" alone forces the model to decide for itself where billing ends and pricing begins, differently each run. One sentence settles it: "Billing: charges, invoices, refunds, duplicate payments." The borderline items — the ones that actually cause the drift — then get decided by your sentence instead of the model's mood. Spend your effort here, because the definitions are the classifier, not the label names.
Write definitions that exclude, not just include. "How-To Question: the user asks how to do something that works as designed — nothing is broken" keeps it clear of "Bug" far better than two cheerful positive definitions would. If your domain has synonyms, name the canonical label and fold the variants into it — "cost and plan-pricing both map to your canonical pricing_plan" — so the model collapses them instead of spawning new labels. You can anchor a hard border with a single near-miss — "'I was charged twice' is billing, not pricing_plan" — but keep it as a rule illustration, not a template to echo: the definition does the work, and the example just points at where the line runs.
Step 3: Choose single- or multi-label, and give the misfits a lane
Decide up front whether each item gets exactly one label or may get several, because it changes the edge-case rules. Single-label suits routing, where one item goes to one place: "choose the label matching the item's primary purpose; on a tie, prefer the more specific definition." Multi-label suits analysis, where one message can be praise and a bug report and a feature request at once: "return every label that genuinely applies, strongest first — a merely mentioned topic doesn't earn its label." Picking one mode and stating its rule stops the model from hedging with two answers when you need one, or flattening three real signals into one when you need all three.
Then give the model somewhere honest to put what doesn't fit. A closed set with no escape hatch forces every ambiguous or thin item into a real label, which is how wrong labels get made silently — they look processed. Add a label like "needs_review: unclear, thin, or fits no label above — explain why," and instruct the model to choose it over guessing. The Classify Support Tickets setup shows the single-label version with a confidence lane for routing; the Categorize Customer Feedback setup shows the multi-label version, where one message legitimately carries three labels ordered strongest-first. One caveat to keep: needs_review catches what the model recognizes as uncertain — it won't catch the item it is confidently wrong about, which is why Step 5 exists.
Step 4: Ask for the evidence and a reason, and treat confidence as a routing signal
A bare label is unauditable — you can't tell a solid call from a coin flip. Ask the model to return, with each label, the span of text it is labeling (the words that triggered it) and a one-line reason tied to the definition (why this label and not its nearest neighbor). Evidence and reason turn a spot-check from re-reading the whole item into glancing at what the model actually keyed on, and they expose the failure you most want to catch: a confident label attached to a phrase that doesn't support it.
Confidence is useful and easy to over-trust. If you ask for it, read it as the model's self-report of how well the label fits — high/medium/low, or a 0-100 band with the bands defined — not a measured probability of being right. Its real use is triage: auto-accept the confident calls, and send the low-confidence ones to a person. Don't average the scores or treat "confidence 90" as "90 percent correct" — the number is self-assessed, and a model can be confidently wrong. What confidence buys you is the split between what you can automate and what a human should see, not a figure you can put in a report.
Step 5: Carry the same taxonomy into every batch, then spot-check
Consistency across a large job is a mechanical discipline: the same label set, the same definitions, and the same rules go into every call. If you classify 500 rows in ten batches, batch ten has to carry the identical taxonomy batch one used — a template you reuse verbatim, not a prompt you retype and quietly reword. Test the taxonomy on a small labeled sample first: hand-label 20 items yourself, run them through, and compare. The disagreements tell you which definition is fuzzy before you spend the whole batch on it — fixing a border definition after 20 items is cheap, and after 500 it is a re-run.
Then check the output, because a clean-looking labeled file can hide all the drift you set out to prevent. Sort by label and read the edges: do near-duplicate items carry the same label? Did any label appear that isn't in your set — a sign the model invented one despite the rule? Read the needs_review and low-confidence rows, because that is where the taxonomy is thin. And keep in mind what the labels actually are: candidate classifications the model produced from your rules, not verified truth. For anything with real consequences — moderation, compliance, routing that affects a customer — the person who owns that domain reviews the labels before they drive a decision. NewPrompt structures the check; running it, and owning the result, is yours.
Common mistakes
The habits that let a tidy-looking tagging job drift:
- Giving the model no label set. "Tag by topic" invents the taxonomy every run; hand it a fixed, closed list of the only allowed labels.
- Shipping labels with no definitions. Names alone let the model decide the borders, differently each time; one defining sentence per label is where consistency actually lives.
- Leaving no lane for misfits. A set with no needs_review forces ambiguous items into real labels silently; add the escape hatch and tell the model to prefer it over guessing.
- Rewording the taxonomy between batches. Retyping the prompt each call quietly changes it; reuse the exact same label block verbatim so batch ten matches batch one.
- Reading confidence as accuracy. A self-reported "high" is the model's opinion of fit, not a probability that it's right — route on it, don't trust it as truth.
- Trusting the labeled file without a spot-check. Near-duplicate items with different labels, and invented labels, only surface when you sort by label and read the edges.
A worked example: feedback tagged two ways, then pinned down
Watch the same kind of feedback get labeled two different ways, then held to a fixed set.
"Tag by topic" relabels the same meaning every run; a fixed label set with definitions, a needs_review lane, and an evidence field pins each item to a rule you can checkTHE WEAK PROMPT (no label set, no definitions):
"Tag these customer feedback items by topic."
THE SAME KIND OF ITEMS, TWO RUNS:
run 1 labels: payment | bug | UX | pricing | performance
run 2 labels: billing | error | onboarding | plan cost | slow
-> "payment" vs "billing" and "pricing" vs "plan cost" are the same
meaning under different labels; any count built on them is unstable.
THE ITEMS:
1. I was charged twice this month.
2. The export button fails on large reports.
3. Love the dashboard, but onboarding was confusing.
4. Can I pay annually instead of monthly?
5. The app is slow after I upload a CSV.
A FIXED TAXONOMY (labels + definitions + rules):
Use ONLY these labels:
billing -- charges, invoices, refunds, duplicate payments
pricing_plan -- plan options, annual vs monthly, upgrade/downgrade
bug -- a feature fails or errors
performance -- slow, timeout, or lag on upload/export
onboarding -- setup or first-use confusion
praise -- positive feedback with no specific issue
needs_review -- unclear, thin, or fits no label above
Rules:
- Use the canonical labels only; never invent a new one.
- Multi-label only when the item clearly holds two separate issues.
- If unsure, use needs_review and say why.
- Return the evidence span and a one-line reason with each label.
THE SHAPE YOU WANT BACK:
[
{ "id": 1, "labels": ["billing"],
"evidence": "charged twice", "reason": "duplicate charge, not a plan question",
"confidence": "high" },
{ "id": 4, "labels": ["pricing_plan"],
"evidence": "pay annually instead of monthly", "reason": "plan cadence, not a billing error",
"confidence": "high" },
{ "id": 5, "labels": ["performance"],
"evidence": "slow after I upload a CSV", "reason": "slowness on upload; no feature failure",
"confidence": "medium" }
]
NEXT: run this yourself on a 20-row sample first, compare it to your own
labels, and tighten any definition the disagreements expose.Where this fits in NewPrompt
Classifying is one job in the structured-output family, and it earns its keep by staying narrow: assign an item to a label you defined in advance. The Data Classification Prompt builds the classification prompt — label set, definitions, edge-case and ambiguity rules, and the output contract. The Text Classification Prompt lays out the anatomy to learn from; Classify Support Tickets and Categorize Customer Feedback are the single-label and multi-label versions worked end to end. Each gives you a prompt you run in your own AI tool — NewPrompt writes the rules, and the labeling happens on your side, on your own data.
Two neighbors are easy to confuse with this one. Pulling values that are literally in the text — a date, a total, a name — is extraction, not classification: extraction reads an answer out of the source, while classification chooses one from a set you defined before the text ever arrived. And making the output parse as valid JSON is a format question; keeping "bug" and "error" from becoming two names for one thing is a meaning question, and the meaning is what drifts. This guide is about the meaning — the taxonomy, and holding it steady.
When the classification has to feed automation — a router, a triage queue — the AI Classification Workflow wraps this step in the two that make it safe to build on: isolate the exact unit to classify first, and afterward validate that the returned label is one you actually allow before anything downstream acts on it. That validation checks the label is in your set and the shape is right; it doesn't check the label is correct. That is still the spot-check, and still yours.
A taxonomy is a ruler, and its whole value is not changing length between the things you measure. Labels that drift — "pricing" one batch, "cost" the next — give you numbers that look precise and mean nothing, because each batch was measured against a different scale. Fixing the set, defining every label, and carrying the same rules into every run is what keeps the ruler the same length from the first item to the last. The model applies the rules; whether the ruler held, and whether any given label is right, are what you check.