How to Extract Data From Documents With AI Without Losing Evidence
AI pulls the fields you asked for, but hands back a flat list with no way to tell which values it read from the document and which it guessed. Here's how to make each extracted value carry its source quote, location, and a review flag — so you can check the result instead of trusting it.
Try the Long Input FormatterWhen the extraction is right but you can't prove it
You give the AI a vendor contract and ask for the fields you care about — vendor, renewal date, notice period, contract value, governing law. Back comes a tidy list: "Renewal date: June 1. Notice period: 30 days. Auto-renewal: yes." It looks done. Then you try to use it and the questions start: June of which year? Where in the 40 pages did the notice period come from? Did the model read "auto-renewal" off a clause, or infer it? And the governing law it filled in so confidently — is that actually in the document, or the kind of value that's usually there? You can't tell, because the output is values and nothing else.
That's the gap this guide closes. The problem isn't only that AI sometimes invents data — it's that a flat result gives you no way to check the data it did return. A correct value and a lucky guess arrive in the same shape, with the same confident tone, and no pointer back to the text. You can't make document extraction perfectly accurate with a prompt. But you can make it reviewable: require a source quote, a location, and a review flag alongside every value, so each field can be checked against the document in seconds instead of re-read from scratch. This is about traceability of what the model extracted — a different job from stopping it inventing what isn't there, and the two pair well.
Why plain extraction leaves you blind
Ask for values and you get values — which sounds like exactly what you wanted until you have to trust them. A bare extracted record hides four different problems behind one clean surface:
- Read and guessed look identical. "Delaware" pulled from a governing-law clause and "Delaware" guessed because contracts usually name a state land in the field the same way. Without evidence, you can't separate them.
- No location means re-reading. To confirm a value, you go hunting through the document for the line it came from — the exact work the extraction was supposed to save.
- One tone covers every value. The model states a shaky reading and a certain one in the same voice, so nothing tells you which fields deserve a second look.
- Conflicts vanish. When a figure appears twice with different values, or a date is ambiguous, the model quietly picks one and moves on — the disagreement never reaches you.
Step 1: Ground the source so every value can cite a location
Evidence needs something to point at. Before you extract anything, package the document so its parts are addressable and the model is told to stay inside them. Wrap the source in clear delimiters, label each section, and state the rule up front: answer only from the delimited text, cite the section for anything you report, and say so plainly when the source doesn't cover a field.
The Long Input Formatter does this in your browser: it wraps your pasted text verbatim between explicit source markers, inserts [§N — title] section labels so any value can name where it came from, and — in its strict grounding mode — tops it with the discipline this whole approach rests on: a claim the model can't cite is a claim it shouldn't make, and a gap is answered with "the source does not say" rather than a plausible fill. It also tells the model to ignore any instructions hiding inside the document. It doesn't parse a PDF, upload a file, or read anything for you — it structures the text you paste so the extraction that follows can be traced back to a section.
Step 2: Define the fields — and the evidence each one carries
Now decide what to pull, and pull more than the value. A field in a source-backed extraction isn't a single cell; it's a small record: the value, a short quote that supports it, the location it came from, a confidence level, and a review flag for anything uncertain. Design the fields with that envelope from the start, because bolting evidence on afterward means re-running everything.
The Extraction Prompt Generator builds the extraction spine — your field list, per-field reading rules, and a missing-data behavior (leave empty, null, "unknown", or skip) so absent values get a defined marker instead of a guess. One honest limit to know: it names the information to pull and how to handle gaps, but it has no built-in evidence, quote, location, or confidence field — those you add as ordinary fields with your own descriptions. The Extract Contract Information with AI resource is a worked contract-extraction prompt you can start from: it pulls a register of contract fields under strict no-inference rules and returns "unknown" for absent terms, so the schema-and-missing-data half is already shaped — you extend each field into the full envelope. To pin that envelope as a strict, typed structure the output has to obey, the JSON Output Prompt Builder turns it into a contract with a type on every field and an example the model matches. None of these run the model or produce the evidence themselves; they assemble the prompt you run in your own AI assistant.
Step 3: Require a short, exact quote — not a paraphrase
The evidence field only works if it's verbatim. A paraphrase — "the contract allows early termination" — is the model's reading of the text, which is the thing you're trying to check, not proof of it. Ask instead for a short exact span copied from the source: enough words to be findable and to actually support the value, not a whole paragraph. A quote you can Ctrl-F in the document is a quote you can verify in one motion.
The Prepare Contracts for AI Review resource is a worked example of this discipline applied to a document: it packages contract clauses under strict grounding and asks the model to list obligations, deadlines, and caps while citing the section for each, preferring a verbatim quote over a paraphrase and reporting a missing term as an explicit "the source does not say" rather than assuming what a typical contract would include. That's the pattern to carry into extraction — every filled value tied to the exact words behind it.
Step 4: Carry the location, especially across a long document
A quote tells you what the model relied on; a location tells you where to find it. Require both. For a short document the section label from Step 1 is enough — "§4 Term & Renewal" points a reviewer straight at the clause. As documents get longer, the location field is what stops a quote from being unfindable in eighty pages.
This matters most when the document is too big to send in one message and you have to break it up. Whatever label each part carries — a section number, a page, a chunk marker — becomes the location each extracted value cites, so a field pulled from part seven still says so and stays traceable to it. The section labels the Long Input Formatter inserts are built for exactly this: they travel with the text, so a value's location isn't a guess about where it probably came from but the label that was actually next to it. Splitting the document itself is a separate job with its own pitfalls; the point here is narrower — carry the label through into the evidence so nothing you extract loses its address.
Step 5: Flag conflicts and low confidence instead of resolving them silently
Some fields are clean; some aren't, and the ones that aren't are where trust breaks. Give the model an approved way to say so: a needs_review flag it sets when a field is ambiguous, when the same value appears in two places with different content, or when it had to lean on a weak reading. A conflict routed to review is a small manual check; a conflict resolved silently is a wrong value you find out about later.
Confidence works alongside it, with one caveat to state plainly in the prompt and in your own head: a confidence level the model reports is self-reported, not a calculated probability. It's useful as triage — sort the low-confidence and flagged fields to the top of your review queue — but it is not a measure of correctness, and a field marked "high" still earns a look when it drives a real decision. Treat confidence as the model's own hint about where to look first, never as a verdict.
Step 6: Check the structure, then check the evidence yourself
A source-backed extraction gives you two checks, and they catch different things. First, structure: did every record come back with its value, quote, location, and confidence fields, in the right types, with the missing marker where it belongs? The AI Output Validator runs that check in your browser — paste the output and the expected shape, and it flags a missing field, a wrong type, or a dropped record, then hands you a repair prompt to send back. The Validate Structured Output from AI resource is a worked example of that loop on an extraction response: it catches a missing required field and an invented one and fixes them without touching what was already right.
But structure is where the tool stops. It confirms that an evidence_quote field exists; it can't tell you the quote is real, appears in your source, or actually supports the value beside it — that's content, not format, and no browser check verifies it. So the second pass is yours: read each value against its cited quote and location, and lean hardest on the flagged and low-confidence ones. This is also the honest boundary of the whole method — the evidence makes each value fast to review; it doesn't prove the value is correct, and for anything with legal, financial, or compliance weight the sign-off stays with you or a qualified reviewer, not the extraction.
Common mistakes
A few habits quietly strip the evidence back out of an evidence-backed extraction:
- Asking for values only. The most common one — a clean list of fields with nothing behind them is exactly the untraceable output the whole approach exists to avoid.
- Accepting a paraphrase as the quote. "The clause permits it" is the model's interpretation, not the source. If the evidence field isn't verbatim, it proves nothing.
- Making the quote optional. If evidence is only required "where available," the model drops it on precisely the values it was least sure of — the ones you most needed to see.
- Skipping the location. A quote with no address is findable in a one-page memo and lost in a long contract; without it, verifying means re-reading.
- Reading confidence as correctness. A self-reported "high" is a hint about where to look first, not a guarantee — treating it as a verdict is how a confident wrong value slips through.
- Mistaking evidence for verification. A cited quote that doesn't actually support its value still needs your eyes; the citation makes the check fast, it doesn't perform it.
A worked example: renewal terms from a vendor contract
Say you're pulling renewal terms out of a vendor contract. On the left is what plain extraction returns; on the right, the same fields with an evidence envelope — including one term the contract never actually states, now visible instead of invented.
The same contract fields, as flat values vs a source-backed envelope// Plain extraction — values only, nothing to check them against:
{
"vendor_name": "Acme Analytics",
"renewal_date": "June 1", // which year? read, or guessed?
"notice_period": "30 days", // from where in the contract?
"auto_renewal": true, // stated in a clause, or inferred?
"governing_law": "Delaware" // the contract never says this
}
// Source-backed extraction — each value carries its evidence:
{
"renewal_date": {
"value": "2026-06-01",
"evidence_quote": "This Agreement renews on June 1, 2026 unless terminated",
"location": "§4 Term & Renewal",
"confidence": "high",
"needs_review": false
},
"notice_period": {
"value": "30 days",
"evidence_quote": "written notice at least thirty (30) days before the renewal date",
"location": "§4 Term & Renewal",
"confidence": "high",
"needs_review": false
},
"governing_law": {
"value": null,
"evidence_quote": null,
"location": null,
"confidence": "n/a",
"needs_review": true,
"missing_reason": "not_stated_in_source"
}
}Where this fits in NewPrompt
One document with an evidence envelope is something you build by hand. Turn document extraction into a standing job and the AI Data Extraction Workflow gives it a fixed order to repeat — ground the source, spell out each field and how it marks a gap, lock the output shape, and run the structural check on what comes back. The Build an AI Document Processing System with AI project goes wider, treating extraction as one stage among intake, classification, and deployment; its own stage for scoring extraction accuracy against a set of known-correct outputs is where "are these values trustworthy" gets settled before anything runs unattended — the volume version of the per-field review this guide does by eye. What NewPrompt supplies throughout is the structure: the citable source, the field spec, the typed contract, the format check. It never reads your document, runs the model, writes the evidence, or confirms a quote is true; you run the prompt in your own assistant or API and hold each value against the source it points to.