Handle Unknown vs Empty vs Not-Applicable in AI Output
Ask AI to extract fields with "use null if missing" and one null means missing, false, a stated "none", and a real zero — valid JSON, wrong meaning. Handle unknown vs empty vs not-applicable in AI output: give each field a value state so absence gets classified, not collapsed.
Build the Extraction PromptThe null that means five different things
You ask the AI to extract some fields into JSON and, being careful, you add "use null if missing." It comes back clean and parseable. Then you look closer: data_retention_days is null, sso_available is null, incident_contact is null, active_incidents is null. But those nulls aren't the same thing. Retention was genuinely absent from the source. SSO isn't null at all — it's false on one plan and true on another. The incident-contact field said "none listed," which is a stated value, not a missing one. And active incidents was zero, which is a number, not an absence. One null is now standing in for "missing," "false," "explicitly none," and "zero" — four different facts flattened into the same empty cell. The JSON is valid. The meaning is wrong.
The trouble is that "empty" in an AI output is never one thing, and a single catch-all like null erases the distinctions that matter downstream. A field can be present, present-but-empty, missing from the source, not applicable to this record, genuinely unknown, or a real zero or false — and a spreadsheet, a database, a support-triage rule, or a dashboard reads each of those differently. Handling this means giving the model a value state to attach to each field, so absence gets classified instead of collapsed. This guide is how to handle unknown vs empty vs not-applicable in AI output: define the states, make each field report its own, and keep zero and false from disappearing. NewPrompt's Extraction Prompt Generator is where you build the extraction prompt with those per-field rules. The boundary: NewPrompt doesn't validate your data, parse or upload your source, or write anything into a database — it helps you build the prompt; you run it on your own source, and the value state the model assigns is a candidate you review, not a verified fact.
Why "use null if missing" quietly loses information
"Use null if missing" sounds like a safe default, and for a single field with one kind of absence it's fine. The problem is that real sources produce several kinds of absence at once, and one null can't tell them apart — so the model, given no finer instruction, maps all of them to the same empty value, and the distinctions are gone before they reach your data. Here is what a single catch-all loses:
- Zero becomes "no data." "Active incidents: 0" is a real, meaningful number, but a model told to null out anything that looks empty can turn a true zero into a null — and now you can't tell "none" from "unrecorded."
- False becomes null. "SSO is not offered on the Starter plan" is an explicit false, not a missing value; collapsed to null, a real negative answer reads as an unanswered question.
- Not-applicable looks like missing. A field that doesn't apply to this record (a delivery slot for a digital product) is different from one that applies but wasn't provided — but null hides which is which.
- Explicitly-stated absence looks the same as unrecorded. A source that says "none listed" has told you something; a source that's simply silent has not. Both become null, and the difference — often the whole point — vanishes.
- Unknown looks like decided. When the model isn't sure, a bare null reads as a confident "there's nothing here," with no flag that a human should look.
- The consumer can't recover the meaning. Downstream, one null has to serve every reader — the code, the analyst, the triage rule — and each guesses its meaning differently, so the same empty cell drives three different wrong decisions.
Step 1: Name the value states before you extract
Before you write the extraction, decide what your states are — the closed set of things "not a plain present value" can mean for your data. A practical set: present (a real value is there), empty_but_present (the field exists but holds nothing, like a stated "none"), missing_from_source (the value could exist but the source doesn't give it), not_applicable (the field doesn't apply to this record), unknown (the model can't determine it), and needs_review (something's ambiguous enough that a human should decide). You won't need every state for every project — a simple form might use only present, missing_from_source, and unknown — but naming the set up front is what turns "empty" from a guess into a labeled decision the model has to make on purpose.
The states only work if you also say when each applies, because the model's instinct is to reach for the easiest one. Spell out the rules: a value the source states, even "none" or "0", is present; a value the source is silent about is missing_from_source; a field that structurally doesn't apply is not_applicable; a value the source hints at but doesn't state is unknown, not a guess. The difference between not_applicable and missing_from_source is the one that trips models most — "doesn't apply here" versus "should exist but isn't given" — so define that boundary explicitly, with an example, rather than trusting the model to infer it.
Step 2: Give each field a value_state, not just a value
The mechanism is simple: instead of returning a bare value per field, return an object — the value plus its value_state from your defined set. Writing active_incidents as {"value": 0, "value_state": "present"} says something a bare 0 or a bare null can't: this is a real zero, stated in the source, not an absence. The value carries the data; the value_state carries what the data's emptiness or presence means. This is the move that lets a downstream consumer treat missing_from_source and not_applicable differently, because they finally arrive as different things instead of two identical nulls.
Building that structure into the prompt is what the Extraction Prompt Generator is for — you define the fields and the rules for each, and it assembles the extraction prompt with the field list, the extraction rules, and the missing-data contract in one place. The JSON Output Prompt Builder is the companion for pinning the shape when you want the value_state enum written into the output contract itself, so every field comes back as the same value-plus-state object. Each builds a prompt you run on your own source in your own AI tool; they shape what the model returns, they don't run the extraction or check that the states are right.
Step 3: Protect zero and false — they are values, not gaps
Two specific values get destroyed by careless emptiness-handling more than any others: 0 and false. Both look, to a model optimizing for a clean output, like "nothing here" — and both are real answers. Zero incidents is not the same as an unrecorded incident count; "not offered" is not the same as "not mentioned." Write the rule explicitly: if the source states a value, including 0 or false, return it with value_state present — never convert a stated zero or a stated negative into null, empty, or missing. This single rule prevents a whole class of silent data loss, because it's exactly the case where "looks empty" and "is empty" diverge.
The false case has an extra trap: a negative statement often carries a scope that a lone boolean loses. "SSO is not offered on the Starter plan" is false for Starter and, two sentences later, true for Enterprise — so a single sso_available field forced to one boolean throws away half the answer. When a value depends on a plan, a tier, a date, or a segment, that's a signal to split the field or flag it for review, not to pick one value and drop the condition. A false that's true somewhere else isn't a boolean; it's a boolean per context, and collapsing it is how a plan-specific fact becomes a flat, wrong one.
Step 4: Separate missing-from-source, not-applicable, and unknown
These three are the absences most worth keeping apart, because they call for different downstream actions. Missing_from_source means the value plausibly exists but the source didn't provide it — the action is to go find it. Not_applicable means the field doesn't apply to this record at all — the action is to leave it, correctly, blank forever. Unknown means the model couldn't determine it from what it had — the action is a human check. Collapse them to one null and every one of those actions becomes a guess. Tell the model to use each precisely: silence in the source is missing_from_source; a structural non-fit is not_applicable; genuine model uncertainty is unknown or needs_review — and none of them is an excuse to invent a value.
The Missing Data in AI Extraction resource is a worked example of the representation half of this — the four behaviors a field can take when it isn't in the text (leave empty, return null, return unknown, or skip the key), and which consumer each one suits. The constant across all of them is the rule this whole guide depends on: never invent or guess a value for a missing field. A value state only means something if the model is honest about which state a field is really in — a fabricated value with a confident present state is worse than an accurate unknown, because it hides the gap instead of flagging it. The resource is a template you adapt; whether the model classified an absence correctly is something you check against the source.
Step 5: Carry the evidence and review the state assignments
A value state is a claim about the source, so make the model show its work: alongside value and value_state, have it return the evidence — the quote or location the state is based on — and a short reason. A field returned as value null, value_state empty_but_present, evidence "Incident contact: none listed," reason "field is mentioned but no contact is given" is a row you can audit in seconds; the same field as a bare null is a row you have to reopen the source to understand. The evidence is what turns the value state from the model's assertion into something you can confirm or overturn, which matters because the model gets these wrong — it will call a not_applicable a missing, or a missing an unknown, in ways only a look at the source reveals.
This classification is one stage of a larger extraction discipline — the AI Data Extraction Workflow runs the full path (bound the source, extract the fields, force a clean JSON shape, and validate before anything flows downstream), and the value-state policy lives in the extract-and-validate steps. NewPrompt gives you the prompts and the structure; running the extraction, checking the state assignments against the source, and deciding how each state maps into your database or spreadsheet all happen on your side. The output is a candidate structured draft, not verified data — the model can misclassify a state, and for anything high-impact (legal, medical, financial, security, compliance), the value states and the values behind them need a domain owner's review before they're imported or acted on. Valid JSON with the wrong value state is still wrong; the states make the meaning reviewable, they don't make it correct.
Common mistakes
The habits that flatten real distinctions into one empty value:
- Using null as a catch-all. "Use null if missing" collapses missing, not-applicable, empty, zero, and false into one cell; give each field a value_state instead of one overloaded null.
- Letting zero or false become null. A stated 0 or a stated "not offered" is a present value; write the rule that any value the source states is returned as present, never emptied.
- Confusing not-applicable with missing. "Doesn't apply to this record" and "applies but wasn't provided" need different downstream actions; define and separate them explicitly.
- Collapsing a context-specific value into one field. When a value differs by plan, tier, or date, splitting the field or flagging it beats forcing one value and dropping the condition.
- Treating unknown as decided. Model uncertainty should surface as unknown or needs_review with a flag, not as a confident null a reviewer will skip past.
- Trusting valid JSON as correct data. NewPrompt doesn't validate the values or the states — parseable output can still carry the wrong meaning, so check the state assignments against the source before you rely on them.
A worked example: extracting vendor security fields
Watch "use null if missing" flatten a vendor profile into four identical nulls that hide a zero, a false, a stated "none," and a real gap, then a value-state prompt keep each distinction — with the evidence attached.
"Use null if missing" flattens a vendor profile into four identical nulls that hide a real zero (active incidents), an explicit false (no SSO on Starter), a plan-specific true (SSO on Enterprise), a stated "none listed," and a genuine gap; a value-state prompt returns each field as value-plus-state-plus-evidence so present / missing_from_source / empty_but_present / not_applicable stay distinct — a candidate you check against the source before importTHE SOURCE (vendor security profile):
"Acme stores customer data in the EU. Acme is SaaS-only. SSO is not
offered on the Starter plan. The Enterprise plan includes SSO. Data
retention is not specified. Incident contact: none listed. Active
incidents: 0."
THE WEAK ASK, AND WHAT IT GIVES BACK:
ask: "Extract vendor security fields as JSON. Use null if missing."
{
"data_region": "EU",
"sso_available": null, <- but it's false (Starter) AND
true (Enterprise)
"data_retention_days": null, <- missing from source
"incident_contact": null, <- source says "none listed"
"active_incidents": null <- but it's 0, a real value
}
four nulls, four different meanings, all flattened
A VALUE-STATE PROMPT:
Extract each field as { value, value_state, evidence_quote,
reason (when the state isn't obvious) }.
value_state is one of: present | empty_but_present |
missing_from_source | not_applicable | unknown | needs_review
Rules:
- A value the source STATES, including 0 or false, is present.
- Field named but stated AS none/empty (e.g. "none listed")
-> empty_but_present, with the quote.
- Field named but the value is NOT DISCLOSED (e.g. "not
specified") -> missing_from_source: it likely exists to find.
- Field not mentioned at all -> missing_from_source.
- Field that cannot apply to this record -> not_applicable.
- If a value differs by plan/tier, split the field; do not
collapse it into one.
- Never invent a value; if unsure, use unknown or needs_review.
PART OF WHAT COMES BACK (a candidate you review):
{
"data_region": {
"value": "EU", "value_state": "present",
"evidence_quote": "stores customer data in the EU" },
"sso_available_starter": {
"value": false, "value_state": "present",
"evidence_quote": "SSO is not offered on the Starter plan" },
"sso_available_enterprise": {
"value": true, "value_state": "present",
"evidence_quote": "Enterprise plan includes SSO" },
"data_retention_days": {
"value": null, "value_state": "missing_from_source",
"evidence_quote": "Data retention is not specified",
"reason": "field named but value not disclosed -- likely exists to find" },
"incident_contact": {
"value": null, "value_state": "empty_but_present",
"evidence_quote": "Incident contact: none listed" },
"active_incidents": {
"value": 0, "value_state": "present",
"evidence_quote": "Active incidents: 0",
"reason": "zero is an explicit value, not missing" },
"on_prem_region": {
"value": null, "value_state": "not_applicable",
"evidence_quote": "Acme is SaaS-only",
"reason": "on-premise region cannot apply to a SaaS-only vendor" }
}
NEXT: you check each value_state against the source, decide how your
database maps them (is missing_from_source a null or a follow-up
task?), and -- for security data -- have the owner review it. Valid
JSON was never the hard part; the meaning is.Where this fits in NewPrompt
Handling value states is part of designing a structured output, and NewPrompt gives you the tools for the prompt, not a data pipeline. The Extraction Prompt Generator builds the extraction prompt where the per-field rules and the missing-data contract live; the JSON Output Prompt Builder pins the output contract when you want the value_state enum baked into the shape; and the Missing Data in AI Extraction resource shows the four ways a field can represent an absence and which consumer each suits. The AI Data Extraction Workflow is the larger path this sits inside — bound the source, extract, force clean JSON, validate before downstream. Each builds a prompt or shows a pattern you run on your own source, in your own AI tool.
This guide sits among the structured-output guides as the one about what an empty value means. Stopping AI from inventing missing data is the rule underneath it — never fabricate a value for an absent field; this adds the next question: given that a field is absent, which kind of absent is it? Making AI return valid JSON is about syntax the parser accepts; this is about semantics the syntax can't see — a null is valid JSON and can still be the wrong answer. And designing an extraction schema decides which fields to pull; value states decide how each field represents presence and absence within that schema. The through-line: parseable is not the same as meaningful, and the meaning lives in the states.
An empty cell is a word with no context, and "empty" is the most overloaded word in a dataset. A blank in a spreadsheet can mean "we checked and it's zero," "we never checked," "this doesn't apply here," or "someone deleted it" — four sentences compressed into one absence of ink, and the reader supplies whichever meaning they expect. A value state is the missing context written back in: it turns the blank from something the next person has to interpret into something the data already says. The AI can propose which kind of empty each field is, faster than reading every source by hand — but which blank means what, and whether the model read the source right, is a judgment you make against the source, not one the empty cell can make for you. A dataset full of confident blanks is a dataset full of questions no one wrote down.