As of November 2, 2026. Documented capabilities and the implementation proposals below are separate. Examples use a fictional document. No paid API evaluation or model-accuracy benchmark was run for this article.
Parsing success is the beginning
Suppose an LLM extracts numbers from research documents. The response is JSON and value is numeric. Is it ready for a database? A model can change 12.5 million into 12.5 billion without violating a numeric type. It can also put last quarter’s figure into the current quarter’s field.
Structured output is useful infrastructure. Receiving a correctly shaped result and receiving a source-correct result are different checks. A system that reliably stores incorrect numbers can pass every syntax check.
Anthropic documents schema-constrained JSON outputs separately from strict tool-input validation, with supported JSON Schema features and exceptions. Adopting these features does not establish business meaning or source accuracy. Structured outputs documentation
Separate three checks
The proposed sequence is format, business rules, and evidence.
| Check | What it establishes | What can remain wrong |
|---|---|---|
| Format | Parseability, required fields, types, allowed keys | Values may disagree with the source |
| Business rules | Units, periods, ranges, cross-field constraints | Plausible values may lack evidence |
| Evidence | Agreement with a registered source passage and value | The source itself or its interpretation |
JSON Schema’s required and additionalProperties express mandatory keys and whether extra keys are allowed. They do not determine whether a revenue figure is true. Application validation remains necessary alongside the provider’s supported schema features. JSON Schema object reference
Original design diagram. A failure at any check holds the write. Evidence matching establishes agreement with a source, not that the source is true.
Let the model select evidence IDs
Free-form source URLs can be invented or point to irrelevant pages. An alternative is to assign document and passage IDs during collection and let the model return only IDs from the supplied set. The application owns the URLs, retrieval timestamps, and raw-file hashes.
Assume a fictional input document, demo-report, has this independently checked record:
{
"passage_id": "p1",
"quote": "Revenue for Q3 was USD 12.5 million.",
"metric": "revenue",
"period": "Q3",
"value": "12.5",
"unit": "USD million"
}This is not a new answer key generated by the extraction model. It must have been checked against the document by a reviewer or obtained with a trusted parser. Comparing one model’s extraction with its own generated answer key is not independent validation.
The model proposes the relevant record in the requested format. Unknown IDs are rejected. Check that quoted text exists, but a substring match does not establish that a passage supports the model’s interpretation. Negation, table headers, and consolidated versus standalone accounts still require interpretation.
A minimal comparison layer
This Python example compares an extraction candidate that has already passed schema validation. It makes no API call. The evidence registry is independently checked under the assumption above.
from decimal import Decimal, InvalidOperation
evidence = {
("demo-report", "p1"): {
"metric": "revenue", "period": "Q3",
"value": "12.5", "unit": "USD million",
"quote": "Revenue for Q3 was USD 12.5 million.",
}
}
def verify(candidate, registry):
key = (candidate["document_id"], candidate["passage_id"])
source = registry.get(key)
if source is None:
return "unknown_evidence"
for field in ("metric", "period", "unit", "quote"):
if candidate[field] != source[field]:
return "evidence_mismatch"
try:
value = Decimal(candidate["value"])
reference = Decimal(source["value"])
except (InvalidOperation, TypeError, ValueError):
return "invalid_number"
if not value.is_finite() or not reference.is_finite():
return "invalid_number"
return "matched" if value == reference else "value_mismatch"
candidate = {
"document_id": "demo-report", "passage_id": "p1",
**evidence[("demo-report", "p1")],
}
assert verify(candidate, evidence) == "matched"
assert verify({**candidate, "unit": "USD billion"}, evidence) == "evidence_mismatch"
assert verify({**candidate, "value": "125"}, evidence) == "value_mismatch"
assert verify({**candidate, "passage_id": "p9"}, evidence) == "unknown_evidence"The amount is converted from a string into Decimal. Strings 12.50 and 12.5 can represent the same number; million and billion are rejected as different units. Automatic unit conversion needs its own explicit rules and verification. Required keys and types are assumed to have been checked earlier, so this is not a complete server handler for arbitrary external JSON.
matched has a narrow meaning: agreement with registered evidence. Accounting definitions, source freshness, and later corrections remain separate questions. Free-form answers without an independent reference cannot be automatically approved using this comparison alone.
Classify failures before retrying
A network timeout can justify bounded retries. Repeatedly requesting an answer that lacks evidence can instead pressure the system to produce an unsupported answer.
Record empty documents, unknown evidence, unit conflicts, and insufficient support as distinct states and hold the write. Provider refusals and truncated responses must also be distinguished from normal extraction. Anthropic recommends allowing uncertainty and checking citations and claims to reduce hallucinations; these techniques do not eliminate the problem. Hallucination reduction guidance
Even a validated record should not grant external action authority. Storing an extraction, publishing a report, and placing an order need separate permissions and approval conditions. This makes the evidence layer in the AI research stack article more concrete.
Evaluate the approval decision too
Include normal documents, documents with no answer, figures repeated across periods, and tables with mixed units. Establish human-reviewed answers and hold conditions before comparing models, prompts, and parsers on the same set.
- Format pass rate: schema-compatible results across all requests.
- Exact-answer rate: correct value, unit, and period among answerable documents.
- Abstention behavior: holds on unanswerable inputs and unnecessary holds on answerable inputs, measured separately.
- Incorrect approval rate: wrong answers among approved records.
- Cost and latency per approved record: include failures, retries, and review work.
Record denominators and document counts so comparisons remain interpretable. This article did not run that evaluation and makes no model ranking for accuracy or cost.
Begin with one document type and one field. Expand once the system can trace an answer to its source, stop when evidence is missing, and explain which check caught an error.
References
- Anthropic structured outputs: format constraints and supported features.
- JSON Schema objects: required fields and extra properties.
- Anthropic hallucination reduction: uncertainty, citations, and grounding.
- The evidence registry and approval metrics are application-design proposals in this article.
