Your pipeline needs a model to classify a support ticket and return a category and priority. You ask for JSON. It replies:
Sure! Here's the classification:
```json
{ "category": "billing", "priority": "high" }
```
Let me know if you need anything else!Now you are writing a parser that strips prose, finds the code fence, extracts the JSON, and handles the times it emits "priority": "High", or invents a category you never defined, or produces malformed JSON entirely.
This is the gap between a text generator and a component in a system. Closing it properly is the difference between a demo and something you can build on.
Three approaches, in increasing reliability
Diagram, as text
- Document leads to Model with constrained decoding
- Model with constrained decoding leads to Schema-valid JSON, always
- Schema-valid JSON, always leads to Values appear in source?
- Values appear in source? leads to Human review queue (no)
- Values appear in source? leads to Cross-field arithmetic holds? (yes)
- Cross-field arithmetic holds? leads to Human review queue (no)
- Cross-field arithmetic holds? leads to Accept automatically (yes)
Ask and parse
Put "respond only with JSON" in the prompt, then parse defensively with retries.
It works most of the time, which is precisely the problem: a 3% failure rate is invisible in testing and constant in production. Every downstream consumer needs error handling for a shape you nominally guaranteed.
Use it only when nothing better is available.
Schema-guided generation
Supply a JSON Schema and ask the provider to conform. Most now support this, and reliability improves substantially — the model is steered towards the shape rather than merely asked.
Better, and still fundamentally a strong preference rather than a guarantee unless enforced at the decoding layer.
Constrained decoding
The one that actually solves it.
Recall from the serving lesson that generation picks one token at a time from a probability distribution. Constrained decoding masks the tokens that would violate the schema before sampling. If the grammar permits only "billing", "technical" or "account" at this position, no other token can be selected.
The output is not checked afterwards — it is impossible to generate anything invalid. Enum values are guaranteed, required fields are guaranteed, types are guaranteed.
This is available from providers offering strict structured output, and directly in self-hosted inference servers via grammars.
| Approach | Guarantee | Cost |
|---|---|---|
| Prompt and parse | None | Retries, defensive parsing |
| Schema-guided | Strong preference | Occasional failures |
| Constrained decoding | Structural certainty | Slight decode overhead, less flexibility |
Designing schemas models handle well
Structural validity is not semantic correctness. A schema can be satisfied by nonsense, so the schema itself is a design surface.
Use enums wherever the value set is closed. This is the single biggest quality gain — it eliminates an entire class of invented values, and with constrained decoding it becomes impossible rather than merely unlikely.
{ "priority": { "enum": ["low", "medium", "high", "urgent"] } }Include an explicit escape value. Without one, a model forced to choose will pick the least-bad option rather than admit uncertainty. Add "unknown" or "other" and you convert silent wrong answers into a signal you can route to a human.
Keep them shallow. Deeply nested structures degrade quality noticeably. Flatten where you can.
Describe every field. Field descriptions are instructions the model reads — "confidence" is ambiguous, "Confidence from 0 to 1 that this classification is correct" is not.
Ask for reasoning before the answer, not after. A field ordered first is generated first, so the model can reason into it and condition the answer on that reasoning. Placing it after the answer means it rationalises a decision already made.
{ "properties": { "reasoning": {...}, "category": {...} },
"required": ["reasoning", "category"] }Where this changes architecture
Reliable structure is what lets a model be a component rather than a chat feature.
Tool calling is structured output underneath — the schema is the function signature, which is why the tool-use lesson leans on validation so heavily.
Extraction pipelines turn documents into rows. Guaranteed shape means you can write them straight to a database instead of into a quarantine table.
Routing and classification become dependable branches. A downstream switch on an enum is safe; a switch on free text is not.
Evaluation gets far easier. Comparing structured fields is exact; comparing prose requires judgement, as the evaluation lesson covers.
Costs worth knowing
Some flexibility is lost. A rigid schema cannot express "this document did not contain that information" unless you gave it a way to. Design nullable fields and escape values deliberately.
Quality can dip on hard reasoning. Heavy constraints can interfere with a model working through a difficult problem. The reasoning-first field pattern mitigates this; for genuinely hard tasks, consider generating freely and extracting structure in a cheap second pass.
Streaming is awkward. Partial JSON is not parseable, so you either wait for completion or use an incremental parser. For user-facing generation this matters; for backend pipelines it rarely does.
Schema changes are API changes. Consumers depend on the shape — version it like any interface.
Check your understanding
Common mistakes
Prompting for JSON and hoping. Works in testing, fails continuously at volume.
Free-text where an enum belongs. Invites invented values and unsafe branching.
No escape value. Forces a confident wrong answer instead of an honest one.
Reasoning after the answer. Produces rationalisation rather than reasoning.
Trusting structure as correctness. Validate referenced entities and ranges.
Deeply nested schemas. Measurably degrades quality; flatten.
Fine-tuning for format. Constrained decoding is cheaper and stronger.
Practice
Design the extraction schema for pulling structured data from invoices, where fields are sometimes genuinely absent and sometimes present but unreadable. Explain what makes a schema easy for a model to satisfy.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
The distinction between 'absent' and 'unreadable' is the interesting part.
Show a strong answer
Model the three states explicitly, because collapsing them loses the information a downstream process needs:
- Present and read — the value.
- Absent from the document — the invoice genuinely has no purchase order number.
- Present but not readable — it is there, but the scan is poor.
A nullable field cannot distinguish the last two, and they demand different handling: absent is final, unreadable should route to a human. So each extracted field becomes an object with a value and a status enumeration, rather than a bare nullable.
What makes a schema easy to satisfy:
- Flat rather than deeply nested. Every level of nesting is another place structure can go wrong, and deep schemas measurably reduce accuracy.
- Enumerations instead of free strings wherever the set is known — currency, status, document type. This converts a generation problem into a selection problem.
- Descriptive field names that carry meaning:
invoice_total_including_taxrather thantotal. The field name is an instruction, and it is read. - Few required fields. Marking everything required forces the model to invent values for fields the document does not contain — the schema is then satisfied and the data is wrong, which is the worst outcome.
- No fields requiring computation. Do not ask for
total_minus_discount; ask for both and compute it in code, where it is correct by construction.
The validation that must follow anyway: dates parse and are plausible, amounts are non-negative, line items sum to the stated total. Schema conformance says the shape is right; only these checks say the content is. The line-item sum in particular catches a large share of extraction errors and costs nothing.
Your extraction pipeline produces valid JSON 100% of the time but the data is wrong about 8% of the time. Design the response.
Write your own answer first, on paper or in a note. Then compare.
Show a hint
Structural validity is solved. The remaining problem needs a different kind of check.
Show a strong answer
Recognise what has been solved and what has not. Constrained decoding removed parse failures entirely — that is why the figure is 100%. It provides no guarantee about correctness, so the 8% is untouched by anything schema-related, and adding more schema constraints will not move it.
Three layers of response:
- Verification against the source. For every extracted value, check it actually appears in the document — string match for identifiers and amounts, fuzzy match for names. A value that does not occur anywhere in the source was invented, and this catches the most damaging errors cheaply. This is the highest-yield check and it requires no model.
- Cross-field consistency. Line items must sum to the total; the tax must be consistent with the rate; the due date must follow the issue date. Internal arithmetic is a strong signal, and inconsistency is nearly always an extraction error rather than a genuinely odd invoice.
- Route by confidence in the checks, not by the model's self-reported confidence. Documents passing every check proceed automatically; those failing any go to a human queue with the failure highlighted. This converts an 8% silent error rate into a review rate, which is a fundamentally different operational problem.
What I would measure to improve it: categorise the 8%. Are the errors concentrated in one field, one document layout, one supplier's template, or poor scans? Extraction errors are almost never uniformly distributed, and the concentration usually points at something fixable — a layout that needs its own prompt, or a preprocessing step for low-quality scans.
What I would resist: asking the model to check its own work in a second pass. It is correlated with the first attempt, so it agrees with its own errors, and it doubles the cost to do so.
Key takeaways
- Prompting for JSON gives no guarantee; schema-guided generation gives a strong preference; constrained decoding makes invalid output impossible to generate.
- It works by masking tokens that would violate the grammar before sampling.
- Use enums for closed value sets — the largest single quality gain — and always include an escape value so uncertainty is expressed rather than hidden.
- Put reasoning fields before answer fields, since generation order is reasoning order.
- Structure is not truth — semantic validation of ids, ranges and referenced entities remains yours.
- Reliable structure is what lets a model be a component: tool calling, extraction, routing and evaluation all depend on it.
- Prefer constrained decoding over fine-tuning for format — format is a decoding problem.
Further reading
- Brandon Willard and Rémi Louf, "Efficient Guided Generation for Large Language Models" (arXiv, 2023) — constrained decoding via finite-state machines.
- Saibo Geng et al., "Grammar-Constrained Decoding for Structured NLP Tasks" (EMNLP 2023).
- Yaniv Leviathan, Matan Kalman and Yossi Matias, "Fast Inference from Transformers via Speculative Decoding" (ICML 2023) — the sampling machinery constrained decoding builds on.
- Anthropic and OpenAI tool-use documentation — schema design guidance for function calling.
- Stephanie Lin, Jacob Hilton and Owain Evans, "Teaching Models to Express Their Uncertainty in Words" (TMLR, 2022) — why self-reported confidence is poorly calibrated.