The model produces a probability distribution over possible next tokens. A decoding strategy turns that distribution into one selected token.
The strategy affects variation. It does not turn an unsupported answer into a supported one.
- Schema die
The schema constrains fields such as action, amount, accountId, and reason.
- Generated proposal
The example refund has amount 9000 and account A-17. Syntactic validity does not prove that this refund is allowed.
- Semantic latch
Check the account, amount limit, and supporting reason against application state.
- Authority gate
Authorize, request approval when required, or reject with a typed error before execution.
Valid JSON can describe an unauthorized action. The application checks meaning after syntax.
How to read this model
An inspection fixture with a form die, a policy latch, and a release gate. The object is a teaching analogy. Inspecting a part highlights its role; it does not execute or simulate the software. Cabinet trays are records, workpieces are data, and latches are checks. Unless an explanation states a quantity, compartments and sample workpieces do not denote data counts. Part dimensions are not a quantitative scale.
Greedy and sampled decoding
Greedy decoding selects the highest-probability token at each step. Sampling draws from the distribution. Provider implementations may expose several controls:
- Temperature rescales logits before probabilities are calculated. Lower values concentrate probability on likely tokens. Higher values flatten the distribution.
- Top-p keeps the smallest set of tokens whose cumulative probability reaches a threshold, then samples from that set.
- Top-k keeps a fixed number of the highest-probability candidates.
- Stop sequences end generation when configured text appears.
Exact behavior and allowed ranges depend on the model API. A temperature of zero is often more repeatable, but it does not promise identical output across model versions, infrastructure changes, or providers.
Structured output constrains shape
A structured-output feature asks the model to produce data that matches a declared schema. A tool call is a common example. The model emits a tool name and arguments rather than executing the operation itself.
{
"type": "object",
"properties": {
"invoiceId": { "type": "string", "pattern": "^inv_[0-9]+$" },
"reason": { "type": "string", "minLength": 1 }
},
"required": ["invoiceId", "reason"],
"additionalProperties": false
}
Schema validation can prove that both fields exist and that invoiceId has the requested form. It cannot prove that the invoice exists, belongs to the caller, or may be refunded.
Validate in layers
Production code needs separate checks:
- Syntax. The output parses.
- Schema. Types, required fields, and structural constraints match.
- Semantics. Identifiers exist and values make sense together.
- Authorization. The caller may perform this action on this resource.
- Execution policy. The action satisfies approval, budget, and risk rules.
Returning valid JSON solves only the first part of this sequence.
Repair has limits
An application can ask the model to repair invalid output, but every repair consumes time and tokens. A bounded retry is reasonable for a recoverable format error. Repeated semantic errors usually point to an unclear schema, a poor tool description, missing context, or a task the selected model does not handle reliably.
Record invalid attempts in evaluation data. Do not hide them behind unlimited repair loops.