Decision models: accuracy converges, calibration does not

A decision model takes unstructured text plus a set of typed questions and returns one constrained answer per question, each with a probability distribution, in a single forward pass. It never generates text. TypeSafe AI shipped the first commercial one, Jev, on September 15. The interesting part is not the model. It is that an open replication appeared four days later and told us which half of the claim is hard.

Part of a series on sizing models for constrained tasks: sizing the guard model, the schema is the contract, and how agents handle structured I/O.

What you actually send it

You send a state, which is a string or a JSON object, and a map of questions. Three question types, per the primitives docs:

• Choice — pick one of up to 255 options you supply, each with a description. Returns the winner, a probability per option, and a confidence number.
• Score — a position on up to 10 ordered levels you describe. Returns a score that can land between levels, plus the distribution.
• Noul — P(yes) for a statement.

Every question is evaluated against the same state in parallel, so a request with seven questions costs about the same wall-clock as one. Current model is jev-1.13. Input is $0.042 per million tokens and output tokens are free. Context is 64k per request, with 32k covering the state plus the single longest question (Models).

The option names and descriptions are the only instruction the model gets. It never sees your question ID, which Reticle found catches almost everybody: naming a field safe_to_publish contributes exactly zero.

Two places to put the taxonomy

Anything with a closed output space has to decide where the label set lives, and there are only two answers.

In the weights. Fine-tune a classifier so the categories are the output head. This is what OpenAI’s privacy-filter does, and what I am doing to it in privacy-filter customization — retargeting a 1.5B-parameter token classifier at Snowflake’s 47 semantic_categories. Cheap per call, fast, runs inside your own boundary. Changing the taxonomy means retraining.

In the request. Ship the label set as criteria on every call. TypeSafe is emphatic that this is the only option they offer: “Jev is not fine-tuned or LoRA-adapted with customer data… the same weights serve every account.” You shape answers through the state and the criteria, never the weights. Changing the taxonomy is a code edit.

The second bet only works if inference is cheap enough that re-sending the taxonomy thousands of times a day is free. At $0.042 per million input tokens it is. That is the whole product thesis, and the company is named after the founder’s version of it: Jev is for William Stanley Jevons, on the expectation that each order of magnitude off the cost of intelligence buys orders of magnitude more uses.

The controlled comparison nobody expected

Four days after launch, Jared Palmer published Kev under Apache 2.0: a LoRA adapter (r=16, 33.8M trainable parameters) plus a pointer head on Qwen3.5-4B-Base, serving TypeSafe’s own /v1/systemone contract so their SDK works against it with a changed base_url. Training took 56 minutes on one H100. No Jev outputs were used.

It is, in other words, the weights-side bet run directly against the request-side bet, on the same frozen items. That makes the model card the only controlled measurement of Jev that exists. On the out-of-domain development partition (764 records, six never-trained sources):

Out of domain	Kev-4B as served	Jev
Accuracy	0.797	0.857
Brier	0.264	0.211
ECE	0.041	0.049
Confident errors (p ≥ 0.9 and wrong)	2.6%	3.7%
Coverage at ≤ 5% error	0.57	0.70

Read the last row first. Coverage is the share of decisions you can automate while holding the error rate under 5%. Kev gets 0.57. Jev gets 0.70. That is a 23% difference in how much work the thing takes off your desk, and it is much larger than the six-point accuracy gap above it.

Kev’s ECE is better than Jev’s, which looks like it contradicts the row underneath until you see how it got there: Palmer fits a temperature (T = 2.14) on in-distribution development rows and bakes it into the head, so every loader gets calibrated probabilities by default. Temperature scaling is a monotone transform. It never changes an answer, so it repairs the average miscalibration without improving the ordering of confidences — which is what coverage and Brier measure. The raw logits score 0.299 Brier and 6.9% confident errors. A post-hoc fix moved the summary statistic and left most of the gap.

That is the finding. Accuracy converges quickly and cheaply. Calibration does not.

What “cannot hallucinate” is worth

Close to nothing, and the Hacker News thread spent most of its 509 comments establishing it. The model cannot emit an invalid type because the type is the set of options you handed it. It can still pick the wrong one. Almeida conceded that directly — “because these models are probabilistic, it’s also possible to be confidently wrong” — while defending the word on the grounds that we do not say a random forest hallucinates. Fine. The launch post putting 0% on a hallucination chart is measuring schema conformance, and constrained decoding already gives you that.

His reply on constrained decoding is the substantive claim, and the only real technical disclosure the company has made:

constrained decoding (OpenAI-style structured outputs) make models dumber unfortunately — the short+dense version is that simply masking logits is insufficient because if ever a model was assigning probability to an invalid token, the model is by definition confused. you’d be better off erroring IMO

That is an argument for a native output head over your options rather than a masked vocabulary, and it is exactly what Palmer’s pointer head implements. It is also testable, and Sean Goedecke partly tested it: prefilling "choice": " and generating one constrained token on Qwen2.5-1.5B-Instruct got him a 2–3x speedup over ordinary structured output. His verdict is that there is no substantial moat here. The Kev numbers mostly agree on accuracy and mostly disagree on calibration.

The number TypeSafe did not publish

There is no expected calibration error, Brier score, or reliability diagram anywhere in TypeSafe’s material, for any task. For a company whose differentiator is calibrated probabilities, that is the conspicuous omission, and their anti-benchmark policy does not cover it — ECE is not a leaderboard, and reporting one is not hill-climbing.

What they published instead is workflow evals: four business workflows, every model given the same harness, scored against reference answers made by averaging GPT-6 Astra and Claude Fable 5.1 at high thinking. Averaged over the four, Jev ties Sonnet 5 at 67.8% for $0.0004 and 0.4 s per case against Sonnet’s $0.1174 and 78.1 s. That is 1/290th the cost, and it is real.

Two things temper it. The reference is a big-model consensus rather than ground truth, so “67.8%” means “agrees with the consensus as often as Sonnet 5 does,” and Opus 5 only reaches 73.1% against it. And the mean hides the jaggedness: 76.0% on customer service against Sol’s 78.3%, but 61.8% on invoice processing against Sol’s 79.1%. Seventeen points, on the workflow full of amounts and line items.

To their credit, their own jaggedness page predicts that failure before the eval shows it. The model does not count, reads dates as text rather than ordered quantities, and has score levels that are “weak in numerical calibration.” It also admits the probabilities are not coherent across questions: asked whether a customer wants a refund, a Noul returns 0.72 and its negation returns 0.47, summing to 1.19. Calibrated in aggregate against outcomes, not coherent as a probability measure. You cannot carry a threshold from a Noul to a Choice, and you cannot do algebra on the outputs.

That page is the most credible document of the launch, and publishing it was the right call.

Coverage at an error budget, not F1

Here is where I would change my own project rather than theirs.

My privacy-filter success criterion is macro-F1 ≥ 0.80 on a hand-labeled holdout across all 47 categories. That is a forced-choice metric. It scores the model as though it must name a category every time, and it discards the one thing the model already knows — which of its own predictions it should not have made.

TypeSafe’s SEC filings cookbook is the cleanest demonstration I have seen of what that costs. One Choice question over 75 SIC industry groups, 60 annual reports. Forced to name a group every time, it gets 39/60. Split at confidence 0.9: the confident half is right 27/30, the unsure half 12/30. Report the unsure half one level up, as the SIC division the group sits in, and 40% becomes 70%. Total useful answers: 48/60. One request per document, no second call, no second model.

Snowflake classification already has that hierarchy. Every classified column gets both a SEMANTIC_CATEGORY (the fine label, US_PASSPORT) and a PRIVACY_CATEGORY (IDENTIFIER, QUASI_IDENTIFIER, or SENSITIVE), per the classification docs. Backing off from the first to the second when the distribution is flat is the same move, available for free.

So the metric I want is coverage at an error budget: what share of columns can be auto-tagged at a 1% wrong-tag rate, with the remainder answered coarsely or escalated. It is the number Palmer reports, it is the number that distinguishes Kev from Jev, and it is the number a governance team would actually ask for. Macro-F1 answers a question nobody in the pipeline is asking.

The mechanism argument applies too. A tagging pipeline that depends on someone noticing the model was unsure is not a pipeline. A threshold that routes low-confidence columns to a coarser tag holds without anyone remembering.

Where the comparison breaks

Jev does not replace a token classifier, and the reason is in TypeSafe’s own docs rather than any critique. A Choice selects among options you supply, so something upstream has to find the candidates. Their extraction cookbook runs a recall-tuned regex to find candidate emails, phone numbers, and amounts, then asks Jev which one plays which role. It names the limit itself: emails and phone numbers have regexes, and “a name does not, so its candidates have to come from a roster you already have, or from a named-entity recognizer or an LLM that proposes them.”

For PII redaction, finding the span is the job. So there are two different surfaces here that look alike and are not:

Surface	Shape	Right tool
Span detection in unstructured text	Label every token	Token classifier
Column classification from name and samples	Pick one of 47	Choice question

The second is a live slot. Snowflake’s AI mode for classification currently fills it with openai-gpt-5-mini. A decision model is a closer fit for that shape and roughly three orders of magnitude cheaper per column.

Two other things rule the hosted version out for my project specifically. Everything moved to Snowflake ML Jobs so training data and checkpoints stay inside the account boundary, and an external API undoes that. And the latency target is p95 ≤ 150 ms per document, against Jev’s 0.3–0.5 s per case measured in its own evals.

Opacity is the real cost, and it is priced by the slot

Simon Willison’s objection is the sharpest thing said about this model class: a decision model is a step further into the black box. No reasoning trace, no cited span, just a float. “If Jev marks something as spam, which content signals tipped it off?” Reticle reaches the same place from operations — “a wrong answer and a right answer look identical from the outside,” and the only replayable artifact is the question text, the option descriptions, and the state blob, all logged verbatim or the call is gone.

My position is that this is disqualifying in some slots and free in others, and the test is who reads the output.

Where a machine consumes the decision, opacity costs nothing you were not already paying. A routing choice, a rerank, a triage gate — nobody asks a BM25 score to explain itself either, and confidence plus a fallback path is the whole contract.

Where a person acts on the decision, it is disqualifying. A SEMANTIC_CATEGORY tag drives a masking policy, and an auditor asking why a column was tagged US_PASSPORT is not satisfied by 0.94. A token classifier answers that question structurally: it fired on a span, and the span is the evidence. That is a second reason the weights-side bet keeps its seat in governance, independent of the trust boundary — and it is not an argument the calibration numbers can settle either way.

One correction to the received wisdom, though. Reticle’s “Jev cannot abstain” is about option-set design, not the model. Palmer measures Jev answering only 9% of deliberately unknowable items at confidence ≥ 0.9. It does signal not-knowing, provided you gave it somewhere to put the uncertainty. If your option set has no “not enough information,” the set is the problem.

Key takeaways

• A decision model puts the taxonomy in the request; a fine-tune puts it in the weights. Only the first survives a policy change without retraining. Only the second stays inside your trust boundary.
• Accuracy is the cheap half. A LoRA plus pointer head on a 4B open base, 56 minutes on one H100, reaches 0.797 out of domain against 0.857.
• Calibration is the expensive half. Same comparison: Brier 0.264 against 0.211, and 0.57 of decisions automatable at a 5% error budget against 0.70.
• Temperature scaling fixes the summary statistic, not the ordering. Kev’s ECE beats Jev’s and its coverage does not, because a monotone transform cannot reorder confidences.
• “Cannot hallucinate” means schema conformance. Constrained decoding already gives that. The substance is calibrated probabilities over the constrained space at $0.042 per million tokens.
• Coverage at an error budget is the metric. Macro-F1 scores a classifier as if it must always answer, which throws away the confidence signal that decides whether to act.
• Enumerable labels still do not imply a small model. Jev scores 0.840 on MMLU-Pro, which settles the parameter-count speculation in TypeSafe’s favor and confirms what the guard model argument predicted: the evidence sets the size, not the output shape.

The launch numbers are all vendor-produced and the reference answers come from two competitors’ models. The one number I trust is the one TypeSafe did not publish and someone else measured anyway.