jev and the case for models that decide instead of talk

if you have built anything with llms in production, you have probably written a prompt that ends with something like "respond in json with a label and a confidence score between 0 and 1." and you have probably shipped it, watched the confidence field hover around 0.9 for almost everything, and quietly stopped looking at it.

that number was never a probability. it was text the model generated because the prompt asked for a number. the model is very good at producing plausible-looking numbers. it is not, by default, any good at producing numbers that match how often it is actually right.

this is the problem jev, a decision model from typesafe, is built around. i spent some time with bijit ghosh's architectural teardown of jev and the public material around its training method, rlcd. this post is my attempt to explain the architecture from first principles, separate what is known from what is inferred, and, most importantly, show how you would actually use calibrated decisions in a production system.

what jev actually is

here is the publicly described part, stripped of marketing:

• it is built to be called by software, not chatted with. you give it context and a set of typed questions. it returns structured decisions.
• every answer comes with a probability distribution over the allowed options, not a sentence explaining its reasoning.
• multiple questions are evaluated in parallel against the same context.
• it is trained with rlcd (reinforcement learning for calibrated decisions), where the stated goal is that a decision returned at 0.8 is correct about 80% of the time.

in other words, it behaves less like a chatbot and more like a very well-read classifier that can take arbitrary questions at runtime. some people have started calling this category "system one" models: fast, intuitive judgments rather than slow, step-by-step reasoning.

why "confidence: 0.91" in text is not a probability

it is worth being precise here, because this is the entire reason the architecture looks the way it does.

when a chat model writes "confidence": 0.91, what happened mechanically is that it picked the token 0, then ., then 9, then 1, each because it was a likely continuation of the text so far. the model has a real probability distribution inside it, but that distribution is over tokens, and the number it typed is just another token sequence. nothing in standard training ties the digits it writes to its actual hit rate.

on top of that, instruction tuning and rlhf reward answers that humans prefer, and humans tend to prefer confident answers. so the stated confidence drifts upward. you end up with a model that says 0.9 for things it gets right 70% of the time and things it gets right 99% of the time, and you have no way to tell those apart.

for a chat ui, that is fine. for a system that automatically refunds money, closes tickets, or flags fraud, it means you cannot set a threshold. and a decision you cannot put a threshold on is a decision you cannot safely automate.

the architecture, layer by layer

the reconstruction that best fits the public behaviour has four parts. a pretrained causal transformer supplies the knowledge. what changes is how context goes in and how answers come out.

┌────────────────────────── shared context ──────────────────────────┐
│  "merchant reports 3 failed payouts in 24h, account age 2 years,   │
│   last payout succeeded on monday, error code PSP_TIMEOUT ..."     │
└──────────────────────────────┬─────────────────────────────────────┘
                               │  encoded once per layer (keys/values cached)
          ┌────────────────────┼────────────────────┐
          ▼                    ▼                    ▼
   ┌─────────────┐      ┌─────────────┐      ┌─────────────┐
   │ q1: queue?  │      │ q2: urgent? │      │ q3: human   │
   │ [payments,  │      │ [yes, no]   │      │   review?   │
   │  risk, ops] │      │             │      │ [yes, no]   │
   └──────┬──────┘      └──────┬──────┘      └──────┬──────┘
          │  attention mask: branches see context + themselves only
          ▼                    ▼                    ▼
   readout over options  readout over options  readout over options
          │                    │                    │
          ▼                    ▼                    ▼
 {payments: .81,        {yes: .64, no: .36}   {yes: .22, no: .78}
  risk: .15, ops: .04}

1. encode the shared context once

in a normal llm call, if you ask three questions about the same document in three separate calls, the model reads the document three times. the expensive part of reading is computing the attention keys and values for every token at every layer.

the jev-style approach computes those keys and values for the shared context once, then lets every question reuse them. this is the same idea as prefix caching in inference servers like vllm, but designed into the product rather than bolted on as an optimisation.

the savings are easy to estimate. take a 2,000-token context and five questions of about 30 tokens each:

• five separate calls: 5 × (2,000 + 30) = 10,150 tokens of prefill
• shared prefix: 2,000 + (5 × 30) = 2,150 tokens of prefill

that is roughly 4.7x less prefill work, and the gap widens as you add questions. it is not free: each question still has to attend to all 2,000 context tokens, so attention cost per question remains. but you stop paying to re-encode the same evidence over and over.

2. isolate the questions from each other

here is a subtle problem. if you put five questions after the same context in one sequence, question four can see questions one to three. now the answer to "is this urgent?" might be influenced by the fact that you just asked "should a human review this?", which leaks your own intent into the model's judgement.

the fix is an attention mask. each question branch is allowed to see the shared context and its own tokens, and nothing else. conceptually:

              context   q1   q2   q3
  context        ✓      ✗    ✗    ✗
  q1             ✓      ✓    ✗    ✗
  q2             ✓      ✗    ✓    ✗
  q3             ✓      ✗    ✗    ✓

every branch gets an independent answer, and because the branches do not depend on each other, they can all run in the same forward pass.

inferred, not confirmed: black-box tests show answers do not influence each other, but the same behaviour would appear if typesafe simply ran separate calls with a shared cache. the mask is the most plausible explanation, not a documented one.

3. read the answer directly, skip generation

this is the part that makes the whole thing fast. a chat model produces answers token by token, and each new token has to wait for the previous one. that sequential decoding loop is where most of your latency goes.

a decision model does not need to write anything. once the question branch has been processed, the model already has everything it needs to score each allowed option. it reads a probability for payments, risk and ops directly from its internal representation, normalises them, and stops. one forward pass, no decoding.

your application code then turns that into whatever shape it needs: a typed enum, a json object, a database write.

4. what we genuinely do not know

• model size and type. some have speculated jev is a mixture-of-experts model because it is fast. speed alone cannot tell you that; batching, hardware and caching all affect latency.
• the exact readout mechanism. it could be the language-model head scoring option tokens, or a separately trained head. the outputs would look similar from outside.
• the rlcd algorithm. only its goal is public.

calibration, explained properly

a model is calibrated if, across all the times it says 80%, it is right about 80% of the time. across all the times it says 30%, it is right about 30% of the time. that is all calibration means.

it is different from accuracy. a model can be very accurate and badly calibrated (always says 99%, right 90% of the time), or mediocre and perfectly calibrated (honestly tells you when it is guessing). for automation, the second is often more useful, because you know exactly which decisions to trust.

how you measure it

the standard metric is expected calibration error (ece). you sort predictions into confidence buckets (0 to 10%, 10 to 20%, and so on), compare average confidence to actual accuracy in each bucket, and take a weighted average of the gaps.

a reported ece of 0.031 on mmlu, which has been cited for jev, means the average gap between stated confidence and actual accuracy was about 3 percentage points. that is good. but read the caveat: if most predictions sit in the 95%+ bucket, the score tells you very little about how the model behaves at 60%, which is exactly the zone where your human-review threshold will live.

how you train for it

you reward the model with a proper scoring rule: a loss that is minimised only when the predicted probability matches the true likelihood. two classic ones:

• log loss, −log p(correct answer). saying 99% and being wrong is punished enormously, so overconfidence becomes expensive.
• brier score, the squared difference between the predicted probabilities and what actually happened (1 for the true option, 0 for the rest). gentler on confident mistakes, but still rewards honesty.

the difference between rlcd and rlhf, as described publicly, is what the reward is tied to. rlhf rewards outputs that human raters prefer. rlcd reportedly rewards probabilities that agree with observed outcomes. typesafe has not said which scoring rule it uses, so treat any specific loss as an educated guess.

there is also a cheap post-hoc trick worth knowing: temperature scaling. you divide the model's raw scores by a single learned number before the softmax. it makes probabilities sharper or softer without changing which answer ranks first. it is often enough to fix most of the overconfidence in an off-the-shelf model, and you can fit it on a few hundred labelled examples.

from probability to action: the part that actually matters

calibrated probabilities are only useful if you turn them into decisions with clear economics. this is where most teams stop too early: they pick 0.8 as a threshold because it feels safe.

the right threshold comes from the cost of each kind of mistake. say you are deciding whether to flag a transaction for fraud review:

• a false alarm costs about ₹50 of analyst time
• a missed fraud costs about ₹2,000 on average

if the model says the probability of fraud is p, flagging costs (1 − p) × 50 in expectation, and not flagging costs p × 2,000. you should flag whenever the first is smaller, which works out to:

flag when  p  >  cost_false_alarm / (cost_false_alarm + cost_miss)
                =  50 / (50 + 2000)
                ≈  0.024

you should flag anything above about 2.4%. that is wildly different from the 80% most teams would have guessed, and it is only a sensible rule if the probabilities are calibrated. with an overconfident model, a "2%" might really be 15%, and the maths falls apart.

in practice i like a three-zone policy: automate confidently in both tails, and send the uncertain middle to a human. the zones should be set from the cost maths above and then adjusted based on how much review capacity you actually have.

you can build a version of this today

you do not need jev to start working this way. here is a minimal toolkit using open models and standard python.

score fixed options instead of generating text

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained(MODEL_NAME)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAME).eval()

@torch.no_grad()
def option_probs(context: str, question: str, options: list[str]) -> dict[str, float]:
    prefix = tok(f"{context}\n\nQuestion: {question}\nAnswer:", return_tensors="pt").input_ids
    scores = []
    for option in options:
        opt = tok(" " + option, add_special_tokens=False, return_tensors="pt").input_ids
        ids = torch.cat([prefix, opt], dim=1)
        logp = torch.log_softmax(model(ids).logits[0, :-1], dim=-1)
        n = opt.shape[1]
        # log-probability of each option token, given everything before it
        scores.append(logp[-n:].gather(1, ids[0, -n:, None]).sum())
    probs = torch.softmax(torch.stack(scores), dim=0)
    return dict(zip(options, probs.tolist()))

two caveats. this re-reads the context for every option, so in production run it behind an inference server with prefix caching enabled. and multi-token options get penalised for length, so keep option labels short and similar in length (single tokens are ideal).

measure calibration on your own data

import numpy as np

def expected_calibration_error(confidence, correct, n_bins=10):
    confidence = np.asarray(confidence)
    correct = np.asarray(correct, dtype=float)
    edges = np.linspace(0, 1, n_bins + 1)
    ece = 0.0
    for lo, hi in zip(edges[:-1], edges[1:]):
        in_bin = (confidence > lo) & (confidence <= hi)
        if in_bin.any():
            gap = abs(correct[in_bin].mean() - confidence[in_bin].mean())
            ece += in_bin.mean() * gap
    return ece

fix overconfidence with temperature scaling

from scipy.optimize import minimize_scalar

def fit_temperature(logits, labels):
    logits, labels = np.asarray(logits), np.asarray(labels)

    def nll(t):
        z = logits / t
        z = z - z.max(axis=1, keepdims=True)
        log_probs = z - np.log(np.exp(z).sum(axis=1, keepdims=True))
        return -log_probs[np.arange(len(labels)), labels].mean()

    return minimize_scalar(nll, bounds=(0.05, 10.0), method="bounded").x

turn probabilities into actions

def act_threshold(cost_false_alarm: float, cost_miss: float) -> float:
    return cost_false_alarm / (cost_false_alarm + cost_miss)

def route(p: float, act_above: float, ignore_below: float) -> str:
    if p >= act_above:
        return "act"
    if p < ignore_below:
        return "ignore"
    return "human_review"

what i would test before trusting any decision model

whether it is jev or something you built, i would not put it on an automated path without these checks:

1. calibration on your data, not a benchmark. collect a few hundred labelled examples from real traffic and plot accuracy against confidence. pay most attention to the band around your threshold.
2. option sensitivity. reorder the options, rename them, add a near-duplicate. if the decision flips, your labels are doing more work than your context.
3. errors above the automation line. count the mistakes the system would have made without a human. that number, not overall accuracy, is your real risk.
4. drift. log every prediction alongside the eventual outcome. calibration that holds in march can quietly degrade by june when your traffic mix changes.

where this fits, and where it doesn't

decision models are a great fit for the high-volume, bounded judgments that sit inside agent workflows: routing, triage, eligibility checks, "does this need a human?", "did this tool call succeed?". these are exactly the calls where chat models are slow, expensive and overconfident.

they are not a replacement for reasoning models. if the task needs a plan, a multi-step investigation, or an explanation a person will read, you still want a model that thinks out loud.

the interesting architecture, to me, is the combination: a reasoning agent that plans and acts, calling a fast calibrated decision model at every fork where it would otherwise guess. the agent gets cheaper and faster, and for the first time you get a principled answer to the question every ops leader eventually asks: "how do we know when to let it run on its own?"

my plan for adopting any of this is boring on purpose: pick one bounded routing workflow, run the model in shadow mode, log predictions next to outcomes for a few weeks, and only then let it act above a threshold derived from real costs. expand only when the evidence says so.

sources and further reading

• inside jev, architecture of a decision model by bijit ghosh, the architectural reconstruction this post builds on
• typesafe jev: rlcd vs rlhf (mindstudio)
• rlcd: reinforcement learning for calibrated decisions (towards ai)
• agentic design patterns, for where decision calls sit inside larger agent systems