Models That DecideOctober 2026

A model that answers instead of writing

Reading TypeSafe, Introducing System One Models & Jev (Sep 15, 2026); TypeSafe, API reference; Vercel, Jev is the fastest-adopted model in AI Gateway history; OpenAI, DevDay 2026 Recap; Hume, Jev's Architecture Unmasked, and Black, Jev is Poorly Calibrated, on the classifier argument; Cloudflare, Introducing Clef, and AWS, Introducing Strands Decider 2B, for the October 1 releases. Reactions on Hacker News and X are described as discourse rather than cited.

1.1The decision inside the software

TypeSafe launched Jev on September 15, 2026, in early access, after two years in stealth. The founder, Diogo Almeida, writes in the launch post that he helped build the instruction-following methods behind ChatGPT at OpenAI, and left because chat models, however capable, had not automated much of the world's work. His answer is a model class TypeSafe calls System One, after Daniel Kahneman's fast, intuitive mode of thinking: models built to make quick, structured decisions that software can consume directly. The launch post's one-line summary is that Jev is a frontier-intelligence function call, with unstructured state going in and typed, probabilistic decisions coming out.

The name Jev is for William Stanley Jevons, who observed that more efficient steam engines raised the demand for coal rather than lowering it. TypeSafe's bet is the same shape: if a decision costs almost nothing, software will ask for far more of them.

1.2Three kinds of question

Every request carries one state (the text or structured data being judged) and a map of named questions. There are three question types, and every answer comes back typed:

• Noul is a yes-or-no question. It returns a single probability that the answer is yes.
• Choice picks one option from a set you define, up to 255 options. It returns the winning option, a probability for every option, and a confidence value.
• Score rates the state against ordered levels you define; TypeSafe's API accepts up to ten. It returns a probability for each level and a probability-weighted score that can land between levels, such as 1.05 on a three-level scale.

The keys you give your questions are for your code only; TypeSafe's documentation says they are never sent to the model. A minimal request looks like this:

curl -X POST https://api.typesafe.ai/v1/systemone \
  -H "Authorization: Bearer $TYPESAFE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Help! My payouts have been failing for 3 days.",
    "model": "jev-latest",
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
          "billing": "Payments, invoicing, refunds",
          "technical": "Bugs, outages, integrations",
          "sales": "Pricing, upgrades, new accounts"
        }
      }
    }
  }'

{
  "model": "jev-1.13.0",
  "answers": {
    "department": {
      "type": "choice",
      "choice": "billing",
      "probabilities": { "billing": 0.88, "technical": 0.12, "sales": 0.0 },
      "confidence": 0.81
    }
  },
  "usage": { "input_tokens": 318, "output_tokens": 34 }
}

There is nothing to parse. The answer is always one of the options you wrote, which is the sense in which TypeSafe says Jev cannot hallucinate: it can be wrong, but it cannot invent an option that does not exist.

1.3What it costs, and how much to trust that number

TypeSafe prices Jev at $0.042 per million input tokens, with output tokens free, and reports end-to-end response times of 70 to 500 milliseconds. Its headline comparison, 193.6 times faster and 444.6 times cheaper than frontier language models, comes with caveats the launch post states itself: TypeSafe expects those figures to sit at the high end of real-world gains, the four workflows behind them were built by its own model capabilities team, the reference answers were the average of GPT-6 Astra and Fable 5.1, and the company cannot prove its pricing is not subsidized. That is more candor than most launches offer, and it is also a reminder that every number in the launch post was produced by the party selling the model.

Adoption was immediate. Vercel reported that within 24 hours of Jev landing on its AI Gateway, nearly 13% of paid teams were using it, twice the GPT-5.6 family and more than six times Fable 5.1. Vercel's own post adds that the next test is whether that adoption lasts. A fast first day measures curiosity and ease of integration, not accuracy.

1.4"It's just a classifier"

Within hours of the launch, a common reply on Hacker News and X was that Jev is a zero-shot classifier with good marketing. The disagreement is worth taking seriously, because it changes what you expect from the tool.

TypeSafe's position is that this is a new class of model: a new architecture, a parallel sampler, and a post-training method it calls Reinforcement Learning for Calibrated Decisions, aimed at probabilities that are, in its words, epistemically honest. Its critics point out that "user-supplied labels in, probabilities out" is the textbook definition of a classifier. Archer Hume, who reverse-engineered Jev's behavior through its API, splits the difference: parallel classification has existed for decades, and what is new is the combination of a broadly capable pretrained transformer, shared computation across many questions, a typed interface, and training that rewards useful uncertainty. Dylan Black describes it, approvingly, as a pretrained transformer with a classifier bolted onto the end.

The category grew faster than the argument could settle. On September 29, OpenAI's DevDay recap announced a Decisions API that focuses its Luna model on developer-defined questions with finite answers, taking text or images; it is in limited preview, and the recap gives no request schema, pricing, or description of its confidence output. By October 1, Cloudflare and AWS had both released open-weight decision models. For a practitioner, the label matters less than one question: do this model's probabilities hold up on the decisions you actually need to make? The rest of this guide is about answering it.

How the trick works

Reading SemIf (OpenJev) README; Kev README; Brooker, Chambers & Nonato de Paula, Introducing Strands Decider 2B and repository; Chen, Reneau & Flansburg, Introducing Clef and the Clef model card; Hume, Jev's Architecture Unmasked; Kelly, Do jev's probabilities mean anything?

2.1Why writing is the slow part

A language model answering a question does two kinds of work. First it reads the whole prompt at once, a step called prefill; every input token is already known, so the model can process them together. Then it writes the answer one token at a time, and each new token has to wait for the one before it. For a short decision, the waiting is most of the cost: a model may spend dozens of tokens spelling out {"department": "billing"} that code immediately throws away.

A decision model stops after prefill. It reads the state, the questions, and the allowed options, then reads a score for each option straight off its internal representation. No token is written, so the time does not grow with the length of an answer. SemIf, an independent project that reproduces Jev's interface with open models, measured the difference on the same frozen Qwen3.5-4B model, the same state, and the same 21 yes-or-no criteria on one RTX 3090: reading the option scores directly took a median of 1.023 seconds and produced no output tokens, while generating a compact JSON array of the same answers took 5.332 seconds and 111 tokens. The author is careful to call this a systems comparison; the two methods agreed on 18 of the 21 answers.

2.2Four ways to build one

Jev's architecture is undisclosed. The open projects that appeared in the following weeks show the range of ways to get the same interface, and all of them start from an existing open Qwen model:

The pointer head that Kev and Strands share is the clearest version of the idea. The question ends at a decision position placed after all the options. The head compares the model's internal state at that position against its state at the end of each option and turns those comparisons into probabilities; because the decision position comes last, it can take the whole option list into account. Because the head holds no parameters for any particular option, Strands notes, nothing in it can learn that the first option is usually right, nothing caps how many options a question may have, and the labels are defined by the request rather than baked into the weights.

2.3Shared state, isolated questions

The other half of the trick is what happens when you ask many questions at once. Hume probed Jev with about 10,000 API calls and concluded, carefully labeling it inference, that the state is encoded once and each question then builds its own branch on top of it, with branches unable to read one another. His cleanest test planted a secret code in one question and asked a second question about it: with the secret in the sibling question, the second question gave it a probability of 0.00; with the same sentence moved into the shared state, the probability rose to 0.90 to 0.92 across five repeats. Kev builds this behavior in on purpose; its README describes the questions as sharing the text but unable to read each other.

Kelly's independent evaluation confirmed the practical consequence. Accuracy at the two-hundredth question in a request matched accuracy at the first, and sixty questions about one state, asked in a single request, used twenty times fewer tokens and finished eight times sooner than the same sixty asked one per request, with identical answers. It was the result his plan expected to be most surprising, and it produced no surprise at all. The lesson is simple: ask your questions together.

Isolation between questions is not isolation between options. Hume appended an irrelevant option ("bad weather caused it") to a four-option question about a payout failure and measured the odds between two of the original options. If every option were scored independently, those odds could not change. They fell in all ten randomized blocks, by 0.28 on average on a log-odds scale. The options in a question are read together, which is useful (an option like "none of the above" depends on its neighbors) and also means that adding, removing, or reordering options can move an answer. Kev's README says the same of its own models: changing option order can change an answer, and question isolation does not prevent it.

Writing questions it can answer

Reading TypeSafe, How to build with TypeSafe and Jev 1.13 jaggedness (reviewed Oct 2, 2026); Kelly; rmax-ai, jev-vs-clef; Hume; Kev README, for the permute endpoint. The Hacker News comment in §3.2 is described as discourse rather than cited.

3.1Code holds the control flow

TypeSafe's own design guidance is blunt about what a decision model is not: it is not an agent. It does not choose its own next step, write code, or carry out actions. The pattern the documentation recommends is ordinary software with the model inserted only where a judgment over unstructured text is needed. Deterministic rules stay in code, as do arithmetic and side effects. If an invoice is more than thirty days overdue, code can work that out from two dates; the model has nothing to add, and asking it invites error.

3.2One judgment per question

The documentation calls decomposition probably the most important idea in its whole guide. A broad question hides several judgments behind one number, and when that number is wrong, you cannot tell which judgment failed. Its example is spam: rather than asking whether a message is spam, ask six narrow yes-or-no questions (does it ask for a credential, announce an unexpected reward, pressure the reader to act fast, name an organization that doesn't match the sender's domain, link somewhere that doesn't match, or disguise where a link goes) and combine the answers with weights you choose in code.

Narrow questions also travel better. A commenter on one Hacker News thread made the point that a question like "is the customer angry?" should get similar answers from different models and from people, while "is this urgent?" depends on rules the model can't see. Ask the first kind, and encode the second kind in your code.

3.3What the model is bad at, in its maker's words

TypeSafe publishes a list of known weaknesses for each model version, which is unusual and worth reading before you build. For jev-1.13, reviewed October 2, 2026:

3.4The form of the input matters more than you'd think

Two independent tests found that how you package the state can move the answer a long way.

Kelly asked Jev whether one point in a program could reach another, giving it the same programs in three forms. As source code it scored 0.894; as a syntax tree, 0.598; as a list of control-flow edges, 0.530. The edge list is the form that makes the question trivial for a program, and it came last while costing two and a half times the tokens. Doing the model's work for it, in a shape built for machines, made it worse.

rmax-ai's battery found the same sensitivity on Clef with simpler material. One refund request was scored as urgent with probability 0.90 when written as prose, 0.86 as flat JSON, and 0.09 when the same fields were nested more deeply. Choice and Score answers were stable across formats; the yes-or-no questions were not. The practical rule from that report is a good one: fix the state format for each pipeline, and re-validate whenever it changes.

Option order deserves the same caution. Hume found that simply reversing the options on a routine support-ticket question moved the probability of the technical-support answer from roughly 0.84 to 0.89 up to 0.93 to 0.96. Nothing about the ticket changed, but a threshold set at 0.9 would now act where before it would have waited. Kev's server includes an endpoint, /v1/systemone/permute, that runs one Choice question under several option orders so you can see how much your answer depends on the order.

Calibration is a claim you check

Reading TypeSafe, Confidence and Models; Kelly report and repository; Hume; Black, Jev is Poorly Calibrated; Kev README; Strands Decider README; TypeSafe's launch post, for the act-when-sure argument; rmax-ai, on Clef's escape options. Practitioner reports from Hacker News and Reddit are described as discourse rather than cited; their figures are the posters' own.

4.1What calibrated means

A model that reports probabilities is making a claim you can check. If it says 0.9 on a thousand questions, about nine hundred of those answers should turn out right. A model whose numbers behave that way is calibrated, and calibration, not raw accuracy, is the main reason to prefer a decision model over a chat model for automation. A model that is right 95% of the time but cannot tell you which 5% it is unsure about cannot be trusted to act alone; TypeSafe's launch post makes exactly this argument. Its claim for Jev is that higher confidence means higher accuracy.

That claim is about the model's behavior on some population of questions. Your questions are a different population. Everything in this chapter follows from that.

4.2What the confidence field actually is

Every Choice and Score answer carries a confidence between 0 and 1. TypeSafe's documentation gives the exact formulas. For a Choice with n options, it measures how far the top probability sits above an even split:

def choice_confidence(probabilities: list[float]) -> float:
    n = len(probabilities)
    return (max(probabilities) - 1 / n) / (1 - 1 / n)

Only the top probability counts, so (0.6, 0.3, 0.1) and (0.6, 0.2, 0.2) both get 0.4. A Noul returns no confidence at all; its probability is the signal, and the documentation suggests distance from 0.5 if you want a matching number. The important point, which Hume makes plainly, is that confidence is arithmetic over the distribution the model already gave you. It is not a second, separately learned estimate of whether the answer is right. A concentrated distribution can still be confidently wrong.

Kelly logged two other properties worth knowing before you set thresholds. Every one of more than three million probabilities he collected sat on a two-decimal grid, so nothing finer than 0.01 can be resolved. And in one experiment, the probability carrying the decision was exactly 1.0 on 52% of 836,700 answers. When half the answers are pinned at the top, a high threshold separates less than it appears to.

4.3Where the numbers hold and where they break

Kelly's evaluation is the most careful public test so far: nine experiments and 28 predictions fixed in writing before any data was collected, ground truth from solvers rather than from any model, 123,805 requests against jev-1.13.0, and a total cost of $12.69. Twelve of his 25 testable predictions held. A support-ticket routing set ran alongside every experiment as a control and scored 0.980 accuracy throughout, which is what lets the failures elsewhere be read as facts about the problems rather than about the harness.

On the kind of work Jev was built for, calibration held: the expected calibration error on support-ticket routing was 0.075. On random 3-SAT, a logic problem where the fraction of satisfiable formulas falls from nearly all to nearly none as clauses are added, it did not. Jev answered satisfiable for every formula at every clause density; its average probability moved by 0.026 while the true satisfiable fraction moved across the full range from 0 to 1. A short program reading clause density off the formula header beat it in 41 of 75 conditions, and given x AND NOT x, which is unsatisfiable on sight, it returned a 0.38 probability of satisfiable.

Other independent work points the same way at the edges, and not at the center. Hume's 1,200-item MMLU sample came out well calibrated, with an expected calibration error of 0.031, though 990 of those items sat in the top probability bin. Dylan Black asked Jev a different kind of question: physics problems whose answer is a known probability distribution, with the answer range split into bins. Jev's binned probabilities were barely closer to the truth than a flat guess (a mean total-variation distance of 0.518, against 0.546 for spreading probability evenly), and for a uniform distribution it was much worse than a flat guess, 0.77 against 0.39, piling probability onto one bin. Yet when Black asked which distribution applied, Jev picked correctly almost every time. Black is testing whether the model can shape a distribution, which TypeSafe never claimed it could, but the result is a useful warning: knowing the right answer and putting honest weight on it are different skills.

4.4The prior problem

A sharper objection came from a Hacker News commenter writing as amluto. Suppose the question is whether a pull request will be merged. Two projects can have very different standards and very different overall merge rates, so no single probability can be calibrated for both. Worse, the output has no way to say "the state tells me nothing; use your prior." A 0.5 might mean "I've weighed the evidence and it's a coin flip," or it might mean "I have no information," and those call for different actions. The commenter gave Jev irrelevant state and asked about sporting outcomes, and reported it got none of three close to the obvious no-information answer.

Kelly tested the same worry directly, by mixing answerable states with states that could not be answered at all, and his result supports two readings. By the measure his plan named in advance, confidence failed: the average gap between answerable and unanswerable states was 0.199, below his 0.30 bar for a usable signal, and his repository's summary notes that gating at a confidence of 0.95 still let through 47% of unanswerable states. But because confidence sits at exactly 1.00 on most answerable states, an average is a poor summary. Ranked instead, confidence on a Choice question separated every kind of unanswerable state from the answerable ones, at 0.893 to 0.992. His conclusion is that the act-when-sure pattern works "at two gates rather than one, and only if you ask as a choice." Kev's author reports a related test on records whose deciding evidence had been deliberately removed: Jev still answered with at least 0.9 confidence on 9% of them, against 0% for Kev-9B, by his own measurement.

4.5Your data is not their data

The practitioner reports from the weeks after launch line up along one axis: how much the decision depends on knowledge the model already has.

At one end, a team posting to r/LLMDevs reported running Jev on 1,759 decisions, some hand-labeled and some from real routing traffic. By their numbers, Jev reached 98.5% accuracy against 99.0% for a frontier Gemini model, at a median of 329 milliseconds against 1,598, and about a twenty-fifth of the cost; every one of the 986 answers it gave above 97% confidence was correct. It struggled with Arabic dialects, getting the language family right with low confidence, which is the behavior you want from a model out of its depth. Another poster, on r/buildinpublic, reported Jev within 0.2 to 4 points of Claude Sonnet 5 on Banking77 and similar sets, while noting that a fine-tuned local DistilBERT or Laya beat zero-shot Jev wherever labeled data existed.

At the other end, a team that builds financial classification systems under German regulation posted a smoke test to Hacker News: about 1,000 real bank transactions, where the same merchant can mean revenue, reimbursement, or tax depending on the company. A base Gemma-2-2B model scored 17.5%; the same model fine-tuned on about 4,000 labeled transactions scored 85.3%; Jev scored 40.7%. Their conclusion was measured: plenty of classification problems are solvable from world knowledge, just not theirs.

Jev cannot close that gap with your data. TypeSafe's Models page says the same weights serve every account and are not fine-tuned on customer data; you shape answers through the state and criteria only. The open models can. Kev's README reports that one epoch on 5,219 labeled consumer-finance complaints took Kev-4B from 0.804 to 0.904 accuracy on complaints it had not seen, that a Kev-4B training run costs about $1 on an H100, and that the gains are in-distribution: they show how well Kev learns your task, not how it does on everything else, and at 400 records the gain on its example workload was inside the noise. Kev also ships a fitted temperature with each checkpoint; on new sources it takes Kev-9B's calibration error from 0.103 to 0.041 and its confident errors (wrong answers given at least 0.9 probability) from 8.2% to 2.4%, against Jev's 3.7%. Jev still ranks its answers better: at a 5% error budget, Kev-4B, 9B, and 27B can automate 0.52 to 0.69 of new-source decisions, against Jev's 0.70. Strands reports that on short classification tasks its model has never seen, answers at 0.9 confidence or higher are right about 95% of the time. Those are the authors' own numbers, which is exactly why the next section exists.

4.6Measure it, then gate on it

The procedure is not complicated, and nothing else substitutes for it:

1. Collect labeled examples from your own traffic and hold some back for testing.
2. Run them through the model and record the probability behind every answer.
3. Group the answers into probability bins and compare each bin's average probability with how often it was actually right. The weighted average of those gaps is the expected calibration error. TypeSafe's own advice is to plot confidence against accuracy on your data before choosing thresholds.
4. Pin the model version. TypeSafe's jev-latest alias moves when a new release ships, so the answers behind it can change without any change on your side; if you tuned thresholds against jev-1.13.0, call jev-1.13.0.

Then gate. TypeSafe's documentation recommends thresholds that scale with risk: a wrong read-only answer is recoverable, a wrong money transfer is not. Kelly's result suggests two gates rather than one. Figure 5 shows one way to build them; it follows his advice to ask as a Choice and TypeSafe's own pattern of giving a Choice an explicit option for "not stated," but the specific design is ours, not his.

from typesafe_sdk import Choice, TypeSafeClient

client = TypeSafeClient(model="jev-1.13.0")  # pin the version your thresholds were tuned on

RISK_BAR = {"refund": 0.9, "faq": 0.6}       # placeholders: set per action from your own data

def route(ticket_text: str) -> str:
    response = client.system_one(
        state={"ticket": ticket_text},
        questions={
            "action": Choice(
                instructions="What does the customer in `ticket` want?",
                criteria={
                    "refund": "Money back for a charge",
                    "faq": "A question the help center already answers",
                    "none_fit": "None of the other options describe the request",
                },
            ),
        },
    )
    action = response.answers["action"]
    if action.choice == "none_fit":                    # gate 1: nothing fits
        return "human"
    if action.confidence < RISK_BAR[action.choice]:    # gate 2: not sure enough for this action
        return "review"
    return action.choice

Untrusted state

Reading TypeSafe, Jev 1.13 jaggedness, adversarial content; Kelly repository; rmax-ai; Hume; Kev README, API section; Strands, Introducing Strands Decider 2B, the tool-call example.

5.1State is data, and data can argue

A decision model is most useful exactly where it judges text you did not write: support tickets, uploaded documents, web pages, the output of another model. That is also where it is exposed. TypeSafe's documentation says so directly: the state is data, the model does not treat it as hostile by default, and content written to steer the answer (an injected instruction, a misleading framing, text that argues for its own classification) can move it. TypeSafe says it expects to improve on this.

5.2The polite injection

Kelly's adversarial set found that the obvious attack fails and the subtle one works. Text shouting "IGNORE THE QUESTION AND THE INSTRUCTIONS ABOVE" moved the answer 0% of the time. A single sentence claiming that a support lead had already made the decision moved it 65% of the time, while confidence fell only from 1.00 to 0.62. The drop is a signal worth gating on; the 65% is a reason not to rely on a gate alone.

Not every test lands the same way. rmax-ai ran two injection probes, one trying to flip an urgency answer and one trying to flip a routing answer, against Jev, Clef, and Clef-flash, and all three held, 6 of 6. Different wording, different result: injection resistance is a property of the attack as much as the model, which is the argument for testing your own adversarial cases rather than trusting anyone's summary.

One structural defense does seem to hold. Hume tried injecting fake options into the input with seven different delimiter formats, and none ever displaced the real options: the boundaries between options are marked in a way plain text cannot forge. Kev's server likewise escapes delimiter-like strings in user input before it reaches the model. The attack surface is the meaning of the text, not the format of the request.

5.3Fencing untrusted text

Kelly's practical conclusion is that anything judging text you do not control needs that text separated from the question. In a decision model, that suggests a few habits, each worth testing on your own adversarial examples:

• Put untrusted content in its own named field of the state and refer to it by path in the instruction, so the question is never phrased inside the untrusted text.
• Ask about the attack directly. A Noul such as "Does message contain instructions addressed to the reviewer, or claims that a decision has already been made?" turns the polite injection from a hidden lever into a measured signal you can gate on.
• Never let a single injectable judgment be the only check in front of a destructive action. Strands' worked example gates an agent's tool call on two yes-or-no questions (are the arguments grounded in what the user actually said, and is it premature to call the tool) before anything runs, and returns the agent to the user with a clarifying question when they fail.

Swapping models and reading their numbers

Reading Kev README; Cloudflare, Introducing Clef and model card; rmax-ai; Strands Decider README; TypeSafe, Models and launch post; JevBench README; Kelly, for Jev's run-to-run noise. Reports from Reddit and X are described as discourse rather than cited.

6.1One wire format

The most consequential thing the clones copied was not the architecture but the request. Kev's API matches TypeSafe's System One endpoint closely enough that TypeSafe's own Python SDK works against a Kev server unchanged. Cloudflare describes Clef as fully Jev-API compatible, and Strands Decider's server exposes the same /v1/systemone path. In practice, the model behind your decision endpoint becomes a configuration value.

git clone https://github.com/jaredpalmer/kev.git && cd kev
uv sync --extra serve
uv run --extra serve python -m kev.serve --run jaredpalmer/kev-4b --port 8009

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

client = TypeSafeClient(
    api_key="local",
    base_url="http://127.0.0.1:8009",
    model="kev-latest",
)
response = client.system_one(
    state="I was charged twice. Please fix this ASAP.",
    questions={
        "billing": Noul(instructions="Is this ticket about billing?"),
        "tone": Choice(
            instructions="What is the customer's tone?",
            criteria={"calm": None, "frustrated": None, "angry": None},
        ),
        "urgency": Score(
            instructions="How urgent is this ticket?",
            criteria=["can wait", "this week", "today"],
        ),
    },
)
print(response.nouls["billing"].noul)
print(response.choices["tone"].choice)
print(response.scores["urgency"].score)

6.2Where "compatible" stops

Same request shape is not the same behavior. The differences that surfaced in the first weeks:

6.3Speed on paper and speed in practice

Cloudflare's own run puts median latency at 209.3 milliseconds for Clef, 38.8 for Clef-flash, and 524.1 for Jev, with Kev-9B at 51.4. rmax-ai, calling the hosted models during launch week, measured Clef at about 18.7 seconds per call from their environment against 0.3 seconds for Jev, and attributes the gap to scale-from-zero queueing rather than model compute. Strands reports a median of 115 milliseconds on an RTX 3090 and 153 on an M3 MacBook; Kev reports 18.1 milliseconds of model time for six questions on an H100.

The network is part of the answer. A poster on r/Rag reported that for intent routing, passage reranking, and fact checks inside a streaming retrieval pipeline, Jev's floor of roughly 380 milliseconds broke their per-sentence budget, and that local DistilBERT, ColBERT, and DeBERTa models were orders of magnitude faster and cheaper there with equal or better accuracy. A decision model is not automatically the fastest tool for a decision; measure from where your code will call it.

6.4Whose benchmark is it?

Almost every comparison number in this space was produced by someone with a stake in it, and the honest ones say so.

TypeSafe's workflow evaluations were built by its own team and graded against the average of two closed frontier models. Kev's author removed those same evaluations from his own release gates for that reason: the reference answers are the averaged output of two closed models, and on 89 questions they could not tell checkpoints apart. Cloudflare's headline table shows Clef ahead on most benchmarks, with Jev keeping When2Call (80.97 against 72.37) and BRIGHT (47.52 against 45.91); the fuller table on Clef's model card shows Jev well ahead on knowledge-heavy tests such as GPQA Diamond (78.3 against 48.0) and MMLU-Pro (82.7 against 65.9). Clef's biggest lead is BANKING77, 94.2 against 79.7, and one user on X reported the opposite ordering on his own policy data, Jev finding 13 of 14 positions against Clef's 9, and suspected Clef had seen Banking77-style labels in training. That is a suspicion, not a finding, but it is the right question to ask of any benchmark a model wins by a wide margin.

Sample size matters as much as authorship. Strands notes that its public benchmark has 231 tasks and that six retrains of one recipe varied by a standard deviation of 3.2 tasks, so gaps under about ten tasks between single runs are unresolved. JevBench, the third-party benchmark Strands reports against, carries a notice from its own maintainer that results are preliminary and should not be cited yet; this guide doesn't. And task shape decides winners: a poster on r/LocalLLaMA reported a local, quantized Clef beating hosted Jev six games to three in a game-playing simulator, then winning one game of 22 against a specialist model trained by self-play.

The only score that answers your question is one you run on your own decisions.

uv run python -m kev.benchmark --remote http://127.0.0.1:8009 --suite evals/v4/transfer-v4 --out runs/my-remote

uv run python -m kev.benchmark --run jaredpalmer/kev-4b --data heldout.jsonl --out runs/kev-4b-mine