The short answer
TypeSafe publishes one set of numbers for Jev: its own workflow evals and the speed and price multiples in the launch post. Independent builders have published at least twenty more benchmarks since launch on 15 September 2026. Most come with code and raw outputs you can rerun. The biggest is the Jev Decision Index, multimodalart’s community board on Hugging Face, where Jev 1.14 scores 60.24 on its Full score in edition 0.4.1, 41st of 229 rows, with Fastino’s GLiDE 1.2 first at 71.52. The newer Artificial Analysis Decision Index, a separate board, ranked Jev first of 11 hosted decision models at 57.5 at launch. On version 2.1, with six tests, it ranks Jev 1.14 ninth of 19 at 66.2 as of 12 November 2026. Nace.AI’s Drex Max v1.1 and SubQ Snap 1.2 lead at 72.6.
Speed is the result the independent tests agree on most. Measured medians for single calls land between 154 ms and 422 ms in the single-author tests, and 92 ms per task for Jev 1.14 on the Artificial Analysis board. Every LLM they were compared with was slower.
Accuracy depends on the task. Jev matched or beat the alternatives on topic classification, intent routing, reranking and agent-failure attribution. It lost clearly to Claude Haiku 4.5 on phishing, to OpenAI structured outputs on medical-note extraction, and to plain classifiers trained on labelled data. It also gets worse outside English.
What the evidence doesn’t give you: a peer-reviewed study, a large test repeated by several independent groups, or any test run on your data. Most numbers below come from one run by one author on jev-1.13.0.
TypeSafe’s own numbers
These are vendor claims. They come from TypeSafe’s own harness and have not been reproduced.
The launch post gives 70ms to 500ms end to end for Jev, against 3 to 329 seconds for frontier models, and “40x-200x faster” for System One shaped queries. The post says the headline “193.6x faster, 444.6x cheaper” figures come from its workflow evals, and that TypeSafe expects “these are on the higher end of real world gains.” The speed and pricing guide covers what those ranges leave out.
The workflow evals site runs four tasks: security incidents, agent-trace observability, invoice processing and customer service. Every model answers the same narrow questions, and code turns the answers into an action. Averaged across the four, read on 24 September 2026:
| Model, workflow mode | Accuracy | Cost per case | Seconds per case |
|---|---|---|---|
| Jev | 67.8% | $0.0004 | 0.4 |
| terra | 67.9% | $0.0304 | 10.1 |
| sonnet 5 | 67.8% | $0.1174 | 78.1 |
| opus 5 | 73.1% | $0.1761 | 37.8 |
| sol | 74.1% | $0.0836 | 23.3 |
| haiku 4.5 | 53.6% | $0.0195 | 12.5 |
Model names are as the site labels them. The reference answers are “an average of the responses of GPT-6 Astra and Claude Fable 5.1, both at high thinking,” so accuracy means agreement with two other models, not with ground truth. Per task the picture moves: Jev scored 61.8% on invoice processing against 78.4% for Opus 5, and 76.0% on customer service against 72.4%.
The Jev Decision Index
The Jev Decision Index is the largest independent board for Jev and the models built to rival it, by number of models. It is a Hugging Face Space published by the account multimodalart, which calls it community-maintained and not affiliated with TypeSafe. As of the data file generated on 22 October 2026 it holds 229 rows: Jev 1.14 and 228 other entries. Its edition is 0.4.1. Edition 0.4, on 19 October 2026, rebuilt the headline score, and 0.4.1, on 21 October, added more and harder private tests in new domains, so new-domain scores are lower than before. Before that, edition 0.3.2 on 12 October fixed a scoring bug that had left three same-skill test sets out of the score for 37 models.
It sends the same requests to every model. The public suite is 37 benchmarks in five areas, each scored so that 0 is random guessing and 100 is perfect. An unanswered request counts as wrong. Since edition 0.4 the headline Full score takes 60% from private tests of the same skills and 40% from private tasks in new domains. The board still runs and shows the public benchmarks, but they no longer count toward the Full score. Nobody outside the Index sees the private tests, so a model cannot train on them, while a model tuned on public training splits can look better on the public suite than it really is. Open entries run on one RTX PRO 6000. Some vendors submit their own public runs, and the Index runs the private tests.
Jev 1.14’s Full score is 60.24, from 59.28 on the private same-skill tests and 48.26 on the new-domain tasks. Its public score is 60.7. Sorted by Full score, it is 41st of 229. The board’s own rule puts it in a shared tenth place: models within 0.9 points of the first model in their group share a rank. The older Jev 1.13 keeps its own row at 59.22, 48th. Fastino’s GLiDE 1.2 leads alone at 71.52, 11.28 points ahead of Jev 1.14. The hosted GLiDE that Fastino launched used a thinking step. GLiDE 1.2 answers in a single pass throughout and costs $0.04 per million input tokens. Jeffan-31b-1710 is second at 69.75, then baselabs mammoth-1 (27B) at 68.44 and Perplexity Decider v1.2 (27B) at 67.98.
| Rank by Full score | Model | Full score | Public only | Calibration error (ECE) |
|---|---|---|---|---|
| 1 | Fastino GLiDE 1.2 | 71.52 | 71.99 | 0.020 |
| 2 | Jeffan-31b-1710 | 69.75 | 73.13 | 0.047 |
| 3 | baselabs mammoth-1 (27B) | 68.44 | 75.34 | 0.041 |
| 4 | pplx-decider v1.2 | 67.98 | 67.74 | 0.055 |
| 35 | Torchcast Decision 27B | 61.49 | 65.10 | 0.024 |
| 38 | Quyet-1.0-Large | 60.77 | 60.79 | 0.078 |
| 41 | Jev 1.14 | 60.24 | 60.70 | 0.080 |
| 46 | Kev 27B | 59.56 | 56.69 | 0.022 |
| 48 | Jev 1.13 | 59.22 | 57.96 | 0.078 |
| 61 | Surogate Rune 26B-A4B v3 | 56.42 | 58.28 | 0.120 |
| 68 | Clef | 54.92 | 61.71 | 0.040 |
| 97 | Drex 1.5 | 45.78 | 58.08 | 0.090 |
| 101 | Clef-omni | 45.04 | 56.54 | 0.040 |
| 103 | Clef-flash | 44.43 | 56.15 | 0.025 |
| 118 | Kev 9B v2 | 40.91 | 41.54 | 0.045 |
| 145 | Tev1-4B-experimental | 33.73 | 29.51 | 0.104 |
| 153 | d1-3B | 31.12 | 48.99 | 0.047 |
| 181 | Strands Decider 2B | 20.72 | 25.41 | 0.044 |
| 209 | d1-omni-600M | 11.75 | 17.90 | 0.128 |
| 211 | Von 1.3 | 11.65 | 13.86 | 0.216 |
| 215 | GLiNER2.5-Decide | 10.14 | 11.48 | 0.088 |
| 216 | Laya | 9.21 | 6.19 | 0.140 |
| 221 | CLM-v0.1-8B | 6.49 | 7.41 | 0.323 |
The rank column is the board’s own order by Full score. It counts every row separately, so it shows no shared places. Smaller Kev and Tev1 sizes score lower still: Kev 4B r10 34.93, Kev 0.8B r15 16.30 and Tev1-0.8B 12.47. ECE is expected calibration error, the average gap between stated probability and actual hit rate, where lower is better; the glossary entry explains it. The Index computes it over the benchmarks that have a per-field right or wrong answer, 31 of them for Jev 1.14.
Public and private scores can disagree a lot. baselabs mammoth-1 has the top public score, 75.34, and is third on the Full score. Clef scores 61.71 on the public benchmarks, above Jev 1.14’s 60.7, and is 68th on the Full score. Calibration is separate from accuracy: in this table GLiDE 1.2’s 0.020 and Kev 27B’s 0.022 are the lowest errors, and CLM’s 0.323 is the highest. The open models people compare with Jev mostly sit far below it. Laya and CLM score under 10, and GLiNER2.5-Decide 10.14.
Caveats.
- Editions do not compare. Edition 0.3 changed the suite from 0.2.1: GSM8K was rebuilt, ForecastBench was retired and the Full score was added. Edition 0.4 then rebuilt the Full score from private tests alone, and 0.4.1 made the new-domain tests harder. A 0.2.1 figure, such as the 57.91 for Jev in Nace’s run, or the 60.11 Jev had in the 0.3.1 file of 10 October, is not on the same scale as a 0.4.1 figure.
- Not every model on this site is on the board. The hosted d1, Mercury Decide, Solar Decide, Span-01, Decider 1, Kai, Celeris-1 Decision, Microsoft-Decision-1, SubQ Snap, H2O-Lightning-4B and the OpenAI Decisions API have no row. Only the open d1-3B and d1-omni-600M do. Six of them, the hosted d1, Mercury Decide, Celeris-1 Decision, Microsoft-Decision-1, SubQ Snap and the OpenAI Decisions API, are on the Artificial Analysis Decision Index below. Some board names are close to models on this site but are different models: Decision-2.0-Kai-0.6B is not Hanzo’s Kai, span-02 is not Span-01, and H2O Lightning 31B is not H2O-Lightning-4B.
- Some rows are vendor-submitted. Nace.AI submitted Drex’s public run, and Nace says Drex trained on the benchmarks’ training splits, so its public score of 58.08 carries that caveat. The Full score of 45.78 rests only on private tests Nace could not train on. Liquid submitted the public runs for both d1 models. Many newer rows, GLiDE 1.2 and Perplexity Decider v1.2 among them, are also vendor-submitted public runs, which the Index adopted after rerunning the model and comparing its answers. Perplexity declared that v1.2 trained on the official training splits of several Index benchmarks and that it picked the checkpoint partly on Index results.
- One maintainer’s harness. The Index publishes its code and rules, but it is not peer reviewed, and the private tests cannot be inspected by anyone else. Ranks within 0.9 points reflect sampling noise as much as skill.
The same board figures appear on the Clef, Jev vs Clef and Jev alternatives pages.
The Artificial Analysis Decision Index
Artificial Analysis, a company that benchmarks AI models, launched the Artificial Analysis Decision Index v1.0 on 13 October 2026 and had moved it to version 2.1 by 12 November 2026. The names are close, but it is a separate board from the Jev Decision Index, with its own tests. Version 2.1 combines six evaluations with these weights:
- AA-Paperclip (30%): 1,500 resumes, receipts, invoices and research papers, each with decisions to make and facts to extract. AA scores each document’s 15 answers as one decision.
- EvalSafe-O*NET-AA (15%): judgements about workplace documents, based on TypeSafe’s O*NET benchmark.
- When2Call (10%): NVIDIA’s tool-calling test. The model has to decide whether to call a tool, ask for more information, decline or answer directly.
- AA-Omniscience-Decision v1.1 (15%): private knowledge questions adapted from AA-Omniscience, where “I don’t know” counts as better than a wrong answer.
- HLE-MultiChoice v1.1 (15%): the text-only multiple-choice questions from Humanity’s Last Exam, a hard reasoning test, with an option to decline.
- AA-Symmetry v1.1 (15%): checks that a decision holds when its options are moved or reworded, or its question is negated.
It also records the cost of a million tasks and the time per task. AA names a host for each of the 19 decision models on it. Open-weights models you run yourself, such as Kev and Laya, are not on it.
| Rank | Model | Index score | Cost per 1M tasks | Time per task | Calibration error (ECE) |
|---|---|---|---|---|---|
| 1 | Drex Max v1.1 | 72.6 | $103 | 283 ms | 0.038 |
| 2 | SubQ Snap 1.2 | 72.6 | $137 | 162 ms | 0.046 |
| 3 | GLiDE v1.2 | 72.5 | $780 | 284 ms | 0.057 |
| 4 | mammoth-1 (Mammoth 27B) | 71.4 | $264 | 106 ms | 0.034 |
| 5 | d1-ultra | 70.6 | $133 | 301 ms | 0.021 |
| 6 | pplx-decider v1.2 | 69.8 | $69 | 229 ms | 0.061 |
| 7 | SubQ Snap 1.1 | 67.9 | $137 | 144 ms | 0.074 |
| 8 | elephant-1 (Elephant 9B) | 66.4 | $92 | 79 ms | 0.051 |
| 9 | Jev 1.14 | 66.2 | $156 | 92 ms | 0.054 |
| 10 | Gemini Decision 1 | 66.0 | $345 | 211 ms | 0.013 |
| 11 | Microsoft-Decision-1 | 62.1 | $143 | 274 ms | 0.124 |
| 12 | Mercury Decide | 62.1 | $70 | 245 ms | 0.171 |
| 13 | Celeris-1.1 Decision | 62.0 | $14 | 61 ms | 0.104 |
| 14 | Drex 2.0 | 61.6 | $200 | 255 ms | 0.045 |
| 15 | d1 (hosted) | 60.8 | $63 | 151 ms | 0.120 |
| 16 | Clef | 60.3 | $863 | 354 ms | 0.071 |
| 17 | OpenAI Decisions API (GPT-6 Luna) | 60.2 | $339 | 206 ms | 0.110 |
| 18 | pplx-decider Flash v1.2 | 60.1 | $196 | 134 ms | 0.150 |
| 19 | Clef-omni | 57.8 | $529 | 192 ms | 0.084 |
Scores are AA’s, read from its data on 12 November 2026 and rounded to one decimal. Rows level at one decimal are in AA’s unrounded order. Version 2.1 has no rows for Jev 1.13, pplx-decider v1.1 or Clef-flash, and adds Drex Max v1.1, Drex 2.0, Mammoth 27B, Gemini Decision 1 and SubQ Snap 1.2. Its Celeris row is Celeris-1.1 Decision, a newer version than the one this site covers. AA ran Microsoft-Decision-1 through OpenRouter. The ECE column is this site’s own calculation from AA’s published calibration bins, with every answer weighted equally. AA publishes the bins as a chart, not a single error figure. Version 2.1 scores do not compare with version 1.0’s. At launch version 1.0 ranked Jev 1.13 first of 11 decision models at 57.5, and on 23 October 2026 d1-ultra led its 17 at 65.2.
The top three are within 0.2 points of each other, and they lead on AA-Paperclip, at 72.8, 71.6 and 71.4. No other decision model reaches 70 there. SubQ Snap 1.2 has the best When2Call accuracy of the 19, 95.6%. Mammoth 27B has the best non-hallucination rate, 96.2, and the best AA-Symmetry score, 88.2. Jev 1.14 has the best EvalSafe-O*NET-AA accuracy, 87.7%, on a test AA based on TypeSafe’s own benchmark. Gemini Decision 1 has the best knowledge accuracy, 44.8%, and Decider v1.2 the best Humanity’s Last Exam score, 13.8%. Celeris-1.1 Decision is the fastest and the cheapest, at 61 ms and $14 per million tasks, against 92 ms and $156 for Jev 1.14.
Calibration. AA’s chart compares each model’s accuracy with the probability it gave its chosen answer. On this site’s calculation, Gemini Decision 1 has the lowest calibration error of the 19, 0.013, then d1-ultra at 0.021. Mercury Decide has the highest, 0.171. Most decision models are right less often than their probability says. Gemini Decision 1 and d1-ultra are the exceptions. Computed per test from AA’s bins, every decision model has its largest gap on Humanity’s Last Exam. Jev 1.14’s error is 0.37 there, against 0.11 on AA-Symmetry, 0.06 on When2Call, 0.04 on the knowledge questions and on EvalSafe-O*NET-AA, and 0.03 on AA-Paperclip.
Against language models. Version 2.1 runs language models through the same six tests and has them answer with probabilities. Claude Haiku 5.5 at low reasoning scores 67.2, one point above Jev 1.14, but takes about 3.0 seconds per task against 0.09 and costs $861 per million tasks against $156. GPT-6 Luna at low reasoning scores 68.8. The top score on the board, 76.5, is Claude Haiku 5.5 at maximum reasoning, at $6,218 per million tasks and about 32 seconds per task.
Against the Jev Decision Index. The two boards disagree at the top. Version 2.1 of the Artificial Analysis board puts Nace.AI’s Drex Max v1.1 first, just ahead of SubQ Snap 1.2 and GLiDE v1.2, and Jev 1.14 ninth. In the 22 October 2026 file of the Jev Decision Index (edition 0.4.1), Fastino’s GLiDE 1.2 is first and Jev 1.14 41st. They test different things: six tests weighted toward documents, against private tests that never get published, with 37 public benchmarks shown beside them. Artificial Analysis calls hosted APIs, while the Jev Decision Index runs open entries on its own GPU and takes some public runs from vendors. The two scores are not on the same scale. This site’s ranking of AI decision models blends both Indexes and JevBench into one System One Score, and reads only version 2.1 rows from Artificial Analysis.
Independent benchmarks at a glance
Each number below was read at the linked source on 24 September 2026, or appears with a quote and link in the claims field of the directory entry.
Classification and extraction
| Benchmark | Who ran it | Task | Compared with | Headline result | Sample and method |
|---|---|---|---|---|---|
| OpenRouter Ori Eval | OpenRouter | Label requests as one of 30 task types | GPT-5.6 Luna, DeepSeek V4.1 Flash, Qwen3.8 Flash, GLM 5.3 Flash | Jev median 154 ms against 860 ms for the next fastest. OpenRouter says accuracy was within a handful of cases across all five | 200 synthetic cases, sequential. Posted on X, raw data not published |
| Jev vs Claude | Ben Greenberg, dev.to | Classify evidence as satisfied, not satisfied or insufficient | Claude Sonnet 5, high reasoning | Jev 100.0% at 378ms median. Sonnet 5 99.0% at 3,554ms median | 102 submissions run three times, 306 decisions. One bounded task |
| jevbench | dhruvmehra | SST-2, AG News, Banking77 | Claude Sonnet 5, GPT-5 mini, Laya, fine-tuned DistilBERT, BART-MNLI | Jev 95.4% on SST-2 (Sonnet 5 95.6%), 84.3% on AG News (Sonnet 5 89.6%), 76.4% on Banking77 (Sonnet 5 77.4%, DistilBERT 88.0%). Jev p50 376ms to 389ms | 500 examples per dataset. Author notes about ±2.5 points of noise at that size |
| sysone-bench | instax-dutta | Nine suites, same bytes to both models | Laya, Qwen2.5-1.5B | Jev best or tied on six of nine suites, including moderation 0.989 against Laya 0.833. Laya best on AG News (0.940 vs 0.910) and MNLI (0.983 vs 0.867) | 751 states. Several suites are curated by the author, not public |
| jev-vs-open-decision-models | elcronos | Zero-shot topic and emotion | Laya, PrismNLI-0.4B, logistic regression trained on labels | Jev 0.793 on tweet_topic against 0.633 and 0.632. On dair-ai/emotion Jev and Laya tie at 0.587. Logistic regression with labels beats every zero-shot system on all four datasets | Frozen protocol, 2,000 rows on the main emotion run. Jev’s ECE ranged 0.063 to 0.281 across datasets |
| Phishing bench | anisselbd | Should an email agent click the link? | Claude Haiku 4.5 | Haiku 81.3%, Jev 62.6%. Jev p50 239 ms against 687 ms. $0.038 against $0.462 per 1,000 emails | 2,000 emails from PhishNChips v5.2. McNemar test, intervals on every figure. A two-line regex scored 91.6% |
| Smoking extraction | vclic | Ten typed fields from outpatient notes | OpenAI chat-latest structured outputs |
All ten fields right: OpenAI 98.7%, Jev 92.4%. Jev $0.18 against $24.54 per 1,000 notes, median 0.375 s against 1.782 s | 1,000 synthetic notes. Author calls these pipeline results on synthetic data |
| Spam eval | bitnovus | Ham, spam or phishing, zero-shot | TF-IDF logistic regression trained on labels | Jev 98.64%, regression 98.87% | 5,733-email test. Adding link targets and Reply-To raised Jev from 93.62% to 97.98% |
| Jev vs fine-tuned Laya | Alexander Ollman | Banking77 plus five public workflows | Laya before and after fine-tuning | Banking77: Jev 80.0%, fine-tuned Laya 79.4%. Fine-tuned Laya beat Jev on four of five other tasks | All 3,080 Banking77 test questions. Laya trained on 1,001 examples |
| Classifier eval | Malte Ubl | An existing internal classifier eval | Gemini 2.5 Flash Lite | “saturated the eval” and ran 6x faster | One post on X. Eval not published |
Ranking and retrieval
| Benchmark | Who ran it | Task | Compared with | Headline result | Sample and method |
|---|---|---|---|---|---|
| jev-rerank-bench | anessbelbati | Rerank 30 BM25 results | Cohere Rerank 4 Pro and Fast, zerank-2, DeepSeek V4.1 Flash | nDCG@10: Jev rubric 0.692, Cohere Pro 0.691. Author says no winner. Jev 422 ms against 844 ms | 8 English datasets, 1,617 scored questions. Weighting each query equally puts Cohere ahead |
| Finnish legal rerank | laguagu | Rerank two Finnish legal corpora | Voyage rerank-2.5, GPT-4.1 mini, GPT-5.6 Luna, Laya | MuPLeR-fi recall@1: Jev 97.0%, Voyage 95.5%. On the harder private corpus Voyage led, 58.3% against 56.9% | 200 and 84 queries. The second corpus is not public |
| Search rerank eval | zhuyansen | Rerank an agent-skills catalog | bge-m3, text-embedding-3-small, BM25 | Jev alone did not beat bge-m3 (+0.012 nDCG@10, interval crosses zero). Jev fused with bge-m3 was best, +0.090 | 164 queries, 9,831 labelled pairs. Measures and removes judge bias |
Security and agents
| Benchmark | Who ran it | Task | Compared with | Headline result | Sample and method |
|---|---|---|---|---|---|
| jev-sec-bench | Gaurav Gosain | Prompt injection, vulnerable code | No other model | Injection: 96.5% accuracy, ROC-AUC 0.9927, p50 325ms. Vulnerable half ranked above its secure twin in 178 of 200 pairs | 662 labelled messages, 200 code pairs. Raw per-sample output committed |
| Agent failure attribution | TokenTrim | Who&When Pro: which agent broke a run, and why | Paper figures for GPT-5.4, Claude Sonnet 4.6, GLM-5, Qwen3.5-122B | Error type F1: Jev 23.7, next best 22.2. Jev also led on agent and step | 6,257 traces, official scorer. Jev picked from listed options while the LLMs generated, so only the error-type column is like for like |
| Qwen on Cerebras vs Jev | iammrduncan | Seven synthetic app workloads | Qwen 3.8 27B on Cerebras | p50 176 ms for Jev against 215 ms. Estimated cost $0.011919 against $0.310581 | About 480 requests each. Author says it is not a controlled speed ranking |
Other tasks
| Benchmark | Who ran it | Task | Compared with | Headline result | Sample and method |
|---|---|---|---|---|---|
| Headline A/B bench | Gaurav Gosain | Pick the winning Upworthy headline | Rules of thumb | 64.5% on 10,984 tests, 74.7% where the gap was decisive, 53.1% where there was no real gap | Design fixed on one split, then run once on the other. Found and removed a position bias |
| jev-acento | marcosmartinez | Same tasks in English and Spanish | Jev against itself | Spanish text cost 3.0 to 6.4 points of accuracy on all four datasets. Spanish instructions did not help | 19,200 calls, pre-registered, model version pinned |
| Adversarial evaluation | Will Kelly | Nine experiments, 28 predictions fixed in advance | Jev against its own claims | ECE 0.075 on ticket routing. On random 3-SAT it answered satisfiable for every formula | 123,805 requests for $12.69 |
| LLM Chess | Maxim Saplin | Pick a legal move each turn | The LLM Chess leaderboard | Rank 59, Elo 242.9±117.5, next to o4-mini-medium at 240.3 | 80 games at about $0.0015 each |
Games and head-to-head demos
These are single matches or short runs. They show speed and behaviour, not general quality.
- Pokemon Showdown against Opus 5: Sid Arya reported that Jev won one battle, spending $0.0029 and 37s against $2.35 and 6m 29s for Opus 5. One game.
- Snake, local Laya against cloud Jev: 86.5 decisions per second for Laya on a laptop against 3.2 for Jev, whose calls went over the cloud API.
- Tetris, Jev against Laya-MLX: the author reports Laya at about 84ms per decision and Jev winning all three rounds.
- Pong on Show HN (item 49754516): the submitter reported Jev at 227ms average against 3.5s for GPT-5.6 Sol. The Jev vs GPT comparison has the details.
What the evidence shows
Speed holds up. The single-call medians above are all under half a second, measured from outside TypeSafe’s network, which fits the vendor’s 70ms to 500ms range. Bigger requests take longer: sysone-bench measured 925 ms to 1,068 ms for a five-question call. The multiples vary a lot because they depend on which LLM was chosen and how much it was allowed to reason.
Accuracy is task-shaped. Jev does well when the answer is a fixed label about text it can read directly: topics, intents, relevance, injection attempts. It does worse when the task needs extraction of exact values, counting or multi-step logic. The 3-SAT result and the invoice workflow are examples.
Calibration varies by task too. Reported ECE runs from 0.026 on SST-2 to 0.281 on an emotion dataset. The headline-bench and sec-bench authors found that higher confidence meant more correct answers. The phishing bench found Haiku better calibrated than Jev. Expected calibration error (ECE) is the average gap between stated probability and actual hit rate; the glossary entry explains how it is computed.
Labels still win. Where authors trained a plain classifier on a few hundred to a few thousand labelled examples, it usually matched or beat zero-shot Jev. Jev’s advantage is that you can start without labels.
Context and wording move the score. Telling Jev what the assistant was for raised injection accuracy from 89.7% to 96.5%. Adding link targets raised spam accuracy by more than four points. Naming each subject instead of numbering it took one of Will Kelly’s tests from 0.420 to 1.000.
Jev vs Opus
Two public data points compare the two directly. On TypeSafe’s workflow evals, Opus 5 averaged 73.1% against Jev’s 67.8%, at $0.1761 and 37.8 seconds per case against $0.0004 and 0.4 seconds. In the Pokemon Showdown post, Jev won the one battle played. The Jev vs Claude comparison covers the Sonnet 5 test and when to use each.
Jev 1.13 jaggedness
TypeSafe publishes a jaggedness page for each version: a list of what the model does badly. For jev-1.13 it lists literal interpretation, unreliable counting, weak arithmetic and date ordering, multi-hop indirection, distraction from irrelevant state, and no defence against prompt injection inside the state. It also says a yes/no question and its negation are not guaranteed to sum to 1.
The independent results line up with that list. The 3-SAT failure is multi-hop logic. The “decision already taken” injection that moved 147 of 200 tickets in Will Kelly’s run is injection inside the state. The glossary entry explains the term.
What has not been measured
We found no peer-reviewed study of Jev as of 24 September 2026. Most runs above cover one model version and one run, and several use synthetic or author-curated data.
Two sources are left out of the tables. Near Here’s event-validation test against Mistral and Gemini is listed in the directory, but its page refused our requests, so its figures are not repeated here. A 23 September post on sotaaz.com often turns up in searches for Jev benchmarks. It tests laya, openjev and NanoJev on BANKING77, TREC and AG News, but its author says they had no Jev API access, so it holds no Jev numbers.
Cloudflare’s 1 October 2026 launch post for Clef compares Jev with its own models on 10 Jev Decision Index tasks and four of TypeSafe’s workflow evals. Jev scores higher on When2Call, BRIGHT and agent trace observability, and lower on the rest. Cloudflare ran those numbers itself and is a competitor, so they are not in the tables above. Jev vs Clef lays them out.
How to benchmark Jev on your own data
A public benchmark tells you how Jev did on someone else’s task. To decide whether to use it, you need your own numbers.
- Collect a few hundred real inputs with known answers. Several authors above used 500, and the Janus tool’s author suggests starting with a few hundred.
- Send the state and questions you would use in production, at production length. Keep the instructions in English.
- Run the model you would otherwise deploy on the same inputs. For classification that is often a small LLM or a fine-tuned encoder, not a frontier reasoning model.
- Record accuracy, the full latency distribution under your real concurrency, and cost.
- Check calibration. Bucket answers by probability and compare each bucket’s average probability to how often it was right. The RLCD guide walks through this under “How to check calibration on your own data.”
- Pick a confidence threshold from the results and route everything below it to a person or a larger model. That is confidence gating.
- If your options appear in a list, ask each question twice with the order swapped. The headline bench found a position bias of about 7 points.
Tools in the directory can do some of this for you. abhixhek/jevcal fits thresholds to your labelled data and fails CI when a model update breaks them.
FAQ
Is there an official Jev benchmark?
TypeSafe’s workflow evals site is the only one. It covers four tasks and grades against the averaged answers of GPT-6 Astra and Claude Fable 5.1. Averaged across the four tasks, Jev scored 67.8% at $0.0004 and 0.4 seconds per case. Everything else on this page comes from independent authors.
How fast is Jev?
Independent times range from 92 ms per task for Jev 1.14 on the Artificial Analysis Decision Index version 2.1 to 422 ms in the reranking bench, all measured over the network. TypeSafe’s own range is 70ms to 500ms end to end. Latency grows with the size of the state you send, so time it on your own inputs.
Is Jev more accurate than Claude Opus?
Not on TypeSafe’s own evals, where Opus 5 averaged 73.1% against Jev’s 67.8%. Jev took 0.4 seconds per case on those tasks against 37.8, and cost $0.0004 against $0.1761. On single narrow tasks, independent tests have Jev matching or beating Claude models. On phishing, Claude Haiku 4.5 beat it by almost 19 points.
Is JevBench a real benchmark?
jevbench is a GitHub project by dhruvmehra that runs Jev, Laya, two LLMs and two BERT models over SST-2, AG News and Banking77, with 500 examples each. It is reproducible with one OpenRouter key. It is a community project, not a standard maintained by TypeSafe.
Does Jev’s confidence mean anything?
In several tests, yes: higher confidence meant more correct answers, and the headline bench found 78.5% accuracy above 0.80 confidence against 51.7% below 0.60. Calibration still varies by task and language, and the phishing bench found it worse than Haiku’s. Measure it on your data before you set a threshold.
Where does Jev do badly?
Counting, arithmetic, date ordering, multi-step logic and long states full of irrelevant text, per TypeSafe’s jaggedness page. Independent tests add non-English input, exact value extraction and injection text that claims a decision was already made. See the Jev model page for the full profile.