Skip to content
System One

Jev benchmarks: every public evaluation in one place

Jev has been measured by TypeSafe itself and by at least twenty independent builders. The independent runs mostly agree on speed: times between 92 ms and 422 ms per call, faster than every LLM they were compared with. They disagree on accuracy. Jev wins on topic classification, routing, reranking and agent-failure attribution, loses to Claude Haiku 4.5 on phishing and to OpenAI on structured extraction, and trails models trained on labelled data. The largest independent board by number of models, the Jev Decision Index, ranks Jev 1.14 67th of 301 rows on its Full score, which since edition 0.4 counts only private tests. Its 60.24 shares 13th place under the board's tie rule and sits 13.62 points behind the leader, MachinePulse's karpo-decision-31B. The Artificial Analysis Decision Index ranked Jev first of 11 hosted decision models at its launch on 13 October 2026. On its version 2.1, Jev 1.14 is 11th of 22, and Nace.AI's Drex Max v1.1 leads. None of the tests is peer reviewed, and most are single runs by one author.

Updated

The short answer

TypeSafe publishes one set of numbers for Jev: its own workflow evals and the speed and price multiples in the launch post. Independent builders have published at least twenty more benchmarks since launch on 15 September 2026. Most come with code and raw outputs you can rerun. The biggest is the Jev Decision Index, multimodalart’s community board on Hugging Face, where Jev 1.14 scores 60.24 on its Full score in edition 0.4.1, 67th of 301 rows in the file of 12 November 2026, with MachinePulse’s karpo-decision-31B first at 73.86. The newer Artificial Analysis Decision Index, a separate board, ranked Jev first of 11 hosted decision models at 57.5 at launch. On version 2.1, with six tests, it ranks Jev 1.14 11th of 22 at 66.2 as of 14 November 2026. Nace.AI’s Drex Max v1.1 and SubQ Snap 1.2 lead at 72.6.

Speed is the result the independent tests agree on most. Measured medians for single calls land between 154 ms and 422 ms in the single-author tests, and 92 ms per task for Jev 1.14 on the Artificial Analysis board. Every LLM they were compared with was slower.

Accuracy depends on the task. Jev matched or beat the alternatives on topic classification, intent routing, reranking and agent-failure attribution. It lost clearly to Claude Haiku 4.5 on phishing, to OpenAI structured outputs on medical-note extraction, and to plain classifiers trained on labelled data. It also gets worse outside English.

What the evidence doesn’t give you: a peer-reviewed study, a large test repeated by several independent groups, or any test run on your data. Most numbers below come from one run by one author on jev-1.13.0.

TypeSafe’s own numbers

These are vendor claims. They come from TypeSafe’s own harness and have not been reproduced.

The launch post gives 70ms to 500ms end to end for Jev, against 3 to 329 seconds for frontier models, and “40x-200x faster” for System One shaped queries. The post says the headline “193.6x faster, 444.6x cheaper” figures come from its workflow evals, and that TypeSafe expects “these are on the higher end of real world gains.” The speed and pricing guide covers what those ranges leave out.

The workflow evals site runs four tasks: security incidents, agent-trace observability, invoice processing and customer service. Every model answers the same narrow questions, and code turns the answers into an action. Averaged across the four, read on 24 September 2026:

Model, workflow mode Accuracy Cost per case Seconds per case
Jev 67.8% $0.0004 0.4
terra 67.9% $0.0304 10.1
sonnet 5 67.8% $0.1174 78.1
opus 5 73.1% $0.1761 37.8
sol 74.1% $0.0836 23.3
haiku 4.5 53.6% $0.0195 12.5

Model names are as the site labels them. The reference answers are “an average of the responses of GPT-6 Astra and Claude Fable 5.1, both at high thinking,” so accuracy means agreement with two other models, not with ground truth. Per task the picture moves: Jev scored 61.8% on invoice processing against 78.4% for Opus 5, and 76.0% on customer service against 72.4%.

The Jev Decision Index

The Jev Decision Index is the largest independent board for Jev and the models built to rival it, by number of models. It is a Hugging Face Space published by the account multimodalart, which calls it community-maintained and not affiliated with TypeSafe. As of the data file generated on 12 November 2026 it holds 301 rows: Jev 1.14 and 300 other entries. Its edition is still 0.4.1; the 22 October file had 229 rows. Edition 0.4, on 19 October 2026, rebuilt the headline score, and 0.4.1, on 21 October, added more and harder private tests in new domains, so new-domain scores are lower than before. Before that, edition 0.3.2 on 12 October fixed a scoring bug that had left three same-skill test sets out of the score for 37 models.

It sends the same requests to every model. The public suite is 37 benchmarks in five areas, each scored so that 0 is random guessing and 100 is perfect. An unanswered request counts as wrong. Since edition 0.4 the headline Full score takes 60% from private tests of the same skills and 40% from private tasks in new domains. The board still runs and shows the public benchmarks, but they no longer count toward the Full score. Nobody outside the Index sees the private tests, so a model cannot train on them, while a model tuned on public training splits can look better on the public suite than it really is. Open entries run on one RTX PRO 6000. Some vendors submit their own public runs, and the Index runs the private tests.

Jev 1.14’s Full score is 60.24, from 59.28 on the private same-skill tests and 48.26 on the new-domain tasks. Its public score is 60.7. Sorted by Full score, it is 67th of 301. The board’s own rule puts it in a shared 13th place: models within 0.9 points of the first model in their group share a rank. The older Jev 1.13 keeps its own row at 59.22, 76th. MachinePulse’s karpo-decision-31B, a Gemma-4-31B-it fine-tune with no page on this site, leads at 73.86, 13.62 points ahead of Jev 1.14. Fastino’s GLiDE 27B is second at 73.37 and shares first place under the tie rule. It replaced the GLiDE 1.2 row, which led the 22 October file at 71.52. Of the other models with pages here, H2O-Lightning-31B v1.3.0 is 16th at 69.92, Jeffan-31b-1710 17th at 69.75, baselabs mammoth-1 (27B) 22nd at 68.44 and Perplexity Decider v1.2 (27B) 26th at 67.98.

Rank by Full score Model Full score Public only Calibration error (ECE)
1 karpo-decision-31B (MachinePulse) 73.86 75.05 0.048
2 Fastino GLiDE 27B 73.37 72.49 0.021
17 Jeffan-31b-1710 69.75 73.13 0.047
22 baselabs mammoth-1 (27B) 68.44 75.34 0.041
26 pplx-decider v1.2 67.98 67.74 0.055
29 pplx-decider v1.3 67.67 69.72 0.042
60 Torchcast Decision 27B 61.49 65.10 0.024
64 Quyet-1.0-Large 60.77 60.79 0.078
67 Jev 1.14 60.24 60.70 0.080
74 Kev 27B 59.56 56.69 0.022
76 Jev 1.13 59.22 57.96 0.078
94 Surogate Rune 26B-A4B v3 56.42 58.28 0.120
104 Clef 54.92 61.71 0.040
147 Drex 1.5 45.78 58.08 0.090
152 Clef-omni 45.04 56.54 0.040
154 Clef-flash 44.43 56.15 0.025
168 Kev 9B v2 40.91 41.54 0.045
202 Tev1-4B-experimental 33.73 29.51 0.104
212 d1-3B 31.12 48.99 0.047
249 Strands Decider 2B 20.72 25.41 0.044
280 d1-omni-600M 11.75 17.90 0.128
282 Von 1.3 11.65 13.86 0.216
286 GLiNER2.5-Decide 10.14 11.48 0.088
287 Laya 9.21 6.19 0.140
293 CLM-v0.1-8B 6.49 7.41 0.323

The rank column is the board’s own order by Full score in the file generated on 12 November 2026. It counts every row separately, so it shows no shared places. Every score in this table that was also in the 22 October file is unchanged. The ranks fell because the file has 72 more rows, many of them above Jev. Smaller Kev and Tev1 sizes score lower still: Kev 4B r10 34.93, Kev 0.8B r15 16.30 and Tev1-0.8B 12.47. ECE is expected calibration error, the average gap between stated probability and actual hit rate, where lower is better; the glossary entry explains it. The Index computes it over the benchmarks that have a per-field right or wrong answer, 31 of them for Jev 1.14.

Public and private scores can disagree a lot. baselabs mammoth-1 shares the top public score, 75.34, with Copernicus-57B, and is 22nd on the Full score. Clef scores 61.71 on the public benchmarks, above Jev 1.14’s 60.7, and is 104th on the Full score. Calibration is separate from accuracy: in this table GLiDE 27B’s 0.021 and Kev 27B’s 0.022 are the lowest errors, and CLM’s 0.323 is the highest. The open models people compare with Jev mostly sit far below it. Laya and CLM score under 10, and GLiNER2.5-Decide 10.14.

Caveats.

  • Editions do not compare. Edition 0.3 changed the suite from 0.2.1: GSM8K was rebuilt, ForecastBench was retired and the Full score was added. Edition 0.4 then rebuilt the Full score from private tests alone, and 0.4.1 made the new-domain tests harder. A 0.2.1 figure, such as the 57.91 for Jev in Nace’s run, or the 60.11 Jev had in the 0.3.1 file of 10 October, is not on the same scale as a 0.4.1 figure.
  • Not every model on this site is on the board. The hosted d1, Mercury Decide, Solar Decide, Span-01, Decider 1, Kai, Celeris-1 Decision, Microsoft-Decision-1, SubQ Snap, H2O-Lightning-4B and the OpenAI Decisions API have no row. Only the open d1 models do: d1-3B, d1-omni-600M, d1-2B and d1-Omni-2B. Six of them, the hosted d1, Mercury Decide, Celeris-1 Decision, Microsoft-Decision-1, SubQ Snap and the OpenAI Decisions API, are on the Artificial Analysis Decision Index below. Some board names are close to models on this site but are different models: Decision-2.0-Kai-0.6B is not Hanzo’s Kai, span-02 is not Span-01, and H2O Lightning 31B is not H2O-Lightning-4B.
  • Some rows are vendor-submitted. Nace.AI submitted Drex’s public run, and Nace says Drex trained on the benchmarks’ training splits, so its public score of 58.08 carries that caveat. The Full score of 45.78 rests only on private tests Nace could not train on. Liquid submitted the public runs for both d1 models. Many newer rows, Perplexity Decider v1.2 among them, are also vendor-submitted public runs, which the Index adopted after rerunning the model and comparing its answers. Perplexity declared that v1.2 trained on the official training splits of several Index benchmarks and that it picked the checkpoint partly on Index results.
  • One maintainer’s harness. The Index publishes its code and rules, but it is not peer reviewed, and the private tests cannot be inspected by anyone else. Ranks within 0.9 points reflect sampling noise as much as skill.

The same board figures appear on the Clef, Jev vs Clef and Jev alternatives pages.

The Artificial Analysis Decision Index

Artificial Analysis, a company that benchmarks AI models, launched the Artificial Analysis Decision Index v1.0 on 13 October 2026 and had moved it to version 2.1 by 12 November 2026. The names are close, but it is a separate board from the Jev Decision Index, with its own tests. Version 2.1 combines six evaluations with these weights:

  • AA-Paperclip (30%): 1,500 resumes, receipts, invoices and research papers, each with decisions to make and facts to extract. AA scores each document’s 15 answers as one decision.
  • EvalSafe-O*NET-AA (15%): judgements about workplace documents, based on TypeSafe’s O*NET benchmark.
  • When2Call (10%): NVIDIA’s tool-calling test. The model has to decide whether to call a tool, ask for more information, decline or answer directly.
  • AA-Omniscience-Decision v1.1 (15%): private knowledge questions adapted from AA-Omniscience, where “I don’t know” counts as better than a wrong answer.
  • HLE-MultiChoice v1.1 (15%): the text-only multiple-choice questions from Humanity’s Last Exam, a hard reasoning test, with an option to decline.
  • AA-Symmetry v1.1 (15%): checks that a decision holds when its options are moved or reworded, or its question is negated.

It also records the cost of a million tasks and the time per task. AA names a host for each of the 22 decision models on it, as of 14 November 2026. Open-weights models you run yourself, such as Kev and Laya, are not on it.

Rank Model Index score Cost per 1M tasks Time per task Calibration error (ECE)
1 Drex Max v1.1 72.6 $103 283 ms 0.038
2 SubQ Snap 1.2 72.6 $137 162 ms 0.046
3 GLiDE v1.2 72.5 $780 284 ms 0.057
4 mammoth-1 (Mammoth 27B) 71.4 $264 106 ms 0.034
5 d1-ultra 70.6 $133 301 ms 0.021
6 pplx-decider v1.3 70.4 $68 289 ms 0.064
7 pplx-decider v1.2 69.8 $68 229 ms 0.061
8 SubQ Snap 1.1 67.9 $137 144 ms 0.074
9 d1.1 (hosted) 66.5 $63 253 ms 0.059
10 elephant-1 (Elephant 9B) 66.4 $92 79 ms 0.051
11 Jev 1.14 66.2 $156 92 ms 0.054
12 Gemini Decision 1 66.0 $345 211 ms 0.013
13 pplx-decider Flash v1.3 64.1 $34 178 ms 0.112
14 Microsoft-Decision-1 62.1 $143 274 ms 0.124
15 Mercury Decide 62.1 $70 245 ms 0.171
16 Celeris-1.1 Decision 62.0 $14 61 ms 0.104
17 Drex 2.0 61.6 $200 255 ms 0.045
18 d1 (hosted, older row) 60.8 $63 151 ms 0.120
19 Clef 60.3 $863 354 ms 0.071
20 OpenAI Decisions API (GPT-6 Luna) 60.2 $339 206 ms 0.110
21 pplx-decider Flash v1.2 60.1 $196 134 ms 0.150
22 Clef-omni 57.8 $529 192 ms 0.084

Scores are AA’s, read from its data on 14 November 2026 and rounded to one decimal. Rows level at one decimal are in AA’s unrounded order. Version 2.1 has no rows for Jev 1.13, pplx-decider v1.1 or Clef-flash, and adds Drex Max v1.1, Drex 2.0, Mammoth 27B, Gemini Decision 1 and SubQ Snap 1.2. Between 12 and 14 November it went from 19 decision models to 22 by adding Decider v1.3, Decider Flash v1.3 and d1.1, so every model below sixth place moved down. AA marks d1.1 as its default d1 row and keeps the older d1 row. Its Celeris row is Celeris-1.1 Decision, the current version. AA ran Microsoft-Decision-1 through OpenRouter. The ECE column is this site’s own calculation from AA’s published calibration bins, with every answer weighted equally. AA publishes the bins as a chart, not a single error figure. Version 2.1 scores do not compare with version 1.0’s. At launch version 1.0 ranked Jev 1.13 first of 11 decision models at 57.5, and on 23 October 2026 d1-ultra led its 17 at 65.2.

The top three are within 0.2 points of each other, and they lead on AA-Paperclip, at 72.8, 71.6 and 71.4. No other decision model reaches 70 there. SubQ Snap 1.2 has the best When2Call accuracy of the 22, 95.6%. Mammoth 27B has the best non-hallucination rate, 96.2, and the best AA-Symmetry score, 88.2. Jev 1.14 has the best EvalSafe-O*NET-AA accuracy, 87.7%, on a test AA based on TypeSafe’s own benchmark. Gemini Decision 1 has the best knowledge accuracy, 44.8%, and Decider v1.3 the best Humanity’s Last Exam score, 15.3%. Celeris-1.1 Decision is the fastest and the cheapest, at 61 ms and $14 per million tasks, against 92 ms and $156 for Jev 1.14.

Calibration. AA’s chart compares each model’s accuracy with the probability it gave its chosen answer. On this site’s calculation, Gemini Decision 1 has the lowest calibration error of the 22, 0.013, then d1-ultra at 0.021. Mercury Decide has the highest, 0.171. On average every decision model is right less often than its probability says. Gemini Decision 1 and d1-ultra miss by less than a percentage point. Computed per test from AA’s bins, every decision model has its largest gap on Humanity’s Last Exam. Jev 1.14’s error is 0.37 there, against 0.11 on AA-Symmetry, 0.06 on When2Call, 0.04 on the knowledge questions and on EvalSafe-O*NET-AA, and 0.03 on AA-Paperclip.

Against language models. Version 2.1 runs language models through the same six tests and has them answer with probabilities. Claude Haiku 5.5 at low reasoning scores 67.2, one point above Jev 1.14, but takes about 3.0 seconds per task against 0.09 and costs $861 per million tasks against $156. GPT-6 Luna at low reasoning scores 68.8. The top score on the board, 76.5, is Claude Haiku 5.5 at maximum reasoning, at $6,218 per million tasks and about 32 seconds per task.

Against the Jev Decision Index. The two boards disagree at the top. Version 2.1 of the Artificial Analysis board puts Nace.AI’s Drex Max v1.1 first, just ahead of SubQ Snap 1.2 and GLiDE v1.2, and Jev 1.14 11th. In the 12 November 2026 file of the Jev Decision Index (edition 0.4.1), MachinePulse’s karpo-decision-31B is first, Fastino’s GLiDE 27B shares first, and Jev 1.14 is 67th. They test different things: six tests weighted toward documents, against private tests that never get published, with 37 public benchmarks shown beside them. Artificial Analysis calls hosted APIs, while the Jev Decision Index runs open entries on its own GPU and takes some public runs from vendors. The two scores are not on the same scale. This site’s ranking of AI decision models blends both Indexes and JevBench into one System One Score, and reads only version 2.1 rows from Artificial Analysis.

Independent benchmarks at a glance

Each number below was read at the linked source on 24 September 2026, or appears with a quote and link in the claims field of the directory entry.

Classification and extraction

Benchmark Who ran it Task Compared with Headline result Sample and method
OpenRouter Ori Eval OpenRouter Label requests as one of 30 task types GPT-5.6 Luna, DeepSeek V4.1 Flash, Qwen3.8 Flash, GLM 5.3 Flash Jev median 154 ms against 860 ms for the next fastest. OpenRouter says accuracy was within a handful of cases across all five 200 synthetic cases, sequential. Posted on X, raw data not published
Jev vs Claude Ben Greenberg, dev.to Classify evidence as satisfied, not satisfied or insufficient Claude Sonnet 5, high reasoning Jev 100.0% at 378ms median. Sonnet 5 99.0% at 3,554ms median 102 submissions run three times, 306 decisions. One bounded task
jevbench dhruvmehra SST-2, AG News, Banking77 Claude Sonnet 5, GPT-5 mini, Laya, fine-tuned DistilBERT, BART-MNLI Jev 95.4% on SST-2 (Sonnet 5 95.6%), 84.3% on AG News (Sonnet 5 89.6%), 76.4% on Banking77 (Sonnet 5 77.4%, DistilBERT 88.0%). Jev p50 376ms to 389ms 500 examples per dataset. Author notes about ±2.5 points of noise at that size
sysone-bench instax-dutta Nine suites, same bytes to both models Laya, Qwen2.5-1.5B Jev best or tied on six of nine suites, including moderation 0.989 against Laya 0.833. Laya best on AG News (0.940 vs 0.910) and MNLI (0.983 vs 0.867) 751 states. Several suites are curated by the author, not public
jev-vs-open-decision-models elcronos Zero-shot topic and emotion Laya, PrismNLI-0.4B, logistic regression trained on labels Jev 0.793 on tweet_topic against 0.633 and 0.632. On dair-ai/emotion Jev and Laya tie at 0.587. Logistic regression with labels beats every zero-shot system on all four datasets Frozen protocol, 2,000 rows on the main emotion run. Jev’s ECE ranged 0.063 to 0.281 across datasets
Phishing bench anisselbd Should an email agent click the link? Claude Haiku 4.5 Haiku 81.3%, Jev 62.6%. Jev p50 239 ms against 687 ms. $0.038 against $0.462 per 1,000 emails 2,000 emails from PhishNChips v5.2. McNemar test, intervals on every figure. A two-line regex scored 91.6%
Smoking extraction vclic Ten typed fields from outpatient notes OpenAI chat-latest structured outputs All ten fields right: OpenAI 98.7%, Jev 92.4%. Jev $0.18 against $24.54 per 1,000 notes, median 0.375 s against 1.782 s 1,000 synthetic notes. Author calls these pipeline results on synthetic data
Spam eval bitnovus Ham, spam or phishing, zero-shot TF-IDF logistic regression trained on labels Jev 98.64%, regression 98.87% 5,733-email test. Adding link targets and Reply-To raised Jev from 93.62% to 97.98%
Jev vs fine-tuned Laya Alexander Ollman Banking77 plus five public workflows Laya before and after fine-tuning Banking77: Jev 80.0%, fine-tuned Laya 79.4%. Fine-tuned Laya beat Jev on four of five other tasks All 3,080 Banking77 test questions. Laya trained on 1,001 examples
Classifier eval Malte Ubl An existing internal classifier eval Gemini 2.5 Flash Lite “saturated the eval” and ran 6x faster One post on X. Eval not published

Ranking and retrieval

Benchmark Who ran it Task Compared with Headline result Sample and method
jev-rerank-bench anessbelbati Rerank 30 BM25 results Cohere Rerank 4 Pro and Fast, zerank-2, DeepSeek V4.1 Flash nDCG@10: Jev rubric 0.692, Cohere Pro 0.691. Author says no winner. Jev 422 ms against 844 ms 8 English datasets, 1,617 scored questions. Weighting each query equally puts Cohere ahead
Finnish legal rerank laguagu Rerank two Finnish legal corpora Voyage rerank-2.5, GPT-4.1 mini, GPT-5.6 Luna, Laya MuPLeR-fi recall@1: Jev 97.0%, Voyage 95.5%. On the harder private corpus Voyage led, 58.3% against 56.9% 200 and 84 queries. The second corpus is not public
Search rerank eval zhuyansen Rerank an agent-skills catalog bge-m3, text-embedding-3-small, BM25 Jev alone did not beat bge-m3 (+0.012 nDCG@10, interval crosses zero). Jev fused with bge-m3 was best, +0.090 164 queries, 9,831 labelled pairs. Measures and removes judge bias

Security and agents

Benchmark Who ran it Task Compared with Headline result Sample and method
jev-sec-bench Gaurav Gosain Prompt injection, vulnerable code No other model Injection: 96.5% accuracy, ROC-AUC 0.9927, p50 325ms. Vulnerable half ranked above its secure twin in 178 of 200 pairs 662 labelled messages, 200 code pairs. Raw per-sample output committed
Agent failure attribution TokenTrim Who&When Pro: which agent broke a run, and why Paper figures for GPT-5.4, Claude Sonnet 4.6, GLM-5, Qwen3.5-122B Error type F1: Jev 23.7, next best 22.2. Jev also led on agent and step 6,257 traces, official scorer. Jev picked from listed options while the LLMs generated, so only the error-type column is like for like
Qwen on Cerebras vs Jev iammrduncan Seven synthetic app workloads Qwen 3.8 27B on Cerebras p50 176 ms for Jev against 215 ms. Estimated cost $0.011919 against $0.310581 About 480 requests each. Author says it is not a controlled speed ranking

Other tasks

Benchmark Who ran it Task Compared with Headline result Sample and method
Headline A/B bench Gaurav Gosain Pick the winning Upworthy headline Rules of thumb 64.5% on 10,984 tests, 74.7% where the gap was decisive, 53.1% where there was no real gap Design fixed on one split, then run once on the other. Found and removed a position bias
jev-acento marcosmartinez Same tasks in English and Spanish Jev against itself Spanish text cost 3.0 to 6.4 points of accuracy on all four datasets. Spanish instructions did not help 19,200 calls, pre-registered, model version pinned
Adversarial evaluation Will Kelly Nine experiments, 28 predictions fixed in advance Jev against its own claims ECE 0.075 on ticket routing. On random 3-SAT it answered satisfiable for every formula 123,805 requests for $12.69
LLM Chess Maxim Saplin Pick a legal move each turn The LLM Chess leaderboard Rank 59, Elo 242.9±117.5, next to o4-mini-medium at 240.3 80 games at about $0.0015 each

Games and head-to-head demos

These are single matches or short runs. They show speed and behaviour, not general quality.

What the evidence shows

Speed holds up. The single-call medians above are all under half a second, measured from outside TypeSafe’s network, which fits the vendor’s 70ms to 500ms range. Bigger requests take longer: sysone-bench measured 925 ms to 1,068 ms for a five-question call. The multiples vary a lot because they depend on which LLM was chosen and how much it was allowed to reason.

Accuracy is task-shaped. Jev does well when the answer is a fixed label about text it can read directly: topics, intents, relevance, injection attempts. It does worse when the task needs extraction of exact values, counting or multi-step logic. The 3-SAT result and the invoice workflow are examples.

Calibration varies by task too. Reported ECE runs from 0.026 on SST-2 to 0.281 on an emotion dataset. The headline-bench and sec-bench authors found that higher confidence meant more correct answers. The phishing bench found Haiku better calibrated than Jev. Expected calibration error (ECE) is the average gap between stated probability and actual hit rate; the glossary entry explains how it is computed.

Labels still win. Where authors trained a plain classifier on a few hundred to a few thousand labelled examples, it usually matched or beat zero-shot Jev. Jev’s advantage is that you can start without labels.

Context and wording move the score. Telling Jev what the assistant was for raised injection accuracy from 89.7% to 96.5%. Adding link targets raised spam accuracy by more than four points. Naming each subject instead of numbering it took one of Will Kelly’s tests from 0.420 to 1.000.

Jev vs Opus

Two public data points compare the two directly. On TypeSafe’s workflow evals, Opus 5 averaged 73.1% against Jev’s 67.8%, at $0.1761 and 37.8 seconds per case against $0.0004 and 0.4 seconds. In the Pokemon Showdown post, Jev won the one battle played. The Jev vs Claude comparison covers the Sonnet 5 test and when to use each.

Jev 1.13 jaggedness

TypeSafe publishes a jaggedness page for Jev: a list of what the model does badly. The jev-1.13 version listed literal interpretation, unreliable counting, weak arithmetic and date ordering, multi-hop indirection, distraction from irrelevant state, and the fact that it does not treat the state as hostile by default. It also said a yes/no question and its negation are not guaranteed to sum to 1.

The independent results line up with that list. The 3-SAT failure is multi-hop logic. The “decision already taken” injection that moved 147 of 200 tickets in Will Kelly’s run is injection inside the state. The glossary entry explains the term.

What has not been measured

We found no peer-reviewed study of Jev as of 24 September 2026. Most runs above cover one model version and one run, and several use synthetic or author-curated data.

Two sources are left out of the tables. Near Here’s event-validation test against Mistral and Gemini is listed in the directory, but its page refused our requests, so its figures are not repeated here. A 23 September post on sotaaz.com often turns up in searches for Jev benchmarks. It tests laya, openjev and NanoJev on BANKING77, TREC and AG News, but its author says they had no Jev API access, so it holds no Jev numbers.

Cloudflare’s 1 October 2026 launch post for Clef compares Jev with its own models on 10 Jev Decision Index tasks and four of TypeSafe’s workflow evals. Jev scores higher on When2Call, BRIGHT and agent trace observability, and lower on the rest. Cloudflare ran those numbers itself and is a competitor, so they are not in the tables above. Jev vs Clef lays them out.

How to benchmark Jev on your own data

A public benchmark tells you how Jev did on someone else’s task. To decide whether to use it, you need your own numbers.

  1. Collect a few hundred real inputs with known answers. Several authors above used 500, and the Janus tool’s author suggests starting with a few hundred.
  2. Send the state and questions you would use in production, at production length. Keep the instructions in English.
  3. Run the model you would otherwise deploy on the same inputs. For classification that is often a small LLM or a fine-tuned encoder, not a frontier reasoning model.
  4. Record accuracy, the full latency distribution under your real concurrency, and cost.
  5. Check calibration. Bucket answers by probability and compare each bucket’s average probability to how often it was right. The RLCD guide walks through this under “How to check calibration on your own data.”
  6. Pick a confidence threshold from the results and route everything below it to a person or a larger model. That is confidence gating.
  7. If your options appear in a list, ask each question twice with the order swapped. The headline bench found a position bias of about 7 points.

Tools in the directory can do some of this for you. abhixhek/jevcal fits thresholds to your labelled data and fails CI when a model update breaks them.

FAQ

Is there an official Jev benchmark?

TypeSafe’s workflow evals site is the only one. It covers four tasks and grades against the averaged answers of GPT-6 Astra and Claude Fable 5.1. Averaged across the four tasks, Jev scored 67.8% at $0.0004 and 0.4 seconds per case. Everything else on this page comes from independent authors.

How fast is Jev?

Independent times range from 92 ms per task for Jev 1.14 on the Artificial Analysis Decision Index version 2.1 to 422 ms in the reranking bench, all measured over the network. TypeSafe’s own range is 70ms to 500ms end to end. Latency grows with the size of the state you send, so time it on your own inputs.

Is Jev more accurate than Claude Opus?

Not on TypeSafe’s own evals, where Opus 5 averaged 73.1% against Jev’s 67.8%. Jev took 0.4 seconds per case on those tasks against 37.8, and cost $0.0004 against $0.1761. On single narrow tasks, independent tests have Jev matching or beating Claude models. On phishing, Claude Haiku 4.5 beat it by almost 19 points.

Is JevBench a real benchmark?

jevbench is a GitHub project by dhruvmehra that runs Jev, Laya, two LLMs and two BERT models over SST-2, AG News and Banking77, with 500 examples each. It is reproducible with one OpenRouter key. It is a community project, not a standard maintained by TypeSafe.

Does Jev’s confidence mean anything?

In several tests, yes: higher confidence meant more correct answers, and the headline bench found 78.5% accuracy above 0.80 confidence against 51.7% below 0.60. Calibration still varies by task and language, and the phishing bench found it worse than Haiku’s. Measure it on your data before you set a threshold.

Where does Jev do badly?

Counting, arithmetic, date ordering, multi-step logic and long states full of irrelevant text, per TypeSafe’s jaggedness page. Independent tests add non-English input, exact value extraction and injection text that claims a decision was already made. See the Jev model page for the full profile.

Related guides