Two kinds of work sit here. One tests decision models, with benchmarks that measure accuracy and calibration, meaning whether a stated confidence matches how often the answer is right. The other uses a decision model as a judge that grades text, agent actions or other model output. A judge built on a decision model returns a typed score or choice, so a test suite can compare numbers instead of reading verdicts. The entries in this hub run on Jev and other decision models, and Laya appears in several. Examples include the Jev Decision Index leaderboard, which runs many open decision models through one question suite, abhixhek/jevcal, which fits a confidence threshold to your own labeled data, and DanRWilloughby/snifftest, a prose linter. Before you trust a benchmark, check who ran it, what data it used, whether the raw outputs are published, and whether the author sells one of the models. Treat any leaderboard as one view, and look for the method before the ranking.
64 entries
Showing 1 to 48 of 64
Vercel CTO reports Jev saturated an existing classifier eval that had used Gemini 2.5 Flash Lite and ran about 6x faster. Original post; TypeSafe's quote-tweet is a separate item.
Malte Ubl (@cramforce)OpenRan Jev against an existing classifier eval that previously used Gemini 2.5 Flash Lite on x.com220k viewsSpeed vs Gemini 6x faster
Cloudflare's live leaderboard for decision models on the Jev Decision Index suite, with Clef, Clef-flash, Jev, Kev-9B and Laya plotted by score and latency. Cloudflare built and ran it, so treat the ranking as vendor-run.
Cloudflare (@ritakozlov)41k viewsOpenDecision Model Leaderboard on clef-evals.workers-ai-mle.workers.dev
A benchmark that has language models play chess against a random-move player and the Komodo Dragon engine, scoring both chess skill and whether the model keeps making legal moves without hallucinating.
Personal-assistant agent built with Vercel's eve framework and 100 mocked tools, served through OpenRouter. Measures how many steps the agent needs when the LLM picks the tool itself versus when Jev picks it via a 101-option Choice.
A Python evals and guardrails library for agent traces that uses Jev-style decision models instead of an LLM judge, running Jev through TypeSafe or Vercel, or Kev or Laya locally on a Mac. All checks for a trace go out as one request in a few hundred milliseconds.
openlayer-aiOpenjevals on github.com81 starsVariance vs LLM 92-913x
Runs Qwen 3.8 27B on Cerebras and Jev side by side across seven synthetic workloads, plus a local Needle 3 evaluation on the same contracts. Records validated outputs, mistakes, latency, token usage and estimated cost.
A prose linter that flags signs of AI-generated writing, combining zero-dependency countable rules with one judgment model call.
An independent, evidence-based map of where Jev's calibrated-decision claim holds up and where it breaks down, built from real API-call receipts rather than a leaderboard.
Measures what a Jev classifier's confidence score is actually worth against real outcomes, and sets the threshold for handing a decision to a human based on what a mistake costs.
Fits a per-question confidence threshold to your accuracy target on your own labeled data, checks it on a held-out split, reports how much traffic still needs an LLM, and fails CI when a model update breaks the locked thresholds. Publishes no Jev numbers on purpose.
An independent benchmark of Jev against Laya on 40 Chinese support-ticket classifications, plus a confidence-gated cascade that calls Laya first and escalates to Jev, matching Jev's 78% accuracy at 1.8x the speed.
Benchmark comparing Jev against Cohere Rerank 4, ZeroEntropy zerank-2 and a chat-model baseline across 14 datasets, publishing every raw API response and bootstrap ranges on each gap.
Reproducible calibration and selective-risk benchmarks for Jev decisions inside DSPy workflows. Runs offline for confidence gating, latency and cost modelling without live API calls.
Graded relevance eval of a Jev score rerank against BM25, bge-m3 and other rankers over the Agent Skills Hub catalog, with 164 queries, 9,831 labelled pairs and the judge-circularity bias measured. Jev alone does not beat a good embedding ranker, but fusing the two wins even with Jev removed from the judging.
A reproducible benchmark running Jev, Laya and four other classifiers over three public datasets, reporting accuracy, calibration, latency, throughput and cost side by side. Jev answers one choice question per example through the Decisions API.
Recursive tournament engine that ranks texts such as poems, startup pitches and cold emails using Jev for pairwise judgments, Elo rating mechanics and Swiss matchmaking, with real-time WebSocket streaming.
Reproducible comparison of Jev and Claude Haiku 4.5 on 2,000 phishing emails, asking whether an email agent should click the link. Haiku wins on accuracy, 81.3% against 62.6%, while Jev is faster and cheaper, and a logistic regression over Jev's five signal questions reaches 95.1%.
Benchmark that tests whether models can forecast federal motion-to-dismiss rulings from the judge's written record, scored with claim-defendant micro-Brier metrics and clustered intervals. Ships a Jev integration under integrations/jev alongside the frontier-model workflows.
Runs Laya and Jev on 751 byte-identical questions across nine suites; Jev sweeps triage, guardrails and moderation, Laya wins agnews and mnli, and both miscalibrate on emotion.
Measures on your own labelled data where a small model's confidence stops paying off, then routes each decision to Jev or a larger model at the threshold it found rather than a default. On 500 Banking77 examples Jev was 77.8% accurate but 90.7% confident, and threshold 0.67 reached 80.2% accuracy for $0.1033.
Asks how far written category definitions get you before you need a labeled training set. On a 5,733-email ham, spam and phishing test Jev reached 98.64% against 98.87% for TF-IDF logistic regression. A separate 18,514-email spam run scored 98.3% against 98.4%, and averaging both reached 99.2%.
Reproducible eval harness and field note for a Jev Ultrafast research-browser session: 11 baseline cases, human and quant stress suites, QC grades, a suite runner and a report generator. It drives an upstream checkout of browser-use/jev-ultrafast at a pinned commit rather than forking it.
Benchmarks Jev on the 6,257 text traces of the Who and When Pro agent-failure-attribution set, predicting the responsible agent, the decisive step and the error type. Scored with the official whowhen_eval scorer against the paper's LLM baselines, at about $1.28 for the full run.
Two blind security benchmarks for Jev on public corpora: 662 labelled prompt-injection messages and 200 matched vulnerable-code pairs, run through jev-go with raw per-sample output committed. At a plain 0.50 cut the injection run reached 96.5% accuracy, ROC-AUC 0.9927 and p50 325 ms.
Evaluates writing against separate named checks, such as hedging, em dash overuse, clarity and passive voice, using Jev for each check with its own verdict and confidence signal.
A frozen 64-case benchmark of Feishu-style workplace messages, scoring Jev and Laya on a single choice question and on a four-question workflow. Jev scored 100 and 98.44 percent correct, Laya scored 31.25 and 28.12 percent, on the same requests.
Compares GLiNER 2.5, GLiFormer, GLiClass, Laya, Von and Jev on zero-shot information extraction and classification in one web UI.
Zero-shot classification benchmark across four datasets: Jev leads decisively on topic tasks, and Laya's advertised edge over Jev does not reproduce, tying on emotion and trailing elsewhere.
Measures how well Jev spots real secret credentials in file snippets. Each case is one Noul question and the probability is compared with the expected label, with no regex matching or provider verification. Reports accuracy, AUC, recall, precision, Brier score, review-band counts and latency.
A zero-shot benchmark of Jev, self-hosted Kev-9B and self-hosted Laya on 171 Moroccan Darija reviews in Arabic script and Arabizi. Jev scored 79.5 percent accuracy, Kev-9B 71.9 percent, and Laya 53.8 percent, tying an always-positive baseline.
A self-hosted app that runs VirusTotal data on IP addresses, domains and file hashes through Jev, Von and Laya so you can compare the models against each other on the same labelled dataset.
hc-nolanOpenIOCArena on github.comCompares Jev with Laya before and after task-specific fine-tuning across Banking77 and five other public workflows, plus a chat-moderation study. Fine-tuning raised Laya's Banking77 accuracy from 51.3 to 79.4 percent, close to Jev's 80.0 percent.
Alexander-OllmanOpenJev vs fine-tuned Laya on github.comBenchmarks Jev against Laya on 14 PostgreSQL statements with deliberately planted bugs, scoring which model catches more of the SQL defects.
Measures whether Jev reranks two Finnish legal corpora better than Voyage, GPT-based rerankers and open-weights Laya over identical shortlists and questions. Batched Jev reached 97 percent recall at 1, matching the shortlist ceiling on the MuPLeR-fi corpus.
A browser maze game where every move is a typed choice answered by a locally run Laya model. Correct moves are marked green and overruled ones red, testing whether Laya's confidence score predicts whether its own move was right.
kspviswaOpenChakravyuha on github.com
Paired comparison of Jev and OpenAI structured outputs on 1,000 synthetic outpatient notes, both given the same note, policy, candidate values and ten typed questions. OpenAI got 98.7% of records fully correct against Jev's 92.4%, while Jev cost about 133 times less with 5.2 times lower mean latency.
Benchmarks Jev against Luna, Haiku and Gemini at choosing the next action in games with explicit states, legal actions and measurable outcomes. Each game is a state machine where code owns the rules and the model only picks among the options it is given, with every model reached through Vercel AI Gateway.
hegargarciaOpenhegargarcia/jev-playground on github.comPre-registered independent check of Jev's calibration and type-safety claims, with the protocol frozen and hashed before the first call. On CLINC150 the chosen-option probabilities were calibrated (ECE 0.0204), on Banking77 they were not (ECE 0.0936, systematically overconfident), and across 8,576 responses there were zero type errors.
jourdanlabsOpenjourdanlabs/assay-001 on github.comTests whether Jev can pick the winner of real headline A/B tests across 10,984 Upworthy randomized experiments. Reported 64.5% overall and 74.7% on decisive pairs.
Gaurav-GosainOpenjev-headline-bench on github.comFive-task timed benchmark for browser agents plus almond-fastloop, a roughly 200-line agent wiring Chrome DevTools state to a bounded Jev decision and an executor. Leaderboard as of 2026-09-17 shows fastloop at 13.8s total versus Codex 66.2s and Claude (Cowork) 100.3s.
A Rust CLI that reads completed Claude Code and Codex sessions, scores them with Jev, and suggests specific changes to your agent instructions, skills and tools based on where work went well or got stuck.
An independent, reproducible audit of how well Jev performs in Spanish, with a run of 19,200 calls across four datasets showing measurably lower accuracy when the same instructions are given in Spanish instead of English, plus a CLI to repeat the comparison on your own labelled data.
marcosmartinezOpenmarcosmartinez/jev-acento on github.comA pre-registered evaluation of Jev's pairwise hawkishness rankings of FOMC chair press-conference openings, checked against same-day federal funds rate changes and an independently published scoring project.
maybern-tripp-smithOpenmaybern-tripp-smith/fedjev-bench on github.comAn adversarial evaluation of Jev with nine experiments and 28 predictions fixed before any data was collected. One run covered 123,805 requests over 138 minutes for $12.69, with five failures, plus a 13-rule prompting guide.
A 49-task, 8-site browser-agent benchmark. Jev paired with a small model for argument generation solved 100% of tasks at far lower cost than screenshot-based computer use, and adding WebMCP tool exposure roughly doubled Jev's own solve rate from 25 to 49 tasks.
idan levin (@0xidanlevin)OpenWindTunnel: the WebMCP benchmark on webmcp.com239k viewsSolve rate 25/49 to 49/49Cost vs Astra 112x lower
Video Tetris benchmark: Laya-MLX runs fast on a MacBook at about 84ms per decision but loses to Jev in all three rounds.
Tony Dinh (@tdinh_me)OpenTetris Bench: Jev vs Laya-MLX on x.com73k viewsLaya latency ~84 msResult Jev won 3/3
OpenRouter benchmarked Jev with its Ori Eval judging harness against popular LLMs and found Jev over 5 times faster than the next fastest model, with its slowest requests still beating every other model's median.
OpenRouter (@OpenRouter)OpenOpenRouter benchmarks Jev's speed against other models (X) on x.com60k viewsSpeed vs next model >5x fasterJev median latency 154 ms
Artificial Analysis scores 11 hosted decision models on knowledge, reasoning, tool-calling and safety tests, with cost and time per task, and runs language models on the same tests. Jev 1.13 leads the decision models at 58; Claude Haiku 5.5 (Low) scores 61 but takes 1.4 seconds per task against 0.13.
Artificial Analysis (@ArtificialAnlys)45k viewsOpenArtificial Analysis Decision Index: 11 hosted decision models against language models on artificialanalysis.aiNothing matches those filters
Clear a filter to widen the search, or send us the thing you were looking for.
Benchmark entries follow one pattern. Pick a labeled set, ask every model the same questions, and report accuracy and calibration. anisselbd/jev-phishing-bench compares Jev with Claude Haiku 4.5 on phishing emails, and bitnovus/jev-spam-eval asks how far written category definitions get you before you need training data. Some entries publish the raw responses so you can rerun them. Compare their questions before you compare their results.
Judging entries use score for quality, choice for pairwise picks and noul for pass or fail checks. AI Elo Ranker by opaielsheikh uses pairwise Jev judgments with Elo ratings and Swiss matchmaking to rank texts. hobbs/jev-turn-analysis scores finished Claude Code and Codex sessions and suggests changes to agent instructions. In both cases the output is a typed value, so a test suite can compare it against a threshold or a baseline run.
Watch the limits. Small test sets and synthetic data can flatter a model, and a model that leads on one task can trail on another. The ikkun article compares one direct Jev call per row with many separately scored dimensions on three tasks. Question design matters as much as model choice. Vendor-run numbers should be read as the vendor's own.
Yes. Entries such as openlayer-ai/jevals and AI Elo Ranker use a decision model to grade output and rank texts. The judgment comes back as a typed score or choice with a confidence value, so tests can compare it directly.
Run it on labeled data of your own and compare stated confidence with how often it is right. abhixhek/jevcal fits a confidence threshold to your accuracy target and checks it on a held-out split.
The Jev Decision Index and the Cloudflare Decision Model Leaderboard both run several decision models through shared question suites. Check who built each suite and read the method before relying on a ranking.
