Skip to content
System One

TokenTrim/jev-agent-failure-benchmark

Benchmarks Jev on the 6,257 text traces of the Who and When Pro agent-failure-attribution set, predicting the responsible agent, the decisive step and the error type. Scored with the official whowhen_eval scorer against the paper's LLM baselines, at about $1.28 for the full run.

Jev vs gpt-5.4
Image from github.com

TokenTrim/jev-agent-failure-benchmark on GitHub

Stars
2
Forks
0
Language
Python
License
Apache-2.0
Last push
17 Sep 2026

Read from GitHub on . Counts change daily.

How it uses Jev

from ..candidates import candidate_agents, candidate_steps
from .base import Prediction

DEFAULT_ENDPOINT = "https://api.typesafe.ai/v1/systemone"
_BLOCK_LINE = re.compile(r"^- (?P<code>\S+): \*\*(?P<name>.+?)\*\* - (?P<desc>.*)$")

# Same task definition the official LLM prompt puts in its header, so Jev is
# judged against the same framing the baselines get (a fairness fix, not a tuned
# boost). Verbatim intent from whowhen_eval.prompts.all_at_once.
_TASK_FRAMING = (
    "You are an expert at diagnosing failures in agentic systems. The transcript below is an "
    "agentic system that attempted to answer a user question and failed because of a decisive "
    "error somewhere in the transcript. The first decisive error is the step that most directly "
    "causes the system to go wrong and eventually produce an incorrect answer."
)

View in src/jevbench/backends/jev.py

Open on GitHub

PrimitivesChoice

More like this

Keep browsing