vinilana/jev-eval-agent on GitHub
- Stars
- 105
- Forks
- 11
- Language
- HTML
- Last push
- 17 Sep 2026
Read from GitHub on . Counts change daily.
jev-eval-agent is a benchmark built to answer one specific question: when a personal-assistant agent has 100 mocked tools available, does it take fewer steps to finish a task if the LLM picks which tool to call itself, or if Jev picks it first? The agent runs on Vercel’s eve framework with models served through OpenRouter, and the whole comparison is a single environment variable: llm-direct shows the LLM all 100 tools on every step, while jev-classifier calls Jev before each model step with the conversation state and exposes only the tool Jev picked.
In classifier mode, Jev answers two questions from the same state on every step: a 101-option choice, over the 100 tools plus “respond to user,” and a noul question asking whether every requested action already appears in what’s been done so far. If Jev picks “respond to user” but the noul question scores below a confidence threshold (0.5 by default), the router blocks that reply and exposes the next-best tool instead rather than letting the agent stop early. Every decision, the choice, its confidence, the done score, the top five candidates, latency, and whether this gate fired, gets logged to a JSONL trace file and shown in a dashboard timeline.
Results are committed to the repository, so the dashboard opens with the author’s numbers without running anything; rerunning the six tasks needs OpenRouter and TypeSafe keys.
How it uses Jev
const DONE_THRESHOLD = Number(process.env.JEV_DONE_THRESHOLD ?? 0.5);
/** Ask Jev which tool to call next. */
export async function chooseNextTool(messages: readonly ModelMessage[]): Promise<RouterDecision> {
const state = buildState(messages);
if (process.env.JEV_STUB === "1") return stubDecision(state);
const started = performance.now();
const { answers } = await getClient().systemOne({
state: JSON.parse(JSON.stringify(state)) as EntryType,
questions: {
next_tool: choice(INSTRUCTIONS, CRITERIA),
done: noul(DONE_INSTRUCTIONS),
},
});
const latencyMs = Math.round(performance.now() - started);View in agent/lib/jev-router.ts
