openlayer-ai/jevals on GitHub
- Stars
- 81
- Forks
- 3
- Language
- Python
- License
- MIT
- Last push
- 23 Sep 2026
Read from GitHub on . Counts change daily.
Reported by the author
- Variance vs LLM
- 92-913x
92x to 913x the score variance of Jev
github.com
jevals is a Python library for evaluating and guardrailing agent traces using typed decision models instead of a general LLM as judge. All the checks for one trace go out as a single request that the project says costs a few thousandths of a cent and returns in a few hundred milliseconds, cheap and fast enough to run on every trace, including inside the agent’s own loop rather than as a separate offline pass. It works with Jev over the TypeSafe or Vercel APIs, with Kev or Laya running locally on a Mac, or with a regular chat LLM as a slower, more expensive fallback. Traces can be passed in as OpenAI chat format, Anthropic content blocks, or LangChain message objects directly.
The project’s stated motivation is that an LLM-as-judge setup like Ragas is expensive by design: faithfulness alone costs two LLM calls, answer relevancy costs three plus embeddings, and context precision costs one call per retrieved chunk, which the authors calculate works out to six to eleven round trips and several seconds for four metrics on a single sample. That cost is why most teams eval only a small fraction of traffic, if any, rather than running checks on the request path. The authors also cite a LangChain comparison in which GPT and Claude judges showed 92 to 913 times the score variance of Jev on identical traces.
How it uses Jev
self.api_key = api_key or os.environ.get("TYPESAFE_API_KEY")
self.base_url = (base_url or os.environ.get("TYPESAFE_BASE_URL") or "https://api.typesafe.ai").rstrip(
"/"
)
self.model = model or os.environ.get("TYPESAFE_MODEL")
self.name = name or "jev"
self.timeout = timeout
self.zero_data_retention = zero_data_retention
self.price_per_m_input = JEV_PRICE_PER_M_INPUT if self.name == "jev" else 0.0
self.price_per_m_output = 0.0
if not self.api_key:
raise BackendError("TYPESAFE_API_KEY is not set")
self.timeout = timeout
async def evaluate(self, state: Any, questions: dict[str, Question]) -> BackendResponse:View in src/jevals/backends/typesafe.py
