Skip to content
System One

abhixhek/jevcal

Fits a per-question confidence threshold to your accuracy target on your own labeled data, checks it on a held-out split, reports how much traffic still needs an LLM, and fails CI when a model update breaks the locked thresholds. Publishes no Jev numbers on purpose.

jevcal lint and demo output in a terminal
Image from github.com

abhixhek/jevcal on GitHub

Stars
10
Forks
0
Language
Python
License
MIT
Last push
18 Sep 2026

Read from GitHub on . Counts change daily.

How it uses Jev

from ..spec import Question
from . import Provider, ProviderError

DEFAULT_BASE_URL = "https://api.typesafe.ai"
RETRY_STATUSES = {429, 500, 502, 503, 504, 529}


class TypeSafeProvider(Provider):
    name = "typesafe"

    def __init__(self, model: str = "jev-latest", api_key: str | None = None, base_url: str | None = None,
                 max_retries: int = 5, timeout: float = 60.0) -> None:
        self.model = model
        self.api_key = api_key or os.environ.get("TYPESAFE_API_KEY")
        self.base_url = (base_url or os.environ.get("TYPESAFE_BASE_URL") or DEFAULT_BASE_URL).rstrip("/")

View in src/jevcal/providers/typesafe.py

Open on GitHub

Use casesConfidence-gated actions

More like this

Use cases this is tagged with