Skip to content
System One

sutro-sh/jev-align

A CLI from Sutro for building AI functions with Jev: it finds uncertain examples, has you label them, and uses GEPA to improve the function.

sutro-sh/jev-align on GitHub

Stars
284
Forks
23
Language
Python
License
Apache-2.0
Last push
20 Sep 2026

Read from GitHub on . Counts change daily.

jev-align is a CLI, run as jeva or jev-align, for turning a Jev prompt into a tuned classifier through active learning rather than manual prompt editing. Point it at a local CSV, Parquet, or JSONL file, and it walks the dataset in rounds: evaluate every row, measure which ones Jev is least certain about, pull a sample of those plus a random audit sample for the user to label by hand, then feed the accumulated labels and optional rationales into GEPA to propose a better version of the function. After each round it shows the score, how certainty shifted, and a diff of the proposed change, and the user can accept, reject, rewind, or resume later.

It supports four task shapes: binary true/false, multiclass with exactly one fixed label, multilabel with zero or more labels, and score, one level from an ordered rubric. Every proposal is a suggestion the user reviews, not an automatic swap; the tool is explicit that a higher training score never accepts a change on its own, and that labels, whether human, synthetic, imported, or agent-assisted, need reviewing to whatever standard the use case demands. Finished functions can be shared as portable definitions through ai-functions.dev.

How it uses Jev

async def evaluate(story: Story) -> Prediction:
    async with semaphore:
        result = await client.system_one(
            state=story.state(), questions=questions
        )
    if isinstance(candidate, MulticlassCandidateSpec):
        answer = result.answers["alignment"]
        prediction = Prediction(
            story_id=story.id,
            probabilities=dict(answer.probabilities),
            choice=answer.choice,
            confidence=float(answer.confidence),
            resolved_model=getattr(result, "model", None),
        )
    elif isinstance(candidate, ScoreCandidateSpec):

View in src/jev_align/jev.py

Open on GitHub

More like this

Keep browsing