sutro-sh/jev-align on GitHub
- Stars
- 284
- Forks
- 23
- Language
- Python
- License
- Apache-2.0
- Last push
- 20 Sep 2026
Read from GitHub on . Counts change daily.
jev-align is a CLI, run as jeva or jev-align, for turning a Jev prompt into a tuned classifier through active learning rather than manual prompt editing. Point it at a local CSV, Parquet, or JSONL file, and it walks the dataset in rounds: evaluate every row, measure which ones Jev is least certain about, pull a sample of those plus a random audit sample for the user to label by hand, then feed the accumulated labels and optional rationales into GEPA to propose a better version of the function. After each round it shows the score, how certainty shifted, and a diff of the proposed change, and the user can accept, reject, rewind, or resume later.
It supports four task shapes: binary true/false, multiclass with exactly one fixed label, multilabel with zero or more labels, and score, one level from an ordered rubric. Every proposal is a suggestion the user reviews, not an automatic swap; the tool is explicit that a higher training score never accepts a change on its own, and that labels, whether human, synthetic, imported, or agent-assisted, need reviewing to whatever standard the use case demands. Finished functions can be shared as portable definitions through ai-functions.dev.
How it uses Jev
async def evaluate(story: Story) -> Prediction:
async with semaphore:
result = await client.system_one(
state=story.state(), questions=questions
)
if isinstance(candidate, MulticlassCandidateSpec):
answer = result.answers["alignment"]
prediction = Prediction(
story_id=story.id,
probabilities=dict(answer.probabilities),
choice=answer.choice,
confidence=float(answer.confidence),
resolved_model=getattr(result, "model", None),
)
elif isinstance(candidate, ScoreCandidateSpec):
