vclic/smoking-extraction-benchmark

Paired comparison of Jev and OpenAI structured outputs on 1,000 synthetic outpatient notes, both given the same note, policy, candidate values and ten typed questions. OpenAI got 98.7% of records fully correct against Jev's 92.4%, while Jev cost about 133 times less with 5.2 times lower mean latency.

Accuracy comparison — Image from github.com

vclic/smoking-extraction-benchmark on GitHub

Stars: 0
Forks: 0
Language: Python
Last push: 18 Sep 2026

Read from GitHub on 24 Sep 2026. Counts change daily.

How it uses Jev

from .shared import prediction

ENDPOINTS = {
    'typesafe': 'https://api.typesafe.ai/v1/systemone',
    'openai': 'https://api.openai.com/v1/responses',
}
OPENAI_INSTRUCTIONS = (
    'Evaluate every typed question against the supplied state. For each question, '
    'choose exactly one option from its criteria. Treat candidate calculations as '
    'possibilities requiring verification against the note, not established facts. '
    'Return answers with type choice and the selected choice. Follow the extraction '
    'policy in state. Do not include probabilities or commentary.'
)

View in smoking_eval/providers.py

Open on GitHub

Use casesStructured extraction

Introducing System One Models and Jev (Hacker News launch thread)

The 1,978-point, 513-comment launch thread. The top comment calls the speed comparison misleading, since Jev only returns structured output, and disputes the cannot-hallucinate claim because a valid type can still hold a wrong value. Others note ordinary LLMs can already be forced into structured output.

albelfioOpen

DiscussionIntent and model routing+3

DSPy: programming, not prompting, language models

Declares typed input/output Signatures for LLM modules and optimizes the underlying prompts and weights against a metric. Its Signature abstraction is the closest widely-used open equivalent of Jev's typed-question interface. Star count is GitHub's rounded display figure.

stanfordnlp38k starsOpen

AlternativeStructured extraction+3

Outlines: structured outputs for LLMs

Guarantees valid structured output during generation by constraining decoding to a grammar or schema. The mechanism HN commenters repeatedly cited as already covering Jev's cannot-produce-a-type-error guarantee on ordinary LLMs. Star count is GitHub's rounded display figure.

dottxt-ai16k starsOpen

AlternativeStructured extraction+2

Instructor: reliable JSON from any LLM

Pydantic-based library that extracts typed, validated structured data from LLMs across providers with automatic retries. The established way teams get Jev-style typed answers today, at LLM latency and cost. Star count is GitHub's rounded display figure.

567-labs14k starsOpen

AlternativeStructured extraction+1

GLiNER: generalist lightweight NER

Compact encoder model that takes the label set as input at inference time and extracts arbitrary entity types zero-shot on CPU. Same classifier-that-takes-its-categories-as-input shape, for extraction rather than decisions. Star count is GitHub's rounded display figure.

urchade3.9k starsOpen

AlternativeStructured extraction+2

vercel-labs/ai-cli

Vercel Labs CLI for generating from the terminal. Its evaluate command takes a state plus typed questions and defaults its evaluation model to typesafe-ai/jev on the AI Gateway, with -m jev as the shorthand and an --input flag for text or JSON state.

vercel-labs813 starsOpen

ToolStructured extraction

Menu

vclic/smoking-extraction-benchmark