Semantic reranking with System One models

The problem

Search over a large corpus usually runs in two steps. A fast method such as BM25, a keyword scoring formula that ranks documents by shared words, or a vector search over embeddings, which are numeric representations of text that put similar meanings near each other, cuts thousands of documents down to a shortlist. Then something more careful picks the winner from that shortlist.

The fast step is good at not losing the answer and bad at ordering it. TypeSafe’s cookbook runs BM25 over 3,565 court opinion passages for 40 legal queries: the correct passage made the top 30 for every single query, and landed first for 5% of them. That gap is what reranking closes.

Reranking scores each candidate against the query on its own, then sorts by that score. A general-purpose language model can do it, but you have to invent a scoring scale, prompt it to apply the same standard every time, and accept that repeated calls give different numbers for the same pair. Jev returns a calibrated probability instead, from a question you write once.

What the state looks like

Each request holds one query and one candidate together, so the question is about the pair. State is the content being judged, sent once per request, and here that means as many requests as you have candidates. No request sees another candidate, which is what keeps the scores comparable across the shortlist.

Name the two parts so the instructions can point at them directly.

{
  "query_excerpt": "...the court applied the same standard to a claim of qualified immunity raised at summary judgment...",
  "candidate_passage": "We hold that government officials performing discretionary functions generally are shielded from liability..."
}

The questions you ask

One question per pair. It’s a Noul: a yes/no question that comes back as a single number between 0 and 1, the probability the answer is yes. That number is the score you sort on.

from typesafe_sdk import Noul, NoulCriteria, TypeSafeClient

client = TypeSafeClient()

is_cited_source = Noul(
    instructions=(
        "The query excerpt comes from a US federal court opinion and was written "
        "immediately around a citation to a precedent; the citation itself has been "
        "removed. Could the candidate passage be from that cited precedent?"
    ),
    criteria=NoulCriteria(
        true=(
            "The candidate passage states or establishes the specific rule, standard, "
            "holding, or fact pattern that the query excerpt attributes to its removed citation."
        ),
        false=(
            "The candidate passage is merely on a similar topic or doctrine; it does not "
            "supply the specific proposition the query excerpt relies on."
        ),
    ),
)

response = client.system_one(
    state={"query_excerpt": query, "candidate_passage": candidate},
    questions={"is_cited_source": is_cited_source},
)

score = response.answers["is_cited_source"].noul   # a float from 0 to 1

The shape here is a Noul, not a Score, and the recipe name invites the opposite guess. A Score is Jev’s ordered-rating primitive, where you write a sentence per level and get back a weighted position on that scale. Reranking doesn’t need levels. It needs one comparable number per pair, and a Noul gives exactly that without any scale to invent. The criteria block is what makes the number mean the same thing on every candidate: true and false spell out what each end of the range stands for, and Jev applies those same criteria to every pair.

Sorting is one line of ordinary code:

reranked = sorted(shortlist, key=lambda c: nouls[c], reverse=True)

Decision policy

Reranking changes the order of the shortlist and nothing else, so the policy sits downstream of it. The noul is a score, so the question is how much of the reordered shortlist you pass on and whether anything gets dropped outright. Cut by position when your generator has a fixed context budget: keep the top five, ignore the rest. Cut by value when a low score should mean “nothing here answers this”, and set that floor from labelled queries rather than intuition, since a floor tuned on this question won’t transfer to a differently worded one. TypeSafe’s jaggedness notes are explicit that thresholds don’t carry between question shapes, so a number tuned on a Noul means nothing to a Choice asking the same thing.

On the CLERC test the reordering moved the correct passage up at every cut point: top-1 from 5% to 18%, top-5 from 15% to 35%, top-10 from 38% to 62%. Those are TypeSafe’s own figures, and the cookbook records them against jev-1.12 rather than the current jev-1.13. Those 1,200 calls used 1.5 million input tokens and cost $0.0645 in total, which follows from Jev’s $0.042 per million input tokens with output free. Reranking can only reorder what the fast search already found, so if the right answer isn’t on the shortlist, widen the shortlist rather than tuning the reranker.

The cookbook asks one question per pair for clarity. A real system asks several in the same request, since TypeSafe reports no speed cost for additional questions, which is how RAG passage filtering screens for relevance and hostile text in the same call that scores relevance. blink is a public project that reranks this way.

When not to use this

Don’t rerank a shortlist of thousands. Cost and latency scale linearly with candidates, so the fast search step has to do real work first. Don’t put a whole document in as one candidate either: accuracy drops as the state fills with material unrelated to the judgment, so chunk to passages.

Keep the question single-minded. Jev reads instructions literally, and a question carrying a double negative or asking about a property of a property loses accuracy, so “could this be the cited source” beats “is this not one of the irrelevant results”. Candidates pulled from the open web are untrusted text, and Jev has no default defence against a passage written to argue for its own ranking, so screen before or after you sort. Anything numeric in the ranking, recency weighting or a blend with the BM25 score, belongs in code.

FAQ

Does this replace a cross-encoder reranker?

It occupies the same slot. A cross-encoder is a model that reads the query and one candidate together and outputs a relevance score, which is structurally what’s happening here. The difference is that you write the criteria in plain sentences rather than fine-tuning on labelled pairs, and you get a calibrated probability rather than an uncalibrated score.

How do I blend the noul with my BM25 or vector score?

In code, after both numbers exist. Normalise each to a common range and weight them according to what your evaluation set says, the way TypeSafe’s composite scoring pattern handles multi-dimension ratings. Don’t ask Jev to do the blending: arithmetic is a documented weak spot, and a weighted sum is something code computes exactly.

Why does the criteria block matter so much?

It fixes what the number means. Without true and false descriptions, “could this be relevant” drifts between candidates and the scores stop being comparable, which is the same drift that makes a prompted chat model unreliable here. Writing both ends explicitly lets one question apply the same standard across a whole shortlist.

Can I rerank with a Score instead?

You can, and the cookbook doesn’t. A Score makes sense when your ranking has named tiers you want attached to outcomes, as in entity alignment, where three levels map onto three actions. For pure ordering, levels add a scale you have to define and defend, and the Noul already gives you one comparable number.

Examples in the wild