Goal
A keyword search such as BM25, which ranks documents by how many query words they share, is fast enough to sweep a whole corpus but poor at ordering what it finds. Re-ranking fixes the order: it compares the query against each candidate on the shortlist one at a time and sorts by the result. Here Jev does the comparing, one call per pair, which is the standard shape for semantic re-ranking.
State shape
One call sees one pair. The state carries the query and a single candidate passage under named keys, so the question can talk about each side without ambiguity.
{
"query": "How long does an employer have to report a workplace injury?",
"passage": "An employer must notify the carrier within 10 days of learning of an injury."
}Thirty candidates means thirty calls. They are independent, so fire them concurrently rather than in a loop that waits. Batching several questions into one call is a different move, covered in the fan-out recipe, and it does not apply when each call needs a different state.
Questions
One Score: a question that places the state on an ordered scale you describe in words, and returns a position on that scale plus a probability for every level. A Score fits because relevance has a middle. The passage that gives background without an answer sits between the one that answers outright and the one that is off topic, and a yes/no cannot express that.
Five levels, numbered by their position in the array starting at 0:
Level 0 is off topic. Level 1 is the same subject area with nothing bearing on the query. Level 2 is background around the query without an answer. Level 3 answers part of it, or answers it indirectly. Level 4 answers it directly and in full inside the passage itself. A Score takes between 2 and 10 levels, and five gives enough separation to sort a shortlist without asking the model to split hairs it cannot split.
The returned score is a probability-weighted mean of the level numbers, so it lands between levels. That is what makes it sortable: two passages both sitting “mostly at level 3” separate at 3.12 and 3.61.
Code
from typesafe_sdk import Score, TypeSafeClient
RELEVANCE = Score(
instructions="How well does this passage answer the query?",
criteria=[
"Off topic: the passage does not address the query at all.",
"Same subject area, but nothing here bears on the query.",
"Background around the query, without an answer to it.",
"Answers part of the query, or answers it indirectly.",
"Answers the query directly and in full, inside this passage.",
],
)
query = "How long does an employer have to report a workplace injury?"
shortlist = [
{"id": "p-41", "text": "An employer must notify the carrier within 10 days."},
{"id": "p-17", "text": "Workers compensation pays medical costs after an injury."},
{"id": "p-88", "text": "The commission publishes annual claim volume statistics."},
]
def rate(client, passage):
response = client.system_one(
state={"query": query, "passage": passage["text"]},
questions={"relevance": RELEVANCE},
model="jev-1.13.0",
)
answer = response.answers["relevance"]
return {"id": passage["id"], "score": answer.score, "confidence": answer.confidence}
with TypeSafeClient() as client:
rated = [rate(client, passage) for passage in shortlist]
ranked = sorted(rated, key=lambda row: (-row["score"], -row["confidence"]))
for row in ranked:
print(row["id"], round(row["score"], 2), round(row["confidence"], 2))Decision policy
Sorting is half of it. The other half is refusing to hand a generator three weak passages dressed up as an answer, and that refusal lives in your code.
KEEP_LEVEL = 2.5 # on a 0 to 4 rubric, below this is background at best
TOP_K = 3
SHAKY = 0.5 # the rubric did not fit this pair cleanly
def select(ranked):
kept = [row for row in ranked if row["score"] >= KEEP_LEVEL][:TOP_K]
if not kept:
return {"passages": [], "fallback": "say_no_answer_found"}
if kept[0]["confidence"] < SHAKY:
return {"passages": kept, "fallback": "answer_with_hedge"}
return {"passages": kept, "fallback": None}
print(select(ranked))Sample output
Illustrative, not a recorded run. This is the answer for one pair; a thirty-candidate shortlist produces thirty of these.
{
"model": "jev-1.13.0",
"answers": {
"relevance": {
"type": "score",
"score": 3.49,
"legend": {
"0": "Off topic: the passage does not address the query at all.",
"1": "Same subject area, but nothing here bears on the query.",
"2": "Background around the query, without an answer to it.",
"3": "Answers part of the query, or answers it indirectly.",
"4": "Answers the query directly and in full, inside this passage."
},
"probabilities": { "0": 0.01, "1": 0.03, "2": 0.08, "3": 0.22, "4": 0.66 },
"confidence": 0.71
}
},
"usage": { "input_tokens": 118, "output_tokens": 20 }
}Pitfalls
- The official cookbook does not use a Score here. It scores each query-candidate pair with a Noul, a yes/no question answered as a probability from 0 to 1, asking whether the candidate could be the cited source, and sorts on that probability. A Noul gives a smooth 0 to 1 spread with no rubric to write, which separates near-ties better. The Score version earns its place when you also want a cutoff that means something to a human, since “keep anything at level 3 or above” is a rule a domain expert can read and argue with. Pick the Noul when ordering is all you need.
- Scores are comparable only within one rubric. Change a level description and every stored score from the old wording becomes noise. Version the rubric alongside the index.
- Do not read exact quantities off a score. Jev is documented as weak at arithmetic, and a returned 3.49 is a weighted average of level positions, not a measurement of anything. Threshold it, sort by it, and stop there.
- Ties at the top happen, especially when two passages say the same thing in different words. Break them on
confidence, then on the original search rank, so the order stays stable across runs. - A whole document as the passage buries the relevant sentence in irrelevant detail, and documented accuracy falls as that detail grows. Re-rank the same chunks you indexed, then check the surviving ones with the citation check recipe before quoting them.
FAQ
Why not rank the whole shortlist in one call?
Each call would then see every candidate, which is a large state full of detail irrelevant to any one comparison, and that is a documented weak spot. Pair scoring keeps each judgment independent and the state small. Thirty calls cost more tokens than one, though input runs at $0.042 per million.
How many levels should the rubric have?
Between 2 and 10 are allowed. Five works because each level is something you can point at in a real passage. Ten levels usually means two adjacent descriptions nobody can tell apart, which flattens the probability distribution, lowers confidence, and gives you a number that moves for no reason you can trace.
Does the confidence value tell me the ranking is right?
No. It tells you how concentrated the probability was across your levels for that one pair. A confident 1.0 and a confident 4.0 are both confident. Use it to break ties and to spot pairs where the rubric did not fit, not as a quality score for the ranking as a whole.