Self-consistency checks with System One models

The problem

You set a threshold at 0.5 and an answer comes back at 0.48. Next run, on the same document, it comes back at 0.52. Nothing in the document changed, but the claim just moved from the deny queue to the pay queue.

That is what a self-consistency check measures: how far an answer moves when nothing that should matter has moved. TypeSafe publishes two cookbooks doing exactly this. The noul version runs a 14 item rubric over one auto-insurance claim 15 times. The choice version runs an 8 item moderation rubric over one borderline post 15 times. Both compare Jev against several LLMs at temperature 0 and at the API default.

TypeSafe’s own numbers from those runs: a mean per-question probability standard deviation of 0.0102 for Jev on the noul rubric, below every LLM probability condition they tested. On the choice rubric, Jev repeated its plurality label 90.8% of the time, and flipped on 2 of the 8 questions. Low spread is not zero spread, and that is the useful finding. A question whose answers span 0.43 to 0.53 crosses a 0.5 threshold however small the spread.

What the state looks like

Send the same state every time, byte for byte, with one throwaway field that differs per run so each call is distinct rather than served from a cache. The cookbooks call it uid.

{
  "uid": "b41f8c2e",
  "claim": {
    "claim_id": "CLM-55029",
    "incident_date": "2026-06-28",
    "driver": "Sam M.",
    "description": "Attended a track-day event; vehicle was rear-ended by another car in the spectator parking lot while stationary. Not on the circuit.",
    "amount_claimed": 3250.0
  },
  "policy": {
    "deductible": 500.0,
    "exclusions": ["track/competitive driving", "drivers not listed on the policy"]
  }
}

The borderline material is deliberate: a track day loss, but stationary in the car park. That is where answers move.

The questions you ask

One call answers the whole rubric. Repeat the call, keep every answer, then look at the spread.

from statistics import mean, pstdev

from typesafe_sdk import Noul, TypeSafeClient
from secrets import token_hex

client = TypeSafeClient(model="jev-latest")

RUBRIC = {
    "covered": "Is the loss covered under the policy's collision coverage?",
    "exclusion": "Does a policy exclusion apply to this loss?",
    "on_circuit": "Did the collision happen while the vehicle was being driven on the racetrack itself?",
    "listed_driver": "Was the driver listed on the policy?",
}

samples = {key: [] for key in RUBRIC}
for _ in range(15):
    response = client.system_one(
        {"uid": token_hex(4), **CLAIM},
        {key: Noul(instructions=text) for key, text in RUBRIC.items()},
    )
    for key in RUBRIC:
        samples[key].append(response.nouls[key].noul)

spread = {key: (mean(values), pstdev(values)) for key, values in samples.items()}

Every check here is a noul, a yes/no question whose answer is one probability between 0 and 1, phrased so that yes always means the thing you are checking for is true. Keeping the polarity uniform is what makes the rows comparable. The choice cookbook does the same with labels, tracking how often the winning label repeats. The primitives guide sets out what each shape returns.

Decision policy

Turn the middle into an explicit outcome rather than pretending the threshold is sharp. The noul cookbook treats anything from 0.30 to 0.70 as uncertain and routes it to a person, keeping the underlying probability visible. The choice cookbook requires a top probability of at least 0.60 before acting on a label.

LOW, HIGH = 0.30, 0.70

def route(probability: float) -> str:
    if probability >= HIGH:
        return "auto_approve"
    if probability <= LOW:
        return "auto_deny"
    return "human_review"

TypeSafe reports that adding that 0.60 floor on the moderation rubric lifted label agreement to 99.2% while still acting automatically on 74.2% of answers. You buy agreement with coverage, at an exchange rate you measure on your own rubric.

The code owns all of this. Jev returns a number; the escalation, the payout and the audit record are yours. Set each band from measured spread rather than intuition: run the rubric 15 times over real documents and put the cut line well clear of where the answers cluster. When an item will not separate, rewrite the question rather than shaving the threshold. The confidence-gated action recipe has the routing scaffolding.

When not to use this

One documented limit governs this whole page. Jev does not guarantee structural invariants between related questions, and TypeSafe publishes the jaggedness list showing it. On the ticket “I’m not happy with the fit. What are my options here?”, the question “is the customer asking for a refund” scored 0.22 as a noul and 0.01 on the yes option of a yes/no choice, with the choice reporting 0.97 confidence in “no”. Same question, two shapes, two different numbers.

The negation identity fails too. On “I was charged twice for the same order. Can someone look into this?”, a noul for refund came back 0.72 and a noul for “asking for something other than a refund” came back 0.47. They sum to 1.19.

So do not use repeated sampling to police an identity the model never promised. A threshold tuned on a noul does not transfer to a choice. A question and its negation are two separate questions. And a choice over options answers which option wins, while one noul per option asks whether each is true on its own terms.

Repeated sampling also tells you nothing about the documented weak spots on Jev’s jaggedness list. A rubric item that asks Jev to count occurrences or compare two dates can be perfectly stable and wrong every time. Stability measures spread, not correctness, so keep a labelled set alongside it.

FAQ

How many repeats do I need?

The published cookbooks use 15 per condition, which is enough to see whether an item clusters tightly or scatters across a threshold. Start there, on 20 to 50 documents that reflect the borderline cases you actually see. More repeats sharpen the estimate of the spread, but they do not tell you anything new about items that were already stable.

Does low spread mean the answer is right?

No. Spread and accuracy are separate properties. A question can return 0.83 every single time and still be answering something other than what you meant, which is what the documented literal reading failure looks like in practice. Measure spread to choose thresholds, and measure against labelled examples to find out whether the question is asking the right thing.

Why does a noul disagree with a yes/no choice?

They are different questions with different output shapes, and TypeSafe states directly that the identities you would expect between them are not guaranteed. The documented example scores 0.22 as a noul against 0.01 on the choice’s yes option. Pick one shape per decision, tune your threshold on that shape, and never carry the number across.

What should I do with an item that keeps flipping?

Rewrite it before you touch the threshold. A flipping item usually hides two judgments in one question or leaves a boundary case unstated, both of which are on the documented failure list. Split it into two literal questions and combine them in code, or spell the boundary out in the criteria, then re-measure the spread.

Examples in the wild