Confidence-gated actions with System One models

The problem

An automated system that can’t say “I’m not sure” will act confidently on its worst reads. That’s fine when the mistake is showing the wrong screen. It’s not fine when the mistake moves money, deletes a record, or sends an email to a customer’s boss.

Most classifiers give you a label and nothing else, so teams bolt on heuristics: if the input was short, ask the user; if it matched a keyword, trust it. Jev reports its own uncertainty directly. Every Choice and Score answer carries a confidence value from 0 to 1, computed from the probability spread the answer already returns. A tight spread means a clear read. A flat one means the model couldn’t separate the options, and that’s information your code can branch on before anything irreversible happens.

What the state looks like

State is the content being judged, sent once per request. For a gated action it’s usually a single utterance plus the small amount of context that tells you what acting would mean. Keep the rest out: Jev’s accuracy drops as unrelated detail grows around the thing being judged, and a flat, noisy state produces exactly the low confidence you’re trying to interpret.

{
  "utterance": "yeah go ahead and send that one through",
  "pending_action": "transfer of $4,200 to savings",
  "channel": "voice",
  "session_verified": true
}

The questions you ask

One Choice is often the whole call. The question asks what the user wants; the confidence that comes back with it is the second axis.

from typesafe_sdk import Choice, TypeSafeClient

client = TypeSafeClient()

response = client.system_one(
    state=utterance,
    questions={
        "intent": Choice(
            instructions="What action is the user requesting?",
            criteria={
                "check_balance": "Check the balance of an account",
                "approve_transfer": "Approve the pending transfer request",
                "other": "Something else",
            },
        ),
    },
)

action = response.answers["intent"]
action.choice          # the winning label
action.probabilities   # every label with its probability, summing to 1
action.confidence      # 0 to 1, how concentrated that spread is

A Choice is a pick-one question over labels you write, and it’s the right shape when exactly one action can follow. Confidence collapses the probability spread into one number so you can threshold without doing the maths yourself, and you still get the full spread if you’d rather compute your own measure.

A Score works the same way when the judgment is an ordered rating rather than a pick: you write a sentence per level, the returned number is the probability-weighted mean of the level positions starting at 0, and it carries its own confidence. A Noul, the yes/no shape that returns a probability from 0 to 1, has no confidence field at all, because that probability already is the uncertainty measure. The primitives guide sets out where each one fits.

Decision policy

Jev returns a label and a number. Every threshold below, and every function the branches call, is code you wrote and can change without touching the question.

if action.confidence < 0.6:
    route_to_support_agent(account_id)

elif action.choice == "check_balance":
    show_balance(account_id)                    # low stakes, 0.6 is enough

elif action.choice == "approve_transfer":
    if action.confidence > 0.85:
        approve_transfer(account_id)
    else:
        ask_user_to_confirm("Just to confirm: approve this transfer?")

else:
    route_to_support_agent(account_id)

That’s the documented three-band shape. High confidence acts automatically, middling confidence confirms or flags or gathers more, low confidence doesn’t act at all. The floor catches anything the model reports as uncertain, and above the floor each action sets its own bar according to what a wrong call costs. Reading out a balance at 0.6 is recoverable. Approving a transfer at 0.6 is not, so it either clears 0.85 or gets a confirmation prompt.

TypeSafe’s own examples use different numbers in different places: a floor of 0.5 with destructive operations above 0.9 on the confidence page, a floor of 0.6 with transfers above 0.85 in the routing pattern. The spread is the point. The docs say directly that correct thresholds depend on your domain and the model’s performance on your use case, so start conservative, log every gated decision with its confidence, and move the numbers once you can see where the wrong answers actually cluster. A threshold tuned on a Choice does not carry over to a Noul asking the same thing.

jev-trader is one public example of gating a real side effect this way.

When not to use this

Low confidence sometimes means the question was wrong rather than the input ambiguous. Jev reads instructions literally, so a scoping word or a negation you meant loosely gets taken at face value, and criteria that contradict the instruction confuse it. When confidence is low across your whole traffic, reword before you retune thresholds.

Gating also can’t rescue a question Jev shouldn’t be answering. Don’t gate arithmetic, date comparison, or counting: those are documented weak spots, and a confident wrong number is worse than an uncertain one. Don’t treat confidence as a defence against hostile input either, because a crafted message can move both the answer and its confidence together. Screen adversarial content separately, as in LLM guardrails, and keep the gate for ambiguous input. If your state is large and mostly irrelevant, expect flat spreads that mean nothing more than “too much noise”.

FAQ

What exactly is confidence measuring?

It’s a statistic computed from the probability distribution the answer already gives you, collapsed into one number so you can threshold on it directly. For a Choice it describes the spread across your options; for a Score, the spread across your levels. A flatter distribution means lower confidence, which usually means none of your options clearly won.

Why doesn’t a Noul have a confidence value?

The noul is already the measure. A Noul returns one number between 0 and 1, the probability the answer is yes, so 0.5 says on its own that the model can’t separate yes from no. A second uncertainty number on top would describe the same thing twice. Choice and Score need one because their answers hide the spread.

Where should I set my first threshold?

Higher than feels necessary, then lower it with evidence. Log the confidence on every decision alongside what actually turned out to be right, and you’ll see where errors concentrate after a few thousand calls. TypeSafe’s published examples range from 0.5 to 0.9 depending on how destructive the action is, which is a starting range rather than a recommendation.

Does low confidence mean the answer is wrong?

No. It means the model couldn’t separate the options cleanly, which happens when the input is ambiguous, when two of your labels overlap, or when the state doesn’t contain enough to decide. The answer may still be correct. That’s why the middle band confirms or flags rather than discarding the reading outright.

Examples in the wild