Gate a destructive action on confidence

Q: Why gate on confidence instead of on the winning probability?

Both work, and probabilities sits on the answer alongside confidence. Confidence is derived from the whole distribution, so it drops when the mass spreads out even if one label still leads. Ask for a specific label’s probability when the question is how sure about this one, and confidence when the question is how clear this was at all.

Goal

Reading an invoice and refunding one are not the same risk, so they should not clear the same bar. This recipe asks Jev what the user wants and whether they actually confirmed it, then uses the certainty on that answer as a second axis. The answer says what; the confidence says whether to act, which is the whole idea behind confidence-gated actions.

State shape

A billing assistant reading a chat turn. Include the turn and the offer it might be answering, because “yes, go ahead” means nothing on its own.

{
  "user_message": "Yes, go ahead and refund the 4 September charge of $49.",
  "assistant_last_offer": "I can refund the $49 charge from 4 September. Confirm?",
  "account_tier": "team"
}

One state per request, and both questions below read the same one.

Questions

Two questions in one call. The first is a Choice, which picks exactly one label from a list you write and returns a probability for every label plus a single confidence value from 0 to 1 summarising how concentrated those probabilities were. Four actions: show_invoice, resend_receipt, issue_refund and close_account, ordered here from harmless to irreversible.

The second is a Noul, a yes/no question answered as a probability between 0 and 1. It asks whether this message confirms something already offered rather than starting a fresh request. Noul answers carry no separate confidence field, because the probability already is the certainty.

Two questions because the gate has two parts. High confidence that the user means “refund” is not permission to refund if they were only asking a hypothetical. Asking both in one request costs one call, and TypeSafe states there is no speed cost for additional questions.

The Choice half is the same move as routing a support ticket; what changes is that acting on the answer here spends money.

Code

from typesafe_sdk import Choice, Noul, NoulCriteria, TypeSafeClient

ACTIONS = {
    "show_invoice": "Wants to see an invoice or a charge. Nothing changes.",
    "resend_receipt": "Wants a receipt emailed again. Nothing changes.",
    "issue_refund": "Wants money returned for a charge already taken.",
    "close_account": "Wants the account shut down and billing stopped.",
}

QUESTIONS = {
    "action": Choice(
        instructions="What is the user asking the assistant to do?",
        criteria=ACTIONS,
    ),
    "confirmed": Noul(
        instructions="Does this message confirm an action the assistant already offered?",
        criteria=NoulCriteria(
            true="It approves a specific action that was put to the user.",
            false="It is a first request, a question, or an ambiguous reply.",
        ),
    ),
}

turn = {
    "user_message": "Yes, go ahead and refund the 4 September charge of $49.",
    "assistant_last_offer": "I can refund the $49 charge from 4 September. Confirm?",
    "account_tier": "team",
}

with TypeSafeClient() as client:
    response = client.system_one(state=turn, questions=QUESTIONS, model="jev-1.13.0")

action = response.answers["action"]
confirmed = response.answers["confirmed"].noul
print(action.choice, round(action.confidence, 2), round(confirmed, 2))

Decision policy

Nothing above moved a cent. The thresholds below do, and they are the part you own. The floor of 0.6 and the 0.85 bar on money both come from TypeSafe’s worked examples, which the docs are explicit are starting points rather than settings.

FLOOR = 0.6
NEEDS = {"issue_refund": 0.85, "close_account": 0.95}
CONFIRMED_AT = 0.9


def dispatch(account_id, action, confirmed):
    if action.confidence < FLOOR:
        return hand_to_human(account_id, reason="unclear intent")
    required = NEEDS.get(action.choice)
    if required is None:
        return run_read_only(account_id, action.choice)
    if action.confidence >= required and confirmed >= CONFIRMED_AT:
        return run_and_log(account_id, action.choice)
    return ask_user_to_confirm(account_id, action.choice)


dispatch("acct-4417", action, confirmed)

Sample output

Illustrative, not a recorded run. The Choice answer carries confidence; the Noul answer does not.

{
  "model": "jev-1.13.0",
  "answers": {
    "action": {
      "type": "choice",
      "choice": "issue_refund",
      "probabilities": {
        "show_invoice": 0.04,
        "resend_receipt": 0.02,
        "issue_refund": 0.92,
        "close_account": 0.02
      },
      "confidence": 0.9
    },
    "confirmed": { "type": "noul", "noul": 0.94 }
  },
  "usage": { "input_tokens": 134, "output_tokens": 24 }
}

Pitfalls

The 0.9 on the Choice and the 0.94 on the Noul are not the same kind of number and do not share a scale. TypeSafe documents a Noul at 0.22 where a yes/no Choice on the same question put 0.01 on yes. Tune the two thresholds against separate labelled sets.
“Don’t refund it yet” can still land on issue_refund. Literal reading of negations and scoping words is the first item on the documented jaggedness list for Jev 1.13, which is exactly why a destructive branch also checks the Noul.
Confidence is not accuracy. It describes how concentrated the probabilities were, so a wrong answer can arrive at 0.95. Editing the option list shifts that distribution too, which moves confidence on messages you were already handling well. Add an action, replay a labelled set, reset the bars.
Comparing “4 September” against a billing period is arithmetic on dates, which Jev reads as text rather than as ordered quantities. Pull the date out as a Choice or a field and compare it in code.
A gate is not a screen. If the message reaching this call came from an untrusted channel, run the prompt injection screen first, because a crafted turn can raise confidence on the action an attacker wants.

FAQ

Why gate on confidence instead of on the winning probability?

Both work, and probabilities sits on the answer alongside confidence. Confidence is derived from the whole distribution, so it drops when the mass spreads out even if one label still leads. Ask for a specific label’s probability when the question is how sure about this one, and confidence when the question is how clear this was at all.

Where do these threshold numbers come from?

The 0.6 floor and the 0.85 bar on a transfer are from TypeSafe’s confidence-routing page, and its confidence page uses 0.5 and 0.9 for the same shape. Both say the right values depend on your domain and your traffic. Start conservative, log every gated decision with its numbers, and move the bars from that log.

Should the model ever call the tool itself?

No, and it cannot. Jev returns typed values, not function calls or text, so the dispatch table above is the only thing that touches an account. That separation is what makes an audit possible: the numbers Jev returned and the rule your code applied are two records you can inspect after the fact.

Menu