Skip to content
System One

Calibrated Probability

A calibrated probability is a number from a model that means what it says: across all the answers given at 80%, about 80% are right. Jev returns one for every question, a distribution over the options for Choice and Score and a single yes probability for Noul.

The distinction worth holding onto is between a number that ranks and a number that predicts. A softmax output from an ordinary classifier ranks options fine but often runs hot, clustering near 1.0 whether or not the model deserves it. A calibrated probability is meant to be read as a frequency, so 0.7 means seven times in ten.

For a Choice question the probabilities cover the listed options and sum to 1. The docs’ ticket routing example returns {"choice":"returns","confidence":1.0,"probabilities":{"shipping":0.0,"returns":1.0,"billing":0.0}}. A Noul returns one number: {"type":"noul","noul":0.99} for “Is the customer asking for a human agent?”

One limit is documented. TypeSafe’s jaggedness page for jev-1.13 says there is no guarantee that the probabilities for a statement and its negation add to 1 when you ask them as separate questions. Calibration applies within a single answer’s distribution, so building a rule that subtracts one question’s probability from another’s will not behave. Choice, Score and Noul covers which shape each question returns.

Related terms