Skip to content
System One

Calibration

Calibration is the match between a model's stated probabilities and how often it turns out to be right. A calibrated model that answers 80% across a large batch of questions is correct on about 80% of them. Calibration is about the honesty of the numbers, separate from raw accuracy.

Accuracy and calibration come apart. A model can be right 95% of the time and still be badly calibrated if it says 99% every time, and a model that is right only 60% of the time can be perfectly calibrated if it says 60%. The second kind is more useful to software, because a threshold in your code does what you expect.

The docs’ bug severity example shows the shape. A ticket about the export button crashing in Safari gets levels 0 to 2, and the answer comes back with probabilities of 0.0, 0.7 and 0.3, a score of 1.3, and a confidence of 0.54. If the model is calibrated, then across many tickets that look like this one, roughly 30% really are blocking issues with no workaround. That number is what a rule like “page someone if the blocking level clears 0.25” is standing on.

TypeSafe’s stated training objective, RLCD, is aimed squarely at this property. No calibration measurement for Jev has been published by TypeSafe or anyone else as of 2026-09-18, so how well it holds is untested outside the company. Confidence gating is where the property gets spent.

Related terms