Composite scoring with System One models

The problem

“Rate this lead out of 10” produces a number nobody can argue with, because nobody can see what went into it. When the top of the list looks wrong you have no lever to pull. Was the model weighting budget too heavily? Did it ignore timing? You cannot tell, so you rewrite the prompt and hope.

TypeSafe’s composite scoring pattern splits the judgment instead. Each dimension gets its own question with its own written rubric. Each answer comes back as a number on that dimension’s scale. Your code normalises them and applies the weights, which means the weights live in a file you can diff, not in a sentence you have to re-tune by feel. The docs make the same point about their resume example: the visibility into how the final score is calculated matters more than the ranking itself.

Two open projects lean on this shape: jev-trader scores trading signals, and Advocaat scores legal material.

What the state looks like

State is what the questions get judged against, and here it is one candidate record per request. Keep it to the fields the rubric actually reads. Accuracy falls as the state fills with material that has nothing to do with the decision.

{
  "company": "Northwind Logistics",
  "employees": 420,
  "inbound_note": "We're replacing a homegrown routing tool before our Q1 peak. Ops lead has sign-off up to $80k. Two other vendors already shortlisted.",
  "last_touch": "demo attended, 6 people from ops and finance"
}

The questions you ask

One request, one score per dimension. A score places the content on an ordered scale you describe in words, with between 2 and 10 levels, and returns the probability-weighted mean of the level numbers.

from typesafe_sdk import Score, TypeSafeClient

client = TypeSafeClient()

response = client.system_one(
    LEAD,
    {
        "budget": Score(
            instructions="How clear is the evidence that this buyer has budget for a purchase of this size?",
            criteria=[
                "No mention of money at all",
                "Interest expressed, no budget discussed",
                "A budget range is implied",
                "A figure is named without approval",
                "An approved figure and a named approver",
            ],
        ),
        "urgency": Score(
            instructions="How much time pressure is this buyer under?",
            criteria=[
                "No timeline mentioned",
                "Exploring, no date attached",
                "A quarter or season named",
                "A hard deadline with a consequence attached",
            ],
        ),
        "fit": Score(
            instructions="How well does the described problem match what this product does?",
            criteria=[
                "Different problem entirely",
                "Adjacent, would need heavy customisation",
                "Core use case with some gaps",
                "Exactly what the product is built for",
            ],
        ),
        "authority": Score(
            instructions="How close is the contact to the person who signs the contract?",
            criteria=[
                "Unclear who is involved",
                "An individual user with no stated authority",
                "A team lead who influences the decision",
                "The budget holder is directly involved",
            ],
        ),
    },
)

Write the levels as descriptions a colleague could apply, not as adjectives. “An approved figure and a named approver” tells the model where the boundary sits. “Very high budget” does not. The primitives guide covers how the returned number relates to the levels you wrote.

Decision policy

Normalise, weight, then act. Each dimension divides by its own top level index, since the four questions above do not all have the same number of levels.

budget = response.scores["budget"].score / 4
urgency = response.scores["urgency"].score / 3
fit = response.scores["fit"].score / 3
authority = response.scores["authority"].score / 3

priority = (0.30 * budget) + (0.20 * urgency) + (0.35 * fit) + (0.15 * authority)

if priority > 0.70 and response.scores["fit"].confidence > 0.6:
    assign_to_rep(LEAD, priority)
elif priority > 0.45:
    queue_for_nurture(LEAD, priority)

The weights are the part you will change most often, and changing them costs nothing because the dimension scores are already stored. Re-rank last quarter’s leads under new weights without a single new request.

Score and choice answers both carry a confidence value from 0 to 1, so a dimension the model is unsure about can be gated separately rather than quietly dragging the composite up or down. TypeSafe documents a three-band policy: act when confidence is high, confirm or gather more in the middle, route to a person when it is low. The example thresholds in the docs range from a 0.5 floor up to 0.85 or higher for consequential actions, and the docs are explicit that the right numbers depend on your domain. The composite lead scoring recipe has a worked version, and confidence-gated actions covers the routing side.

When not to use this

Do not read a score as a measurement. TypeSafe says directly that jev-1.13’s score levels are weak in numerical calibration, so a 2.4 does not mean “40% of the way from level 2 to level 3” in any recoverable sense. Use it to check a threshold or to order a list, and do not reconstruct a quantity from it.

Anything already in your database should never be a score question. Headcount, contract value, region and days since last contact are numbers you already have, and a score over them wastes a call and loses precision. Weight them directly.

Arithmetic across dimensions belongs in code, which is what the pattern is for. Counting is out for the same reason it is out everywhere else in Jev.

Watch the literal reading trap in your rubrics. A level written as “no significant budget concerns” gets read at face value, negation and all, and negations are on the documented failure list. Write each level as the positive thing that must be true.

And a long record full of unrelated history costs accuracy. Select the fields the rubric reads before you send anything.

FAQ

How many dimensions should I use?

Enough to separate the cases you actually confuse, which is usually four to six. Every dimension you add is another set of level descriptions to maintain and another weight to defend. Start with the ones a human reviewer names out loud when explaining a decision, measure whether each one moves the ranking, and drop the ones that never do.

Does splitting the judgment cost more?

Barely. All the dimensions travel in one request against one state, and TypeSafe’s fan-out pattern notes there is no speed cost for additional questions. Input is billed at $0.042 per million tokens with output free, so the document you send dominates the bill rather than the number of questions you ask about it.

How do I choose the weights?

Fit them to outcomes you already have. Score a few hundred historical records, compare the composite against what actually happened, and adjust until the ordering matches. Because the per-dimension scores are stored, this is an offline exercise on a spreadsheet rather than a loop of model calls, and you can keep several weightings for different segments.

Can I mix in a yes/no check?

Yes. A noul returns one probability from 0 to 1 and slots into the same arithmetic, which is useful for gates rather than gradients: is this an existing customer, is the contact a competitor. Keep gates separate from graded dimensions in the formula, since a gate usually wants to zero the whole score rather than shift it.

Examples in the wild