Structured extraction with System One models

The problem

Start with the constraint, because it changes the whole design. Jev is not trained to generate text. TypeSafe says so plainly in the jaggedness list for jev-1.13: you can force generation by chaining choices, but it works badly and runs slowly. So the usual extraction move, “read this invoice and return the total as JSON”, is not available.

Turn it around and the problem gets easier. A regex can find every money-shaped string in an invoice. What it cannot do is tell you which one is the total the customer owes and which one is a credit already applied. That is a reading comprehension question with a small fixed answer set, which is the shape a choice question fits.

The side effect is worth stating. Because the options are spans that code already found, the answer is a copy of one of them. There is no digit to transpose and nothing to invent, which removes an entire class of hallucination from the pipeline. Projects like commit-miner use this shape to pull structured fields out of unstructured history.

What the state looks like

State is the content the questions get judged against. Send the document, and send the candidate list as part of the question rather than the state, since the candidates are the options.

{
  "document_id": "INV-2087",
  "text": "Invoice INV-2087.\nSubtotal: $1,200.00\nSales tax: $115.50\nTotal due: $1,315.50\nA $50.00 courtesy credit from last month has already been applied."
}

Four money strings in there and a regex finds all four. Nothing in the digits says which is which. The words around them do, and that is the part worth a model.

The questions you ask

Find in code, pick with a choice, copy the pick.

import re
from decimal import Decimal

from typesafe_sdk import Choice, Noul, TypeSafeClient

MONEY_RE = re.compile(r"[$€£¥]\s?\d[\d,]*(?:\.\d{2})?")
NONE = "none"

client = TypeSafeClient()
candidates = list(dict.fromkeys(MONEY_RE.findall(INVOICE["text"])))

response = client.system_one(
    INVOICE,
    {
        "total": Choice(
            instructions="Which amount is the total the customer must pay?",
            criteria={c: None for c in candidates} | {NONE: "None of these is the requested value."},
        ),
        "is_credit": Noul(
            instructions="Does this invoice show a credit or refund applied to the customer?"
        ),
    },
)

picked = response.choices["total"].choice
amount = Decimal(re.sub(r"[^\d.]", "", picked)) if picked != NONE else None

Tune the regex to over-find. Recall is what matters, because anything it misses cannot be picked. The none option does real work: without it the model must pick something even when the document never states the value, and a wrong pick is harder to catch than an abstention.

A noul, a yes/no question returning one probability from 0 to 1, handles the attributes that go with the value. Is this amount a credit or a charge? Which decimal convention does the document use? In TypeSafe’s cookbook run the credit question came back at 0.01 on the invoice total and 0.99 on the courtesy credit, so the code knew the sign of each number it parsed.

Dates get the same treatment, in parts. The month, the day and the year are each a small closed set, so each becomes its own choice with an explicit “not stated” option, and code assembles the parts into a real date. Never ask Jev to compare two dates or work out a duration.

Decision policy

Code owns the copy, the normalisation and everything downstream. The model only ever selects.

REVIEW_BELOW = 0.60

answer = response.choices["total"]
if answer.choice == NONE:
    queue_for_review(INVOICE, reason="no candidate matched")
elif answer.confidence < REVIEW_BELOW:
    queue_for_review(INVOICE, reason="low confidence", value=answer.choice)
else:
    post_to_ledger(amount)

A choice answer carries a confidence value from 0 to 1 alongside the full probability distribution over the options, so you can gate on it. TypeSafe’s date cookbook flags anything under 0.60 for a person and reports the lowest confidence among the parts it used, which means one weak answer on the year sends the whole date to review. That is the behaviour you want on a field that becomes a payment.

Set the floor where the review volume is bearable and the escapes are rare. Two candidates that both look plausible will split the probability mass between them, which shows up as low confidence rather than as a silent coin flip. TypeSafe is explicit that the right threshold depends on your domain, so measure yours on documents you have already labelled. The confidence-gated action recipe covers the routing.

When not to use this

Do not ask for text back. Summaries, rewritten clauses, normalised addresses and generated field values are all outside what Jev does, and chaining choices to fake it is slow and unreliable.

Counting the candidates or summing them is out too. Counting is a documented weak spot, arithmetic belongs in code, and the decimal parsing in the snippet above is deliberately code-side for exactly that reason.

Date comparison is the same trap. Extract the parts and do the calendar work yourself, since “is this deadline before that one” is arithmetic wearing a text costume.

Two practical ceilings. A choice takes at most 255 options, so a document with more candidates than that needs narrowing in two stages: pick the section first, then the span inside it, which is the same idea as hierarchical classification. And some values have no regex at all. A person’s name is not a pattern, so its candidates have to come from a roster you already hold or a named entity recogniser.

FAQ

What if the regex misses the value entirely?

Then Jev cannot return it, and with a none option in the criteria it will usually say so rather than pick a wrong candidate. That failure is visible, which is the point of building it this way. Tune the pattern for recall and let the model discard the extra matches, since over-finding costs you nothing but a few more options.

How do I extract a value with no obvious pattern?

Produce the candidates some other way and keep the same second step. A roster of customer names, a named entity recogniser, or a generative model proposing spans all work. Jev then picks which one the question asks for, so the selection stays typed and the returned value is a verbatim copy of something that was in the document.

Can one request extract several fields at once?

Yes, and it should. Each question is scored independently against the same state, and TypeSafe’s fan-out pattern notes there is no speed cost for additional questions. Send the total, the credit, the currency and the invoice number together, with each field’s own candidate list as its own criteria, and read them off one response object.

Why not just use an LLM with structured outputs?

An LLM writes the value, so it can write a value the document does not contain. Here the answer space is the set of spans your code found, so a wrong answer is a wrong selection rather than an invented string, and you can trace it back to a real position in the source.

Examples in the wild