Skip to content
System One

Check a citation against its source with a noul

An LLM cites a document for a claim. This recipe checks the citation in two steps: a plain string match that catches quotes missing from the source, then one noul question that returns the probability the quoted section actually supports the claim. Your code turns that probability into a verdict.

Goal

Take one claim, one quote and the section of a source document the quote came from, and decide whether the source really backs the claim. Two failures matter: a quote that is nowhere in the document, and a quote that sits there word for word while its surrounding text says nothing about the claim. The first is a string comparison, the second is one question to Jev, the decision model this recipe targets. The wider problem is covered under citation verification.

State shape

State is the content you send for judging, and every question in the call sees the same state. Send an object rather than one blob of text, so the claim and the section stay separate and the question can name them. Keep the section to the part of the document the quote came from, because accuracy drops as unrelated text piles up around the answer.

{
  "claim": "A validator that is not named in a token's audience must reject the token.",
  "section": "4.1.3.  \"aud\" (Audience) Claim\n\n   The \"aud\" (audience) claim identifies the recipients that the JWT is intended for. If the principal processing the claim does not identify itself with a value in the \"aud\" claim when this claim is present, then the JWT MUST be rejected."
}

Questions

One noul, which is a yes/no question that comes back as a single number from 0 to 1: the probability the answer is yes. The question is “does the section support the claim?” and the criteria spell out both sides, so “yes” means the section states the claim or directly implies it, and “no” covers both silence and contradiction.

A noul returns no separate confidence value. The probability is the certainty measure, so 0.5 is the model telling you it cannot decide, while 0.94 and 0.06 are both strong answers pointing in opposite directions. That is why the thresholds below come in a pair, one high and one low, rather than a single cut.

A noul rather than a choice, because the downstream action is binary: you publish the citation or you do not. If you also want “contradicts” and “says nothing” separated, use a choice instead. The pitfalls section covers that trade.

Code

import re
import sys

from typesafe_sdk import Noul, NoulCriteria, TypeSafeClient

SECTION = """4.1.3.  "aud" (Audience) Claim

   The "aud" (audience) claim identifies the recipients that the JWT is
   intended for.  If the principal processing the claim does not identify
   itself with a value in the "aud" claim when this claim is present, then
   the JWT MUST be rejected."""

CITATION = {
    "claim": "A validator that is not named in a token's audience must reject the token.",
    "quote": "the JWT MUST be rejected",
}


def normalize(text: str) -> str:
    """Collapse whitespace and fold curly quotes so a quote survives line wraps."""
    folded = text.translate(str.maketrans({"“": '"', "”": '"', "’": "'"}))
    return re.sub(r"\s+", " ", folded).strip()


if CITATION["quote"] and normalize(CITATION["quote"]) not in normalize(SECTION):
    print("fabricated: the quote is not in the source")
    sys.exit(0)

client = TypeSafeClient()
response = client.system_one(
    state={"claim": CITATION["claim"], "section": SECTION},
    questions={
        "supports": Noul(
            instructions="Does the section support the claim?",
            criteria=NoulCriteria(
                true="The section states the claim or directly implies it is true",
                false="The section says nothing about the claim or states the opposite",
            ),
        )
    },
    model="jev-1.13.0",
)

print(response.answers["supports"].noul)

Decision policy

The model hands back a probability. Everything that happens next belongs to your code, including the thresholds, which you tune on your own documents rather than copying from here.

SUPPORTED = 0.80  # publish without a human looking
REVIEW = 0.40     # below this, treat the citation as unsupported


def verdict(quote_found: bool, supports: float) -> str:
    if not quote_found:
        return "fabricated"
    if supports >= SUPPORTED:
        return "verified"
    return "review" if supports >= REVIEW else "unsupported"


probability = response.answers["supports"].noul
label = verdict(True, probability)

if label == "verified":
    publish_citation(CITATION)
elif label == "review":
    queue_for_human(CITATION, probability)
else:
    strike_citation(CITATION, label)

Sample output

Illustrative, not a recorded run. The shape matches the documented response type, and a noul answer carries no confidence field.

{
  "model": "jev-1.13.0",
  "answers": {
    "supports": { "type": "noul", "noul": 0.94 }
  },
  "usage": { "input_tokens": 268, "output_tokens": 9 }
}

Pitfalls

  • The official cookbook uses a choice with three options, supports, contradicts and says_nothing, plus the same string match for missing quotes. A choice separates a source that argues the opposite from a source that never mentions the subject, and it gives you a confidence value that a noul does not. This recipe collapses those two failures into one low probability. Take the choice version when your reviewers need to know which kind of wrong they are looking at.
  • Never move a threshold from one primitive to the other. TypeSafe documents a case where the same question scored 0.22 as a noul and 0.01 as the yes option of a choice, so 0.40 tuned here means nothing on a choice. The same warning applies when you reuse this screen’s cut-off in an injection screen.
  • The string match is exact after whitespace and quote-mark folding. A quote that an LLM truncated or lightly reworded comes back as fabricated. Fuzzy matching is the fix, and it is your code’s job, not the model’s.
  • Sending the whole document instead of the one section costs accuracy. Jev’s documented weak spots include large state full of irrelevant detail, and a 60,000 character RFC around a two-line quote is exactly that. Narrow the section first, with a scored rerank if you are picking it out of a retrieval set.
  • A claim with a scoping word in it (“must”, “only when”, “unless”) gets read literally. If your claims routinely carry conditions, test that wording before you trust the numbers.

FAQ

Why does a noul have no confidence value?

Choice and score answers return a separate confidence between 0 and 1 because the selected label hides the shape of the distribution behind it. A noul has nothing to hide: the number it returns is already the probability of yes. A result of 0.5 says the model cannot separate the two outcomes, which is what a low confidence means elsewhere.

Can I check several citations in one call?

Not against different sources. One call carries one state, and each citation brings its own claim and section, so each is its own request. Several claims resting on the same section can share a call as separate nouls, since extra questions reuse the one copy of that section and add no round trip.

What should I do about a quote the model never sees?

Citations that name a section without quoting anything skip the string match and go straight to the model, with the named section as state. The cookbook does the same. Those cases deserve a lower auto-accept bar than quoted ones, since nothing has been verified mechanically before the model reads it, so route more of them to a person.

Recipes using the same primitives