Semantic code linting with System One models

The problem

Half of what a reviewer enforces is not in the linter. “Don’t swallow an exception and carry on.” “Money never touches a float.” Those rules live in a style guide and in the heads of two senior engineers, and they get enforced when someone happens to be reading carefully.

TypeSafe’s use-case map lists semantic code linting as a fit: write your team’s conventions as checks, run them in CI, flag what they catch. Jev works here because a review question has a typed answer. A noul (a yes/no question that comes back as a single probability between 0 and 1) maps onto “does this hunk break rule 4” much better than a regex does.

The economics help. Input costs $0.042 per million tokens and output is free, so a diff of a few hundred lines is a rounding error per pull request. jev-review is one open implementation of the idea.

What the state looks like

State is the content you send in one request. There is one state per request and every question is scored against it independently, so the unit of work is a single hunk: the changed lines plus the file path plus the rule text you want applied. Sending the whole repository would cost accuracy, because Jev’s documented failure modes include a state padded with detail the question does not need.

{
  "file": "billing/refunds.py",
  "hunk": "@@ -44,6 +44,9 @@\n-    charge = gateway.refund(order_id)\n+    try:\n+        charge = gateway.refund(order_id)\n+    except GatewayError:\n+        charge = None",
  "convention": "Never discard a gateway failure. Log it or re-raise it."
}

The questions you ask

Each rule becomes its own noul, and a score rates how much damage the hunk does if it ships. They all travel in one call.

from typesafe_sdk import Noul, Score, TypeSafeClient

RULES = {
    "swallowed_error": "Does this hunk catch an exception and continue without logging it or re-raising it?",
    "silent_default": "Does this hunk substitute a default value for a failed lookup, hiding the failure from the caller?",
    "float_money": "Does this hunk perform arithmetic on a currency amount held in a floating point variable?",
    "stale_comment": "Does a comment in this hunk describe behaviour that the changed code no longer has?",
}

client = TypeSafeClient(model="jev-latest")

questions = {name: Noul(instructions=text) for name, text in RULES.items()}
questions["blast_radius"] = Score(
    instructions="If this hunk ships as written, how bad is the worst realistic outcome?",
    criteria=[
        "Cosmetic only",
        "Confusing to the next reader",
        "Wrong behaviour in an edge case",
        "Wrong behaviour on the normal path",
    ],
)

response = client.system_one(HUNK_STATE, questions)

A noul is the right shape for a rule because the answer really is yes or no and you want the grey area quantified rather than rounded away. A score handles severity, which is ordered rather than binary: you describe the levels in words and get back the probability-weighted mean of the level numbers, so a hunk sitting between “confusing” and “wrong in an edge case” comes back as 1.6 rather than being forced to one side.

Bundling the rules costs nothing. TypeSafe’s fan-out pattern states there is no speed cost for extra questions, which is why the parallel questions recipe sends every check in one request instead of looping.

Decision policy

The model returns numbers. Your CI script decides what happens, and that split matters more here than in most places, because a flaky check that blocks merges gets switched off within a week.

severity = response.scores["blast_radius"].score / 3  # top level index is 3

for rule in RULES:
    probability = response.nouls[rule].noul
    if probability > 0.85 and severity > 0.6:
        request_changes(rule, probability)
    elif probability > 0.6:
        post_review_comment(rule, probability)

Two bands, not one. A high probability on a rule that only matters cosmetically gets a comment; a high probability on something that changes behaviour on the normal path asks for changes. The thresholds above are a starting point, and TypeSafe is explicit that the right numbers depend on your domain. Run the checks in report-only mode over a few hundred merged pull requests first, then set the cut lines where the false positives stop being annoying.

A noul has no separate confidence field. The probability is the certainty, so confidence gating on a noul means gating on the number itself. Choice and score answers do carry a confidence value from 0 to 1, which is what you would gate the severity score on.

When not to use this

Anything that counts. “How many times is this helper called?” is a documented weak spot: Jev recognises the shape of an answer rather than tallying, and the error grows with the size of the thing being counted. Your parser already knows, so ask it.

Arithmetic and version comparison belong in code too. So does anything date-shaped, like “is this deprecation notice past its removal date”, because Jev reads dates as text rather than as ordered quantities.

Rules with several hops in them degrade. “Does this function call something that writes to a table that another service owns?” asks about a property of a property. Resolve the call graph yourself and ask a direct question about what you found.

And treat the diff as untrusted input. State is data, and Jev has no default defence against text written to steer it, so a contributor can put an instruction in a comment. Screening for that is a separate pass, closer to LLM guardrails.

FAQ

Can this replace my existing linter?

No, and it should not try. Anything a parser can decide exactly belongs in the parser, where it is deterministic and free. Jev earns its place on the rules you had to write in prose because no grammar captures them. Run both, with the fast deterministic checks first and the semantic pass over whatever survives.

How do I stop it flagging the same thing every build?

Key each finding on the rule name plus a hash of the hunk, store the ones a reviewer dismissed, and skip them on later runs. Because Jev answers each question against the state alone, the same hunk gives you a stable answer, so a dismissal stays dismissed until the code around it actually changes.

What does a run cost on a busy repository?

Input is billed at $0.042 per million tokens and output is free, so cost tracks the size of the diffs you send rather than the number of rules. Ten rules over a 400 line diff is one request. TypeSafe’s own measurement puts end to end latency at 70ms to 500ms, which keeps the check off the critical path.

Should the model ever fail the build on its own?

No. Have it write findings, and let code apply the policy that turns a finding into a blocked merge. That separation gives you somewhere to put per-repository thresholds, an allowlist for known exceptions, and an audit trail showing which probability produced which outcome when someone asks why their pull request was held.

Examples in the wild