Feature extraction for machine learning with System One models

The problem

You have a table with an outcome column and a free-text column, and the text plainly carries signal that the numbers do not. A tasting note predicts a critic’s rating. A claims narrative predicts fraud. Gradient boosting wants a matrix of floats, so the text usually gets reduced to word counts or dropped.

Embeddings give you hundreds of numbers that mean nothing on their own. A question gives you one number with a name on it, and a column called “how much this note dwells on tannin structure” is something a domain expert can check.

TypeSafe’s autoresearch cookbook builds this end to end on 2,000 wine reviews, predicting a critic’s score on an 80 to 100 scale. Their published run reports these held-out errors, measured on 800 reviews the loop never saw, where lower is better: 3.09 for predicting the training average, 2.47 for CatBoost reading the note as word counts, 2.15 for asking Jev the score outright and rescaling it, 1.87 for 18 questions from a single proposal call, and 1.77 after five rounds of refinement. Those numbers are TypeSafe’s own measurement on their own dataset, so treat the ordering as the transferable part rather than the magnitudes.

What the state looks like

One row per request. Send the text the questions read and the structured fields that give it context, and leave out the rest, since accuracy falls as the state fills with detail the questions do not need.

{
  "review_id": 41207,
  "variety": "Nebbiolo",
  "note": "Rose petal and tar on the nose, giving way to sour cherry. The tannins are grippy and still closed, needing five years at least. Long, savoury finish."
}

The outcome column stays out of the state. Showing the model the label you are predicting would leak it into the features.

The questions you ask

Each feature is one question.

import numpy as np
from typesafe_sdk import Noul, NoulCriteria, Score, TypeSafeClient

INTENSITY = [
    "Not present in this note at all",
    "Barely present, mentioned once in passing",
    "Present at a moderate level",
    "Present strongly, the note dwells on it",
    "Dominant, the note is largely about this",
]
PRESENCE = NoulCriteria(
    true="The note states this or clearly implies it",
    false="The note gives no indication of this",
)

client = TypeSafeClient(model="jev-latest")

response = client.system_one(
    REVIEW,
    {
        "tannin_focus": Score(instructions="How much does this note dwell on tannin structure?", criteria=INTENSITY),
        "oak_focus": Score(instructions="How much does this note dwell on oak or barrel character?", criteria=INTENSITY),
        "ageing_advice": Noul(instructions="Does the note advise keeping the wine before drinking it?", criteria=PRESENCE),
    },
)

levels = np.arange(len(INTENSITY))
probs = np.array([response.scores["tannin_focus"].probabilities.get(i, 0.0) for i in levels])
mean = float(probs @ levels)
spread = float(np.sqrt(probs @ (levels ** 2) - mean ** 2))
row = [mean, spread, response.nouls["ageing_advice"].noul]

A score question becomes two columns: where the answer sits on the scale, and how spread out the distribution is around it. That second column is the model telling you how ambiguous the note was, and gradient boosting can use it. A noul, a yes/no question returning one probability from 0 to 1, becomes a single column. In TypeSafe’s final run, 29 score questions and 9 noul questions produced 67 numeric columns.

Keeping the distribution rather than the winning level is the part people skip. A note sitting between two levels makes a different row from one sitting squarely on a level, and collapsing to a label throws that away.

Decision policy

There is no decision to gate here. The features feed a downstream model, which owns the decision and its own threshold. What your code owns is the feature set.

importances = model.get_feature_importance()
keep = [name for name, weight in zip(columns, importances) if weight > 0.5]

That is the loop in the cookbook, made concrete. Train, read which columns the model used and which rows it still gets wrong, feed that report to a proposal step that writes new questions, and run again. TypeSafe’s run used an LLM to propose the questions, adding, rewording and dropping features as the report suggested. Most of the gain arrived in the first proposal call, going from 2.47 to 1.87, with four further rounds taking it to 1.77.

Two rules keep the loop honest. Judge every change on a held-out split the proposal step never sees, and keep the features you can explain, because a column named tannin_focus is auditable in a way a coordinate in an embedding is not. If the downstream model drives a consequential decision, put the confidence gating there rather than on individual features.

When not to use this

Anything already in your table. Price, region, vintage year and review length are columns you have. A model call to re-derive them costs money and loses precision, and counting words is a documented weak spot anyway.

Questions requiring arithmetic make poor features. “How many grape varieties are listed” asks Jev to count, and the error grows with the size of the thing being counted.

Anything date-shaped belongs in code, since Jev reads dates as text rather than as ordered quantities.

Do not build a feature that needs several hops through the record, since indirection costs accuracy. And do not reach for this when your text is short, formulaic and well covered by keywords. A bag of words is free and already good at that.

One scale point: this is a request per row. At $0.042 per million input tokens with output free, a 2,000 row dataset is cheap, but a ten million row corpus needs a sampling plan and attention to the documented rate limits.

FAQ

How is this different from composite scoring?

Composite scoring combines the dimensions with weights you write by hand, which keeps the formula readable and needs no training data. Here a supervised model learns the combination from labelled outcomes, which usually fits better but needs those outcomes to exist. Same questions, different owner of the arithmetic.

Why keep the spread column?

It separates “the note clearly sits at level 2” from “the note sits ambiguously between 1 and 3”, which are different facts about the row even though both average to 2. Gradient boosting can split on that column and treat confident and uncertain rows differently. It costs nothing extra, since the distribution comes back in the same response.

Do I need an LLM to propose the questions?

No. A domain expert writing 15 questions gets you most of the way, and the cookbook’s own numbers show the first proposal call carrying most of the improvement. The automated loop earns its place when you have many datasets to cover or when the feedback from model errors keeps suggesting rewordings a person would not have tried.

How stable are these features between runs?

Stable enough to train on. TypeSafe’s consistency cookbook reports a mean per-question probability standard deviation of 0.0102 on a 14 item rubric over 15 repeats, which is small next to the variation between rows. Cache the answers keyed on the row and the question text so a re-train uses the same matrix rather than re-querying.

Examples in the wild