Skip to content
System One

bitnovus/jev-spam-eval

Asks how far written category definitions get you before you need a labeled training set. On a 5,733-email ham, spam and phishing test Jev reached 98.64% against 98.87% for TF-IDF logistic regression. A separate 18,514-email spam run scored 98.3% against 98.4%, and averaging both reached 99.2%.

bitnovus/jev-spam-eval on GitHub

Stars
2
Forks
0
Language
Jupyter Notebook
License
MIT
Last push
18 Sep 2026

Read from GitHub on . Counts change daily.

How it uses Jev

record.update(arm=arm, state_sha256=hashlib.sha256(json.dumps(state, sort_keys=True).encode()).hexdigest())
t = time.monotonic()
try:
    response = await client.system_one({'email': state}, questions, model=MODEL, timeout=60)
    record.update(model=response.model, input_tokens=response.usage.input_tokens, output_tokens=response.usage.output_tokens,
                  choices={k: {'choice': a.choice, 'probabilities': a.probabilities, 'confidence': a.confidence} for k, a in response.choices.items()})
    if response.model != MODEL:
        raise ValueError('Returned model differs from requested model')
except Exception as error:
    record.pop('choices', None)
    record['error'] = type(error).__name__ + ': ' + str(error)[:300]
record['seconds'] = time.monotonic()-t
if arm == 'enriched':
    record.update(link_count=len(rich['links']), links_omitted=rich['links_omitted'], attachment_count=len(rich['attachments']))
return record

View in experiments/jev-context/evaluate.py

Open on GitHub

PrimitivesNoul

More like this

Keep browsing