Skip to content
System One

anisselbd/jev-phishing-bench

Reproducible comparison of Jev and Claude Haiku 4.5 on 2,000 phishing emails, asking whether an email agent should click the link. Haiku wins on accuracy, 81.3% against 62.6%, while Jev is faster and cheaper, and a logistic regression over Jev's five signal questions reaches 95.1%.

anisselbd/jev-phishing-bench on GitHub

Stars
5
Forks
0
Language
Python
Last push
19 Sep 2026

Read from GitHub on . Counts change daily.

How it uses Jev

args = parser.parse_args()

model = env("TYPESAFE_MODEL", "jev-latest")
base_url = env("TYPESAFE_BASE_URL", "https://api.typesafe.ai").rstrip("/")
emails = select_emails(load_emails(), limit=args.limit, sample=args.sample)

if args.dry_run:
    print(json.dumps(build_payload(emails[0], model), indent=2, ensure_ascii=False))
    return

api_key = env("TYPESAFE_API_KEY", required=True)
headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
url = f"{base_url}/v1/systemone"
out = RAW_DIR / f"jev_pass{args.pass_no}.jsonl"
already = done_ids(out)

View in run_jev.py

Open on GitHub

Use casesLLM guardrails

More like this

Use cases this is tagged with