Skip to content
System One

Jev vs Laya: SQL Review Benchmark

Benchmarks Jev against Laya on 14 PostgreSQL statements with deliberately planted bugs, scoring which model catches more of the SQL defects.

DDnim/jev-vs-laya on GitHub

Stars
0
Forks
0
Language
Python
License
MIT
Last push
23 Sep 2026

Read from GitHub on . Counts change daily.

How it uses Jev

    key = os.environ["TYPESAFE_API_KEY"]; out = {}
    for c in CASES:
        t0 = time.time()
        r = requests.post("https://api.typesafe.ai/v1/systemone", headers={"Authorization": f"Bearer {key}"},
                          json={"state": state_of(c), "model": "jev-latest", "questions": QUESTIONS}, timeout=60)
        r.raise_for_status(); a = r.json()["answers"]
        out[c["id"]] = {"model": a["model"]["choice"], "model_p": a["model"]["probabilities"], "effort": a["effort"]["score"], "ms": round((time.time()-t0)*1000)}
    return out

def run_laya(ckpt):
    import laya; ag = laya.load("convaiinnovations/laya", subfolder=None if ckpt == "english" else ckpt); out = {}
    for c in CASES:
        t0 = time.time(); a = ag.predict(state_of(c), QUESTIONS)["answers"]
        out[c["id"]] = {"model": a["model"]["choice"], "model_p": a["model"]["probabilities"], "effort": a["effort"]["score"], "ms": round((time.time()-t0)*1000)}
    return out

View in routing_bench.py

Open on GitHub

PrimitivesChoice

More like this

Keep browsing