lkarlslund/laya.cpp on GitHub
- Stars
- 85
- Forks
- 8
- Language
- C++
- License
- MIT
- Last push
- 23 Sep 2026
Read from GitHub on . Counts change daily.
Reported by the author
- CUDA BF16 speedup
- 1.11-2.71x
1.11–2.71×
github.com
laya.cpp is a standalone C++ implementation for running Laya, an open-weight typed-decision model, without a Python runtime: tokenization, inference and JSON formatting all happen in native code. It supports CUDA and Vulkan on Windows and Linux, and Apple’s Core ML on macOS arm64, across Laya’s english, multilingual and typed-decisions variants, and it ships an HTTP server compatible with Jev’s /v1/systemone endpoint, with automatic request batching.
The author benchmarked it against matching-precision Python across all three models, at batch sizes 1, 2, 4 and 8, using 250 fixed questions per comparison. Results vary a lot by backend: optimized CUDA in FP32 ran 1.35x to 2.48x Python’s throughput, and CUDA in BF16 ran 1.11x to 2.71x. Vulkan results were more mixed, from a slowdown (0.44x on NVIDIA with plain FP32) up to roughly 2.45x on AMD’s compensated FP32 path, depending on the GPU and precision mode. Every measured answer matched Python’s output within a small numeric tolerance (0.0001).
Building it from source needs a C++20 compiler, CMake 3.24 or newer, ICU, nlohmann-json, and whichever GPU backend’s own dependencies you’re targeting; binary releases cover Windows and Linux x64 with CUDA or Vulkan, plus a macOS arm64 Core ML build.
How it uses Jev
print(f'{variant}: {question_count} questions, HTTP batch {size}: exact CLI parity', flush=True)
canonical = 'laya' if variant == 'english' else f'laya-{variant}'
def jev(index):
request = {**cases[index], 'model': 'jev-latest'}
started = time.perf_counter()
got, batch_id, offset = exchange(port, '/v1/systemone', request, metadata=True)
return batch_id, offset, request, got, (time.perf_counter() - started) * 1000
for workers in (1, 2, 4, 8):
started = time.perf_counter()
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
observations = list(pool.map(jev, range(len(cases))))
elapsed = time.perf_counter() - started
latencies = sorted(item[4] for item in observations)
