vercel-labs/ai-cli

Vercel Labs’ ai CLI wraps the Vercel AI SDK for terminal use: generating text, images, video, and audio through one set of commands. Its evaluate subcommand is separate from generation: it takes an input piped through stdin, plus one or more typed questions, and returns a judgment for each.

Three question types are supported. Boolean returns a probability that the statement is true, including confident no answers near zero. Choice returns one of a supplied set of named options along with the model’s distribution across all of them when available. Score returns a fractional position on an ordered list of levels, which Jev computes as a probability-weighted mean. Jev is the default model behind all three, set via typesafe-ai/jev on Vercel’s AI Gateway, with -m jev as shorthand to select it explicitly.

Questions can be passed inline as flags or as a JSON file with fuller instructions and criteria per question, and several questions can run against the same input in one call. The CLI’s own documentation is direct about where this breaks down: Jev is not a reliable calculator, so exact arithmetic, counting, and date comparisons need to happen in code around it, and a reference date has to be supplied explicitly for any question about “today” since the CLI doesn’t add one. Output is always JSON on stdout.

Open on GitHub