llama.cpp Serves Typed Decision Calls Locally With a System One Endpoint

On October 2, 2026, the llama.cpp maintainers documented native decision-model support in the local server: a /v1/systemone endpoint that accepts a state plus typed questions and returns option probabilities in one forward pass, matching the System One request shape popularized by TypeSafe’s Jev API. The Hugging Face blog from the ggml-org account positions the change as infrastructure, not a new model family—builders already collecting open decision checkpoints can point existing clients at a local base URL instead of a hosted classifier.

Why typed questions beat free-form gates

Decision models occupy a different lane from chat LLMs. A generative model spends a forward pass per output token and still needs parsing; a decision model reads the state once and answers only from the schema you supply, with calibrated probabilities attached. Typical agent jobs—route this ticket, gate this tool call, score urgency, confirm a step succeeded—fit that pattern better than free-form prose. llama.cpp’s blog walks three question types: choice (pick among labeled options), score (ordered levels, expected value can fall between ticks), and noul (yes/no probability). Questions batch independently while heavier models can prefetch the state once, which keeps hot-path latency down when an agent asks several gates together.

The supported catalog in the post spans sizes and licenses. Julia-1 (~144M, mmBERT-small, 50+ languages, Apache 2.0) and Laya (~421M, ModernBERT-large, English, Apache 2.0) target millisecond-class text decisions. Kev-4B and lev (~4B on Qwen3.5 bases, Apache 2.0) trade latency for richer English understanding. OpenJev (~27B on Qwen3.8-27B, CC BY-NC 4.0) adds vision so screenshots or document images can enter the state. Median times on a single NVIDIA RTX PRO 6000 are listed from about 3 ms (Julia-1) to about 43 ms (OpenJev) for one question—company-bench numbers that will shift with quantization and hardware. Implementation details point at PR #29818; operators pull current builds via llama.app / llama update and start weights with llama serve -hf ….

Model catalog from Julia-1 to OpenJev

Operational tips in the same post matter as much as the endpoint. Bare option labels confuse small models; short criteria descriptions flip bad routes (the blog’s billing-versus-shipping example). Confidence cutoffs are model-specific—a vague ticket may score 0.25 on Julia-1 and 0.80 on Kev-4B—so teams should calibrate on their own traces before auto-acting. Quantized GGUFs remain first-class (:Q8_0 and friends). Router mode can host several decision models and select per request, with /v1/models listing ids. Vision follows the chat image_url pattern: data URLs in the state or an images array, with projectors downloading automatically for vision-capable checkpoints.

This release sits beside—not on top of—the October 1 wave of new decision weights such as Cloudflare Clef and Strands Decider. Those launches expand the open catalog; llama.cpp’s contribution is making System One a local commodity API so agent runtimes do not need a cloud gate for every typed choice. Cloudflare’s Clef is explicitly called out as next on the integration list. Non-commercial OpenJev remains a license footgun for product teams; Apache-licensed Julia/Kev/lev paths are the safer default for redistribution.

Calibrating confidence on your own traces

Builders wiring agents should keep the hybrid pattern: decision models for bounded gates, generative models for language and multi-step planning.

From an integration standpoint, the curl and Python examples in the blog are enough to spike a gate: POST JSON with state and a questions map, read answers.*.probabilities or choice/noul/score, and branch in application code. Image classification follows the same schema once a vision-capable GGUF is loaded—encode a PNG as a data URL, ask a choice question about document type, and skip OCR when the decision is the only output you need. Teams already running llama.cpp in router mode can add decision GGUFs beside chat models without standing up a second inference stack.

License hygiene belongs in the same checklist as latency. Apache 2.0 Julia, Laya, Kev, and lev are the straightforward self-host path; OpenJev’s CC BY-NC 4.0 tag blocks many commercial deployments even if the vision quality is attractive. Cloudflare’s forthcoming Clef GGUF path—and any other Apache decision weights that land in the Decision models collection—will matter more for product teams than non-commercial 27B checkpoints. Until those land, the practical pattern is small Apache deciders on the hot path and a separate hosted or local chat model for language.

For Writeble’s opensource desk, /v1/systemone is the October 2 infrastructure punchline: typed agent decisions can now stay on-device inside the same llama.cpp server fleets teams already trust for GGUF chat.