A new open-source tool is challenging the way developers extract structured decisions from large language models, replacing text generation with a single logit readout that is dramatically faster and produces calibrated probability distributions instead of brittle point estimates.

The Bottleneck in Getting Structured Answers From LLMs

Most workflows that require an LLM to return structured data rely on generating text tokens until the model reaches an end-of-sequence marker, then parsing the resulting JSON. This approach is slow, token-hungry, and prone to hallucinating fields that do not belong in the schema. For applications that need to score dozens or thousands of items against a fixed set of options, the latency compounds quickly.

Existing browser-based tools like openjev.com demonstrated that a different approach was possible, reading the model's next-token logit distribution at the boundary position after the prompt rather than waiting for generated text. jev-serve, published by developer rreinold on GitHub, brings that same technique to the server side, making it available for any MLX model or OpenAI-compatible API.

How Logit Readout Replaces Text Generation

The core insight behind jev-serve is straightforward. Instead of asking the model to produce a structured answer as text, the tool reads the probability distribution over the vocabulary at the very first token position after the prompt is fed in. For each option in a question schema, it identifies the corresponding first token and applies softmax normalization to obtain a calibrated probability for each choice.

One forward pass per question. No sampling loop, no JSON parsing, no generated tokens, and no possibility of the model inventing fields outside the defined schema. The output is locked to the schema the developer provides, which eliminates an entire class of hallucination errors that plague structured generation pipelines.

The tool exposes a single REST endpoint at /v1/systemone and supports three question types.

  • choice -- select one option from a set, returning the choice plus a probability for each option
  • noul -- return a probability value between 0 and 1 for a yes/no question
  • score -- return an expected ordinal level along with a full probability distribution across levels

Performance Benchmarks and Practical Speed

The performance difference between logit readout and traditional structured generation is substantial. In a benchmark running 20 products through the same four-option choice question sequentially without batching, jev-serve averaged 0.23 seconds per decision while a conventional JSON decoder averaged 7.80 seconds. That is a 34-fold speedup.

The gap widens further with larger option counts and longer expected outputs. At the 95th percentile, jev-serve latency sits around 0.31 seconds compared to roughly 11.2 seconds for the decoder approach. The decoder generates roughly 23 output tokens per question; the logit method generates zero.

Scaled to 10,000 products, the difference becomes a matter of hours versus days: 0.6 hours with jev-serve against 21.7 hours with structured output generation.

Deployment and Backend Options

jev-serve ships with two backend options. The MLX backend runs direct inference on Apple Silicon hardware, and the API backend accepts any OpenAI-compatible endpoint, including local servers like ollama and LM Studio as well as cloud providers.

Installation uses the uv toolchain, with separate feature flags for each backend:

uv pip install -e ".[mlx]"
uv pip install -e ".[api]"

A typical invocation against a local MLX model looks like this:

jev-serve --mlx lmstudio-community/Qwen3.8-27B-MLX-6bit

And against an API backend:

jev-serve --api http://localhost:11434/v1 --api-model gemma4:e4b-mlx

From there, a POST to http://localhost:8008/v1/systemone accepts a state description and a questions object, returning a structured response with answers, confidence values, and full probability distributions.

What This Means for Agent and Workflow Design

For teams building agents or decision pipelines that repeatedly query models with the same question schema, the elimination of text generation removes the single largest latency bottleneck. The approach also provides something that structured JSON generation rarely does: honest uncertainty. A model that returns a 0.51 confidence on a choice is signaling genuine indecision, whereas a generated JSON answer offers no visibility into how confident the model actually was.

The project is licensed under Apache 2.0 and is derived from Jared Palmer's kev project. While the current release targets MLX and OpenAI-compatible APIs, the architecture suggests that any model with accessible logits could plug into the same runtime.