If LLMs Can Decide Without Fine-Tuning, Do We Still Need Models Like Jev?

TypeSafe’s Jev raises a question: can base models already decide?

TypeSafe released Jev in September. It takes context and a question and returns a choice, score, or probability directly. Decisions are treated as a first-class task with calibrated probabilities and efficient batching over shared context. The design makes sense for workflows that need fast routing without generating full responses.

While learning how Jev works I asked whether this capability actually requires fine-tuning. A language model already assigns probabilities to text continuations given context. If the options are expressed in words with meaning, the model should already favor some answers over others after reading the question and choices. A well-trained model might distinguish the better option using its learned distribution alone. Could we read that preference from the logits and choose without another round of training?

I also suspected that specialist models like Jeff and Laya might overfit their fine-tuning tasks. Doing well on familiar benchmarks is one thing. What happens when I give them my own questions?

Reading logits directly from Qwen and Gemma

I tested existing Qwen and Gemma models without any decision fine-tuning or new classification head. The method is simple: feed the model the context, question, and available answers. Then take the logits for the answer labels at the final input position. Use those scores to choose without generating the answer token by token.

This still requires a full forward pass. Input prefixes can be reused where caching allows. It skips the subsequent text generation. These scores let me compare answers but cannot be interpreted directly as the probability that a decision is correct.

I built a set of 300 questions I cared about rather than reusing the specialists’ training sets. The goal was to see whether base model preferences could support useful decisions on my tasks and how much of the specialists’ advantage would transfer.

Results on custom questions

Tests ran on an M2 Max with 64 GiB of memory. Accuracy uses the 300 fixed inputs. Timings cover valid warm calls across repeated runs, excluding model loading and context preparation. Model sizes, quantization, and input formats differ so this is not a controlled comparison of fine-tuning effects.

Inputs rejected by Laya English due to length still count toward its accuracy denominator. All CLM correct results came from fallback decisions.

Gemma’s direct scoring reached 75% accuracy, close to the highest score in this test. It had not solved every problem but was enough to consider using the model I already had. Small specialists can be very fast but useful judgment does not necessarily require additional training.

Task overfitting shows up on the public benchmark

For reference I also tested the public Typed Decisions dataset. Laya Typed Decisions scored about 77% there versus 51% on my questions. That gap strengthened the overfitting suspicion. Tasks, languages, and input conditions all changed together so this comparison alone cannot prove overfitting is the cause. The question I cared about was whether performance on familiar tasks carries over to the tasks I actually need.

Several models could select an answer but struggled to recognize when they should not make a direct choice. My two general-purpose baselines had this problem too. That matters more than comparing percentage points: a workflow needs more than a model that can pick an answer.

Direct scoring versus full generation on the same weights

I compared direct scoring with a full generation pipeline using the same Gemma weights. This was a separate offline test with 53 inputs run twice, so its results should not be combined with the first table.

The full pipeline includes parsing, validation, and repair when needed. Runtimes and caching differ between the two paths. Context preparation for direct scoring takes about five seconds on a cache miss and is timed separately.

The same model’s ability can be accessed in different ways. Getting a judgment need not involve the full generation pipeline. Direct scoring still has limited scope. In this test it did not handle requests that could not be expressed through the available options. The full pipeline was not reliable on those either. The comparison gives a reason to keep a constrained decision path but does not show it can replace all generation capabilities.

When a single model handles both decisions and replies

If tasks are stable and the workflow only needs frequent decisions, a small fast specialist is appealing. I am more interested in a local workflow that already needs a model to write replies.

An incoming sentence might call for a decision or a written answer. The system must distinguish between them before choosing a path. Splitting that work between two models brings routing, switching, fallback, and maintenance overhead. If both models run locally, whether they both need to stay in memory matters too.

If the existing model has to run anyway and can already make useful judgments, keeping decisions and replies in that model may be a better fit even if individual decisions take longer. Jev itself has a design for using decisions to route requests. The question is how much that extra layer benefits a particular workflow once its costs are included. A single model also needs a way to distinguish when to decide and when to reply; using one model does not automatically solve routing.

These experiments have not compared complete one-model versus two-model workflows but they have informed my current choice. I have not added a separate decision model. I changed how I call the existing model: when a choice is needed I read the answer-label logits directly, allowing it to return a decision too.

Independent confirmation arrives

During the time I was writing this post a paper titled LLM-as-Jev: LLMs Are Already Jev-Style Decision Models—When and How to Fine-Tune Them was released. One of its main findings matches what I observed: capable existing language models can make useful decisions directly without additional fine-tuning.