When Simon Willison needed to classify roughly 21,000 Git commits as either maintenance or new development, the obvious approach was to call an LLM for each one. The catch was that at roughly 1.5 seconds per call, the process would take over eight hours of wall time. His solution was a hybrid classifier that uses a statistical model for the easy cases and the LLM only for the ambiguous ones, and it reduced the workload by nearly half while fitting in under 40 lines of Perl.

The Problem with Calling LLMs at Scale

Willison used GPT-5.6 Luna through his llm CLI tool, which can be piped from Perl to read and write responses. The model performed perfectly on a small test set, matching his own labels for every commit. The problem was not accuracy. It was cost and latency at scale. Shelling out to the LLM for each of 21,000 commits meant 1.5 seconds per call, mostly spent on input tokens since the prompt was text-heavy and the response was a single label.

Willison acknowledged that if latency were the only concern, a more direct API call to OpenAI would be faster than routing through the llm CLI and Perl's open2 function. But the deeper question was whether there was a way to avoid calling the LLM at all for the commits that were obviously one category or the other.

The Fast/Slow Classifier Technique

The insight was straightforward. Many commits are easy to classify from their message text alone. A commit titled "fix null pointer in auth module" is clearly maintenance. A commit titled "add email notifications on new direct messages" is clearly new development. Only the ambiguous cases need the LLM's judgment.

To build a fast classifier for these easy cases, Willison chose logistic regression trained on word frequencies from commit messages. The model learns to predict the probability that a given set of words belongs to the "new development" category. When the prediction is very confident, the fast classifier accepts the label and skips the LLM entirely.

In practice, the fast classifier intercepted roughly 46 percent of the classification tasks, effectively halving the number of expensive LLM calls. The thresholds were deliberately conservative: labels were accepted only when the model predicted a probability below 0.08 for maintenance or above 0.92 for new development.

Why Logistic Regression and Not Something More Complex

The choice of logistic regression over a more sophisticated model was deliberate. It can be trained incrementally, one label at a time, using closed-form gradient descent that fits in three to ten lines of code depending on how you count. It requires no advanced mathematics libraries and updates its weights with each new labeled example.

The implementation tracks a set of weights for each word feature, computes log-odds from the weighted sum of features, converts those log-odds to probabilities, and adjusts the weights using gradient descent on each new training example. Regularization pulls all weights toward zero to prevent any single word from dominating the prediction.

Calibration and the Platt Scaler

A practical issue emerged during testing. The logistic regression's probability estimates drifted from true calibration, showing a 0.2 log-odds bias toward the positive label and probabilities that were about 10 percent more extreme than desired. This can happen when regularization is applied or when the training data is still small early in the process.

The fix was Platt scaling, which fits a second logistic regression on top of the first one's raw log-odds output to neutralize the bias. This requires siphoning off about 10 percent of training samples to an independent scaler so that the scaler is not trained on the same data it is evaluating. The combined system of fast classifier and Platt scaler was still instances of the same logistic regression class, with the scaler receiving the raw score rather than the probability.

What the Technique Enables

The approach is not specific to commit classification or even to Perl. The general pattern is reusable: identify a cheap classifier that can handle the easy cases, use an expensive classifier only when needed, and feed the expensive classifier's answers back to train the cheap one. Willison notes that the technique could easily speed up the process by an order of magnitude or more depending on the task and the distribution of easy versus ambiguous cases.

A commenter, Julian Ferrone, pointed to newer models like Jev that are trained to produce decisions alongside confidence scores in a structured schema rather than raw text. These models eliminate the need to instruct an LLM to output only a specific label and can return probabilities directly. But independent benchmarks for these approaches are not yet available, and the logistic regression hybrid remains a practical, implementable solution with existing tools.

The Practical Takeaway

The broader lesson is that for bulk labeling tasks where an LLM is accurate but slow, a two-tier system can substantially reduce cost and time without sacrificing accuracy. The fast classifier does not need to be perfect. It only needs to be confident when it is right and willing to defer to the LLM when it is not. For a project labeling thousands of items, this distinction between easy and hard cases is where the savings live.