A developer has released an open-source tool that addresses a specific but increasingly common problem: how to train a small, fast, local text classification model using an LLM as a teacher rather than collecting thousands of human labels. The project, called shrewd, grew out of an experiment in prompt optimization and evolved into a general-purpose pipeline for distilling LLM judgments into models that run entirely offline.
The Core Idea
The premise is straightforward. A user provides a few hundred hand-labeled examples along with a larger pool of unlabeled text. An LLM acts as a teacher, labeling the unlabeled pool. Those LLM-generated labels then train a small student model that runs locally with no LLM involved in the loop. The result is a lightweight classifier that can operate on a laptop or phone without any internet connection, inference cost, or latency dependency on a remote API.
Two paths are available. The first produces a basic classifier that assigns one label per document, useful when the question is simply "which of these N buckets does this go in." The second builds a decision panel: a fixed set of typed questions, each with choices, yes/no probabilities, or ratings on an ordered scale, all answered in a single pass.
What the Experiments Revealed
The project documents several findings that are valuable even for those who never use the library itself. Prompt optimization using the GEPA framework improved teacher outputs on the development split, but those gains frequently disappeared when tested on held-out data. This suggests that prompt tuning can overfit to the training distribution in ways that do not generalize.
More counterintuitive was the discovery that better teacher labels did not always produce a better student. In some cases, changing the student architecture or giving it more training data yielded more improvement than investing in higher-quality teacher outputs. The author notes that calibration, while helpful for making probabilities match human judgments, sometimes masked a model that could barely distinguish between examples.
On the cost side, the experiments showed that selecting informative rows from a good teacher was more effective than buying cheaper labels or escalating uncertain ones. The pipeline can acquire labels in rounds, picking the rows a quick baseline probe is least confident about, which reduced the number of teacher calls needed by 1.1 to 2.3 times for the same student accuracy.
Prebuilt Decision Panels
To demonstrate the approach, the project ships four prebuilt decision panels built from public data using Fable 5.1 as the teacher. These include SMS spam and scam triage, email spam and phishing detection, an AI-input guardrail for on-device AI features, and a personal-data gate that determines whether text is safe to send off-device.
The panels are not production models to adopt as-is. They serve as examples of what this pipeline can produce. Each has documented limitations. The SMS panel, for instance, ships a fine-tuned encoder model even though a tf-idf student tied it on the labeled test split, because on newer scam patterns the encoder performed substantially better. The personal-data panel has a known gap: it was trained on card numbers written as unbroken 16-digit runs, so spaced or dashed formats score far lower unless those variants are added to the training pool.
Practical Considerations
The project is clear about where this approach fits and where it does not. It is a poor fit for tasks that require reasoning or fresh world knowledge per item, for domains that change quickly, or for workloads where the volume is too low to justify building and maintaining a student model. The saved model answers only the task it was trained for; adding a new question or changing an existing one invalidates the cached answers.
The privacy model is a key selling point. The prompt goes out to the provider as a system message marked for the prompt cache, and every response is cached locally so re-running a stage never costs twice. The final model runs entirely on the user's hardware. However, because the seed and pool text are sent to the provider during training, users with sensitive data are advised to review the provider's data-use terms first.
The broader significance of the project lies in its empirical contribution. By documenting where prompt optimization succeeds and where it fails, where better teachers help and where they do not, the project provides a practical counterweight to the assumption that throwing more LLM capacity at a labeling problem always yields a better outcome. Sometimes, the cheaper path is not the one you would expect.