Personal email agents decide, for every message, what it is, whether it is urgent, whether to draft a reply, create a task, or ignore it. Their error rate is rarely measured: the stored labels were written by the system itself. We present Foyer, a self-hosted personal inbox agent in daily use, and a measurement protocol (stratified sampling, frozen snapshots, thread-grouped splits, pre-registered hypotheses, human labels in the agent's review loop) that yields ground truth before any training. On 739 real Italian/English emails labelled by the owner (231 unbiased, 500 adversarially sampled), we compare nine systems under thread-grouped cross-validation: the production keyword rules, a TF-IDF + logistic-regression baseline, local typed decision models (Tev1, Nimble), a hosted decision model (Nimble v4-max), and hosted generative LLMs (Qwen2.5-7B, Gemma 4 26B/31B). Four results. The production rules score 28% on message type against a 58% majority class: the field shown most prominently carried no information. With ~200 labels, logistic regression has the highest accuracy on every field (type 0.79, urgency 0.74, needs-reply 0.83, is-noise 0.90), though it never predicts the two rarest classes and is statistically tied with Gemma 4 on type; few-shot prompting does not close the gap. The sender's relationship (client, supplier, service) is recovered from other emails of the same contact with 98.6% accuracy where the contact has been seen, and is not inferable from a single message by any model: it belongs in the contact graph, not the prompt. Finally, accuracy is not the product metric: composed into actions, the most accurate system most often ignores emails needing action (18% vs 4% for the best LLM); a cost-chosen threshold on its noise probability halves that rate and makes it the cheapest system.
Measuring before training: a protocol for ground truth
Most personal email agents operate with labels that the system writes about its own behavior. This creates a circularity: error rates are inflated or miscalibrated because no independent ground truth exists. Foyer, a self-hosted personal inbox agent in daily use, changes this dynamic by introducing a measurement protocol designed to produce reliable labels before any training step. The protocol combines stratified sampling of the message stream, frozen snapshots of the agent state at known points, thread-grouped splits that preserve conversation coherence, pre-registered hypotheses about which features predict each decision field, and human labels fed back into the agent's review loop. Together, these steps ensure that the dataset reflecting true agent performance is available before a model is trained on it.
The evaluation uses 739 real Italian/English emails, of which 231 were selected without bias from the natural inbox flow and 500 were adversarially sampled to test edge cases and rare categories. Each message received a human label for four decision fields: message type, urgency, whether a reply is needed, and whether the message is noise. These labels form the reference standard for all subsequent comparisons.
Nine systems compared under thread-grouped cross-validation
Nine systems were evaluated on the same labeled dataset, using thread-grouped cross-validation so that messages from the same conversation thread are kept together either in training or testing, never split across both. The systems fall into three categories: hand-crafted rules, classical machine-learning baselines, and generative LLMs.
The first system is the production keyword rules that currently power the inbox. These rules rely on pattern matching against subject lines, sender prefixes, and body heuristics. Across the test set, they score 28% accuracy on message type against a majority class that occupies 58% of messages. The most prominent field carried effectively no information beyond what the majority class already provides, indicating that the existing rule set is thin where it matters most.
A TF-IDF + logistic-regression baseline was trained on a subset of the labeled messages. With approximately 200 labels — roughly a quarter of the full dataset — this system achieves the highest accuracy on every decision field: 0.79 on message type, 0.74 on urgency, 0.83 on needs-reply, and 0.90 on is-noise. A notable limitation is that the two rarest classes in each field are never predicted by logistic regression, regardless of training set size. Its performance on message type is statistically tied with Gemma 4 26B/31B, and few-shot prompting of generative models does not close the gap.
Two local typed decision models — Tev1 and Nimble — were also evaluated. These models encode domain-specific constraints about which values are valid for each field (e.g., message type must be one of a fixed set). Their accuracy is competitive with the logistic-regression baseline on most fields, but they share the same failure mode of omitting the rarest classes.
A hosted decision model, Nimble v4-max, was run via API. Its accuracy profiles are broadly similar to the local models, with slight gains on urgency classification but comparable performance on the other fields.
Hosted generative LLMs include Qwen2.5-7B and two sizes of Gemma 4 (26B and 31B). These models were prompted in few-shot settings to perform the same four classification tasks. Across the board, their accuracy is lower than the logistic-regression baseline, and the gap does not narrow even when model size is increased from 26B to 31B parameters. The systems also exhibit inconsistent behavior on the two rarest classes, never assigning a positive prediction where the logistic model also abstains.
Sender relationship: a contact-graph problem, not a prompt problem
One question the evaluation addressed is whether the sender's relationship — classified as client, supplier, or service — can be inferred from a single message. The answer, across all nine systems, is no. The relationship is recovered from other emails of the same contact with 98.6% accuracy when that contact has been seen before. When the contact is novel, no model — not even the largest LLM — can infer the relationship from a single message. This finding suggests that sender context is best maintained in a contact graph outside the per-message prompting or classification head.
Accuracy is not the product metric
The final and perhaps most practical result concerns how the systems behave when their decisions are composed into actions. The most accurate system overall — the logistic-regression baseline — most often ignores emails that need action, doing so 18% of the time. The best LLM, by contrast, ignores needing-action emails only 4% of the time. This inverse relationship means that higher per-field accuracy does not translate to higher action coverage.
To address this, the authors introduce a cost-chosen threshold on the logistic model's noise-probability output. By raising the threshold so that the system is more willing to classify a message as needing reply or task, the ignore rate drops to 4%, matching the LLM's rate. At this threshold, the system also becomes the cheapest to run, because fewer expensive LLM calls are needed and the overall pipeline processes more messages per unit time.
Limitations and practical considerations
The evaluation is confined to a single personal inbox in Italian and English; transfer to other languages or email providers is not tested. The 739-message sample, while larger than typical internal studies, represents a snapshot of activity over a limited time window. The nine systems compared are not an exhaustive survey of all possible classifiers or LLM prompting strategies, but they cover the main approaches in use today. The measurement protocol — while rigorous — requires human labeling in the review loop, which may not scale to very high message volumes without automation of the labeling step itself.
Use Measure Before You Train when building or evaluating personal email agents
If you are developing or comparing personal inbox agents, the protocol introduced here — stratified sampling, frozen snapshots, thread-grouped splits, pre-registered hypotheses, human review — gives you ground truth before you train. Logistic regression with ~200 labels achieves the strongest per-field accuracy and, with a noise-probability threshold, the best action coverage at lowest cost. Sender relationships belong in a contact graph, not in per-message prompts. Few-shot LLM prompting does not surpass classical baselines on this task, and the rarest classes remain a blind spot across all systems. Use the measurement protocol to assess any new agent against the same standard, and treat accuracy alone as a product metric only after composing decisions into actions.