Producing high-quality alignment data for large language models remains one of the most expensive bottlenecks in the AI pipeline. The standard approach, manual post-editing, requires annotators to read an entire model-generated response, identify every error, and rewrite the text from scratch. This is slow, cognitively taxing, and produces off-policy data that diverges significantly from the model's own sampling distribution, making it less effective for on-policy supervised fine-tuning. A new tool called onPanda, developed by Lei Yang and colleagues at StepFun and Xiamen University, addresses this problem with a fundamentally different interaction paradigm: instead of rewriting entire responses, annotators correct individual tokens at the point of first error and let the model continue from there.
The locate-correct-continue paradigm
onPanda's core interaction is elegantly simple. While an annotator reads a model-generated response, the system color-codes each token by its generation probability, with greener tokens indicating higher confidence and redder tokens indicating lower confidence. The annotator's job is to locate the first inappropriate token, not to evaluate the entire response. When identified, the annotator either clicks a substitute from the model's top-20 candidate tokens at that position or double-clicks the token and types the correct text via free-form editing.
After the correction, the system truncates everything after that position and continues generation from the corrected prefix. The annotator no longer needs to re-read the entire response to find the next error; the model itself continues generating from the fixed point, and the annotator simply repeats the locate-correct-continue loop along the response until it meets the quality bar. This linear workflow eliminates the repeated review cycles that dominate manual post-editing time.
The key insight is that most corrections are mouse clicks, with brief typing only when the model's candidate set misses the correct direction. In production deployments across thousands of annotation sessions, 97.0% of tokens in the final response are model-generated, 2.1% are selected from candidate tokens, and only 0.9% are manually typed by annotators. Human intervention is sparse and precisely targeted.
Why this preserves on-policy fidelity
The distinction between onPanda and manual post-editing is most apparent in the data quality metrics. Because human touches only a few erroneous positions while the vast majority of tokens are generated by the model itself, either sampled directly from the prompt or continued from a corrected prefix, the resulting SFT data stays close to the rollout model's sampling distribution. In the paper's controlled study, onPanda achieves a perplexity of 1.181 under the rollout model, a relative change of just +0.86% from the baseline of 1.171. By comparison, manual post-editing via POTATO yields a perplexity of 1.596, a 36.31% increase, reflecting the substantial distributional shift introduced by rewriting entire responses from scratch.
This on-policy fidelity matters because post-training on data that closely matches the model's own generation distribution produces more stable and effective updates. Off-policy data from manual editing can introduce exposure bias, where the model learns to generate text that looks nothing like what it actually samples during inference.
Fine-grained supervision as a byproduct
Beyond efficient annotation, onPanda automatically records every token-level correction as structured data. Each correction captures the error position, the substitute token, and the samples before and after the correction. These corrections naturally form paired positive-negative samples: the negative is the ancestor response before the correction, and the positive is the corrected version. The two share the prefix before the correction point and diverge exactly there, so the corresponding token-level supervision is precise in both position and correction direction.
These correction triples can be converted into preference data for reward model or DPO training, or into process reward data for PRM training, based on the step containing the corrected token and its preceding steps. Because positive and negative tokens pair one-to-one at the same position, optimization receives naturally balanced signals, unlike response-level preferences where the comparison is coarse and position-agnostic.
Empirical results: 52% faster, same quality
A controlled study with three annotators labeling 21 image-description prompts compared onPanda against two mainstream paradigms: manual post-editing via POTATO and preference ranking via Argilla. The results are compelling. onPanda achieves a median annotation time of 330 seconds per prompt, 51.5% less than POTATO's 681 seconds and comparable to Argilla's 336 seconds. The efficiency gain comes from two sources: continuation eliminates repeated review of the entire response, and the probability shading helps annotators prioritize low-confidence tokens where errors are most likely.
Crucially, the speed does not come at the cost of quality. onPanda achieves the highest pairwise win rate at 66.7% when judged by GPT-5.5 against the other two methods. An anonymized human comparison further supports this: human evaluators preferred onPanda outputs over POTATO in 54.8% of pairs. All three methods achieve 100% SFT coverage by design, since editing continues until the response qualifies.
The NASA-TLX workload questionnaire confirms the human experience: onPanda scores 3.1 out of 10, compared with Argilla's 5.4 and POTATO's 6.8, with annotators reporting that immediate feedback and probability visualization reduce cognitive load significantly.
Agentic and multimodal support
onPanda extends beyond plain text to structured agent responses and multimodal content. The system uses a response template mechanism that converts bidirectionally between structured messages and the model's native token stream, making reasoning chains, content, and tool-call arguments all directly visible and correctable at the token level. Special tokens such as thinking delimiters and tool-call markers are rendered in the UI and can be corrected just like any other token.
For agent settings, onPanda connects to external environments and tools through the Model Context Protocol (MCP), with adapters wrapping existing harnesses such as Claude Code, Codex, and OpenClaw. Tool calls can be configured to await annotator approval before execution: a problematic call can be executed after its arguments are corrected, or rejected with textual guidance, with rejected trajectories automatically retained as negative samples. This enables interactive trajectory annotation in realistic environments where corrections actually execute tools and obtain real feedback before generation continues.
Panda-CVL: a dataset and benchmark for token-level correction
To support research on this new type of supervision signal, the paper releases Panda-CVL, a Chinese vision-language dataset annotated with onPanda, along with a benchmark for evaluating models' token-level correction capability. The dataset comprises 7,491 annotation sessions stored in the .panda.json format, with 6,839 in the training set and 652 in the test set, filtered from production data to include only publicly licensed images and tasks that evaluate general-purpose capabilities such as visual perception and reasoning.
The benchmark decomposes token-level correction into three subtasks: determine whether the response is acceptable; if not, locate the first inappropriate token; and correct it to an appropriate token, either by selecting from candidates or via free-form editing. Models are evaluated on a find-and-replace format and measured by format compliance, good-answer accuracy, error localization accuracy, correction accuracy, and their harmonic mean F1 score.
The results reveal that token-level correction remains a challenging task even for state-of-the-art models. The best-performing model, GPT-5.5, achieves an F1 of only 17.09%, with a good-answer accuracy of 53.37% and correction accuracy of 10.18%. GPT-6 achieves the highest error localization at 24.46%, but its overall F1 is 16.57%. Seven of nine evaluated reasoning models achieve format scores above 90%, yet correction accuracy remains below 16% for all of them, suggesting that reliable format compliance does not translate into accurate error localization and correction.
Limitations and open questions
OnPanda's approach requires an inference API that supports continuation from a prefix and returns top-k candidate tokens with probabilities, which limits deployment to systems with these capabilities. The free-form editing fallback, while powerful, introduces tokens that deviate from the model's own distribution; these occur only at positions beyond the model's capability, but their presence means the data is not perfectly on-policy.
The benchmark results also highlight that token-level correction is far from solved. With the best F1 at 17.09%, the task remains substantially more challenging than standard generation or classification benchmarks. This opens significant room for research into better correction models and training methods.
What this means in practice
For organizations building and tuning LLMs, onPanda offers a concrete answer to a persistent question: how can annotation scale without sacrificing data quality or on-policy fidelity? The 52% reduction in median annotation time, combined with perplexity-preserving data, makes it a practical tool for production annotation pipelines. The 97% model-generated token ratio at production scale suggests that human annotators serve as precise surgical instruments rather than wholesale editors.
For researchers, the token-level correction data and the Panda-CVL benchmark open a new direction in supervision signals. Rather than treating alignment as a response-level classification problem, token-level correction treats it as a sequence of precise, position-targeted edits. This finer-grained supervision could prove more sample-efficient for post-training, though the low benchmark scores suggest substantial work remains before models can reliably perform this task autonomously.
The onPanda Python package is available on PyPI, and the code, dataset, and benchmark are released under open-source licenses, making it straightforward for practitioners to integrate token-level correction into their annotation workflows and for researchers to explore this new supervision paradigm.