Homomorphic Advantage Operator: Stabilizing Reinforcement Learning Under Fully Homomorphic Encryption Constraints

Privacy-preserving machine learning presents significant deployment challenges on the cloud for intelligent systems with confidential data. Fully Homomorphic Encryption (FHE) offers a compelling solution for secure computation, preserving data confidentiality of cloud computations. However, applying FHE to reinforcement learning (RL) requires replacing non-linear operations with polynomial approximations, which diverge catastrophically due to a unique recursive error phenomenon known as the Bellman drift. This article introduces the Homomorphic Advantage Operator (HAO), a stabilization framework designed to prevent polynomial approximation divergence in FHE-based deep RL. HAO adapts the zero-mean centering projection from advantage-based value estimation directly to temporal-difference (TD) targets. This linear projection annihilates the uniform state-value baseline that drives the Bellman drift, maintaining per-state action rankings while requiring zero additional non-linear multiplicative depth and avoiding expensive ciphertext bootstrapping. The proposed HAO framework was evaluated using a three-tier experimental methodology, including a tabular Markov Decision Process (MDP), an encrypted CartPole environment using real CKKS cryptographic operations, and a 20-node logistics routing benchmark with dense continuous features. The results demonstrate that the proposed HAO strictly bounds network pre-activations within the safe polynomial approximation domain. The proposed HAO RL agents achieved 0% boundary breaches across all random seeds used, whereas regularization alone (L2 weight decay and gradient clipping) breached the bound on 3 of 5 seeds and the unstabilized baseline did so in 83.8% of episodes. Finally, HAO agents improve optimal policy accuracy by 18.0 percentage points in tabular domains and remain stable when DP-SGD-style Gaussian noise is added to the clipped gradients.

Jev-style decision models and community benchmarks

Jev formalizes decision tasks as categorical distributions over discrete candidate sets across three canonical formats: choice (selecting one of K options), noul (binary verification, yes vs. no), and score (ordinal grading). JevBench benchmarks such decision models across intelligence, calibration, inference latency, and computational cost. The benchmark provides 231 publicly released items (48 easy, 72 standard, and 111 hard), while held-out and sealed test suites remain private. In this study, the authors evaluate exclusively on the public items and therefore report diagnostic accuracies rather than official leaderboard standings.

A central design choice among community Jev-style models is whether to update model parameters. Many approaches fine-tune the backbone, frequently introducing custom classification heads or LoRA adapters trained on synthetic questions from teacher models; others evaluate frozen LLMs directly. The resulting performances vary substantially. For instance, Plumb fine-tunes Qwen3.5-4B with LoRA over several iterative rounds of teacher-generated queries and hard-negative mining, achieving 89.6% accuracy on public JevBench items. In contrast, Hopper and reflex, both similarly adapted from Qwen3.5-4B, yield performance comparable to SemIf, which simply extracts option-letter logits from the unmodified frozen model (82.3% and 79.2% vs. 81.0%). This stark contrast raises two fundamental questions: to what extent do pre-trained LLMs inherently possess decision-making capabilities, and under what conditions does fine-tuning yield practical gains?

Architecture-preserving framework for extracting calibrated decisions

LLM2Jev introduces a unified interface that turns existing causal LLMs into Jev-style decision models without altering their architectures, tokenizers, or vocabularies. In contrast to many community Jev-style models that introduce custom classification or pointer heads or reserved tokens that must be retrained for every new backbone, the architecture-preserving design provides three distinct benefits. First, zero architectural overhead per backbone: the same prompting convention and parallel scoring procedure apply out of the box to any causal LLM, allowing newly released models to be evaluated immediately and fine-tuned only when targeted adaptation is required. Second, direct inheritance of base model scaling: as foundation models advance in general reasoning and factual knowledge, their decision-making accuracy improves correspondingly without necessitating new decision-specific training corpora. Third, native multimodal support: multimodal LLMs equipped with vision encoders can perform image classification and visual question answering through the identical interface without multimodal fine-tuning.

LLM2Jev introduces two recipes unified under a single interface. The training-free recipe formats candidate options with bracketed numeric indices [1]…[K], conditions generation on the assistant prefix Best answer: [, and computes joint probabilities over candidate continuation tokens; it requires no labeled supervision, additional parameters, or vocabulary expansions. The fine-tuning recipe operates through the identical interface, updating either full model weights or LoRA adapters: a tree-factorized listwise loss trains the model to discriminate among valid candidates, while three auxiliary KL-divergence penalties (anchors) prevent probability drift across other generation contexts. Under either configuration, the model remains a fully functional causal LLM capable of standard text generation.

Joint candidate scoring accumulates log-probabilities across suffix tokens for each candidate, followed by softmax normalization over all candidate scores. Because candidate tokens are scored after conditioning on the complete prompt, the model contextualizes each alternative relative to the full candidate slate. A known trade-off of joint scoring is susceptibility to option order, reflecting the well-documented position bias in LLMs. This trade-off is mitigated during fine-tuning by randomly shuffling candidate order across training instances, ensuring that numeric identifiers carry no static semantic association.

Because all candidate suffixes are known a priori, they can be evaluated concurrently rather than decoded sequentially. After computing the key-value activations for the prompt prefix, they are cached and evaluated candidate branches in parallel via teacher forcing, mirroring the parallel sampling strategy in Jev. Suffixes of identical length can be batched together. With sufficient batch capacity, even a 151-way classification task requires only one prompt prefill and at most three short forward passes under digit-level tokenizers, or a single forward pass under multi-digit tokenizers.

Zero-shot decision competence and when fine-tuning helps

Evaluating both recipes on Qwen3.5-4B (with thinking mode disabled) and Qwen3-0.6B across six data compositions alongside ablations over anchor strengths and parameter-efficient tuning, the authors find that modern LLMs are inherently effective decision models. Without training, the 4B model matches competitive community Jev-style models built on the same backbone, outperforms letter-logit extraction, naturally scales to arbitrary candidate counts, and exhibits strong probabilistic calibration. Furthermore, multimodal visual decision-making emerges zero-shot through the same interface.

Frozen Qwen3.5-4B resolves 188 of 231 public JevBench items (81.4%). This matches or exceeds dedicated community Jev-style models (SemIf 186, reflex 184). The training-free LLM2Jev recipe outperforms simple option-letter extraction from the same backbone. Prefix-free bracketed numeric indices [1]…[K] let the model scale to any K (tested up to 151-way classification) without architectural changes. Expected Calibration Error and Brier score reported directly from raw next-token probabilities indicate the model's confidence aligns well with empirical accuracy. Qwen3.5-4B (vision-enabled) attains 82.4% on MMBench (CircularEval) and 64.6% on MMStar without any multimodal fine-tuning. Withholding visual input collapses performance to approximately 15% and 28%, confirming genuine visual comprehension. The 0.6B variant shows noticeable accuracy drops when the prompting format changes, whereas the 4B model is robust. The training-free recipe surpasses SemIf by 3.2 percentage points macro-average accuracy, winning 313 instances and losing 154 (p < 10^-12). Gains driven by MMLU +123/−56, MMLU-Pro +96/−41, ARC-Challenge +28/−8.

Fine-tuning provides targeted rather than universal benefits. It yields marked improvements for smaller models and targeted domains (such as many-option intent routing), but offers diminishing returns—and can even induce negative transfer—when applied indiscriminately to capable models on broad benchmarks. Importantly, this targeted adaptation incurs negligible degradation in general model abilities. Fine-tuning is primarily beneficial when training data directly covers specific deficiencies of the base model. Each training mixture improves the tasks it contains (for example, Banking77 improves under all configs; WinoGrande gains when reasoning data are added). Improvements are absent on benchmarks not represented in the training mixture (MMLU, MMLU-Pro, TriviaQA). Model scale governs adaptation utility: compact 0.6B models see large gains when fine-tuned on domain-specific data; the 4B model shows diminishing returns and can even suffer negative transfer if the data are mismatched. MMLU and MMLU-Pro exhibit no advancement despite high lexical overlap with reasoning datasets; the authors argue that shared task format matters more than vocabulary overlap. Format and sequence-length sensitivity must be carefully balanced for reliable adaptation.

Practical fine-tuning recipe and regularization strategies

KL-divergence anchors are crucial for preventing degenerate generation outputs and preserving conversational integrity. Removing the three KL-divergence anchors (mass, output, position) leads to severe generation runaway and conversational drift. Anchors keep the model's generative behavior faithful to the base LLM. Anchor strength λ should match model capacity: larger models tolerate tighter anchors (λ ≈ 0.01–0.1) without degrading performance; smaller models need milder regularization (λ ≈ 0.1) to avoid under-regularization. The overall training objective combines the listwise loss with the regularizers: L = L_tree + λ(L_mass + L_out + L_pos), where λ = 1 by default.

LoRA (rank 16, α = 32) achieves strongest performance on capable backbones (4B) while adding only ~32M trainable parameters. At the 0.6B scale, LoRA and full fine-tuning parity in decision accuracy, but LoRA is far more parameter-efficient. Even with LoRA, the KL anchors remain critical; dropping them hurts both LoRA and full-parameter runs. Training uses AdamW under Fully Sharded Data Parallelism with a conservative learning rate, linear warmup and cosine decay, shuffled candidate ordering, and a single training epoch. An automated validation check halts training if the probability mass on legal tokens drops substantially relative to the base model.

Six training configurations spanning varying data mixtures and step budgets: Intent-100 (3,000 examples, 100 steps, 4B only), Intent-1ep (95,461 examples, 2,984 steps), Reason-9k (9,000 examples, 282 steps), Reason-21k (21,000 examples, 657 steps), Reason-30k (30,000 examples, 938 steps), Reason+Long-12k (12,000 examples, 375 steps).