Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data

A Hypernetwork That Writes User Data Into Model Weights During a Conversation

Language models are frozen after training. When a user supplies facts, gives corrections, or establishes preferences during a conversation, that knowledge lives in the prompt. It gets re-read on every request, consumed context window tokens, and discarded when the session ends. The next conversation starts from scratch. A paper posted to arXiv on September 16, 2026 proposes a different architecture: one that generates feed-forward weights from live data instead of storing them in a fixed parameter bank.

The paper is called "Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data." It was written by Jinli Hu, Ross M. Clarke, Yichuan Zhang, and Jose Miguel Hernandez-Lobato. The core idea is a compact hypernetwork that takes data supplied at run time and produces a low-rank modulation of a shared base network. The base network's stored parameters never change. But the weights that actually execute are generated from whatever the user has told the model during the current session.

Why Mixture-of-Experts Is Not Enough

Mixture-of-Experts architectures have been the primary way to scale model capacity without scaling compute proportionally. An MoE model stores an enormous parameter bank and activates only a fraction of it for each token. That fraction is selected by a router based on the input. The approach has delivered strong results across language, vision, and multimodal tasks.

The limitation is that MoE routers select from a fixed set of experts. The knowledge in those experts was learned during pretraining. A deployed MoE model cannot create a new expert from the data a user provides during a conversation. If a user says "my company uses PostgreSQL 14 with the pgvector extension," that fact goes into the context window. It does not change which experts fire or what weights they contain.

The Infinite-Parameter LLM addresses this by replacing the fixed expert bank with a generated one. Instead of routing to stored experts, a hypernetwork compiles the user's data into weight modulations on the fly. The stored footprint stays constant. The effective parameter space is unbounded because the weights are derived from live input rather than pre-allocated during training.

The Bayesian Belief Over Latent Code

The paper's most important technical contribution is the online update mechanism. Prior hypernetwork approaches read the context once, generate weights, and freeze them. If the user provides new information later in the conversation, the weights do not change. The model either re-reads the new information from the prompt or ignores it.

The Infinite-Parameter LLM carries a Bayesian belief over the generator's latent code. As the conversation proceeds and the user supplies more data, that belief is updated. The effective weights are re-derived from the evolving belief at each step. This means the model's behavior changes as the session progresses, not just its prompt.

The Bayesian framing gives the system a principled way to handle uncertainty. Early in a conversation, when the user has provided little data, the belief is broad and the weight modulations are conservative. As more data arrives, the belief narrows and the weights specialize. The model can also express uncertainty about conflicting information by maintaining a multimodal belief distribution rather than collapsing to a single point estimate.

What Changes for the User

The practical implications are concrete. First, context window freed. Facts that would normally occupy prompt tokens are carried in the weights instead. For a conversation that spans many turns with accumulating user preferences, this could save thousands of tokens per request. Second, persistence across turns. In-context learning resets when the context window fills or the session ends. Generated weights persist as long as the session is active, without consuming prompt space. Third, better generalization. The paper argues that carrying knowledge in weights rather than the prompt can generalize better than in-context use, because the weight modulations can influence the model's processing of novel inputs in ways that prompt injection cannot.

The tradeoff is compute. Generating weights from live data is more expensive than reading them from a prompt. The paper describes this as amortized: the hypernetwork runs once to generate weights, and those weights apply to every subsequent token in the session. Whether this amortization pays off depends on session length, the complexity of the user data, and the cost of the hypernetwork forward pass relative to the prompt processing it replaces.

Where It Fits in the Hypernetwork Literature

The paper sits in a growing body of work on hypernetworks for LLM adaptation. SHINE, published earlier in 2026, maps context to LoRA adapters in a single forward pass. LLM-HYPER uses language models as hypernetworks for weight generation in cold-start ad ranking. LatentSkill converts textual skill documents into LoRA updates through a skill compiler. A paper called "Model of Models" ran a four-way comparison across zero-shot, in-context attention, test-time gradient adaptation, and hypernetwork-emitted specialist weights.

The Infinite-Parameter LLM differentiates itself through the online Bayesian update. Most prior work generates weights once and fixes them. This paper generates weights continuously throughout the session. The distinction matters for long-running interactions where the user's data accumulates over time, rather than being provided all at once at the start.

An Evaluation Protocol for Weight-Based Knowledge

The paper specifies an evaluation protocol designed to test exactly the claim that weight-carried knowledge outperforms in-context knowledge. The protocol compares three approaches: carrying user data in the prompt (the baseline), generating weights from user data through the hypernetwork, and retrieving relevant information from an external store. The evaluation measures accuracy, generalization to novel inputs, and context window usage across sessions of varying length.

The authors position this as a necessary contribution because existing benchmarks do not distinguish between these mechanisms. A model that scores well on a knowledge-intensive task might be relying on its training data, its prompt, or its retrieved context. The proposed protocol isolates the source of knowledge to determine whether weight generation actually provides an advantage.

Open Questions

The paper raises several open questions. The hypernetwork adds latency to the first token of a conversation. How large is that overhead relative to the savings from freed context window? The Bayesian update requires maintaining a belief state across turns. How does that state scale with conversation length and the complexity of the user data? The weight modulations are low-rank. Is low-rank sufficient for the kinds of knowledge users actually provide in conversations, or does important information get lost in the rank bottleneck?

There is also the question of what happens when the session ends. The generated weights disappear. The model does not learn from the user's data in any lasting way. This is a feature for privacy, but it means every new session starts over. Whether that is acceptable depends on the use case. For a coding assistant that works with the same codebase across sessions, the re-generation overhead might be acceptable. For a customer service agent that needs to remember a customer's history, it might not be.

The paper is a preprint with no reported experimental results yet. The evaluation protocol is specified but not executed. The claims about amortized compute, freed context window, and better generalization remain theoretical. Whether the Bayesian online update actually produces better weight modulations than a single-shot generation, and whether the hypernetwork overhead is worth the context savings, will depend on the experimental results when they arrive.