Text-to-image personalization aims to equip pretrained diffusion models with the ability to learn user-specific concepts from only a few reference images. Existing methods, however, typically train each concept independently or assume that all concepts are available simultaneously. In practical applications, concepts often arrive sequentially, requiring models to continually learn new subjects, objects, or styles while minimizing interference with previously acquired concepts. We address this continual concept personalization setting by extending residual-space adaptation to the stochastic adaptation process of pretrained diffusion models. Our work achieves the highest concept fidelity among competing continual personalization methods while maintaining near-zero forgetting. Beyond these empirical gains, the learned residual-space dynamics reveal where adaptation capacity is concentrated within the diffusion model. Restricting adaptation to the identified layers by our method preserves nearly the same performance while reducing the number of trainable parameters by up to 72%, demonstrating that residual-space dynamics can identify the layers that contribute most to continual concept personalization and directly guide more parameter-efficient adaptation.
Why Continual Concept Personalization Matters
Recent text-to-image models such as Stable Diffusion and FLUX have enabled high-quality image synthesis from natural language descriptions, making generative models increasingly accessible to end users. Their flexibility has further motivated personalization methods that adapt pretrained models to user-specific concepts, subjects, objects, or artistic styles from only a few examples.
Personalization methods commonly rely on parameter-efficient fine-tuning techniques such as LoRA to adapt pretrained diffusion models to new concepts. Most existing approaches perform a separate fine-tuning procedure for every concept, and subsequently merge the resulting adapters into a unified personalized model. In many practical scenarios, however, concepts are acquired sequentially over time as new subjects, objects, or styles become available. Under this setting, whenever a new concept is introduced, the expensive merging procedure must be repeated to reconstruct the final personalized model. As the number of concepts grows, this repeated merging process becomes increasingly inefficient.
This motivates continual concept personalization, where a single model is updated incrementally to incorporate new concepts while preserving previously acquired ones. The main obstacle to continual concept personalization is interference: learning new concepts can degrade performance on previously learned concepts. At the same time, the fixed model capacity must be used parsimoniously to preserve sufficient adaptation capacity for future concepts. Several continual learning methods achieve low levels of interference by constraining the parameter update to a pre-computed safe subspace. The most successful among these rely on layer-wise input statistics to characterize the safe subspace. This procedure cannot be directly extended to stochastic backbones, thus limiting its applicability in image generation tasks.
Related Work: Diffusion Personalization and Continual Adaptation
Text-to-image personalization has been widely studied for adapting pretrained diffusion models to user-specific identities, objects, or styles from only a few examples. Early approaches such as Textual Inversion learn new concepts in the text embedding space while keeping the diffusion model fixed. DreamBooth improves personalization quality by fine-tuning the diffusion model using a prior-preservation objective. Subsequent methods such as Custom Diffusion restrict adaptation to selected cross-attention layers, improving parameter efficiency while supporting multi-concept customization. More recent personalization methods increasingly adopt parameter-efficient fine-tuning techniques such as LoRA. Mix-of-Show introduces ED-LoRA together with structured concept tokens for multi-concept personalization, while LoRACLR improves composition through contrastive alignment of independently trained adapters.
Another line of work reduces interference through the design of the adaptation space itself. Orthogonal Adaptation (OA) constructs approximately orthogonal LoRA subspaces, allowing independently trained adapters to be combined through simple summation with minimal interference. Although originally proposed in a non-continual setting, its orthogonal structure naturally lends itself to sequential adaptation. Within diffusion adaptation, C-LoRA extends Custom Diffusion to a sequential setting by introducing regularization to preserve earlier concepts. Orthogonal Adaptation can also be naturally extended to continual settings through additive composition of orthogonal adapters.
Several methods constrain the parameter update to a safe subspace based on the layer-wise input statistics. Among these, Gradient Projection Memory estimates layer-wise feature covariance from previous tasks and constrains the new updates to its low-energy subspace. The more recent InfLoRA applies the same strategy to LoRA-based fine-tuning, where each task is learned within a fixed low-rank budget. Building upon InfLoRA, a dynamic rank allocation method with simplified updates allows for significant capacity savings. In addition to providing theoretical guarantees for mitigating forgetting, the method demonstrates strong empirical performance on challenging continual language model adaptation tasks.
Although effective, these approaches have been studied primarily in deterministic architectures. Whether residual-space methods remain effective in stochastic diffusion training, where layer statistics are influenced by sampled noise, remains unexplored.
Evaluating Continual Performance: Concept Fidelity and Forgetting
Evaluating continual text-to-image personalization requires measuring not only the quality of generated images but also how well previously learned concepts are preserved throughout sequential adaptation. Unlike discriminative tasks, where forgetting can be directly quantified through accuracy, there is no canonical notion of forgetting for generative models. Existing evaluation protocols are primarily designed for static personalization settings and focus on aggregate measures of text or image alignment. In order to assess the degradation of the concept identity over time, it is important to have metrics that distinguish concept fidelity from general image quality.
To address this problem, the paper introduces a concept-level fidelity measure that quantifies how faithfully a personalized concept is preserved. A generated image may differ substantially from its reference in pose, background, or artistic style while still faithfully depicting the same concept. Concept fidelity therefore measures similarity at the concept level rather than the image level.
To quantify concept fidelity, the paper first measures similarity between individual generated and reference images in a learned embedding space. Given an embedding function F(⋅), the similarity between two images i₁ and i₂ is defined as sim(i₁,i₂) = cos(F(i₁),F(i₂)). While image similarity is defined between individual image pairs, the fidelity of a personalized concept should be assessed from a set of generated images rather than a single sample. The paper generalizes the image-level similarity measure by aggregating pairwise similarities across the generated and reference image sets.
Let G = {g₁,…,gₘ} denote the generated images corresponding to a personalized concept and let I = {i₁,…,iₙ} denote the associated reference images. The concept fidelity score is defined as S(G,I) = 1/(|G||I|) ∑_{g∈G} ∑_{i∈I} sim(g,i). This concept fidelity score extends image-level similarity to a set of generated images, providing a single measure of how well a model preserves a personalized concept.
The paper considers several pretrained embedding models, including CLIP, DINO, and DreamSim, as candidates for computing concept fidelity. DreamSim retrieves images belonging to the same personalized concept despite variations in pose and background, whereas CLIP and DINO retrieve visually similar images from different concepts. The paper adopts DreamSim as the default embedding model throughout the paper.
Using the concept fidelity score, the paper defines continual evaluation metrics that summarize model performance throughout sequential adaptation. Final concept fidelity is defined as Q = 1/N ∑_{k=1}^{N} S(Gₖ(T),Iₖ), which characterizes the average concept fidelity retained at the end of continual training. To investigate the dynamics of forgetting throughout training, let tₖ denote the training stage immediately after learning concept k. The average forgetting trajectory at training stage t ≥ tₖ is defined as F(t) = 1/nₜ ∑_{k=1}^{nₜ} [S(Gₖ(tₖ),Iₖ) - S(Gₖ(t),Iₖ)], where nₜ is the number of concepts that have been learned by stage t. The final forgetting is defined as F = F(T), which measures the average loss of concept fidelity after continual training.
Residual-Space Adaptation for Stochastic Diffusion Models
The paper builds upon the residual-space adaptation framework DyRA, originally designed for LLMs, which constructs layer-wise adaptation subspaces from the input representations. Applying this framework to diffusion models is nontrivial, as these representations depend not only on the input concept but also on the randomly sampled diffusion noise and denoising timestep. The paper first reviews the residual-space adaptation procedure and then develops a stochastic interpretation that justifies applying the framework to diffusion models and motivates the implementation.
A key step of residual-space adaptation is constructing the residual space from layer input representations collected before training each task. For deterministic backbones, this procedure is well defined since each training sample corresponds to a unique layer input. Diffusion models, however, introduce an additional source of stochasticity through the sampled diffusion noise and denoising timestep, so the layer input of a training sample is no longer unique. The paper addresses this by extending the deterministic interference measure to its expectation, considering the stochastic interpretation of residual-space adaptation: projection directions should maximize the expected adaptation ability of the current task while suppressing the expected interference with previously learned tasks.
Since the optimization objective is defined over the training distribution of diffusion noise and denoising timesteps, the collected layer inputs should provide an empirical estimate of the same distribution. Therefore, layer inputs are collected using exactly the same sampling strategy as diffusion training, namely Gaussian noise with uniformly sampled denoising timesteps. This simple step extends residual-space adaptation from deterministic feature representations to stochastic diffusion training by aligning the feature collection procedure with the stochastic optimization objective.
Stable Continual Personalization
This section evaluates whether residual-space adaptation can be successfully extended to continual diffusion personalization. The paper conducts all experiments on Concept101, a benchmark containing 101 concepts spanning 17 semantic classes, including visually similar concepts, making it particularly suitable for studying interference and identity preservation in continual learning. All concepts are learned sequentially in a rehearsal-free setting, where only the current concept data are available during training.
The paper compares DyRA configured with the sigular value threshold τ against three continual LoRA baselines: Trainable-AB, which optimizes both LoRA projection matrices A,B and corresponds to a continual variant of Mix-of-Show; Rank-Fixed-B, which optimizes only the LoRA up-projection A while keeping the randomly initialized down-projection B fixed; and OA, the continual version of Orthogonal Adaptation. Unless otherwise specified, all methods adapt the linear projections in the transformer blocks of both the text encoder and the UNet, and all fixed-rank methods use rank r=4. All experiments are conducted on Stable Diffusion v1.5.
Concept Fidelity and Forgetting Results
Table 1 summarizes the final concept fidelity (Q) and forgetting (F). Trainable-AB achieves competitive concept fidelity but exhibits the largest forgetting, highlighting the trade-off between adapting to new concepts and preserving previously learned ones. In contrast, Rank-Fixed-B and OA effectively suppress forgetting but achieve slightly lower concept fidelity. DyRA maintains negligible forgetting while achieving the highest concept fidelity. These results demonstrate that residual-space adaptation extends effectively to the stochastic training process of diffusion models.
DyRA-0.5 achieves concept fidelity of 0.6454 with forgetting of 0.0002. DyRA-1.0 achieves concept fidelity of 0.6497 with forgetting of −0.0001, indicating actually slightly improved fidelity after all concepts are learned. Trainable-AB achieves concept fidelity of 0.6392 with forgetting of 0.0368, demonstrating significant degradation of previously learned concepts. Rank-Fixed-B achieves concept fidelity of 0.6442 with forgetting of −0.0033, and OA achieves concept fidelity of 0.6435 with forgetting of −0.0023.
The qualitative results are also reflected in the generated images. DyRA remains nearly unchanged as additional concepts are introduced, while Trainable-AB exhibits the largest appearance drift. DyRA maintains high concept fidelity under diverse prompt templates, demonstrating generalization beyond the token-only prompts used during continual evaluation.
Figure 3 reveals how forgetting evolves throughout continual learning. Trainable-AB oscillates around substantially higher forgetting values, indicating persistent forgetting throughout the training process. In contrast, Rank-Fixed-B, OA, and DyRA all fluctuate closely around zero, demonstrating that their low forgetting is maintained consistently over time rather than occurring only at the final evaluation.
Dynamic Rank Improves Parameter Efficiency
The performance gains of DyRA stem not only from restricting adaptation to the residual space, but also from allocating adaptation capacity according to the residual-space structure of each layer. The paper compares DyRA with the fixed-rank OA under different trainable parameter budgets to evaluate the effectiveness of dynamic rank allocation.
Across the entire range of budgets, DyRA consistently achieves higher concept fidelity than OA with comparable numbers of trainable parameters, demonstrating a superior performance–parameter trade-off. Notably, even under high singular-value thresholds, where DyRA has substantially fewer trainable parameters, it outperforms OA with considerably larger parameter budgets. These results indicate that allocating rank adaptively according to the residual-space structure is substantially more parameter-efficient than assigning a uniform rank to every layer.
Layer-wise Specialization: Where Adaptation Capacity Is Concentrated
Beyond empirical performance, the adaptive ranks themselves provide a signal of where adaptation capacity is consumed during continual learning, since additional rank is allocated only when new concepts introduce directions outside the accumulated input subspace. This section analyzes these rank dynamics to understand how adaptation capacity is distributed across the layers and how this information can inform future parameter-efficient adaptation.
Adaptation Capacity Concentration
The paper first analyzes how DyRA distributes adaptation capacity across the adapted layers by averaging the allocated ranks over all 101 concepts. DyRA exhibits clear layer-dependent allocation patterns that closely follow the repeated transformer architecture. In the text encoder, the allocated ranks exhibit a four-layer periodicity that exactly mirrors the repeated key(K), value(V), query(Q), and output projection layers within each transformer block. A similar correspondence is observed in the UNet, where every group of eight adapted layers corresponds to the self-attention and cross-attention projections within a transformer block. Within every block, the two dominant rank peaks consistently coincide with the cross-attention key(K) and value(V) projection layers, which repeatedly receive substantially larger ranks than all other projections.
The repeated concentration of residual rank in the UNet cross-attention K,V projections indicates that these layers consistently consume more adaptation capacity than other adapted modules during continual concept personalization. This observation motivates a closer examination of these layers in the following analysis.
Early Capacity Saturation
The UNet cross-attention K,V projections receive the largest residual ranks. The paper examines whether exhausting the residual capacity of these layers is associated with degraded adaptation to subsequently learned concepts.
Figure 7 summarizes the remaining rank of every adapted layer after learning all 101 concepts. Under DyRA-0.5, all 32 UNet cross-attention K,V projections become fully saturated, whereas no UNet layer saturates under DyRA-1.0. Figure 8 further illustrates representative remaining-rank trajectories. While non-saturated layers retain sufficient residual capacity throughout training, representative cross-attention K,V layers under DyRA-0.5 steadily exhaust their available rank and become saturated around tasks 77–78. In contrast, the corresponding layers under DyRA-1.0 retain residual capacity throughout the entire task sequence.
To assess the effect of this capacity exhaustion, Table 2 compares the final concept fidelity with the average fidelity of the final 20 learned concepts. DyRA-1.0 maintains essentially unchanged performance on later concepts, whereas DyRA-0.5 exhibits a clear late-task fidelity degradation after the cross-attention K,V layers become saturated. Although this evidence is correlational, the consistent agreement between the rank allocation, capacity saturation, and late-task performance suggests that preserving residual capacity in the UNet cross-attention K,V projections is important for sustaining continual concept adaptation over long task sequences.
Layer-Guided Parameter-Efficient Adaptation
The analyses consistently identify the UNet cross-attention K,V projections as the layers that require the largest residual adaptation subspaces throughout continual learning. This suggests that these layers differentiate newly learned concepts through distinct adaptation directions. If this interpretation is correct, restricting adaptation to the identified layers would be expected to preserve most of the continual personalization performance while substantially reducing the number of trainable parameters.
Table 3 supports this hypothesis. Restricting adaptation to only the UNet K,V projections reduces the trainable parameter count by approximately 60% (67.56M → 27.12M) while achieving virtually identical concept fidelity (Q=0.6496 vs. 0.6497) and comparable forgetting. Restricting adaptation further to only the UNet cross-attention K,V projections reduces the trainable parameter count by approximately 72% (67.56M → 18.87M), yet still preserves nearly all continual personalization performance in practice.
Beyond validating the proposed interpretation, these results also provide a structural explanation for previous personalization methods such as Custom Diffusion and C-LoRA, which restrict adaptation to cross-attention K,V projections based primarily on empirical observations of large weight updates during training. In contrast, residual-space dynamics explain the same design choice from the perspective of feature-space utilization, providing a principled interpretation of why these layers are particularly effective adaptation targets.
More broadly, residual-space dynamics provide a principled way to identify where adaptation capacity is most effectively utilized. Rather than relying on empirical heuristics or exhaustive layerwise search, the proposed analysis directly guides targeted adaptation, enabling substantially more parameter-efficient continual personalization while preserving nearly the same overall performance.
What This Means for Practitioners Working with Diffusion Models
For researchers and engineers working with diffusion-based image generation, the paper offers several practical takeaways. First, continual concept personalization is a distinct setting from independent concept training, and methods designed for the latter may exhibit significant interference when concepts arrive sequentially. Second, residual-space adaptation provides a competitive option for continual personalization that mitigates interference while maintaining adaptivity at low cost. Third, the learned residual-space dynamics serve as an efficient probe for identifying the layers that contribute most to continual concept adaptation, providing insights into layerwise specialization and guiding more targeted parameter-efficient fine-tuning strategies.
In practice, the finding that the UNet cross-attention K,V projections concentrate most adaptation capacity suggests that practitioners seeking parameter-efficient personalization can restrict their fine-tuning to these layers without substantial performance loss. The paper demonstrates that restricting adaptation to only the UNet cross-attention K,V projections reduces the trainable parameter count by approximately 72% while still preserving nearly all continual personalization performance. This dramatic parameter reduction can lower computational costs and memory requirements, making personalization more accessible on resource-constrained hardware.
For developers building systems that must incorporate new concepts over time, the systematic evaluation framework introduced in the paper provides a way to measure not only image-generation quality but also concept fidelity and forgetting over time. This is essential for assessing whether a personalization method truly preserves previously learned concepts as new ones are added.
Limitations and Future Directions
Although DyRA achieves strong continual personalization performance, several limitations remain. First, the adaptation capacity of residual-space methods is fundamentally bounded by the width of individual layers. As continual learning progresses, layers with consistently high adaptation demand may eventually exhaust their available residual subspaces, limiting scalability to arbitrarily long task sequences. Second, the method is implemented only on Stable Diffusion v1.5 and Concept101. The residual-space analysis developed in this work is considerably more general than the specific adaptation algorithm, since it relies only on layer input statistics and residual-space dynamics, and it can potentially be applied to other diffusion backbones, transformer-based generative models, or broader neural architectures to identify the layers that contribute most to learning new tasks.
Future work could explore adaptation patterns across architectures to provide further insights into representation specialization and guide the design of future parameter-efficient adaptation methods. Additionally, the paper notes that the adaptation capacity of residual-space methods is fundamentally bounded by the width of individual layers, and developing methods that can scale to arbitrarily long task sequences remains an open challenge.
Key Takeaways
Our work shows that residual-space adaptation extends naturally to stochastic diffusion models, enabling continual concept personalization with stable performance, negligible interference, and strong parameter efficiency. Beyond its empirical performance, the residual-space dynamics observed during continual adaptation reveal how adaptation demand is distributed across the adapted layers. By identifying the layers that consistently require new concept-specific adaptation directions, it provides a structural interpretation of existing adaptation strategies and directly guides more parameter-efficient adaptation. We hope this perspective motivates future work that uses adaptation dynamics not only to improve continual learning algorithms, but also to better understand layer specialization in modern generative models.
Continual concept personalization in diffusion models has moved beyond the assumption that all concepts are available simultaneously. The introduction of residual-space adaptation to stochastic diffusion training provides a new competitive option for learning user-specific concepts sequentially while minimizing interference with previously acquired concepts. The method achieves the highest concept fidelity among competing continual personalization methods while maintaining near-zero forgetting. Beyond these empirical gains, the learned residual-space dynamics reveal where adaptation capacity is concentrated within the diffusion model. Restricting adaptation to the identified layers preserves nearly the same performance while reducing the number of trainable parameters by up to 72%, demonstrating that residual-space dynamics can identify the layers that contribute most to continual concept personalization and directly guide more parameter-efficient adaptation.