Most language models face a fundamental tradeoff: they can learn new information, but they forget old information in the process. CellularFlow, an open-source neural architecture, attacks this problem by replacing the dense feed-forward networks that store knowledge in standard transformers with associative memory banks that keep facts separate from reasoning.

Why standard transformers forget

In a standard transformer, the feed-forward network layers store factual knowledge as distributed weights entangled with the model's reasoning patterns. When the model is fine-tuned on new data, those weights shift, and the old knowledge they encoded degrades. This is catastrophic forgetting, and it is why fine-tuning a language model on new domains without losing its existing capabilities is so difficult.

CellularFlow replaces those feed-forward layers with a Hybrid CMC Layer that splits the work. Multi-Head Associative DNA Memory Banks store factual knowledge in discrete, addressable slots. An Episodic Memory Slot Buffer handles fast writes for new facts. The standard causal self-attention mechanism handles sequence reasoning. By separating storage from reasoning, the architecture lets you update knowledge without disrupting the model's ability to process sequences.

The result, measured on a standardized 62KB multi-domain corpus, is a model with 379,000 parameters that achieves 73.7% accuracy and a perplexity of 2.54. A GPT-mini baseline with 810,000 parameters, more than twice as many, scores 36.4% accuracy with a perplexity of 8.51. CellularFlow gets better results with fewer parameters by using them more efficiently.

Three modes of learning

The architecture supports three distinct modes of updating knowledge, each suited to a different use case. Mode 1, live learning, updates DNA memory values during inference using exponential moving average with zero backward passes. You can feed the model streaming data, and it incorporates the information without any gradient computation. Spherical anisotropy regularization prevents the memory vectors from collapsing during these updates.

Mode 2, selective fine-tuning, freezes approximately 85% of the model's backbone, including projections, embeddings, and layer norms, and trains only the DNA memory banks. This lets the model absorb new domains rapidly while retaining its foundational knowledge. Measured across sequential training on literature, science, history, technical, and poetry domains, Mode 2 achieves 83.9% retention, compared to 61.8% for full fine-tuning of all weights. That is a 22.1 percentage point improvement.

Mode 3, episodic fact injection, writes individual facts into slot-based episodic memory with temporal age decay, facts lose influence as they age, and consolidates the most important facts into DNA banks after each epoch. This is for instant knowledge updates, when you need the model to know a specific fact immediately without retraining.

How the memory architecture works

The Hybrid CMC Layer processes input through two parallel pathways before combining them. The DNA Memory Associative Banks hold long-term factual knowledge in specialized subspaces, where different heads specialize independently across syntax, semantics, and domain knowledge. Gaussian exploration noise prevents dead memory slots during sparse routing, ensuring the memory stays populated and useful.

The Episodic Memory Slot Buffer handles fast writes, accepting new facts immediately without waiting for a training cycle. After each epoch, the most relevant episodic facts get consolidated into the DNA banks, creating a natural hierarchy from short-term to long-term memory.

The two pathways feed into gated memory enrichment, which combines their outputs before passing them to causal multi-head self-attention with rotary position embeddings. The output is the same shape as a standard transformer layer, which means CellularFlow can drop into existing architectures with minimal modification.

What this enables in practice

The practical implication is a model that can learn continuously without forgetting. You can train it on literature, then technical documentation, then medical data, and it retains its capability across all three domains. Standard transformers require careful curriculum design, learning rate scheduling, and often still lose performance on earlier domains as new ones are added.

The live learning mode is particularly interesting for applications where data arrives in real time. A monitoring system, a chatbot, or a coding assistant that learns from ongoing interactions can incorporate new information without retraining. The zero-backpropagation requirement means the update is computationally cheap and can happen during normal inference.

The fact injection mode solves a different problem: getting specific, up-to-date information into the model without fine-tuning. Instead of updating weights through gradient descent, you write the fact directly into episodic memory and let it consolidate naturally. The temporal decay ensures that old facts do not persist indefinitely.

What is available now

The project includes the core architecture, a trainer supporting all three learning modes, an interactive web dashboard for real-time inference and memory inspection, and a CLI playground for exploration. The dashboard runs as a FastAPI server and connects to the model over WebSocket, showing layer-wise episodic slot utilization and allowing live fact injection.

The code is MIT licensed and structured for contribution. The repository includes benchmarks, evaluation harnesses, and extension modules for blockwise attention, compressed KV cache, multi-token prediction, and beaconing. For researchers exploring alternatives to standard transformer architectures for continual learning, CellularFlow provides a working implementation with published results and a clear architectural rationale.