Google's Gemma 4 small models, the E2B and E4B variants, use a technique called per-layer embeddings that exploits a gap between parameter count and compute cost. The "E" in the model names stands for "effective." Gemma 4 E2B lists 2.3 billion effective parameters, but 5.1 billion when you count embeddings. Gemma 4 E4B lists 4.5 billion effective parameters, reaching 8 billion with embeddings. The main transformer stack runs at the smaller number. The extra parameters live in embedding tables that add token-specific information without making the core compute as expensive as a dense model of the same total size.
What PLE does differently from KV sharing
Gemma 4's cross-layer KV sharing reduces the key-value cache, which lowers memory usage during inference. PLE addresses a different problem. It increases parameter efficiency by giving each transformer layer access to additional learned representations that are cheap to compute but expensive to replicate through the standard transformer stack alone. The two techniques are complementary, not alternatives.
In a standard transformer, the same token embedding flows into every layer. Each layer transforms it through attention and feed-forward networks, but the initial representation is fixed. PLE adds a separate path where each layer gets its own slice of embedding information, conditioned on the layer's current hidden state.
How the PLE path works
The PLE vectors are prepared outside the repeated transformer blocks. Token IDs go through a per-layer embedding lookup, producing one vector per layer. Separately, the normal token embeddings pass through a linear projection into the same packed PLE space. These two components are added together, scaled, and reshaped into a tensor with one slice per layer.
Inside the transformer block, attention and feed-forward branches run as usual. The resulting hidden state then gates the layer-specific PLE vector. That gated vector is projected back to the model's hidden size, normalized, and added as an extra residual update. Each layer receives a different slice, so the additional information is not shared across the stack.
The cost is modest. The per-layer embeddings and their projections are small compared to the transformer's weight matrices. The gating operation adds a lightweight computation inside each layer. But the additional parameters give the model more capacity to represent token-specific patterns without scaling up the attention heads or feed-forward dimensions.
Why this matters for small models
Small language models face a tension. Reducing parameters makes them cheaper to run but limits what they can learn. PLE offers a middle path. You keep the transformer stack small, which controls inference cost and memory usage, while the embedding tables provide extra capacity at minimal compute overhead. The effective parameter count reflects the compute the model actually performs. The total parameter count reflects the full set of learned representations available.
The technique is not inherently limited to small models. You could attach per-layer embedding slices to larger architectures as well. But larger models typically have enough capacity through their transformer layers that the marginal benefit shrinks. For big models, mixture-of-experts designs serve a similar purpose, increasing capacity while keeping per-token compute manageable.
An open question
The real test of PLE is comparison. How does Gemma 4 E2B perform against a regular 2.3B model and a regular 5.1B model on the same benchmarks? If the effective parameter count is accurate, E2B should approach the larger model's quality while running at the smaller model's cost. Google has not published that comparison directly, and independent evaluations will take time. The design is sound on paper, but the gap between "this should work" and "this actually works in practice" is where interesting research lives.
What PLE demonstrates more broadly is that the relationship between parameter count, compute cost, and model capability is not fixed. Clever parameter allocation can give a small model access to representations that would normally require a much larger transformer stack. As inference cost becomes the binding constraint for deploying language models, techniques that decouple capacity from compute will matter more than raw scale.