A 3-Million-Parameter Model That Matches Bigger Earth Observation Encoders
Earth observation models face a paradox: they need to handle multiple sensor types (optical, radar, metadata) with missing inputs and varying spatial resolutions, which pushes architectures toward larger sizes. But operational deployment on large satellite archives or constrained hardware demands compact models. MEOX from ECMWF takes the opposite approach: a 2.939 million-parameter encoder with 3.115 million parameters total that outperforms models 87 times its size on several benchmarks. The trick is not just making the model small, but designing it so that sensor-specific processing, validity-aware fusion, and sparse expert routing work together to preserve the flexibility of larger systems within a tight parameter budget.
Delayed Fusion: Process Sensors Independently, Then Merge
The core architectural decision is delayed, validity-aware fusion. Each available sensor (Sentinel-2 optical, ascending Sentinel-1 SAR, descending Sentinel-1 SAR) is processed independently through a shared transformer-MoE block before cross-sensor merging. This is not a simple concatenation. At each spatial position, the model computes a fusion score for every available sensor by combining a learned weight on the normalized patch representation, a sensor-specific bias, and a log-prior based on the fraction of valid pixels in that patch. Softmax produces nonnegative weights summing to one. Sensors that are not supplied are excluded entirely from the computation.
This design has two consequences. First, sensor-dependent routing can learn to favor different experts for radar versus optical data within the shared block, without needing separate parameter sets per modality. Second, the deeper encoder processes only one spatial sequence, not three concatenated streams. With three sensors, the shared pre-fusion block is evaluated three times, but the 14 subsequent encoder blocks process N+5 tokens instead of 3N+5. This saves memory and compute at every layer beyond the first.
Before fusion, each sensor image is divided into nonoverlapping 4x4 patches, producing 256 tokens of width 144 at 64x64 pixels. Each patch token carries four pieces of information: a convolutional adapter embedding of the pixel content, an embedding of the validity mask (initialized to zero so it does not perturb initial representations), a learned sensor identity vector, and a projection of which expected bands are actually available. The validity mask embedding is critical: it lets the model distinguish a missing value from a valid observation equal to the normalization mean. Without it, a nodata pixel looks identical to a genuinely zero-valued pixel.
Metadata enters as four additional tokens: latitude, longitude, month, and ERA5 climate variables (12 temperature and precipitation values). Each metadata token carries a type identifier, a projection of available values, and a missingness signal that increases as more components become unavailable. This explicit missingness representation means omitting metadata produces a deliberate missing-data embedding rather than being interpreted as a zero value.
Sparse Experts with Private Low-Rank Residuals
Each encoder block contains self-attention followed by a MoE feed-forward sublayer. The number of experts grows with depth: three in layers 0-4, four in layers 5-9, and five in layers 10-14. Every token selects exactly two experts via top-2 routing, and both are executed. There is no capacity limit or forced dispatch quota.
The key parameter-efficiency mechanism is shared value and output projections with expert-private rank-8 residual paths. Each expert has a SwiGLU gating projection (private) and two low-rank residual paths: one for the value projection and one for the output projection. The U matrices initializing these residuals are set to zero, so private contributions start at nothing and emerge during training. This lets each expert modify the shared transformations without storing complete private copies of the weight matrices. In the deepest layers, with five experts, output cosine similarities between different experts range from 0.39 to 0.69, indicating differentiation despite the shared backbone.
An auxiliary balance loss encourages uniform expert usage. It measures the product of each expert's average router probability and its fraction of executed assignments. The loss is approximately one under uniform routing and increases when dispatch concentrates. At 0.01 weight, it keeps routing balanced without forcing it. Routing diagnostics on 128 validation samples show normalized entropy of 0.984-0.999 globally (near maximum) but 0.886-0.956 per token, confirming that routing is balanced on average while remaining selective for individual inputs.
Pretraining: Modality-Balanced Reconstruction with Sensor Dropout
MEOX is pretrained on 1.228 million MMEarth64 samples using masked reconstruction. The reconstruction loss is computed only on spatially masked and valid target values. Critically, each sensor contributes one loss term regardless of its channel count: Sentinel-2's 13 bands and Sentinel-1's 2 VV/VH bands receive equal weight. Without this normalization, the 13-band optical target would contribute 6.5 times as many error terms as either SAR target, dominating training.
Sensor dropout teaches the model to handle missing modalities. In 90% of batches, all three sensors are provided. In the remaining 10%, one of six possible subsets is selected uniformly: each individual sensor, each pair, and so on. The reconstruction targets always include all three sensors regardless of what the encoder sees. This forces the model to predict one sensor from the others when it is absent. Each sensor is omitted in roughly 5% of batches.
The decoder is deliberately simple: two dense transformer blocks with SwiGLU hidden width 54, no MoE layers, no cross-attention. It has 176,096 parameters, or 5.65% of the total model. This keeps the encoder as the primary representation learner. Axial rotary position encoding (RoPE) supplies geometry to attention without adding absolute positional vectors to the residual stream, which is important because it allows variable token grids at inference without retraining positional embeddings.
Frozen Transfer: Strong on Cashew and EuroSAT, Mixed Elsewhere
Evaluation uses GEO-Bench v1 with frozen encoder weights and linear or UPerNet heads. At 64 pixels, MEOX reaches 64.42% mean IoU on cashew segmentation and 89.70% average accuracy on EuroSAT, exceeding the strongest reported CSMoE variants by 5.02 and 1.40 points respectively. At 224 pixels, EuroSAT improves to 90.56% AA.
The picture is more nuanced on other tasks. BigEarthNet frozen transfer reaches 55.16% micro-mAP at 64 pixels, trailing larger models. Crop type segmentation is weak at 27.65% mIoU. So2Sat drops 2.43 points when moving from 64 to 224 pixels. Spatial resampling is task-dependent: the 224 grid improves crop type by 4.68 points but reduces So2Sat, likely because different spatial resolutions change the sampling pattern and token count without changing the geographic extent.
BigEarthNet adaptation with finetuning (all encoder blocks enabled, 17,400 router parameters frozen) raises micro-mAP from 57.22% to 72.95% at 224 pixels. This demonstrates that the compact encoder has task-specific capacity to unlock when allowed to adapt, even though frozen transfer is the primary evaluation regime.
Parameter and Compute Budget
The numbers are striking. MEOX has approximately 87 times fewer parameters than CSMoE P=14 and uses 2.88 times fewer training archive pixels. At 64-pixel native resolution, inference costs 1.029G FLOPs for a single Sentinel-2 image, compared to 2.92G for CSMoE P=32 at 224 pixels. At 224 pixels, however, the global attention over 3,136 patches raises MEOX to 51.477G FLOPs, losing the compute advantage. The practical recommendation is clear: 64 pixels is the efficient operating point for MEOX, while 224 pixels demonstrates grid flexibility at higher cost.
Training used AdamW for 50 epochs on an A100 MIG 2g.20gb partition, with approximately 166.9 hours elapsed time. The checkpoint minimizing total validation loss was selected at epoch 31 (reconstruction loss 0.1046). Epoch 50 did not improve, confirming that longer training does not help this architecture.
Metadata Matters, But Not Much
A held-out WorldCover probe isolates the contribution of metadata without rerunning pretraining. All metadata yields 30.45% test mIoU. Removing everything costs 0.64 percentage points. ERA5 climate variables contribute the largest isolated change (-0.29 points). Latitude and longitude removal costs only 0.12 points. Month removal is negligible. Nine of eleven classes improve with metadata relative to none. The 0.64-point benefit is small but consistent, supporting a modest positive role for context and graceful operation without it.
The same probe measures cross-sensor retrieval. Same-sensor F1 is 64.41% for Sentinel-1 and 66.33% for Sentinel-2. Cross-sensor F1 drops to 39-40%, indicating that multimodal reconstruction yields useful sensor-specific semantics but does not fully align radar and optical representations in a shared metric space. This is an honest limitation: the model learns to reconstruct both modalities, but the embedding geometry remains sensor-biased.
Routing Diagnostics: What Experts Actually Do
The paper provides four distinct routing measurements rather than inferring specialization from overlays. Expert participation is measured by execution rate (fraction of tokens selecting an expert among the top-2). Spatial dependence is measured by normalized mutual information between patch position and top-1 expert assignment (0.00189-0.00623, close to the shuffled baseline of 0.00557, showing no fixed spatial routing template). Modality association is visible in the shared pre-fusion block: expert 1 receives 33.05% of S2 tokens but 52.86% of ascending-S1 tokens, confirming sensor-conditioned routing without exclusive assignment. Functional contribution is measured by output suppression: removing a single expert from a layer increases reconstruction loss by 0.002-0.003 in early layers, with unequal effects across experts.
This diagnostic suite is a contribution in itself. Most MoE papers report routing balance and leave it at that. MEOX distinguishes active dispatch, spatial association, modality conditioning, and causal contribution, providing a more complete picture of what the sparse computation actually does.
Limitations and What Comes Next
MEOX does not address random-band-removal robustness, overlapping-window inference for large scenes, or matched ablations against the original Geo-MoE-MAE architecture. The model supports configured sensors with named bands, not arbitrary unseen instruments, which limits generalization to new satellite missions. The crop type and So2Sat results indicate that compact size does not guarantee strong performance on every task. Cross-sensor retrieval shows that multimodal pretraining does not produce a fully aligned radar-optical metric space.
For practitioners, MEOX offers a reproducible compact alternative for multimodal EO representation learning. The code and experiment notebooks are available on GitHub. The frozen-transfer results at 64 pixels are competitive with much larger models, the parameter budget is small enough for edge deployment, and the routing diagnostics make the representation inspectable. The practical trade-off is clear: accept weaker performance on some tasks (crop type, BigEarthNet frozen) in exchange for 87x fewer parameters and 2.8x less training data.