The BabyLM challenge asks a deceptively simple question: can a language model learn grammar from an amount of text a child might actually encounter? Submissions are capped at 100 million words, and evaluated on grammatical minimal pairs, world knowledge, and downstream tasks. Through 2025, every entrant trained on English. The 2026 edition finally opened the door to other languages, and MéTRON-FR, a 125M-parameter GPT-2 trained on 92.47 million words of French, is one of the first to walk through it.
The paper's headline number is 85.97% on QFrBLiMP, a native Quebec-French benchmark of grammatical minimal pairs. That beats the best scores posted by any English, Dutch, or Chinese entry in the same competition: 80.75% on English BLiMP, 81.70% on Dutch BLiMP-NL, 78.60% on Chinese ZhoBLiMP. The model also scores 62.80% on the BabyLM-weighted leaderboard, at or above the re-evaluated Strict-track reference points from prior years. But the real contribution is not the leaderboard position. It is a set of methodological findings showing that the standard evaluation toolkit for child-scale language models is badly broken, and what to do about it.
Why French, and Why Quebec French Specifically
The choice of language is not arbitrary. Prior work (Wasserman, 2026) has shown that morphologically rich languages reach grammatical competence with far fewer tokens than English. French, with its verb conjugations, determiner-noun agreement, and subject-verb agreement, may be easier to learn grammatically from a small corpus than English, which relies heavily on word order and function words. A French-only submission tests whether the architectural innovations driving English BabyLM progress (ELC-BERT, GPT-BERT) are actually the most important variable, or whether language choice matters more.
The native-language angle matters for a different reason: evaluation. Translating an English benchmark into French does not produce an equivalent probe. The translated BLiMP Supplement scores 34% on MéTRON-FR, below chance, because the items contain a systematic lexical-frequency confound. In the "good" sentences, the content words happen to be less frequent in the training distribution than in the "bad" sentences, so the model prefers the ungrammatical option. The score measures translation artifacts, not pragmatic inference. A native benchmark avoids this entirely.
QFrBLiMP itself encodes the prescriptive norm of the Office québécois de la langue française (OQLF), which differs from metropolitan French usage on anglicism-sensitive items. The anglicism-related bucket is the lowest of four at 80.15%, reflecting two mechanisms: internalization of French phonotactic and morphological regularities (which rejects non-French coda clusters), and sensitivity to the OQLF's prescriptive prescriptions (which flags phonotactically licit forms like "matcher" as unacceptable). Morphological and syntactic findings are expected to generalize to metropolitan French; anglicism findings require re-validation.
The Haitian-Creole Oracle: A Corpus-Weighting Heuristic
The training corpus combines three French sources: approximately 2.1 million words from CHILDES French child-directed speech, approximately 24.9 million words from the non-subtitle portion of the French component of BabyBabelLM, and approximately 66 million words from filtered French Wikipedia. But sentences are not sampled uniformly.
The authors oversample sentences containing French lemmas whose cognates survived into Haitian Creole. The reasoning comes from creole-genesis research: creoles form under acute communicative pressure, so lemmas that survive this contact filter are plausibly disproportionately load-bearing for grammatical communication. No Haitian Creole text enters training. The lemmas act as an oracle lexical filter on French source material only, weighting the corpus toward words that are likely to carry grammatical weight.
The oracle is presented as an exploratory heuristic, not a validated contribution. Its effect is not isolated against an unweighted baseline within this paper, though a clean-rebuild ablation (Section 5.1) bounds any oracle-specific gain to within the training-time noise floor. It is the recipe the authors submitted, flagged honestly as uncontrolled.
How the Model Trains and What It Learns
MéTRON-FR is a standard 125M GPT-2 pretrained for five epochs on the assembled corpus. The tokenizer is a 50,000-token byte-pair encoding trained on the French corpus itself. The submitted checkpoint is epoch 3, selected as the empirically located grammatical-competence peak. A 5-seed reproduction (seeds 42 through 46) provides variance estimates.
On QFrBLiMP, accuracy by linguistic phenomenon: syntactic 89.74%, semantic 87.19%, morphological 85.47%, anglicism-related 80.15%. Grammatical accuracy stays within a narrow band of approximately 0.40 percentage points across all five epochs while training loss continues to decline monotonically from epoch 3 onward. Grammatical competence saturates before perplexity saturates at this scale.
On QFrCoLA, the fine-tuned acceptability classifier achieves 69.68% accuracy and an MCC of 0.13 across 5 seeds. For comparison, BERT-base on English CoLA reaches MCC 0.52 with 110 million parameters but approximately 2.5 billion words of pretraining, 27 times more than MéTRON-FR's budget. The 0.13 MCC is positive and above chance, but well below standard-scale performance, characterizing what acceptability-judgment competence is measurable at the BabyLM word budget.
Structural Alignment Tracks Grammatical Competence, Not Training Duration
Bilingual Lexicon Induction provides a geometric measurement of cross-lingual alignment. A frozen orthogonal Procrustes map is learned between MéTRON-FR's French embeddings and two English embedding spaces, using a hand-curated 73-entry bilingual lemma bridge (not part of training). Held-out word-translation precision at rank-1 reaches 68.84% against GPT-2, 18 times above chance.
The decisive contrast is against a matched-architecture English model that failed to acquire English grammar despite training on 6.5 billion tokens, 71 times more than MéTRON-FR. Alignment drops to 25.00% precision at rank-1. The successful target (GPT-2) was trained on roughly 8 billion tokens. Structural alignment tracks acquired grammatical competence, not architecture or training duration. Content-word alignments are rank-1 nearly universally (fuir to flee, donner to give, cassé to broke), and morphological features map structurally (écrivent to write, révélé to revealed).
Cross-Lingual GLUE: Where Translation Helps and Where It Does Not
The cross-lingual GLUE protocol translates English task data into French and applies rank-16 Low-Rank Adaptation (LoRA). LoRA preserves the base model by construction: each per-task adapter applied to the epoch-1 checkpoint yields an identical 83.53% on QFrBLiMP because base parameters are bit-identical and adapter weights do not enter the autoregressive head. This is a methodological choice the authors argue is preferable to post-hoc verification at child scale, where every grammatical-competence parameter is hard-won.
The results reveal a sharp task-type gradient. Relational tasks gain measurably over the English-LoRA baseline: BoolQ +3.45pp, RTE +5.13pp, MRPC +2.89pp. World-knowledge tasks regress: MNLI drops 11.10pp. The discourse-coreference task WSC is statistically inconclusive (minus 1.92pp with standard deviation of 8.88pp). The per-task mean is approximately flat, but the structure is the finding: training-language adaptation helps where the task probes structural relations and hurts where it probes encyclopedic content or extended discourse.
A rank-16 English-LoRA condition isolates the rank effect within a single language and shows no consistent gain over rank-8, supporting the reading that per-task structure is driven by training-language adaptation rather than adapter rank.
Tokenizer Ablation: The Artifact That Looks Like a Result
The most striking methodological finding comes from an ablation that was not planned. The authors initially trained a heavily reallocated corpus ("v2") with 38 million words of EWoK-targeted Wikipedia, 25 million words of instruction data in GLUE format, and a 16K child-directed-speech-trained BPE tokenizer. The v2 model regressed by 7.70 percentage points on GLUE-axiomatic scoring.
To isolate which intervention caused the regression, four additional models were trained on the original v1 corpus, each varying one component. v3d, which used the v3a corpus but swapped in the 16K CDS tokenizer, reproduced the entire v2 collapse: 43.50% versus v2's 43.30%. The format-targeted instructions and EWoK-targeted Wikipedia did not regress when introduced individually. Only the tokenizer swap did.
A second observation compounds the finding. v3a, a clean rebuild of v1 with a nominally identical recipe, scored 4.90pp lower than v1 itself. The only difference was a different pre-tokenization shuffle seed that reordered sentences of an identical corpus. No source material changed. No word budget changed. The score is a property of the tokenizer's vocabulary against the scoring protocol's answer-template tokens, not of task-level competence.
Any method comparing log-probabilities of single-token answers across prompts is vulnerable to this artifact. A benchmark that does not report tokenizer-swap sensitivity cannot distinguish task competence from tokenizer-vocabulary differences.
Dictionary-Axiom Prompting: A Placebo Falsifies the Result
An inference-time approach prepended a French-English vocabulary axiom block to each English benchmark item, producing apparent gains of 5 to 9 percentage points on three GLUE tasks. A placebo control killed the interpretation. Three axiom variants were tested under the same French answer template: targeted (relevant FR-EN pairs), placebo (random unrelated pairs of the same length), and tautology (en=en pairs with zero translation information).
Targeted minus bare: +2.33pp. Placebo minus bare: +2.60pp. Tautology minus bare: +1.60pp. The translation-specific effect (targeted minus placebo) is minus 0.27pp. Random, unrelated axioms produce as much improvement as targeted ones. The semantic content of the axiom block is irrelevant; the gain is structural prompting noise. The original 5 to 9pp effect was further inflated by an answer-template confound (English template without axioms versus French template with axioms); with template held constant, the structural prompting effect is approximately 2pp.
Recommendations for Child-Scale Non-English Evaluation
The paper distills its findings into concrete reporting requirements for child-scale submissions. Two diagnostics should accompany headline accuracy on any single-token log-probability benchmark: tokenizer-swap sensitivity (re-evaluating under a second BPE tokenizer trained on a different distribution) and placebo-controlled prompting (re-evaluating any prompt-prepended scoring under an unrelated-content prompt of matched length). Without them, a benchmark cannot separate task competence from tokenizer-vocabulary or prompt artifacts.
Native-language benchmarks are the preferred evaluation axis. Where a native benchmark exists, it is the cleaner instrument. The gap between QFrBLiMP (85.97%, native) and translated BLiMP (76.28%) is a 9.69-point difference on essentially the same kind of probe, differing only in whether items are native or translated. A placebo-controlled translation pass is a workable second-best when no native benchmark exists.
The model, corpus, bilingual lemma bridge, and all evaluation scripts are available at github.com/adamzwasserman/babylm.