Most language model training happens in two zones: tiny experiments on a single GPU, or massive runs that cost millions. Hugo Vergnes wanted to explore the space between them. His result is a 3.8B-parameter model trained from scratch for under $1,000, scoring 0.384 on the CORE benchmark in 43 hours on rented B200 GPUs.

From NanoGPT Toy to Meaningful Model

Vergnes built little-lm as a config-driven framework for training small decoder-only LLMs, inspired by Andrej Karpathy's nanochat project. Every run is specified by a YAML file covering model architecture, dataset, optimizer, schedule, and callbacks. Components self-register into a global registry, so swapping an optimizer or dataset is a one-line change.

The final model is Llama-style: RMSNorm, RoPE, grouped query attention with 24 query heads and 8 KV heads, relu-squared MLPs, QK-norm, logit softcap, per-layer learnable residual scalars, and ResFormer-style value embeddings. Those value embeddings are notable, accounting for 721.2M of the model's 3.848B parameters, roughly 19% of the total. They come as 14 tables of vocabulary times KV dimension, one on every other layer.

Five Changes That Mattered

An early 858M-parameter run on a single A100 over six days produced a model worse than GPT-2 124M, from 2019. The loss curve went flat after 70% of training. Vergnes identified five problems and fixed them, and the difference between that failed run and the final result is entirely attributable to these changes.

First, a trapezoidal learning rate schedule replaced cosine decay to zero. The model warms up for 5%, holds flat, then linearly cools down over the last portion to 5% of peak. The key insight: the model keeps learning until the end instead of coasting through the tail. The eval loss was still descending at the final step of the 3.8B run.

Second, Muon for matrix parameters and AdamW for everything else. Muon uses Newton-Schulz orthogonalization, which is about 25% slower per step, but at 7 gradient-accumulation steps that cost dilutes to roughly 4% of total runtime. Convergence is significantly faster overall.

Third, ClimbMix instead of FineWeb-Edu for training data. Vergnes calls this a "tremendous jump in convergence speed," consistent with what Karpathy found independently.

Fourth, FP8 training with vocab padding. All three GEMMs (one forward, two backward) run in FP8 with dynamic tensorwise scaling. The vocabulary was padded from 50,257 to 50,304, a multiple of 64, so tensor cores operate efficiently. Together these delivered a 33% throughput improvement.

Fifth, 1024-token context instead of 2048. Halving context roughly doubles batch size at fixed memory, with minimal per-token throughput change. The model is still dominated by MLP compute, a good sign the hardware is being used well.

The Numbers

The final run processed 65.3B tokens on 8x B200 GPUs in 43 hours at a cost of $998. Steady-state throughput was about 480,000 tokens per second. GPU utilization hit 92% SM activity with 40% occupancy, yielding roughly 1,047 TFLOP/s sustained per B200, or about 25% MFU against Blackwell's dense FP8 peak.

For comparison, nanochat d32 on 8x H100s for about $1,000 scored 0.310 on CORE. The little-lm 3.8B at 2048 context scored 0.384, a meaningful improvement at a similar price point. The distributed strategy was plain DistributedDataParallel; at 3.8B on a single node, gradient communication was never the constraint.

What Did Not Work

Several optimizations were tested and discarded. Document-boundary masking with flex attention was elegant but unnecessary; Karpathy also found that cross-document attention under BOS-aligned packing does not hurt much. Best-fit packing replaced it in about 10 lines of code. Liger RMSNorm and RoPE were micro-benchmarked but showed no end-to-end throughput improvement; RMSNorm was actually slower than PyTorch 2.9's built-in version. Nanochat-style initialization started the loss curve marginally lower but converged to the same place within 1,500 steps.

Streaming datasets were also abandoned. Even with healthy network conditions, local shards gave 2-3% more throughput, and occasional network dips cost far more. For runs longer than a few hours, downloading shards once at the start is worth it.

Context Length and the SQuAD Surprise

An ablation on context length revealed an unexpected failure. Three of 22 CORE tasks have prompts that essentially never fit in 1024 tokens. SQuAD was the worst case: 100% of its prompts were cropped. Since SQuAD is a 10-shot task with a median of 1,998 tokens, truncation removed the few-shot examples while keeping the passage and question.

The result was a model that scored zero on SQuAD after training, because it never saw the demonstrations teaching it the expected output format. Worse, the score declined monotonically from 0.1478 at step 2,500 to exactly 0.0000 at step 25,000. An early, high-entropy model occasionally emits something short that accidentally matches the gold answer. As it sharpens, it commits to well-formed continuations and the accidental hits disappear. Getting better at language made it worse at guessing right by accident.

Re-running at 2048 context scored 0.384, nearly all of the gap attributable to context-dependent tasks like this one.

The Bigger Picture

The project sits in a space that is increasingly accessible but still underexplored. Vergnes notes that the cost of training a capable model from scratch keeps dropping, and that good infrastructure pays for itself at the first convergence problem you encounter. The ability to express experiments as a three-line YAML diff rather than a branch changed how quickly he could iterate.

Value embeddings were a useful experiment. They cost essentially no FLOPs, only memory and optimizer state, and delivered the equivalent of about 1,200 extra training steps. CORE moved seven times more than loss did between the two configurations, a reminder that accuracy metrics amplify relative differences when scores are still low.

The full run log, code, and configs are available on Vergnes' site. The project was written in the evenings, debugged on a 5090, and finished on rented B200s. It is a concrete data point that meaningful model training is no longer restricted to research labs or companies with massive compute budgets.