Model architecture shapes the scaling behavior, training efficiency, and inference cost of LLMs, making it a central focus of both industry and academia. In industry, teams such as those behind DeepSeek, Qwen, and Kimi routinely introduce architectural innovations, yet public reports rarely document which ideas succeeded, which were abandoned, and why. In academia, limited compute and inconsistent baselines often restrict evaluations to small-scale, single-point comparisons, making it difficult to establish whether improvements hold at larger scales or translate into broader impact. ArchSpace addresses both gaps as a public platform for Recording, Reviewing, and Referencing Architecture Research: each proposal is recorded with its experiments and reasoning, reviewed by the community, and referenced by later work. To evaluate ideas before costly large-scale training, ArchSpace leverages scaling laws to predict model loss at larger scales from smaller experiments, a practice widely adopted in industry. We provide initial references that make the platform usable: Qwen3.5 and OLMo3 baselines, fitted scaling laws, and a reusable training recipe. Ideas validated through this review can in turn become new baselines, and ArchSpace thereby aims to broaden the range of ideas that shape the next generation of models.
Why Architecture Documentation Matters for LLM Progress
Large language models have become the default interface for many AI-powered products. From chat assistants to code generation tools, the capabilities users expect depend as much on design choices inside the model as on the amount of compute spent during training. Architecture decisions—such as whether to use a mixture-of-experts routing scheme, how attention layers are configured, or whether to incorporate learned positional embeddings—affect not only final performance but also the resources required to reach it. When a new architectural choice yields a measurable gain, teams can accelerate development by building on proven designs. When a choice proves costly or ineffective, others can avoid repeating the same mistake.
Despite the importance of these trade-offs, a systematic record of architectural experiments remains scarce. Industry labs publish occasional blog posts or technical notes, but these artifacts are inconsistent in format and scope. Academic publications focus on evaluation results rather than the design process, and reproducibility packages often omit the “why” behind modeling choices. The result is a fragmented knowledge base where valuable insights are lost, and subsequent work must rediscover what has already been tried.
The Industry Documentation Gap
Leading AI organizations such as DeepSeek, Qwen, and Kimi regularly introduce architectural modifications that shift the scaling frontier. Examples include hybrid attention mechanisms, dynamic token allocation, and progressive deepening of expert modules. These innovations can reduce training FLOPs per accuracy point, lower inference latency, or improve data efficiency. However, the public record of which experiments succeeded, which were abandoned, and the reasons for those outcomes is thin. Company blog entries may celebrate a breakthrough, but they rarely publish negative results or detailed ablations that would allow a rival or a newcomer to emulate the successful parts while sidestepping the dead ends. This selective reporting creates an uneven knowledge distribution: newcomers to the field encounter a stream of headline results without the context needed to judge which directions are worth pursuing.
Academia's Scale Problem
Academic research groups typically operate with far less compute than industry counterparts. Training a single LLM from scratch can cost tens of thousands of dollars in GPU hours, a barrier that limits the number of architectural variants that can be explored. When resources are scarce, researchers tend to converge on a single baseline—often a publicly available model such as Llama or Mistral—and make small, incremental modifications. Each variant is evaluated on a fixed benchmark, and the results are compared against that one baseline. While this approach can demonstrate modest gains, it suffers from two limitations. First, the evaluated range of architectures is narrow; second, the observed improvements may not scale. A change that reduces loss by 5% on a 7-billion-parameter model might vanish when the same design is applied to a 70-billion-parameter model, simply because the interaction between architecture and dataset scale behaves differently at larger widths or depths.
Furthermore, inconsistent baselines across papers make cross-study comparison unreliable. One group might use a checkpoint from the original release, another might fine-tune the baseline on a different subset of data, and yet another might apply a different preprocessing pipeline. These variations obscure whether an observed gain stems from the architectural change or from an unrelated factor such as data curation or hyperparameter tuning.
ArchSpace: A Public Platform for Architecture Research
ArchSpace proposes a structured approach to bridging the documentation divide. The platform treats each architectural proposal as a first-class entity: it records the design choice, the experiment configuration, the training procedure, the resulting metrics, and the authors' rationale for the design. These records are then exposed to community review, allowing other researchers to upvote, critique, or extend the proposal. Later work can reference a recorded proposal as a baseline, creating a chain of accountability and buildability.
The platform operates in three interconnected phases. In the recording phase, a researcher or team submits a description of their architectural idea, accompanied by a minimal reproducible training script, the dataset and data preprocessing steps, and a narrative explaining the expected benefit. This information is stored in a searchable repository, where each entry is assigned a persistent identifier. In the review phase, the community evaluates the submission. Reviewers look at the clarity of the experimental design, the soundness of the claimed advantages, and the thoroughness of the ablations. Accepted proposals receive a badge or tag that signals they have passed a basic quality filter. In the referencing phase, later work can cite a reviewed proposal when selecting a baseline for their own experiments. If a new architectural variant builds upon a previously reviewed design, the connection is explicit, and the provenance of the baseline is transparent.
Recording Proposals with Experiments and Rationale
The recording step is the foundation of the platform. A submission to ArchSpace includes several mandatory fields. First, a clear description of the architectural modification: what change is being proposed, which component of the model it affects, and what theoretical or empirical motivation drives the choice. Second, a reproducible training recipe: dataset, tokenizer, optimizer settings, learning rate schedule, and any regularization techniques. Third, the results of a pilot experiment, typically on a model size an academic lab can afford—such as 70 million or 1 billion parameters—along with the observed loss and any downstream task metrics. Fourth, a candid discussion of outcomes: what worked, what did not, and why the authors believe the design has potential at scale.
By requiring these elements up front, ArchSpace ensures that a submitted proposal carries enough information for another researcher to rerun the experiment without guessing. The inclusion of “what did not work” alongside “what did work” is intentional: negative results are as informative as positive ones when the goal is to narrow the space of promising designs.
Community Review as Quality Filter
Once a proposal is recorded, it enters a review pipeline. The review criteria are deliberately lightweight compared to a full conference paper review, aiming for speed and breadth rather than deep novelty assessment. Reviewers check that the training recipe is fully specified, that the pilot experiment covers a reasonable range of model sizes, and that the rationale for the architectural choice is articulated without overclaiming. Submissions that meet the threshold receive a “reviewed” tag; those that do not are returned to the author with feedback but are not prevented from being resubmitted after revisions.
This review layer serves two purposes. First, it filters out submissions that lack the basic ingredients for reproducibility—missing hyperparameters, ambiguous architectural descriptions, or pilot experiments too small to be informative. Second, it creates a community-sanctioned set of proposals that later workers can trust as starting points. A researcher looking for a new attention mechanism to experiment with can browse reviewed proposals, confident that each has passed a minimal quality gate.
Referencing and Baseline Propagation
The true power of ArchSpace emerges when reviewed proposals become baselines for future work. Suppose a team proposes a novel depthwise separable attention layer, records the experiment on a 1-billion-parameter model, and receives a reviewed tag. Six months later, another group wants to explore whether that attention layer improves multilingual performance. Instead of reinventing the attention module from scratch, they can reference the ArchSpace entry, adopt the same implementation, and report their results relative to that baseline. The provenance is clear: the baseline originates from a recorded and reviewed proposal, and any deviations or extensions are documented alongside the new results.
Over time, this referencing cycle accumulates. Validated ideas spawn new proposals, which in turn are reviewed and referenced. The platform thus functions as a living knowledge graph of architectural choices, each node tagged with its experimental context and review status. This structure addresses the academia scale problem by providing ready-made baselines that have already been vetted for reproducibility and narrative coherence.
Scaling Laws as Evaluation Shortcut
A distinctive feature of ArchSpace is the use of scaling laws to extrapolate pilot experiment results to larger model sizes. The practice of fitting loss versus compute or loss versus parameter count curves is standard in industry, where teams predict whether a promising architectural tweak will remain beneficial when the model is scaled to billions or trillions of parameters. ArchSpace makes this practice accessible to academic groups by providing tools and fitted curves for common architecture families.
When a researcher submits a pilot experiment, they are encouraged to report loss at the trained compute level. ArchSpace then fits a scaling law of the form L = a × N^b + c, where L is the expected loss, N is the number of parameters, and a, b, c are fit parameters. By adjusting N to larger values, the platform can surface a predicted loss for a hypothetical larger model. This prediction is not a substitute for actual training, but it offers a cost-effective signal: if the projected loss degradation is steep, the authors may reconsider pursuing the architecture at scale; if the projection is flat, the architecture may warrant a full-scale run.
The availability of initial scaling law fits is part of the initial references ArchSpace provides upon launch. These fits are derived from the Qwen3.5 and OLMo3 baselines mentioned in the platform's introductory material, giving users a concrete starting point for understanding how loss behaves across parameter sizes for well-established architectures.
Initial Baselines: Qwen3.5, OLMo3, and Beyond
To make the platform usable from day one, ArchSpace ships with a set of reference baselines. The Qwen3.5 baseline represents a state-of-the-art dense model architecture, trained on a large corpus with a standard scaling law profile. The OLMo3 baseline offers an open-source alternative with a different tokenizer and data mixture, allowing researchers to test whether architectural ideas transfer across data and tokenizer choices. Both baselines come with fitted scaling laws, meaning that a user can immediately explore how loss evolves as model size changes, without needing to reproduce the original training runs.
In addition to these two main baselines, the platform includes a reusable training recipe that specifies optimizer warmup, gradient clipping thresholds, and loss scaling settings. This recipe is designed to work out of the box with the provided baselines and can be adapted for new architectural proposals. By supplying these building blocks, ArchSpace lowers the barrier to entry: a researcher who has never trained an LLM from scratch can still submit a proposal by plugging their architectural modification into the reusable recipe and reporting results on a small model.
The platform also encourages contributors to submit their own baselines after running larger experiments. As more baselines accumulate, the scaling law fits become more diverse, covering different tokenizer families, data mixtures, and training regimes. This growth in baseline diversity is one of the long-term goals of the project.
From Validated Ideas to New Baselines
The cyclical nature of ArchSpace—record, review, reference, and then re-record—creates a feedback loop. A proposed architectural change that passes review and proves effective at scale becomes a new baseline. Later submissions can reference this baseline, ensuring that the community builds on a foundation that has already undergone scrutiny and scaling validation. Conversely, proposals that fail to deliver promised gains, or that prove brittle when extrapolated via scaling laws, are documented as cautionary tales. Future researchers exploring similar territory can read the failure narrative and avoid repeating the same design choices.
This loop also helps address the industry documentation gap. When an industry team introduces an innovation that becomes a de facto standard—such as a particular attention variant or a routing scheme—they can contribute a retrospective submission to ArchSpace, documenting what succeeded, what was learned, and how the design evolved. The platform thus becomes a living archive of both successes and lessons learned, accessible to both academic and industrial researchers.
Limitations and Trade-offs
ArchSpace is not a silver bullet for all challenges in LLM architecture research. The scaling law predictions, while useful, are based on empirical fits that may not capture all the nuances of a given architectural choice. Models exhibit phase transitions and unexpected behavior at scales beyond the range of the fit data, so a flat projected loss curve does not guarantee smooth performance at very large sizes. The platform explicitly cautions users against treating scaling law predictions as definitive verdicts; they are signposts, not verdicts.
Another limitation is the review throughput. Because the review process relies on volunteer community members, the time between submission and “reviewed” status can vary. Fast-turnaround reviews are a goal, but they depend on community availability. The platform addresses this by providing a basic reproducibility checklist that submissions must satisfy before they can even enter the review queue, reducing the likelihood of lengthy back-and-forth over missing information.
There is also the question of baseline drift. As new data preprocessing techniques, tokenizer updates, or hardware improvements become common, a baseline that was reviewed six months ago may no longer be directly comparable to a new experiment. ArchSpace mitigates this by requiring each baseline submission to include a full snapshot of the training environment—code version, library versions, hardware specifications—so that re-replication can be attempted under conditions as close as possible to the original.
What It Means in Practice
For a working developer or researcher, ArchSpace offers several practical benefits. First, it provides a catalog of architectural ideas that have been documented with enough detail to be reproducible. Second, the scaling law estimates give a low-cost way to triage which ideas merit a full training run. Third, the reviewed baseline mechanism means that new experiments can start from a vetted starting point rather than from scratch, saving compute and time. Fourth, the platform’s growth means that over time, the body of recorded proposals expands, increasing the chances that a novel idea has a close relative already explored and recorded.
Building on ArchSpace is straightforward. A researcher can browse the repository, filter by architecture type (e.g., attention variant, routing scheme, positional encoding), and download the associated training recipe. If a proposal is reviewed, the researcher can reference it directly in their own submission, creating a transparent chain of provenance. If a proposal is not yet reviewed, the researcher can still use the training recipe and scaling law tools, but should expect that the community review step may add an extra layer of quality assurance before the proposal becomes a referenceable baseline.
For teams in industry, ArchSpace provides a way to externalize internal knowledge. By contributing retrospective submissions that document which architectural experiments succeeded and which were abandoned, teams can contribute to the commons while also gaining visibility into the broader landscape of architectural choices. The platform does not replace internal documentation, but it supplements it with a public, citable record.
# Example: Submitting a proposal to ArchSpace
{
"architecture_name": "RoPE-Shift",
"description": "Rotary positional embeddings with a learnable shift parameter",
"model_size": "1B",
"dataset": "c4",
"optimizer": "adamw",
"learning_rate": 2e-4,
"loss_at_scale": 3.45,
"scaling_law_fit": {"a": 1.2, "b": -0.45, "c": 0.1},
"rationale": "Expected to improve long-context retention without increasing FLOPs",
"experiment_log": "steps=200000, batch_size=128, grad_clip=1.0",
"result_summary": "Loss improved by 3% over RoPE baseline on LM eval, but downstream QA suffered"
}
This JSON-like snippet illustrates the kind of minimal information a proposer would include when submitting an architectural idea to ArchSpace. The fields cover the what, how, and why, and the inclusion of a scaling law fit shows how the platform integrates prediction into the evaluation workflow.
Conclusion
Architecture research is a key driver of LLM progress, yet the knowledge generated remains fragmented and difficult to build upon. ArchSpace tackles this problem head-on by providing a public platform that records architectural proposals with full experimental context, subjects them to community review, and makes them referenceable baselines for future work. The use of scaling laws to extrapolate from small to large models gives researchers a cost-effective triage mechanism, while the initial baselines—Qwen3.5, OLMo3, and a reusable training recipe—lower the entry barrier for newcomers. The cyclical flow of ideas from proposal to review to baseline and back to proposal creates a self-reinforcing knowledge base that has the potential to broaden the range of architectural choices that shape next-generation models. Like any infrastructure project, the value of ArchSpace depends on community participation: the more proposals that are recorded, reviewed, and referenced, the richer the knowledge graph becomes. For anyone interested in the design space of large language models, ArchSpace represents a step toward a more transparent, cumulative, and efficient research practice.
Read the paper on arXiv