ML Performance Tools Break Every Year. What If You Regenerated Them Instead of Maintaining Them?
Machine-learning performance modeling sits at the worst possible intersection in computing. Above it, model architectures mutate monthly: mixture-of-experts routing, latent attention, heterogeneous inference phases. Below it, hardware accelerators and interconnects evolve on their own cadence. A framework that assumed every transformer layer looks the same becomes obsolete the moment someone introduces sparse routing or variable-length sequences. The result is a permanent refactoring tax: engineers patch aging abstractions to absorb each new architecture, accumulate tech debt, and repeat.
A team from MIT, Google, and Stanford proposes a different answer: stop treating code as the durable artifact. Their system, SMART, is a rigorous symbolic performance-modeling library whose main branch contains almost no code. The repository is a DAG of natural-language design docs. Coding sub-agents regenerate the implementation from scratch on every version update. Every human change is a natural-language edit to a doc. Code is a build product, not the source of truth.
Incremental Patching Has a Mathematical Cost
The paper frames the problem precisely. Let S_t be the specification at time t, and let G be a code generator (human or agent). A greenfield build computes C_t = G(S_t). In practice, the generator at time t+1 receives the previous implementation as an extra argument: C_{t+1} = G(S_{t+1}, C_t). The technical debt at each step is the distance between the incrementally patched system and what you would have built from the current spec alone. Because starting over is mentally hard and time-consuming for humans, this debt is in practice far from zero. Every patch introduces compromises that compound.
AI coding agents make this worse through context-window myopia. An agent fed fragmented code snippets cannot see global invariants, cross-module dependencies, or architectural intent. It generates locally plausible code that degrades structural coherence over time. SMART inverts this: sub-agents read only a single design doc, never an aging codebase. The orchestrator discovers the dependency DAG automatically, walks it in topological order, and assigns one sub-agent per doc. Because C_t = G(S_t) is recomputed from scratch on a regular cadence, the incremental debt term is driven to zero by construction.
In practice, a full clean-slate regeneration takes 1.5 to 3 hours. The API cost using Claude Code is around 100 USD, roughly 20% of a weekly usage budget under a standard high-tier plan. Continuous, full-library regeneration is both practical and economically viable.
How the Design-Doc DAG Works
SMART's repository is a folder structure of markdown files composing into a DAG. Some modules must be generated before others: hardware topology and numerics precede the collective-cost models, which precede the model catalog. The edges are machine-discovered, not hand-maintained. Read-only agents analyze the documents and infer dependency edges. Once the graph resolves, an orchestrator walks it topologically and assigns a dedicated coding sub-agent to each self-contained doc.
This architecture yields three benefits. First, bounded context windows: each generation step is scoped to a single document, keeping the LLM's task length manageable and the probability of correct generation high. Second, targeted human iteration: the orchestrator's central log records where sub-agents struggled, giving engineers fine-grained visibility into which docs need prose refinement. Third, dynamic model routing: foundational documents like the core DSL design route to larger, more capable models, while downstream docs can use smaller, cheaper ones.
The Worked-Example Design Style
The authors argue that conventional AI-assisted development focuses on tests and high-level rules, which is a top-down constitution the generator must obey. SMART adds a complementary bottom-up ingredient: worked examples. Writing out how pseudo-code executes on a given input, step by step, with intermediate shapes, intermediate values, and exact closed-form cost expressions, maintains consistency across independent agentic generations.
Like in-context learning benefits from a concrete trace that pins down semantics prose leaves ambiguous, every number-bearing doc ends with a reconciliation anchor: a small preset whose expected outputs are stated exactly and enforced by generated tests. A doc might describe how a 2x2x2 torus with wraparound produces a per-node link count of 3, not 6, and the exact cost of an all-gather of V bytes. These vignettes are executable-in-your-head and serve as in-context demonstrations for the generating agents.
A Minimal, Recursively Defined Operator IR
Reliable regeneration also constrains the artifact being specified. The abstractions must be few, orthogonal, and stable under architecture churn. SMART uses a single recursively defined Op type:
Op:
inputs: List[Tensor] # symbolic shapes
outputs: List[Tensor]
cost: OpCost # SymPy exprs per key cost
rrt: RRT # resource reservation table
params: Union[InnerLoop(n_iter, body: Graph),
GraphParams(graph),
LeafParams(...)]An Op is either an interior node, a loop with a trip count and a body graph, or a plain subgraph, or a leaf. Leaf nodes are where software and system meet: the system specifies an RRT and an OpCost. Targeting TPUs, the leaves are TPU-shaped: an MXU matmul tile, a VMEM tile load, or an ICI collective. The algorithm side composes leaves into loop nests; the system side prices them. Swapping either side, a new attention variant or a new interconnect generation, touches only its own docs.
Models are authored in a thin Python-embedded tracing DSL. Decorated blocks trace into named subgraphs, decorated loops become InnerLoop nodes, and builder calls emit system-priced leaves. Every dimension is a SymPy symbol, so trip counts like T_q/q_blk stay symbolic and one trace serves the entire design space. The flash-attention core, for instance, traces a (B, H, T_q, T_kv) score matrix that never leaves VMEM, with the asymmetric T_q/T_kv making the same nest serve both prefill and flash-decoding. Distribution is expressed as sharding annotations, not hand-placed collectives: tensors name the mesh axes each dimension is sharded on, and a sharded-einsum wrapper infers collectives from operand/output shardings. Only layout-moving collectives like All-to-All in DeepSeekMoE blocks are explicit.
Two Roll-Up Modes: Fast Sweeps and Slow Schedules
Two modes turn a tree of per-op costs into wall-clock time. In fast mode, loops are rolled up coarsely: each leaf's cost is scaled by the product of enclosing trip counts, and simple analytical schedulers model communication/computation overlap as composable transforms in the style of a roofline bound. Evaluation is closed-form and fast enough for sweeps over thousands of design points. In slow mode, each loop is modulo-scheduled into its resource reservation table, software-pipelining the body against per-resource capacity, and the achieved initiation interval rolls up recursively. This yields dependency- and resource-aware schedules for the design points that sweeps flag as interesting.
All cost formulas propagate upward symbolically. Every roll-up produces a closed-form SymPy expression in the free variables of the design space: batch size, sequence length, bandwidths, mesh axes, datatype widths. Numeric binding happens only at the edge, one substitution per design point, so a single symbolic build serves an entire sweep. A design doc can even state the exact expected expression for a collective's cost, and the generated tests assert it.
Validated Against DeepSeek-V3 on TPU Pods
The authors regenerate the library against hand-audited reference models and report round-off precision agreement. Among the validated targets is DeepSeek-V3 serving on a TPU pod slice, a demanding test case involving mixture-of-experts routing, variable-length sequences, and multi-node communication patterns. The symbolic cost expressions produce the same wall-clock predictions as hand-tuned references, confirming that regenerated code from natural-language docs can match expert-crafted implementations.
SMART today comprises 50 design docs totaling roughly 9,000 lines of specification prose, spanning TPU topology, collective cost models, numerics, schedulers, and a catalog of frontier model families including dense, MoE, latent-attention, and robotics/VLA variants. Master is reduced to the docs plus a handful of leaf utilities, and the library is rebuilt by sub-agent orchestration.
Trade-offs and Limitations
The approach has clear boundaries. A 1.5-to-3-hour regeneration cycle is fast enough for weekly cadence but not for interactive development. The 100 USD API cost per build is manageable for a well-funded team but scales with doc count and model choice. Dynamic model routing helps, but the orchestrator still needs to infer correct dependency edges, and errors in the DAG propagate through the generation order.
The worked-example style imposes a documentation burden. Writing step-by-step traces with exact expected outputs is more labor than writing prose alone, and the reconciliation anchors must be kept in sync with the underlying math. The authors frame this as a feature rather than a bug: vague intent must be written into a doc to survive, surfacing ambiguity early. But it raises the bar for contributors who are not performance-modeling experts.
The symbolic IR targets TPUs specifically. Generalizing to GPUs or custom accelerators means defining new leaf ops and pricing models, which the modular architecture supports in principle but the paper does not demonstrate at scale. The modulo-scheduling mode also assumes a fixed resource model; future hardware with dynamic resource allocation would require extending the RRT abstraction.
What This Means for ML Systems Engineering
SMART suggests a shift in how performance tools are maintained. Instead of treating a codebase as the primary artifact and writing docs to explain it, treat the docs as the primary artifact and regenerate the code on demand. This eliminates incremental tech debt by construction, makes every change self-documenting, and lets AI agents work within bounded, well-specified contexts rather than struggling with sprawling codebases.
For teams building performance models, the practical takeaway is: design your abstractions to be regenerable. A minimal operator IR with symbolic cost expressions, a builder DSL that traces without constructing nodes by hand, and worked-example docs with reconciliation anchors form a kit that other domains could adopt. The constraint is that specs must be precise enough for a machine to implement, which is a higher bar than prose for humans but a lower bar than maintaining code that silently diverges from intent.
The broader implication is that AI-native development might not mean humans writing code with AI assistance. It might mean humans writing specifications and AI regenerating code from them on a cadence that keeps the implementation perpetually aligned with the current understanding. SMART is an early demonstration that this is feasible for a non-trivial, real-world system.