Most AI-driven optimizations for large language model inference start with profiling. You measure how an existing system performs, identify bottlenecks, and tune parameters within the constraints of the current stack. This approach works, but it has a fundamental limitation: it can only optimize what already exists. A research team has now published RoofLang, a domain-specific language designed to break out of that constraint and let AI systems architect entirely new inference configurations rather than just polishing old ones.

Why profiling-bound optimization hits a ceiling

Profiling-based feedback confines the search space to the capabilities and performance characteristics of an existing software stack. If the current architecture makes poor design choices, an optimizer that profiles it and tunes within its boundaries will never discover a fundamentally better arrangement. The optimizer can squeeze out incremental gains, but it cannot question the architecture itself.

This is a real problem for LLM inference, where the gap between a well-architected system and a poorly architected one can be measured in multiples, not percentages. The choice of how to manage KV-cache memory, how to batch requests, how to pipeline decode stages, and how to allocate GPU resources across concurrent users all interact in ways that profiling alone cannot fully explore. An optimizer needs the ability to propose structural changes, not just parameter tweaks.

What RoofLang provides

The paper identifies three capabilities required for AI-driven system architecting, and RoofLang implements all three.

First, a general workload representation that abstracts away implementation details. Instead of describing a specific software stack, RoofLang describes the computational patterns and resource requirements of an inference workload in a way that is independent of any particular framework or runtime. This lets the optimizer reason about the workload itself rather than about the quirks of a specific implementation.

Second, a verifiable mutation space. The DSL defines a set of structural transformations that can be applied to an inference configuration, and each transformation is defined precisely enough to be verified. The optimizer can propose changes to the architecture, and the system can confirm that the proposed changes are valid within the constraints of the workload and the hardware. This is what enables exploration of fundamentally different architectures rather than just parameter tuning.

Third, an implementation-independent evaluator. Because the workload representation and mutation space are decoupled from any specific software stack, the evaluator can assess proposed architectures without needing to build and run them. This dramatically accelerates the search process, since the optimizer can evaluate thousands of architectural alternatives without the cost and latency of full system builds.

The DeepSeek V4 finding

The evaluation produced a striking result. RoofLang's analysis revealed that DeepSeek V4-series models could achieve 3.5 to 39.5 times higher peak decode throughput than other representative models. This is a large gap, and the paper argues it is disproportionate to the models' total parameter counts.

The explanation lies in KV-cache design. The key-value cache is the memory structure that stores intermediate attention states during token generation. It is one of the primary bottlenecks in LLM inference, because its size grows with sequence length and limits how many concurrent requests a system can handle. DeepSeek V4-series models use compact KV-cache designs that support larger batch sizes and reduce memory traffic. The result is higher throughput without proportional increases in compute resources.

This finding matters because it illustrates exactly the kind of architectural insight that profiling-based optimization would miss. A profiler looking at a specific system running a specific model would measure the throughput and identify bottlenecks within that configuration. It would not compare the architectural implications of different KV-cache designs across model families, because that comparison requires the implementation-independent representation that RoofLang provides.

The optimizer agent discovers new architectures

The paper describes a persistent optimizer agent that uses RoofLang's mutation space to explore architectural alternatives for DeepSeek V4 Pro on NVIDIA B300 hardware. The agent discovered several new configurations that improved both throughput and interactivity, with gains ranging from 6.23% to 50.1%.

The range is notable. A 6.23% improvement is meaningful in production, where inference costs scale linearly with throughput. A 50.1% improvement represents a qualitative change in what the hardware can support, potentially enabling new use cases or dramatically reducing the cost of existing ones. The fact that the optimizer found multiple configurations across this range suggests that the architectural design space for LLM inference is richer than current practice explores.

What this means for inference engineering

RoofLang represents a shift in how we think about LLM inference optimization. The current paradigm treats inference as an engineering problem where the architecture is fixed and the task is to tune it. The RoofLang paradigm treats architecture as a design variable, something that can be explored and optimized by AI systems with the right abstractions.

For teams running inference at scale, the practical implication is that the current architecture of their inference stack may not be close to optimal. The tools and techniques for optimizing inference have been constrained by profiling-based approaches that assume the architecture is given. RoofLang suggests that significant gains are available to teams willing to let AI systems propose and evaluate structural alternatives.

The paper also raises an important question about model design itself. If compact KV-cache designs produce throughput gains disproportionate to parameter counts, then the research community's focus on parameter count as a primary measure of model capability may be missing a more important variable. Architectural efficiency, not just scale, may determine which models are practical for production deployment.

RoofLang is available as a domain-specific language, and the paper is published on arxiv with associated code and evaluation artifacts. For teams building or operating LLM inference systems, it is worth reading closely, not just for the specific results on DeepSeek V4, but for the broader argument that inference architecture is a design problem, not just an optimization problem.