A new benchmarking service is publishing performance data for AI models across different hardware configurations, aiming to give companies deploying large language models concrete numbers for making infrastructure decisions. Neural Nova's approach focuses on what happens after a model is chosen: how fast it actually runs, how much it costs, and which combination of software and hardware produces the best results for specific workloads.
The service arrives at a moment when the gap between a model's theoretical capabilities and its real-world performance has become a critical variable in deployment decisions. Companies selecting AI models for production workloads face a matrix of choices involving model size, hardware type, serving framework, and configuration parameters. Small differences in any of these can produce large differences in throughput and cost, and the available information about these tradeoffs has been fragmented across vendor marketing, independent benchmarks, and anecdotal reports from engineering teams.
What the Benchmarks Show
Neural Nova's published results cover four model and hardware combinations, each optimized for different use cases. The most dramatic improvement came from Qwen3-235B-A22B, a reasoning model running on eight NVIDIA H100 GPUs using the vLLM serving framework. The optimized configuration produced 138.7% more tokens per second than the baseline, with a 58% reduction in cost per token.
The cost savings figure is particularly significant because it reflects not just raw speed but the efficiency of the entire deployment stack. Running more tokens per second on the same hardware means lower per-query costs, which scales directly to bottom-line expenses for companies serving millions of requests daily.
A second reasoning model, GLM-5.2, was benchmarked on AMD's Instinct MI325X GPUs, a notable inclusion given that most AI infrastructure benchmarks focus exclusively on NVIDIA hardware. The optimized configuration delivered 26.8% more tokens per second with 21.1% cost savings. The smaller improvement margin compared to the Qwen benchmark suggests either a less optimized baseline or inherent differences in how the model maps to AMD's architecture, but the result still demonstrates meaningful gains.
For multi-modal workloads, Gemma-4-31B-it running on a single H100 showed a 66% throughput improvement and 40% cost reduction. This result matters for companies deploying models that process both text and images, a category growing rapidly as businesses integrate visual analysis into their AI workflows.
The fourth benchmark covered GPT-OSS-120B, an open-weight model, on a single H100. The 24.5% throughput improvement and 20% cost savings are more modest but still meaningful for teams running open-source models in production, where even small per-query savings accumulate across large request volumes.
Why This Matters for Deployment
The AI industry has spent the past two years focused largely on model capabilities: which system produces the best answers, handles the most complex reasoning, or generates the highest-quality code. But for companies actually deploying these models, the question is often not which model is most capable but which model can serve their workload at acceptable speed and cost.
These are different questions with different answers. A model that produces slightly better responses but runs at half the throughput may not be the right choice for a customer service application handling thousands of concurrent requests. A reasoning model that excels at complex analysis may be overkill for a summarization task that could be handled by a smaller, faster system.
Neural Nova's benchmarking approach addresses this gap by testing across multiple variables simultaneously. The results do not just say "this model is fast" but "this model, on this hardware, with this software configuration, produces these results." That level of specificity is what engineering teams need when making deployment decisions that will affect performance and cost for months or years.
The inclusion of AMD hardware in the benchmarks reflects a growing interest in alternatives to NVIDIA's dominance in AI compute. AMD's Instinct MI325X represents the company's latest attempt to capture market share in AI training and inference, and real-world performance data from independent benchmarks carries more weight than vendor claims. The 26.8% improvement on AMD hardware suggests that optimization work can narrow the gap between competing platforms, though the gap remains significant.
The Economics of Optimization
The cost savings figures in Neural Nova's benchmarks range from 20% to 58%, depending on the model and hardware. For a company spending $100,000 per month on AI inference, a 58% cost reduction represents $58,000 in monthly savings, or nearly $700,000 annually. Even the more modest 20% improvement on the open-weight model translates to $24,000 per month in savings.
These numbers explain why optimization has become a distinct discipline within the AI infrastructure space. The difference between a default deployment and an optimized one can be tens or hundreds of thousands of dollars per month, depending on scale. Engineering teams that invest in optimization can deliver the same performance at lower cost, or higher performance at the same cost, creating direct business value.
The optimization process itself involves tuning multiple parameters: batch sizes, memory allocation, kernel selection, quantization settings, and serving framework configuration. Each parameter affects performance in ways that interact with the others, creating a large search space that makes manual optimization time-consuming and error-prone. Benchmarking services like Neural Nova compress this search by testing configurations systematically and publishing the results.
The vLLM Factor
All four benchmarks used vLLM as the serving framework, reflecting its growing dominance in open-source AI inference. vLLM has become the default choice for many teams deploying large language models because of its efficiency, flexibility, and active development community. The framework's PagedAttention mechanism, which optimizes how model weights and KV cache are managed in GPU memory, is a key enabler of the throughput improvements documented in the benchmarks.
The choice to standardize on vLLM across all benchmarks provides a consistent baseline for comparing model and hardware combinations. Different serving frameworks can produce different performance characteristics, and mixing frameworks in a benchmark would make it difficult to attribute improvements to specific causes. By holding the framework constant, Neural Nova isolates the effects of model architecture and hardware selection.
For teams evaluating their own deployments, the vLLM baseline suggests that this framework is a reasonable starting point for optimization work. Teams using other frameworks, such as TensorRT-LLM or TGI, may see different results, but the vLLM benchmarks provide a reference point that is increasingly relevant as the framework's adoption grows.
What Comes Next
Neural Nova's initial benchmarks cover a narrow slice of the model and hardware landscape. The AI ecosystem includes hundreds of models across dozens of hardware configurations, and the four combinations tested represent only the most prominent options. Expanding the benchmark suite to cover additional models, hardware platforms, and use cases would increase the service's value to engineering teams making deployment decisions.
The service's focus on production-ready performance rather than academic accuracy metrics reflects a maturation in how the industry evaluates AI systems. Accuracy on benchmark datasets matters, but so does the ability to serve that accuracy at scale, at predictable cost, with acceptable latency. Neural Nova's approach treats these operational characteristics as first-class metrics rather than afterthoughts.
For companies navigating the increasingly complex landscape of AI deployment, the availability of independent, standardized performance data reduces the uncertainty in infrastructure decisions. The difference between a well-optimized and poorly optimized deployment can determine whether an AI application is economically viable, making benchmarking data a critical input into business planning.