vLLM is introducing a new set of hardware-agnostic layers to address a growing tension in its architecture. As frontier model architectures diverge and push the limits of GPU-specific optimizations, the project needs to keep pace with performance demands while continuing to serve users who run models on diverse and older hardware.
The problem: model definitions splitting apart
vLLM has historically maintained a clean abstraction layer that lets the same model logic run across NVIDIA GPUs, AMD GPUs, Intel XPUs, Google TPUs, IBM Spyre, and Huawei Ascend. This worked because the project could keep model definitions relatively simple and rely on torch.compile to fuse and optimize the underlying operations for each platform.
That is changing. Frontier models increasingly ship with bespoke layers and optimized kernels that do not fit into shared abstractions. The attention mechanism alone is diverging: DeepSeek V4 and Kimi K3 achieve million-token context through entirely different approaches. Fitting these models into vLLM's shared layer system requires significant engineering effort, and the new flat model definitions are starting to use custom fusions that are fundamentally incompatible with torch.compile.
This creates a difficult trade-off. The flat models unlock better performance on the latest hardware, but they break the abstractions that out-of-tree accelerator plugins and older GPUs depend on.
Three model definition flavors and why it matters
Today vLLM supports three types of model definitions. The new flat models live under vllm/models/ and use hardware-specific customizations. The legacy models live under vllm/model_executor/models and are gradually being removed. The transformers backend imports models from the transformers library and rewires them to use vLLM's common layers.
All three flavors currently resolve to the same set of shared layer implementations. The transformers backend, in particular, has become the primary path for supporting older and more exotic models. Without torch-compilable layers, performance for those models on GPU would degrade significantly.
The four design principles
vLLM's solution is a new set of hardware-agnostic layers built in-tree at model_executor/hw_agnostic. These layers adhere to four principles. First, they must be fully torch-compilable so that out-of-tree plugins relying on compilation for performance can continue using them. Second, they keep extensibility mechanisms like CustomOp and PluggableLayer so plugins can override implementations when needed. Third, they are isolated from the hardware-specific paths, so development in each direction does not impede the other. Fourth, they use portable implementations via native PyTorch code or DSLs like Triton and Helion.
Performance on real hardware
The project validated the new pathway on NVIDIA H100 GPUs, comparing the transformers backend with USE_HW_AGNOSTIC=1 against the native implementation. On a geometric mean across three recent models, the hardware-agnostic layers achieved total token throughput within 3.4% of the native implementation.
In some cases, the portable implementations actually matched or slightly exceeded the performance of CUDA-optimized libraries like FlashAttention and CUTLASS. This is notable because the hardware-agnostic models are built entirely from portable code rather than platform-specific kernel tuning.
How it integrates today
The hardware-agnostic pathway is already in vLLM's main branch for a limited number of layers. Users can enable it by setting the environment variable USE_HW_AGNOSTIC=1 when running the transformers backend. For example: USE_HW_AGNOSTIC=1 vllm serve google/gemma-4-31B --model-impl=transformers.
The project has validated this with the IBM Spyre out-of-tree plugin on models like Gemma 4, Qwen3, and Granite 4.2. vLLM plans to start including hardware-agnostic models in its CI and gradually switch Spyre to this as its default pathway.
For flat models specifically, the project plans to provide a new model.py file for each model that implements it using the hardware-agnostic layers. Shared layers across multiple models will reside in the shared hw_agnostic directory, while model-specific layers like DeepSeek V4's attention mechanism will stay local to the model's directory but still follow the four design principles.
What this means for the ecosystem
The hardware-agnostic layers represent vLLM's attempt to avoid a fork in its project. Without them, the project risks splitting into two tracks: a frontier-focused track using custom hardware-specific kernels, and a compatibility track that falls behind on performance. Out-of-tree plugin maintainers would face the burden of keeping their own model definitions in sync across transformers, vLLM, and their own codebase.
vLLM's own usage data shows a significant portion of its user base still runs on older GPUs, consumer hardware, or out-of-tree accelerators. The hardware-agnostic layers are designed to ensure that this segment of the user base is not left behind as the project optimizes for the latest Blackwell and CDNA 4 hardware.