vLLM TorchTPU enters the scene as a PyTorch-native TPU backend for the popular LLM serving framework. The project, packaged as vllm-torchtpu, lets teams run vLLM's standard PyTorch model definitions on Google Cloud TPUs without rewriting model code. The goal mirrors what made vLLM successful on GPUs: the same CLI, the same OpenAI-compatible server, the same model definitions, but executing at TPU speed.

Running PyTorch models on Cloud TPUs with no code changes

The technical foundation rests on TorchTPU and torch.compile with a "tpu" backend. Custom kernels written in Pallas handle attention, Mixture-of-Experts, and quantized matmul operations. These include Ragged Paged Attention, dense and sparse MLA, Gated DeltaNet, fused MoE with SparseCore offload, and FP8/INT8/MXFP4 precision paths. The plugin preserves vLLM's familiar API while translating model execution onto TPU hardware.

Developers install the plugin with a single pip command, referencing a PyTorch extra-index URL. The command mirrors a standard vLLM server start, making the onboarding path nearly identical to GPU deployment. A matching Python 3.12 environment on a Cloud TPU VM is required.

pip install \
  --extra-index-url https://download.pytorch.org/whl/cpu \
  vllm-torchtpu

Once installed, starting an OpenAI-compatible server follows the same pattern as any vLLM platform:

vllm serve meta-llama/Llama-3.2-1B-Instruct \
  --tensor-parallel-size 1 \
  --max-model-len 4096

Requests then route through the normal vLLM endpoint, with the backend executing on TPU hardware transparently.

Parallelism schemes compose across single-host and multi-host TPU slices

Scale-out arrives through a combination of parallelism types: tensor parallelism (TP), data parallelism (DP), pipeline parallelism (PP), expert parallelism (EP), and prefill/decode context parallelism. These compose across single-host and multi-host TPU slices, with disaggregated prefill/decode serving as an option. The architecture aims to accommodate large model families and long context lengths without forcing model authors to restructure their code.

A technical constraint appears with prefill context parallelism. When using the --prefill-context-parallel-size flag, each attention chunk splits across the stage's cores. MoE models then require the --enable-expert-parallel flag. This depends on a recent torch_tpu build that compiles stages' kernels and collectives for the stage's devices only. Earlier platform builds that only support world-size meshes will reject such stages. A two-core stage must not map to the two cores of a single chip; developers set VLLM_TPU_DEVICE_ORDER to map each local rank to distinct devices, for example 0,2,1,3,4,6,5,7 on a host with four two-core stages.

Model support spans major open-source families

vllm-torchtpu serves most PyTorch models supported by the upstream vLLM project, with first-class kernel, quantization, and parallelism support for a broad set of model families. These include Qwen3 and Qwen3.5, DeepSeek-V3 and its R1 and V4 variants, Kimi-K2.5 and K3, GLM-5, Llama 3 and 4, GPT-OSS, Gemma 3 and 4, Mixtral, Qwen-VL, and Gemma VL. Diffusion models run via the vllm-omni extension. The project maintains nightly support matrices that track TPU v7x (Ironwood) unit correctness and benchmark status, updated automatically from continuous integration.

Quantization formats available through the Pallas-backed kernels include FP8, INT8, and MXFP4. These reduce memory bandwidth pressure while preserving acceptable accuracy for serving workloads.

What this means for development teams

Teams already invested in vLLM can target TPU hardware without migrating model code or adjusting inference logic. The same speculative decoding, prefix caching, multi-LoRA, and structured output features continue to work through normal vLLM flags. Multimodal inputs, including those handled by vllm-omni, also function without modification. This lowers the barrier for organizations exploring TPU cost-performance for LLM serving, particularly those with existing PyTorch codebases.

The experimental status on older TPU generations (v6e, v5e, v5p) means early adopters should validate workloads against the recommended TPU v7x platform before production deployment. The nightly support matrices provide a machine-readable path to assess compatibility before committing resources.

Contributions and ongoing development

The project welcomes contributions across bug reports, model and feature requests, documentation fixes, and code. Technical discussions occur on the vLLM Forum's TPU support topic, while development coordination happens in the vLLM Developer Slack channel under the #sig-tpu topic. Good first issues are tagged and serve as entry points for new contributors. Testing infrastructure, profiling guides, and benchmarking best practices live in the developer references section of the repository.

vLLM TorchTPU represents the latest step in making LLM serving hardware-agnostic. By bringing PyTorch-native execution to Cloud TPUs without requiring model rewrites, it expands the hardware options available to teams building large-scale serving infrastructure.