Serving LLMs on Tenstorrent Hardware: Inside the vLLM TT Plugin

vLLM Gets Tenstorrent Support Without Forking the Serving Stack

vLLM has added Tenstorrent hardware support through its out-of-tree plugin system, and the interesting part is not that another backend exists. It is that Tenstorrent's mesh architecture forced design choices that expose how much of vLLM's serving stack assumes GPU-shaped hardware. The plugin registers Tenstorrent devices automatically when the TT-Metal runtime is importable, and the serving surface stays identical: same OpenAI-compatible API, same request format, same client code. Everything Tenstorrent-specific lives outside vLLM core.

The plugin ships with support for Llama 3.1 through 3.3, Llama 3.2 Vision, Qwen 2.5 and 3 through 3.6, Qwen-VL variants, Mistral and Mistral 3, Gemma 3 and 4, DeepSeek V3, and GPT-OSS at 20B and 120B. Multimodal models are covered from the start, which is unusual for new backends that typically stay text-only for months. The architecture match works by TT-prefixed convention, so one entry can cover several model releases. TTQwen3_5ForConditionalGeneration serves Qwen3.6-27B, for instance.

Models are not built into the plugin. The classes behind the architecture names live in TT-Metal alongside the runtime. The plugin registers the names, and tt-metal provides what they resolve to. Pointing EXTRA_MODELS_DIR at a directory of bundle folders, each containing a vllm_metadata.json and an adapter class, registers additional architectures at startup without source edits. Setting TT_VLLM_BUILTIN_MODELS=0 narrows the registry to only what was supplied externally.

Why the Mesh Architecture Changes Everything

A Tenstorrent system is a mesh of cores and chips connected by an on-fabric network. A single n150 or n300 card is already a small mesh. A QuietBox is larger. A Galaxy wires 32 Wormhole chips into a topology the runtime configures directly through FABRIC_1D, FABRIC_2D, or FABRIC_1D_RING modes. Programs are compiled and traced against a mesh shape, and the fabric moves data between chips as part of the compiled program rather than as collective calls issued by the host.

This compilation model drives nearly everything downstream. There are no tensor-parallel or pipeline-parallel ranks to configure. A 70B model on Galaxy is not TP=32 processes. It is one program compiled for a 32-chip mesh. MESH_DEVICE=TG replaces --tensor-parallel-size, and the plugin rejects -tp and -pp outright rather than pretending to honor them. The parallelism that best fits the model-mesh combination is implemented in the model code itself.

The unit of work is a whole traced step. Device execution is dominated by replaying a captured trace for a fixed batch shape, which makes homogeneous, shape-stable batches dramatically cheaper than heterogeneous ones. And because the mesh program can carry sampling through to token selection, the token often comes back already chosen, and the host never sees the logits.

Each of these properties conflicts with an assumption somewhere in vLLM's GPU-shaped inference stack. The plugin resolves them entirely through vLLM's extension points. TTPlatform.check_and_update_config() validates configuration, registers model architectures, and swaps in Tenstorrent-owned runtime classes for the worker, scheduler, and sampling path. Device-specific options ride on vLLM's generic additional-config namespace rather than through new CLI flags.

Phase-Based Scheduling: Why Prefill and Decode Stay Separate

Upstream vLLM's V1 scheduler is token-budget based. A request has computed tokens and target tokens, and each step hands out more token work subject to budgets. Prefill and decode are not separate modes, which allows chunked prefill and mixed-progress batches to fall out naturally.

The Tenstorrent path is more constrained. Every scheduling step resolves to one of three outcomes: prefill-only, decode-only, or empty. There are no mixed prefill-plus-decode batches. The reason is traced execution. A step that is uniformly prefill or uniformly decode replays a trace captured for exactly that shape. A step mixing the two would need a shape the trace was never captured for.

This is not a Tenstorrent eccentricity. The largest GPU deployments make the same choice deliberately, running prefill and decode on entirely separate instances. This is called disaggregated serving. The Tenstorrent scheduler applies the same split at step granularity within one engine rather than at instance granularity across a fleet.

Continuous batching still holds in the broad sense. Requests arrive into waiting, may be parked while structured-output grammar compiles, are admitted while other requests remain active, can be preempted back, and complete independently. The restriction is within a device step, not across the request lifecycle. What it costs is interleave granularity. Upstream mixes a prefill chunk and ongoing decode into the same step. The Tenstorrent scheduler alternates, so a decode request waits out each prefill chunk between its own steps, and each mode switch drains the async decode overlap pipeline. Both are scheduling-policy costs, not fundamental limits.

Single-Process Lane Parallelism on Galaxy

Some Tenstorrent models, including Llama 3.3 70B and Qwen3-32B on Galaxy, are served by single-execute generators: one program spanning the entire mesh, executed once per step. There is no submesh to give a second engine process. Standard multi-process data parallelism, which assigns each rank its own devices, simply has nothing to partition.

But these models keep four independent data-parallel KV caches, each on its own submesh. So there is nothing to partition at the process level, and four things to schedule independently. The initial implementation gave each rank its own process, as vLLM normally does. But because the ranks must negotiate the prefill versus decode step type, and there is actually only one mesh submit and readout, the per-rank schedulers needed significant modifications to vLLM core. The extra inter-process scatter and gather on every step cost more than the parallelism won back.

The solution is lane-based parallelism inside one engine process. TTLaneCoordinator owns one independent TTScheduler per lane. Each lane has its own waiting and running queues, its own admission decisions, its own KV cache manager, and its own lane-local block ID space. New requests are assigned to the least-loaded lane and stay bound to it.

Because the device executes all lanes together, the coordinator picks one shared mode per step. If any lane can admit prefill, all lanes run a prefill step. Otherwise, all lanes run a decode step. A lane with no work for the selected mode contributes an empty slice of the merged batch. The coordinator merges per-lane SchedulerOutput objects, the worker builds one merged device input, and the runner splits the result back out by lane. All in one process, with no process-level collectives anywhere.

One subtlety took effort to get right. If a forced prefill step admits zero tokens because of KV pressure while some lane still has running decode work, the step is retried in decode mode. Without that retry, KV pressure drives the coordinator into a no-progress loop: prefill is selected because a lane wants to admit, admits nothing because no blocks are free, and the decode that would have freed those blocks never runs.

The user-facing surface is deliberately boring. The same flags users already know map to whichever topology the model actually needs, in-process lanes for single-execute Galaxy models, ordinary multi-process DP with per-rank submeshes for everything else.

On-Device Sampling and Async Decode

When sample_on_device_mode is set, the mesh program carries sampling through to token selection and returns tokens rather than logits. Requests that cannot use that path, such as those needing logprobs, penalties, allowed-token masks, bad-word filtering, or custom logits processors, fall back to vLLM's own LogitProcessor and sampler path automatically. The plugin decides per batch, returning to the device path when the batch needs nothing the device path cannot express. always_compat_sampling forces the host path for debugging.

The async decode support is gated on a per-model supports_async_decode declaration. If a model has not declared it, the platform disables async scheduling rather than letting a user turn on something unvalidated. Underneath, async here means asynchronous host readback, not a device-side execution thread. The engine submits decode work non-blocking, starts host readback asynchronously, and later waits on those events at finalization. The engine keeps an in-flight queue of depth 2 and fills it before blocking, so the host can overlap readback with the next decode step.

What This Means for Serving Infrastructure

The vLLM TT Plugin demonstrates that vLLM's plugin interfaces are general enough to express hardware that looks nothing like a GPU. A mesh architecture with compiled traces, on-device sampling, and phase-constrained scheduling is a fundamentally different execution model. The plugin implements it entirely outside vLLM core, which means support tracks vLLM's release cadence rather than the plugin author's fork.

For teams evaluating Tenstorrent hardware for inference, the plugin removes the last integration barrier. The same serving setup, the same API, the same monitoring tooling. For the broader vLLM community, the plugin is a stress test of the plugin architecture itself. If a mesh of 32 chips with compiled traces and no tensor-parallel ranks can plug in cleanly, the extension points are doing their job.