FPGA fabric runs Qwen3.5 at 9B parameters with no GPU in sight
A project called llm.vhdl has built a complete large language model inference engine in VHDL that runs on Xilinx Versal FPGAs. The design executes Qwen3.5-class models, 9 billion parameters on a single SQRL FK33 card and 27 billion across two cards, entirely in programmable logic. Weights stay resident in on-card HBM. No GPU. No CUDA. The tokenizer, chat template, and per-layer activations are bit-exact matches to llama.cpp running the BF16 GGUF.
Why put an LLM in FPGA fabric
GPUs dominate inference today, but they carry a heavy bill: high power, expensive cards, and a software stack that changes every quarter. FPGAs offer deterministic latency, lower power per operation, and a hardware description that does not rot when a vendor updates a driver. The llm.vhdl project targets two platforms: the SQRL FK33 with a single XCVU33P die and 8 GiB of HBM, and the SQRL Jungle Cat carrier with two XCVU35P modules, each with 8 GiB HBM, connected over Ethernet. Both cards are commercially available today.
Qwen3.5 hybrid architecture maps cleanly to hardware
Qwen3.5 is not a uniform transformer. Most layers use a Gated DeltaNet linear-attention token mixer. The remaining layers use gated full attention. Every layer ends in a SwiGLU MLP. The llm.vhdl design mirrors this split across five subsystems. Subsystem A is a single INT4 streaming matrix-vector engine that serves every matmul in the model: projections, MLP, and the LM head. Subsystem B implements the Gated DeltaNet path with causal convolution, delta-rule recurrence, SiLU gating, and QK L2-norm. Subsystem C handles gated attention with RoPE, QK scoring, softmax, score-times-V, and an output gate. Subsystem D is a transformer sequencer that reads a per-token descriptor program from HBM and decides, at runtime, which mixer each layer uses. Subsystem E, a tensor-parallel collective for multi-die inference, remains a skeleton for now.
One matvec engine, all state in HBM
The critical design choice is concentration. A single INT4 matvec core (Subsystem A) handles every matrix multiply in the model. The weight streamer feeds INT4 weights and scales from HBM. Banked activation memory stages inputs and outputs. A descriptor-in-memory control plane lets the sequencer issue matvec jobs without host intervention. All persistent state, INT4 weights, KV cache, GDN recurrent state, and activations, lives in the 8 GiB HBM. The host only sends token IDs and receives the argmax ID back over PCIe Gen3 x4 using the XDMA driver.
Two ways to deploy the 9B model
On a single FK33, the entire 9B model fits in one card's HBM. The flow: program the bitstream over JTAG, load the weight image with its manifest, run a context test that gates on CTXTEST_PASS, then launch the web UI. The server is a Python shim that exposes an OpenAI-compatible chat endpoint at http://127.0.0.1:8000/. It is greedy-only, one request at a time, with no prompt cache.
For higher throughput, two FK33s run as a pipeline. Card 0 holds blocks 0 through k-1 and emits the residual. Card 1 holds blocks k through N-1 plus the LM head. The host ferries the residual between cards each token. Measured decode throughput roughly doubles to about 2.5 tokens per second. Each card loads its own manifest slice; the split point is read from the manifest, not configured by hand. The xdma node numbers can swap on reload, so the host identifies card roles by which image contains the LM head.
Validation is bit-exact against llama.cpp
The project does not trust simulation alone. Per-layer activations are captured through llama.cpp's eval callback and compared against a bit-accurate C model of the INT4 fixed-point datapath and against residuals read back from the card. The tokenizer and chat template are bit-exact against llama.cpp. This closes the loop between the hardware datapath and the reference software implementation.
Silicon runs at 75 MHz with headroom to 200 MHz
Both cards have been tested on silicon at 75 MHz only. Nothing above 75 MHz, and nothing at the full 0.85 V VCCINT, has run on hardware yet. Standalone per-block place-and-route ratings show significant headroom. On the FK33 (VU33P -2LV, 0.72 V) the attention MAC array hits 150 MHz, the INT4 matvec engine 185 MHz, and the Gated DeltaNet block 195 MHz. On the VU35P -2 grade at 0.85 V, every shipping block clears 200 MHz: the matvec engine rates 264 MHz, GDN 254 MHz, RMSNorm 282 MHz. Reaching those clocks in the full design is ongoing work.
Jungle Cat 27B target blocked by clock and data plumbing
The 27B model targets the Jungle Cat dual-VU35P platform. The design fits one VU35P at about 46 percent LUT utilization, and every block closes 200 MHz standalone. Two-die bring-up has hit three hard blockers. First, the inter-die GTY Aurora link (8 lanes, quad 126) cannot come up because the carrier's reference clock oscillator sites X1 and X2 are unpopulated. A 156.25 MHz LVDS oscillator and CDCLVD1204 fan-out are identified and await soldering. Second, there is no fast weight-load path. Without PCIe, the only host links are the CoE JTAG bridge at roughly 1 MB/s and BMC Ethernet, far too slow for the 13.5 GB of 27B INT4 weights. A JTAG-AXI probe bitstream is built to measure the real ceiling; a bulk path over Aurora or BMC Ethernet remains an open question. Third, Subsystem E, the tensor-parallel collective, is still a skeleton.
What this means for engineers who want local inference
If you have an FK33 card and a Linux host with Vivado 2023.2, you can run a 9B parameter model today with no GPU, no cloud API, and no model weights leaving your machine. The build flow is documented: build the bitstream (one to one and a half hours), install the XDMA driver, load weights, verify, and serve. The web UI is a copy of the llama.cpp interface. The project is MIT licensed. Third-party components and their licenses are listed in NOTICE. The live development log lives in docs/WORKLOG.md with dated design and debugging notes in docs/.
An open hardware path for sovereign AI
llm.vhdl demonstrates that a modern hybrid-attention LLM can be implemented in synthesizable VHDL, validated bit-for-bit against a software reference, and deployed on commercially available FPGA cards. The remaining work, clock distribution, high-speed weight loading, and multi-die collectives, is engineering, not research. For teams that need deterministic, air-gapped, or simply GPU-free inference, this project is the furthest-along open implementation in hardware.