Fitting a 27B-Parameter LLM into 24 GiB of Laptop Memory
The promise of local LLM inference is straightforward: run capable models on hardware you own, keep your data private, and avoid API costs. The obstacle is equally straightforward. A 27-billion-parameter model in four-bit quantization already consumes about 12.6 GiB for weights alone. Add a key-value cache for long contexts, attention workspaces, and the various runtime state a serving system needs, and a 24 GiB laptop runs out of room fast. The mlx-vlm baseline for Qwen3.8-27B on an M4 Pro MacBook hits its memory guard at 32K input plus 6K output tokens. That's enough for a short conversation, not for the sustained, multi-turn, context-heavy workflows that make local inference worth the effort.
JustFit, a new MLX-based inference runtime, solves this by treating memory management as a runtime scheduling problem rather than a static allocation problem. The system introduces three mechanisms that work together: KVExec for compressed KV cache execution, PhaseSwap for managing component lifetimes, and StateTrans for coordinating state across requests. Together they expand the completed single-request context from 30,720 positions to 212,992, nearly a 7x increase, on the same hardware with no changes to model weights or quantization.
The Memory Budget Equation
The core problem is captured by a simple equation. At any moment, the live memory of the serving system is the sum of reachable weight storage, packed KV backing, recurrent and speculative state, execution workspaces, and other host and allocator storage. Weight compression reduces the first term. KV encoding reduces the second. The third, fourth, and fifth terms are where most serving systems leave money on the table, because they treat them as static allocations rather than managing them dynamically based on what the system is actually doing at each moment.
For Qwen3.8-27B specifically, the architecture interleaves 16 full-attention layers with 48 Gated DeltaNet layers. The full-attention layers use four KV heads of dimension 256. With TurboQuant's four-bit encoding, each retained position stores 16,640 bytes across the attention layers, compared to 65,536 bytes in FP16. At 229,376 positions, that's 3,640 MiB versus 14,336 MiB, a 3.94x reduction. But the compressed cache is only useful if you can reconstruct the floating-point key and value vectors needed for attention computation without holding those reconstructed vectors in memory longer than necessary.
And reconstruction is expensive. A single attention layer's floating key and value vectors at 192K positions require 768 MiB in BF16. If the system reconstructs all layers and holds the results simultaneously, you've just doubled your memory usage, defeating the purpose of compression. The design problem is not just choosing a smaller cache datatype. It's avoiding unnecessary intermediate representations and cross-layer retention.
KVExec: Fused Reconstruction with Layer-Scoped Lifetime
KVExec addresses this by fusing the reconstruction and attention steps. The TurboQuant representation stores each vector as 32 U32 words (eight indices per word) and one FP16 norm. Reconstruction reads the packed page IDs, performs centroid lookup, applies an inverse Hadamard rotation, restores norms, and writes contiguous key and value vectors in the query dtype. These are immediately consumed by scaled dot-product attention.
The critical design choice is that an evaluation boundary completes the attention result before the layer returns. This means the floating-point operands needed by attention are created, used, and released within a single layer's execution. They don't persist across layers as a second representation. The packed cache remains live throughout, and only the current layer's floating operands are materialized.
The implementation specializes Metal kernels to 256-dimensional heads and 256-token pages. Single-query decode uses page-native attention directly on the compressed data. Multi-query prefill uses direct inverse reconstruction when eligible, with a packed verifier as an alternative path and a gather-and-SDPA fallback. This separation lets decode consume compressed state directly while prefill uses an optimized floating-point consumer.
PhaseSwap: Components That Live and Die with Their Owners
The second mechanism recognizes that components small relative to the model can still consume substantial context headroom. The untied LM head occupies 644.14 MiB, equivalent to about 40,591 retained positions in the KV cache. The MTP predictor adds another 215.21 MiB. These aren't additional KV allocations, but they compete for the same fixed memory pool.
PhaseSwap manages component residency based on ownership. A component stays attached as long as any active owner needs it. The LM head, for instance, is held during mixed prefill and decode phases. While one request decodes and another prefills, the head remains attached even during the second request's intermediate prefill chunks, avoiding repeated reconstruction at every scheduling step. Release becomes eligible only when the last relevant owner exits.
Media requests follow the same principle for the vision tower. Text decode cohorts hold the tower queued at the phase boundary. When admitted, a media-embedding lease loads the tower, evaluates the media embeddings, and releases the tower before ordinary text prefill continues. The predictor references the target embedding and head rather than duplicating them, keeping the component graph tight.
This matters in practice. A 644 MiB component that could otherwise block 40K positions of context becomes available for KV storage when no active request needs it. Near the memory limit, these component lifetimes are the difference between completing a request and hitting the guard.
StateTrans: Request Transitions at Safe Boundaries
The third mechanism handles what happens when requests arrive, finish, and interact. StateTrans separates queue arrival from admission from execution. A queued text peer doesn't change the incumbent's execution mode. Once eligible under page and phase budgets, admission waits for a complete speculative round, converts the incumbent from singleton multi-token prediction to autoregressive batching, and reuses the incumbent's target prompt cache. The singleton drafter is released at the round boundary.
For admission, the system reserves physical-page equivalents equal to the ceiling of the prompt length plus the minimum of the output budget and 8,192, divided by 256 (the page size). Shared prefixes use reference counting. The scheduler tracks output beyond the guaranteed portion as surplus and permits bounded bypass of capacity-blocked requests, turning compressed capacity into a resource that can be allocated among requests rather than a per-request maximum.
Completion and cancellation follow a strict ordering: finish the current generation step, evaluate pending token and cache state and page writes, filter departing rows, then decrement page references. A page ID returns to the free list only when its reference count reaches zero after this boundary. This prevents a finished request from blocking new work by retaining page ownership beyond its useful lifetime.
The algorithm is straightforward but critical. A generation row holds the head lease through mixed prefill and decode. When a peer is selected and the incumbent uses MTP, the system converts to AR, preserving the target cache and releasing the singleton drafter. After the peer is admitted and interleaved, if exactly one eligible row remains with no pending or new prompt, MTP is restored using the same target cache. On cohort close, owners are released but pool backing persists.
Results: 212K Positions, Three for Three
On a 24 GiB M4 Pro MacBook running Qwen3.8-27B MXFP4, three independent fresh-process runs complete 196,608 input and 16,384 output tokens. Every run finishes, produces identical output-token sequences, and passes postflight and page-reuse checks. The median decode rate is 4.985 tokens per second, with sampled peaks between 20,960 and 20,977 MiB. The process guard is set at 21,000 MiB, and the smallest margin observed is 23 MiB.
With two concurrent requests, the system completes 2 times (96K + 16K), retaining 229,376 positions in aggregate at 12.55 aggregate tokens per second and a 20,310 MiB peak. A staggered-arrival run achieves the same lengths at 12.51 aggregate tokens per second and 20,247 MiB. A four-request mix, one 128K + 12K stream and three 8K + 12K streams, retains 204,800 positions at 20.78 common-interval aggregate tokens per second.
The throughput picture varies with context length. Short-output probes at 8K, 32K, and 64K input with 64 generated tokens reach 24.31, 19.11, and 14.59 tokens per second respectively. The fitted memory growth per 1K positions is 12.5 MiB for JustFit over 8 to 64K, compared to 169.9 MiB for the mlx-vlm baseline over 8 to 24K. These are process-level slopes, not KV-array compression ratios, and they reflect the system-level cost of serving, not just the cache.
The fixed 32K + 6K workload study traces the runtime's development. Starting from the mlx-vlm baseline, which hits the memory guard, the first completing configuration (TQ4 + MTP) achieves 5.98 tokens per second at a median peak of 18,203 MiB. The final configuration reaches 11.54 tokens per second at 16,374 MiB, a 1.93x decode improvement and 1,829 MiB lower peak. Median prefill throughput rises from 110.84 to 115.00 tokens per second.
An interesting development artifact: segmented KV reaches 13.30 tokens per second, faster than later paged configurations. Paging and lifecycle changes reduce memory and support shared serving state, but at the cost of some throughput. The cumulative context record rises to 229,376 positions across the development configurations, but the individual changes are not one-factor ablations. Different configurations optimize for different operating points.
Prefix Reuse: Making Long Context Practical
Capacity is only useful if you can actually use it. In the exact-prefix test, a new request restores 16,383 cached positions and prefills one new token in 82.314 milliseconds while an incumbent decodes. This demonstrates that unchanged prefixes can be reused instead of recomputed. For a cached prefix of length L and a suffix of delta, the input-side work is the lookup or restore time plus the prefill time for delta given L, not a new cold prefill of L + delta positions.
This matters for repository-scale coding, where source files, tool results, failed attempts, and earlier decisions accumulate over many model calls. Compaction can omit details that later edits require. Hermes requires at least 64,000 context tokens per active tool-using session. Increasing retained capacity and reducing redundant prefill are complementary requirements, and prefix reuse is the bridge between them.
The mixed-arrival test verifies the text transitions: head reconstruction, MTP-to-AR transition, peer prefill, row removal, and MTP re-promotion with the same target cache. Three short requests finish before a blocked large request emits its first token. A cancellation test disconnects a client after 128 stream events, and the cancelled row's pages are reused by survivors. The component trace records one head restore of 0.151 seconds and an MLX-active allocation decrease of 644.14 MiB on one unload.
Reasoning at 200K: 29 of 30 AIME Problems
The integrated TQ4 runtime answers 29 of 30 AIME 2026 problems correctly, generating 696,834 tokens across the suite at 15.04 token-weighted tokens per second. The maximum recorded process footprint is 19,043 MiB. A uniform INT8 comparison answers 28 of 30 at 14.64 token-weighted tokens per second. Both arms reach an output ceiling on one problem, but their other sampled trajectories differ.
This is end-to-end evidence that the compressed representation supports extended generated mathematical reasoning, beyond synthetic capacity tests. The AIME results also show that the memory management overhead doesn't degrade model quality in a meaningful way. A separate Bonsai 27B run with paged TQ4 answers 26 of 30 on a MacBook Air, illustrating compatibility with more aggressive weight compression on a different host.
Limitations: Cold Starts, Speed, and Real Workloads
The 192K cold-input experiment exercises a full cache build followed by 16K generated tokens. It establishes a capacity boundary, not the expected ingestion cost of every agent turn. The repeated single-request limit runs spend about 48 minutes on cold prefill and generate at 4.99 tokens per second. Full attempts take about 103 minutes. These costs favor asynchronous ingestion and repeated reuse of a stable prefix, not interactive 200K-token sessions.
The development study combines changes rather than isolating each mechanism, and no matched PhaseSwap-off/on test is reported. The observed changes are cumulative, making it difficult to attribute specific gains to specific mechanisms. The smallest sampled guard margin is 23 MiB, so everyday service requires more reserve than the limit configuration.
The evaluation uses repetitive text, greedy token selection, and EOS suppression for capacity tests. These are controlled stress tests, not measurements of repository-level task success or compaction frequency. The AIME evaluation provides a reasoning check, but real-world coding tasks involve different patterns of context use, tool calls, and multi-turn interaction that the synthetic benchmarks don't capture.
Apple's newer M5 hardware reports 3.33 to 4.06x lower time to first token on 24 GB systems using GPU Neural Accelerators. Faster prefill and more efficient weight representations complement JustFit's memory management, but the benefit for this specific 27B workload requires measurement on that hardware.
What This Means for Local Inference
JustFit demonstrates that the bottleneck for local LLM serving is not weight storage. With four-bit quantization, the weights fit comfortably. The bottleneck is the runtime state: the KV cache, the execution workspaces, the components that must be attached when active, and the page management that coordinates requests. JustFit's contribution is treating these as dynamic, lifetime-aware resources rather than static allocations.
The practical impact is significant. A developer running Qwen3.8-27B locally can now maintain a 192K-token context window with 16K of output, enough for substantial code generation tasks with full repository context. Multi-turn sessions can retain 229K aggregate positions across two concurrent requests. And the AIME results show that this compressed state doesn't come at the cost of reasoning quality.
For teams building on this work, the key insight is that memory management and quantization are complementary, not competing, strategies. JustFit fixes the model's MXFP4 weights and optimizes everything around them. Future work could combine this with even more aggressive weight compression, like Bonsai's binary and ternary variants, to push the capacity boundary further. The scaling laws aren't established yet, but the architecture of the problem suggests that the same just-in-time state management principles apply regardless of the specific quantization scheme.
Read the paper on arXiv