Pruning LLMs for Smart Homes: What Breaks First

Pruning is the standard trick for making language models cheaper to run. Cut out redundant layers, channels, or experts, and you get a smaller model that runs faster. But when the model is controlling your lights, thermostat, and security camera, the question changes from "how much accuracy do I lose?" to "what specifically breaks, and when does it become dangerous?"

Congjing Zhang, Vashishtha Patil, Henning Lange, and Usman Aleem at Amazon's Alexa Home AI team study exactly this. They evaluate four LLMs across dense Transformer, dense hybrid, and mixture-of-experts architectures, applying depth, width, hybrid, and expert pruning methods. After post-pruning supervised fine-tuning, they evaluate over 19,500 instances from three smart-home datasets and analyze degradation along two dimensions that standard pruning benchmarks ignore: which part of the tool-calling action breaks, and under what task complexity.

Tool Calling Is Not a Single Prediction

The core insight is that smart-home tool calling is not one task. It is four decisions packed into every action. For a request like "dim the bedroom light to 50," the model must predict the operation (set), the device (bedroom.light), the argument (brightness), and the value (50). Get any one wrong and the action fails or does something unintended.

The authors decompose each generated action into these four components and measure accuracy independently. This turns out to matter because pruning does not degrade all four equally. It degrades grounded specificity before schema-level intent.

Operation and argument prediction come from a small, repeated schema. "Set," "turn on," "toggle" are the same set of choices regardless of which device you're controlling. "Brightness," "temperature," "volume" are the same small set of arguments. These are the parts of the action that pruning preserves longest. Device and value prediction, by contrast, require instance-specific grounding in the provided context. Which device in this particular home has the name you're looking for? What value did the user actually request? These require selecting precisely among candidates, and pruning weakens them first.

At 50% dense pruning, device accuracy drops 24.2% while operation accuracy drops 21.1%. Value accuracy drops 20.7% while argument accuracy drops 14.5%. The gaps are consistent across all three datasets and all pruning methods. The model retains the ability to generate a plausible action schema but loses the ability to ground it in the specific home context.

Dense Models Have Cliffs, MoE Models Have Plateaus

The comparison between architecture families reveals a sharp structural difference. Dense models, both the standard Transformer Qwen3-4B and the hybrid Qwen3.5-4B and Qwen3.5-9B, exhibit narrow safe pruning regions followed by steep degradation. At 10% pruning, seven of eight LLM-method combinations lose less than 0.6% accuracy, suggesting that mild pruning mainly removes redundant capacity. But beyond that threshold, performance falls off rapidly.

The MoE model, Qwen3.6-35B-A3B, behaves fundamentally differently. Removing 70% of experts reduces stored parameters from 35B to approximately 12.5B with a nonsignificant minus 0.37% accuracy change. The reason is architectural: REAP pruning removes stored experts while preserving the router and the eight active experts per token, around 3B parameters that actually execute on each input. The sparse MoE design already allocates capacity selectively, so removing unused experts does not impact the active computation path.

The hybrid model shows an interesting middle ground. Qwen3.5-4B, which interleaves Gated DeltaNet linear-attention layers with full-attention layers, is more robust than the same-scale pure Transformer model. The heterogeneous layer types may provide more alternative information pathways. But scaling from 4B to 9B parameters in the hybrid architecture yields no consistent benefit, suggesting that additional parameters do not necessarily correspond to pruning-redundant capacity.

Partially Executable Requests Are the Hardest to Prune

The task-complexity taxonomy reveals that baseline difficulty does not predict pruning sensitivity. The authors categorize requests into five groups: simple (one action, one device), medium (multiple actions or devices), complex (requiring prior context), partially executable (only some actions feasible), and infeasible (no actions feasible).

Partially executable requests are the most sensitive, losing about 10.3% accuracy per additional 10% pruning. These require the model to simultaneously identify which actions are feasible and which are not, execute the valid portion, and reject the invalid portion. This dual execution-and-rejection task is exactly the kind of structured reasoning that pruning degrades fastest.

Simple requests lose 7.7% per 10% pruning. Medium requests lose 7.2%. Infeasible requests are notably more robust. But the robustness of infeasible requests hides a problem: aggressive pruning shifts the model toward refusing to act, even on executable requests.

Over-Refusal: When Pruning Changes the Decision Policy

The most concerning finding is that aggressive dense pruning does not just reduce accuracy. It changes what the model does when it is uncertain. On executable HomeBench requests, unpruned Qwen3-4B refuses 7.7% of the time. At 50% pruning with Angular, refusal rises to 56.2%. With FLAP, it reaches 80.8%. The model stops trying to generate tool calls and defaults to rejection.

The problem is that refusal and correct execution look similar in aggregate metrics. If you only measure how often the model refuses infeasible requests, a deeply pruned FLAP model appears robust, refusing 99.9% of infeasible requests. But it also refuses four out of five valid commands. The model's decision policy has shifted from "try to execute and accept the risk of errors" to "refuse when uncertain," which is a qualitatively different behavior.

This matters for smart-home deployment because a model that refuses to turn on lights when you ask is not a minor inconvenience. It is a fundamental failure of the tool-calling interface. The user cannot distinguish between "the system does not understand me" and "the system is refusing to act," and repeated refusals erode trust in the entire system.

Post-Pruning Fine-Tuning Helps but Does Not Fix Everything

All pruned models undergo supervised fine-tuning after pruning, using the same training mixture across methods. This "healing" step recovers some performance, but the degradation patterns persist. Dense models still show cliffs. Specificity still degrades before intent. Over-refusal still emerges under aggressive pruning.

The healing data is a standard SFT mixture from the training splits of the three evaluation datasets. The authors note that this may not be optimal. Pruning-specific failures, like distractor-rich device resolution, precise value prediction, and the execute-versus-reject calibration, might require targeted training data rather than generic smart-home dialogues.

What This Means for Deploying Pruned Models

The paper's central argument is that aggregate accuracy is insufficient for evaluating pruned models in tool-calling contexts. A model that loses 2% in overall accuracy might have lost 10% in device prediction and 5% in value prediction, which means it is substantially less reliable for the specific decisions that matter most.

The action-component analysis provides a diagnostic framework. Before deploying a pruned model for smart-home control, measure device and value accuracy separately from operation and argument accuracy. If the specificity gap is large, the model is generating plausible but wrong actions, which is worse than generating no action at all.

The complexity-level analysis provides deployment guidance. If your workload includes partially executable requests, be especially cautious with pruning. If your workload is mostly simple or infeasible requests, you have more room to prune. If your workload includes complex requests requiring prior context, test carefully because context-dependent grounding is exactly what pruning weakens.

For MoE models, expert pruning is a promising starting point with a wide safe region. For dense models, pruning should be conservative, testing multiple ratios and criteria. The safe region for dense models ends abruptly, and beyond it the model may not just become less accurate but may change its behavior in ways that are hard to predict from aggregate metrics.

Limitations

All four models belong to the Qwen family. Whether the observed pruning patterns generalize to other model families is unknown. Pruning methods are not evaluated in a fully factorial manner because some methods are architecture-specific. FLAP and 2SSP cannot be applied to Gated DeltaNet-based models without modification. The evaluation is restricted to smart-home tool calling, and some complexity categories are available in only a subset of the three datasets.

The study also does not measure deployment-level efficiency metrics like latency and throughput. A model that retains accuracy under pruning but does not actually run faster because the sparse structure introduces memory access overhead would not achieve the deployment goal.

The Bottom Line

Pruning LLMs for smart-home tool calling is not just about preserving aggregate accuracy. It is about preserving the specific capabilities that make tool calling work: grounding actions in the current home context, selecting the right device and value, and maintaining the execute-versus-reject decision boundary. Dense models have narrow safe regions. MoE models tolerate more. Specificity breaks before intent. And aggressive pruning can make a model refuse to act rather than act incorrectly, which is a failure mode that aggregate metrics miss entirely.

Read the paper on arXiv