Every developer using LLM APIs faces the same arithmetic. Frontier models like Claude Sonnet are reliable, but they cost more and run slower than the smaller alternatives sitting right next to them in the model catalog. Haiku, Gemini Flash, and a growing roster of lightweight open-source models can handle plenty of tasks at a fraction of the price. The hard part is knowing which tasks, at query time, without running both and comparing.
The default-to-frontier problem
The instinct to reach for the most capable model is rational. A failed or wrong answer from a cheap model costs more than the token difference: you burn a retry, add latency, and lose the user's trust. So developers set Sonnet or GPT-4 as the default and move on. The result is predictable. Teams report spending ten to fifty times more on their LLM bill than they would if they routed simple tasks to smaller models.
The question is not whether smaller models can handle simple tasks. They obviously can. The question is how to classify the task before you send it. You need a signal that says "this prompt is complex enough to justify the expensive model" or "this prompt is simple enough to route downstream." That signal does not exist as a single metric. It is a combination of factors that developers are still learning to weigh.
What makes a prompt actually complex
Complexity in this context is not about word count or token length. A short prompt can be brutally hard. "Write a function that implements a lock-free skip list in Rust with linearizability guarantees" is thirty tokens and demands a frontier model. A long prompt can be trivial. "Summarize this meeting transcript in three bullet points" is thousands of tokens and runs fine on a model one-tenth the cost.
The dimensions that matter are reasoning depth, domain specificity, instruction count, and output structure. Reasoning depth is how many logical steps the model must take. A prompt that asks for a comparison of two algorithms requires more depth than one that asks for a definition. Domain specificity is whether the task requires knowledge that lives in the training data of larger models but not smaller ones. Legal reasoning, medical terminology, and obscure programming languages all push toward frontier models.
Instruction count is the number of constraints the prompt imposes. "Write a Python function" is one instruction. "Write a Python function that handles edge cases X, Y, and Z, follows PEP 8, includes type hints, and has a docstring" is five. Each constraint narrows the solution space and increases the chance that a smaller model will miss something. Output structure matters too. JSON, code, and table outputs are less forgiving of errors than free-form text, because a single malformed token breaks the downstream parser.
Heuristics developers are using
There is no standard formula yet, but patterns are emerging in teams that have optimized their routing. One common approach is task taxonomy. You categorize incoming prompts into buckets: summarization, translation, classification, code generation, multi-step reasoning, and creative writing. Each bucket maps to a model tier. Summarization and classification go to small models. Code generation and multi-step reasoning go to frontier models. This is coarse, but it works for most workloads.
A more refined approach is prompt scoring. You extract features from the prompt itself: number of instructions, presence of code, domain keywords, output format requirements, and length. These features feed a lightweight classifier, often a small model itself, that predicts whether the prompt will succeed on a cheaper endpoint. The classifier is trained on historical data: prompts you have already sent, which model handled them, and whether the output passed your quality checks.
Some teams use a hybrid strategy. Send every prompt to a small model first. If the output fails a confidence check, retry on a frontier model. The confidence check can be as simple as a regex validator for structured output, or as involved as a second model grading the answer. The cost of the retry is still less than sending everything to Sonnet, because most prompts succeed on the first try.
The latency angle
Cost is only half the equation. Latency is the other. A frontier model at 80 tokens per second is fine for a chat interface, but painful for a real-time pipeline processing hundreds of requests per second. Smaller models often run two to five times faster, and for tasks where the user does not see the output in real time, that speed difference translates directly into throughput and infrastructure cost.
The tradeoff is not always obvious. A prompt that runs in 200 milliseconds on Haiku but 800 milliseconds on Sonnet is a clear win for the small model. A prompt that runs in 200 milliseconds on Haiku but produces wrong output 30 percent of the time is not a win, because the retry and quality check eat the latency savings.
What the community is building
The ecosystem is responding. Router services like OpenRouter and Martian let you send a prompt to a model selector that picks the cheapest model likely to succeed. LiteLLM provides a unified API across providers, making it easy to swap models without rewriting application code. Some teams are training small fine-tuned models on their own prompt/response data, creating a cheap model that handles their specific workload with high accuracy.
The general direction is clear. The era of "use one model for everything" is ending. The teams that control their LLM costs in 2026 will be the ones that built prompt classification into their routing layer, measured what actually fails on small models, and stopped paying frontier prices for tasks that do not need them.