A startup called Rivvr launched an automation platform this week aimed at one of the messier problems in running large language models in production: keeping latency predictable when traffic spikes hit. The tool targets teams running vLLM, the popular open-source inference engine, and promises to handle the operational overhead that normally keeps engineers up at night.
The 300-passenger problem
The scenario Yevhen Radchenko, Rivvr's founder, describes is familiar to anyone running voice agents or real-time AI services at scale. A delayed flight triggers hundreds of simultaneous calls to an airline's AI voice agent. Requests queue up. Time to first token climbs. Callers sit in silence. An engineer gets paged.
After the incident, the team spends hours tuning configurations, rerunning load tests, and provisioning extra capacity for the next spike. Over-provisioning buys headroom, but that headroom sits idle between bursts, burning compute dollars for nothing.
Rivvr's pitch is that this entire cycle, the reactive firefighting, the idle capacity, the manual tuning, can be automated away.
What the autopilot actually does
The platform lets operators set latency and throughput targets for their models. Rivvr's autopilot then handles the infrastructure work. It load-tests and tunes vLLM kernels, claiming up to 2x higher tokens per second throughput. It monitors metrics in real time and adjusts cluster topology on the fly to keep service level objectives met.
On the cost side, Rivvr claims it can cut AWS bills by 40% to 70%. The mechanism is straightforward: the system migrates workloads to lower-cost Spot instances when possible, switches VM sizes when AWS runs out of capacity on a particular instance type, and increases memory headroom to reduce time to first token when latency starts creeping up.
The platform also handles incident recovery, rollouts, and rollbacks automatically, removing another category of operational work that typically requires human intervention.
Why vLLM teams feel this pain
vLLM has become the default inference engine for many teams deploying open-source models. It handles the low-level details of GPU memory management, batching, and kernel optimization that would otherwise require deep systems expertise. But running vLLM in production introduces its own operational challenges. The engine's performance depends heavily on configuration tuning, GPU memory allocation, and batch size management, all of which need to adapt to changing traffic patterns.
For teams running voice agents, search systems, or any latency-sensitive application, a sudden traffic spike does not just mean slower responses. It means broken user experiences, missed service level agreements, and pages at 3 a.m. The traditional solution is to over-provision, paying for enough capacity to handle the worst case even though the worst case happens rarely.
Rivvr's bet is that automation can replace both the over-provisioning and the manual tuning with a system that adapts continuously, keeping latency within targets while minimizing cost.
The broader infrastructure automation trend
Rivvr fits into a growing category of tools that automate the infrastructure layer for AI workloads. As models move from notebooks to production, teams are discovering that the hard part is not training or even inference, it is the operational plumbing around it. Auto-scaling, cost optimization, incident response, and performance tuning are problems that repeat across every organization running models at scale.
The company is positioning itself as a managed layer on top of vLLM, handling the infrastructure so engineering teams can focus on model selection, prompt design, and application logic. Whether the claimed cost savings and throughput improvements hold up under real-world workloads will determine whether Rivvr can carve out a spot in a space where cloud providers' own auto-scaling tools and managed inference services already compete.