Harvard's MadSys lab published a trove of production data from FreeInference, an open LLM serving platform, and the numbers upend several assumptions about how inference workloads actually behave. The release, described in a paper titled "A Year in LLM Serving," includes a full year of Chutes trace data plus live GPU and serving telemetry, all freely available for research.

What the Dataset Contains

FreeInference is a real inference fleet serving agent and conversational workloads. The team released three layers of data: a one-year trace of Chutes serving traffic, live infrastructure metrics on GPU utilization, memory, and engine performance, and planned KV-cache traces from 13 agent harnesses. The one-year trace captures 3.5 months of accumulated traffic across roughly 12,000 sessions and 205 billion input tokens.

The live telemetry is served through an open API at /api/dashboard for the current snapshot and /api/history for time-windowed queries. Two-minute samples cover the last two hours, 20-minute averages cover the last day, and hourly averages cover the last week. Each chart downloads as CSV, and bulk export of up to 30 days of hourly rollups is available on request.

User IDs rotate every three months, so a single person appears under different identifiers across a year. This is deliberate: the dataset is anonymized for public release while preserving temporal patterns within sessions.

The Surprising Cache Behavior

The headline finding is that 99% of repeat requests from the same user to the same model arrive within 15 minutes. That tight window means a simple LRU cache matches or outperforms more sophisticated eviction policies. The implication is straightforward: if you are designing a KV-cache layer for LLM serving, you do not need complex eviction logic. A straightforward recency-based policy captures almost all the reuse.

Cache-aware routing, where requests are directed to servers that already hold relevant cached state, raises the token hit ratio. The tradeoff is a 5 to 7% load imbalance across the fleet. That is a manageable cost for the cache hit improvement, but it means the routing layer has to tolerate uneven GPU utilization.

Agent Workloads Break the Pricing Model

The agent-serving data reveals a structural problem with how inference is priced. Input and output token prices are largely irrelevant for agent traffic. Cached token reads dominate the cost profile: in 76% of sessions, the cached input is the single largest expense. This means the economics of serving agents hinge on cache efficiency, not on raw compute cost or output volume.

Tool execution is the real bottleneck. Halving tool execution time speeds up a session by 1.38x. By comparison, halving prefill time yields a 1.10x speedup and halving decode time yields 1.16x. The agent is spending most of its time waiting for external calls, not generating tokens. Optimizing the LLM pipeline without addressing tool latency produces marginal gains.

Context Mutations Are Costlier Than They Look

Only 3.41% of requests mutate prior context, but those mutations account for 62.6% of fresh prefill under infinite cache retention. The dominant cause is system prompt changes at 34.8% of fresh prefill, followed by compaction at 7.7% and injection at 6.9%. Dropped turns and tool result drops contribute smaller shares.

The asymmetry is the point: a small number of requests that break the prefix chain force a disproportionate amount of recomputation. For teams building agent frameworks, this means the design of how context is appended, compressed, or mutated has a larger impact on serving cost than the choice of model or hardware.

What Comes Next

The team plans to release Parquet block-hash traces with replay examples and an LLMLCS export format for cache simulators like libCacheSim. A raw prompt dataset with sanitized agent conversations is pending IRB approval. A separate Chutes conversational dataset, focused on chat and roleplay traffic rather than agent tool calling, is also planned.

The datasets are designed for concrete research use: replay prefix-block hashes to evaluate cache eviction policies, benchmark scheduling and batching with recorded request arrivals, analyze context mutations to understand their impact on cache reuse, and study inter-arrival times to guide KV-cache offloading and prefetching. The citation for the one-year Chutes trace is Nixon et al., 2026, arXiv:2608.13573.