Conviva's query engine hit a wall that looked familiar to anyone running data-intensive workloads on Linux: mmap made everything fast until it didn't. Under concurrent production load, their Rust-based engine saw p95 latency jump from roughly 30 seconds to over 150 seconds, not because of CPU saturation or disk limits, but because the kernel's page cache became a shared bottleneck that no single pod could control.

What Conviva's Engine Does

Conviva processes trillions of events daily to diagnose end-user experience. The core engine is built on DataFusion, Arrow, Rust, Rayon, and Tokio. Raw events arrive in a proprietary mostly-numeric format, get stored in the cloud, then copied to local NVMe as Arrow IPC files around 3 to 5 GB each. Arrow IPC's on-disk and in-memory layouts match, which means zero-copy reads with mmap and minimal decode overhead. A typical query pulls 6 columns from 8 batch files, touching about 13 GB of data per day range.

The hardware is a 192-core box with roughly 750 GB of RAM, tested with two disk configurations: a pair of NVMe drives striped via LVM (5.5 GB/s ceiling) and a 32-drive NVMe RAID-0 array (21 GB/s ceiling). During the investigation they ran kernel 5.15, with 6.x in production.

When Mmap Stops Scaling

At light loads, mmap delivered. Queries ran in seconds. The trouble started as concurrency increased. Adding more pods to the same host made things worse, not better. A controlled test comparing one pod versus four pods on the same machine, with identical query load, showed the single pod winning by 41% at peak and over 20% at p95 for 14-day queries.

The root cause was page cache thrashing. Mmap's page cache is implicit shared state. Every process on the host shares one cache, one lock hierarchy, one eviction policy. As pods compete for cache space, the kernel starts evicting pages that active queries still need. The evidence was in the numbers:

  • RSS grew to 98.91% of available RAM, then the kernel began evicting.
  • Major page faults spiked to 1,352 per second once evicted pages got touched again.
  • Minor faults sustained over 2 million per second, each touching cache lines via atomics and trashing L1/L2 caches.
  • Context switches hit 2.1 million per second, 150 times higher than a warm-cache run at 14K per second.

Actual throughput from vmstat during these runs peaked at 3.44 GB/s, about 16% of what the NVMe hardware could deliver with io_uring and O_DIRECT. The gap between what the storage could do and what mmap delivered was the size of the opportunity.

What the Kernel Was Actually Doing

Perf profiling told the story. On cold runs under memory pressure, the kernel function __filemap_add_folio consumed 78% of CPU samples. That function adds pages to the page cache, and it dominated because pages were constantly being evicted and re-inserted. On warm runs, it barely appeared, and the actual query code claimed 45% of CPU instead of 5%.

Off-CPU analysis via bpftrace broke down where threads spent their blocked time: 30.9% on futex waits (threads queued behind other threads' page-fault handlers), 29.3% preempted by the kernel's readahead work, 6.9% on actual disk I/O, and 0.9% on the mmap semaphore itself. The semaphore number understates the real cost because it only captures direct lock waits, not the cascading futex wakes from threads queued behind it.

The picture was clear. Under load, page-cache thrashing and kernel lock contention, not disk I/O, were the bottleneck. Every thread was constantly blocking on page faults, getting descheduled, and rescheduled once pages arrived.

Why O_DIRECT and io_uring Were the Obvious Move

io_uring offers two things mmap does not: submission-completion batching that reduces per-I/O syscall overhead, and the ability to bypass the kernel page cache entirely via O_DIRECT. The team chose compio, a Rust-native io_uring wrapper that provides an executor, futures, and a reactor built around io_uring. The plan was straightforward: submit reads via io_uring with O_DIRECT, coordinate with Tokio, decode Arrow inline, and build their own cache instead of fighting the kernel's.

The first design, called the Batch Materialization Layer, exposed two functions: prefetch(batch, columns) to fire io_uring reads for a set of columns, and materialize(batch, column) to return cached bytes or await an in-flight read. Everything ran on a single thread: accepting calls, submitting compio futures for all requested columns, awaiting completions, decoding bytes into Arrow buffers, populating the cache, and handing materialized columns back to the query engine. In hindsight, one thread doing I/O coordination, Arrow decode, cache management, and the query-facing API was too much. The first cut also fired all roughly 40 column reads at once, one future per column across 8 batches.

The Results: Fewer Major Faults, But Slower

The numbers on Linux told a complicated story. Compared to mmap on cold runs, io_uring with O_DIRECT dropped major page faults from 128,957 to 3,647, a 35x reduction. That validated the core hypothesis: the kernel was no longer thrashing on major page-ins. But minor faults went up 8x, to 8.6 million, and total query time rose from 13.6 seconds to 21.8 seconds. The io_uring implementation was about 60% slower than the mmap baseline.

The culprit was Arrow buffer construction. The team was using arrow-rs's default Buffer::from_slice_ref, which allocates fresh memory and copies bytes into it. Every 4 KiB destination page that memcpy touches triggers a minor page fault, and 8 million minor faults over roughly 13 GB of reads matched that math almost exactly. The Arrow layer was forcing the kernel to redo the memory-management work the team had moved to io_uring to avoid.

Enabling O_DIRECT dropped runtime from 21.8 seconds to about 19 seconds, a real but modest gain. The team also tested on macOS first (which lacks io_uring entirely, using kqueue instead), where total query time was slower but major faults collapsed by 70x, confirming the expected shape of bypassing the page cache.

What This Means for Teams Running Similar Workloads

The headline finding is not that io_uring is slower than mmap. It is that swapping the I/O mechanism without rethinking the entire data path can make things worse. The Conviva team traded one class of fault for another and lost on the trade. The real win from io_uring is not automatic; it requires eliminating unnecessary copies, managing concurrency explicitly, and building a cache layer that does not re-introduce the kernel's problems in user space.

For teams running Arrow-based engines on Linux with NVMe storage, the mmap-to-io_uring migration is not a drop-in replacement. The page-cache bypass is real and valuable, but the decode path, buffer allocation strategy, and I/O scheduling all need to change together. The Conviva team's experience suggests that the second iteration, with O_DIRECT, properly pipelined reads, and Arrow buffers constructed directly from io_uring-owned memory, is where the actual wins live.