November 20, 2026

The Hidden Storage Bottleneck in Agentic AI

The Hidden Storage Bottleneck in Agentic AI
Contact Us

Nelson Duann, Senior VP of Client & Automotive Storage Business

A storage stall can waste the most expensive silicon in an AI rack. A slow KV-cache read forces a GPU to wait or repeat work. A delayed container load holds up an agent's next step. Multiply those delays across hundreds of concurrent agents, and storage becomes a bottleneck for the entire rack. The cost rarely appears on the SSD's price tag. It shows up in expensive compute sitting idle and fewer tasks completed per watt.

The CPU is back. Storage never left.

Arm made a useful argument recently in a sponsored piece on The Next Platform. Agentic AI has pulled the CPU back to the center of the datacenter, and the right way to judge a CPU is no longer a core or socket benchmark. It is how many tasks a rack can finish inside its power and cooling budget.

I agree with that framing. It also makes the case for putting storage at the center of the discussion. Read the piece closely and you notice how often the agent is waiting. It waits on a database. It waits on an API. It waits for a sandboxed tool to come back. Each delay holds up the next step, regardless of how much compute is available.

Storage sits behind many of those waits. The database relies on a storage engine whose reads and writes ultimately reach a drive. The tool's working files and logs land on a drive. KV caches that outgrow HBM and DRAM increasingly spill to flash. Before a node can serve its first task, it has to load its operating system, runtime, container images and model weights from storage.

To me, that puts storage at the center of agentic AI performance. It affects how quickly agents can start, retrieve context and move through each step of a task. Arm's focus on completed tasks per rack gives us a useful way to evaluate that role. Storage is already on the critical path. We need to measure—and design for—it accordingly.

Where storage sits on the agent's critical path

An agent's workflow is a chain of interdependent steps. The model reasons, calls a tool, waits, updates its state, goes back to the model and decides what to do next. The next step usually can't start until the last one finishes, so latency piles up along the chain even when the hardware underneath could run things in parallel.

Storage powers agentic AI: enterprise SSDs and boot SSDs support the loop of retrieving context, model inference, executing tools and updating state

Look at what actually happens inside those waits:

These dependencies involve several storage operations:

  • Retrieval. A RAG lookup hits a vector index. Once the index is bigger than memory, part of every query is a random read from an SSD.
  • State. Agents checkpoint memory, conversation history and intermediate results. Long-running agents write these often, and in small blocks.
  • Tool execution. A sandbox spins up a container, pulls an image, runs code and writes output. Cold-start time is mostly an I/O problem.
  • Database transactions. When an agent books, updates or looks up a record, it is waiting on a storage engine whose commit path ends at a flash device.
  • Inference context. Reloading a previous session's KV cache from storage is far cheaper than recomputing prefill on the GPU, but only if the read comes back quickly.

At low concurrency, each of these shows up as a direct hit to task completion time. At high concurrency the problem changes shape. Hundreds of agents produce a mixed stream of small random reads, metadata operations and bursty writes. Average latency starts to matter less than tail latency, because the slowest I/O in a chain sets the pace for the whole task. A drive that looks fine at queue depth 1 in a quiet lab can behave very differently at 70% utilization with garbage collection running in the background.

That is the storage version of Arm's latency-and-concurrency point, and it deserves the same discipline.

KV cache: from a GPU memory problem to a storage tier

The KV cache is where this gets concrete. Every token a model has processed leaves behind key and value tensors. Agents run long contexts, come back to the same conversation many times, and share system prompts and tool definitions across thousands of sessions. That makes the cache large, and it makes reuse valuable.

HBM is the fastest place to keep it and the most expensive. Host DRAM is the next tier, and with DRAM supply as tight as it has been this year, that tier is not cheap to grow either. The industry's answer, visible in NVIDIA's inference context memory work and in open-source frameworks that tier KV blocks out to flash, is to treat NVMe SSDs as a context tier between DRAM and capacity storage.

KV cache tiering: offloading and reloading inference context across GPU HBM, host DRAM and NVMe SSD to reduce prefill recomputation

The economics are straightforward. If a returning agent's context sits on an SSD, the GPU reads it back instead of running prefill again. Prefill is compute-heavy and lands on the most expensive silicon in the rack. Every recompute avoided is GPU time handed back to decode, which is the same point Arm makes about CPU delays holding up the accelerator.

But a KV-cache SSD is not a general-purpose enterprise drive with a new label on it. The workload is read-heavy and latency-sensitive, with large reads on reload and steady writes as caches get evicted. What matters is predictable read latency under mixed load, throughput per watt, endurance sized to the real eviction rate, and firmware that keeps garbage collection out of the read path. Most of that is decided by the controller, not the NAND.

The boot drive is no longer an afterthought

Boot drives used to be the line item nobody argued about. Small, cheap and rarely touched once the OS had loaded. Agentic infrastructure changes that in three ways.

First, nodes come up and go down far more often. Elastic agent fleets scale with demand, and every node that joins has to boot, load its runtime, pull containers and often stage model weights before it serves anything. Time to first useful task is partly a boot-drive number.

Second, the boot device now carries more than the OS. Logs, telemetry, sandbox scratch space and local caches often end up on it, especially in dense CPU sandbox servers that may have no other local drive. A quiet device becomes one with a steady write load.

Third, agent hosts run hot and run all the time. A boot drive that throttles or stalls under sustained thermal load can hang the host, and a hung host stalls every agent on it. Reliability, power-loss protection, secure boot and encryption, and consistent supply over a five-year fleet life now matter more than the last few cents of BOM cost.

In a sandbox fleet with thousands of CPU nodes, the boot drive gets multiplied by the node count. Small differences in power, latency and failure rate add up fast at rack and row scale.

Measuring storage the agentic way: device, system, rack

Arm suggests evaluating CPUs at the core, system and rack level. The same three layers work for storage, and they match how buyers will actually have to compare drives.

Device level. Measure latency at the percentiles that sit on the critical path, not just averages. That means p99 and p99.9 random read latency at realistic queue depths, read latency while writes and garbage collection are running, and performance once the drive reaches steady state rather than fresh out of the box.

System level. Measure what happens when many agents share a node. Look at mixed read/write throughput under high concurrency, how the drive behaves when KV-cache reloads compete with retrieval and logging, how many host CPU cycles each I/O costs, and how well the drive works with direct GPU data paths that bypass the host.

Rack level. Measure completed tasks inside the power and thermal envelope. Watts per drive matter here, but so do IOPS per watt and how many prefill recomputes the SSD tier saves the GPUs. A drive that draws slightly more power but cuts GPU prefill time can still lower energy per task across the rack.

As with CPUs, the component numbers explain where differences come from, and the end-to-end numbers tell you whether those differences change task completion. Synthetic tools like fio are still useful. They just need to be tied to traces from real agent workloads, with retrieval, checkpointing, sandbox churn and KV-cache tiering all running at once.

The rack is the unit of value

A datacenter has fixed power, cooling and floor space. Inside those limits, the question is how much useful agent work the infrastructure can sustain. Arm frames it as tasks per rack and energy per task. Storage belongs in that equation for a simple reason: a storage stall burns the budget of everything waiting on it.

When a KV-cache read is slow, the GPU sits idle or recomputes. When a sandbox cold start drags, the CPU core assigned to it waits. When a boot drive throttles, the whole host falls behind. None of these costs show up on the SSD's line of the BOM. They show up as lower utilization of the most expensive parts of the rack.

That is why storage controllers matter more in this era than their share of system cost would suggest. Latency consistency, power efficiency and firmware that understands the workload are all controller decisions. At Silicon Motion, that is the lens we bring to our MonTitan enterprise SSD platform and our boot-drive controllers: predictable latency under load, low power, and behavior that holds up at steady state rather than only in a benchmark run.

The agents running in datacenters over the next five years will spend a good part of their lives waiting. Some of that waiting is on the CPU, some on the GPU, and more of it than most people admit is on storage. The infrastructure that wins will be the one that measures all three honestly and shortens the waits that sit on the critical path. At Silicon Motion, our controller expertise supports that goal across enterprise SSDs and boot drives—helping the entire rack turn available power and compute into completed tasks.

Sources

Contact Us