Blog

Where Does GPU Time Go Between the Checkpoint and the First Token?

by Eric Wright, Industry Evangelist

What is your GPU doing while it waits? In training it waits for the next batch or for a checkpoint to finish writing, and in inference it waits for context it already computed once and then evicted. Scott Ekstrom and I spent two hours with the AI Field Day 9 delegates walking that wait from ingest to the next token, and the recordings and session summaries are on the Tech Field Day appearance page. I left convinced that the useful question for a platform team is what the data path does to every GPU-hour, so that you can decide where the next infrastructure dollar goes.

The case we made is that DDN is the data engine under the productive AI stack, meaning ingest, training, checkpointing, inference, KV cache and the retrieval corpus for RAG. Each of those capabilities maps metrics that will affect the quality of AI output such as GPU utilization, time to first token, cost per million tokens, users served per GPU, or the time it takes a model to get from experiment to production.

Why Does a Fast Cluster Still Make Users Wait?

We opened with four training problems and four inference problems. In training, GPUs wait on data at full power, checkpoints pause thousands of GPUs while each one writes, one heavy job slows every other job on a shared cluster, and every expansion costs as much as it adds. In inference, prefill waits on context from file and object layers, agentic loops recompute the same context every turn, a design that’s fine at eight nodes slows down at hundreds, and one tenant’s spike slows every other tenant.

Most of the delegate questions focused on inference as it was the core story for our session, and the reason is the important role of KV cache. Prefill is the work the GPU does to read the prompt before it can write, and it produces a KV cache that decode reads for every output token. Disaggregated inference, where prefill and decode run on different GPUs, can’t work unless that KV cache can move between them. So how big the KV cache is, where it’s kept and how long it lives determine how long a user waits for the first token and how much GPU time goes to repeating work.

Should the Engine Retrieve the KV Cache or Recompute It?

Every time a request arrives, the inference engine makes the same decision, to retrieve context it already computed or to recompute it from the prompt. The cost of retrieval depends on where the KV blocks went after they were evicted from HBM. On the slide we showed, a retrieval takes about 0.001 ms from GPU HBM, about 0.05 ms from CPU DRAM (roughly 50x HBM), about 1 ms from local flash (roughly 1,000x) and about 10 ms from shared storage (roughly 10,000x).

Those figures are approximate orders of magnitude that depend on hardware, network and block size, but they explain the tradeoff. For a short chat turn, recompute is cheap, so the sensible policy is to let those turns recompute and keep HBM free for long-context and agentic work. For a 100K-token document, or an agent that resends the same tool definitions on every step, recompute costs far more than a fetch. Those KV blocks belong somewhere the GPU can reach quickly, so that you can stop paying twice for the same context.

Why Doesn’t Flash Behave Like Memory Out of the Box?

The common approach treats flash as a capacity extension. A page gets evicted, the engine writes it to NVMe, and a later re-reference becomes a random page read. Each decode step references a sparse set of blocks spread across the sequence, so without structure the drive sees random 4 to 16 KB reads at low queue depth. Under that pattern the device runs far below its sequential peak, write amplification and garbage collection pressure rise inside the drive, and effective bandwidth collapses.

The approach we argued for treats HBM, DRAM and flash as tiers with distinct roles, places blocks by reuse probability and accounts for the cost of moving them. It rests on a property of the workload, which is that KV blocks written together (same sequence, same turn, same prefix) are usually read together, and multi-turn and agentic workloads raise that prefix reuse. We described four optimizations that turn that locality into bandwidth.

Access pattern reshaping writes blocks from the same sequence into contiguous LBA ranges and coalesces nearby pages at retrieval, so the drive sees sequential or strided reads instead of scattered page fetches. The network and host data path uses GPUDirect Storage, zero-copy DMA and NIXL-style RDMA registration of GPU memory, so the CPU stays off the data path. Parallelization stripes sequences across many NVMe devices with load-balanced queues, so aggregate bandwidth grows with the drive count. Smarter eviction uses importance and frequency to decide what gets written at all, so low-value blocks never reach flash and both write amplification and later read traffic drop.

Scott made the parallelism point for the delegates because of how impactful that is on the quality and efficiency for the AI lifecycle. Inside Infinia, each NVMe drive has its own metadata and data pool, so one drive running at 100% doesn’t hold back its neighbors, and a client can pull pieces from every drive and node at once. The result is flash that adds bandwidth to the KV cache path as well as capacity, so that you can serve longer contexts and more concurrent sessions without waiting on HBM you can’t buy.

The first session, Thinking Fast and Furious Inferencing with DDN, covers the problem slide, the tier latencies and the flash discussion, including delegate questions on CXL memory pooling and where Infinia sits in NVIDIA’s storage tiers (the G4 shared storage tier).

Thinking Fast and Furious Inferencing with DDN

Where Does Infinia Sit in a Dynamo Serving Stack?

Infinia is the DDN data engine for inference, and the delegates wanted to know exactly where it plugs in. NVIDIA Dynamo runs the serving engines, such as vLLM and SGLang, on the GPU. When the GPU builds a KV cache, it offloads blocks through NIXL, and the Infinia NIXL plugin moves them between GPU memory and Infinia over RDMA, then back again when a later request needs the same context. The plugin ships in the Dynamo prebuilt container, so turning it on is a configuration change on the serving side and the application developer doesn’t have to touch their code.

Underneath, Infinia is a distributed key-value store that presents S3, POSIX and a native KV cache interface in one namespace. The KV cache interface is native, meaning it doesn’t run on top of the S3 or file layer. Infinia stores metadata with the data, so the client gets metadata and data from the same node in one round trip and skips a hop, and a hash of the key locates every piece so the client fetches them in parallel.

Scott was careful to state where Infinia fits. The serving engine owns eviction from GPU memory, and Infinia doesn’t decide which blocks leave HBM. The KV cache dataset is typically protected with one or two copies (RF1 or RF2), because a missing block can always be recomputed, and the tradeoff is lower protection overhead in exchange for an occasional recompute. When a GPU node goes offline its local cache goes with it, and the replacement GPU can reload the prefix KV from Infinia instead of rebuilding it, so that you can reschedule work or replace hardware without every affected user paying for a cold prefill.

We also noted to the delegates that at eight nodes most storage platforms and inference architectures look about the same. The differences show up at dozens or hundreds of nodes and above, under real concurrency, with several tenants sharing the same path, and that’s the scale we design and tune for. The second session, To Infinia and Beyond AI Inferencing with DDN, covers the Dynamo integration, the four optimizations and the metrics discussion.

To Infinia and Beyond AI Inferencing with DDN

What Changes on the Second Call With the Same Context?

The third session, Welcome to the Loop AI Inference Lifecycle with DDN, walked the full inference path on one diagram. It runs from the prompt, instructions, documents, tools and video frames through prefill, KV cache placement and decode to the reload on the next turn.

On the first call, the GPU prefills the full prompt and the KV cache is saved to Infinia alongside the files the prompt came from. On the next call with the same context, Infinia returns that KV cache and the GPU prefills only the part that changed. Because the cache is stored outside the GPU server, it survives a restart or a move to a different GPU. We judge the result on time to first token, time per output token and eval quality, because a fast answer that’s wrong doesn’t help anyone.

We showed three demos built on that pattern, each with a check that proves the answer is right as well as fast. We ran them as recorded replays, because a large-scale inference run doesn’t fit in a two-hour session.

In We Already Read This Book, the knowledge is stored as documents in an object bucket, and the system prompt says to answer only from those documents and to name the file. The first question prefills the instructions and the retrieved passages. A second question pulls different passages, but the instructions are identical, so their KV cache is reloaded from Infinia and the GPU prefills only the new passages. The check is to time both questions to first token and open the file the answer names, so that you can trust a RAG answer and get it sooner on every question after the first.

In Your Agent Has Goldfish Memory, each agent step resends the tool definitions and the notes from earlier steps. We turned GPU prefix caching off for the run, so a faster second call could only mean the KV cache came from Infinia, and we showed the KV cache object in the Infinia bucket next to both timings. Teams often delete those notes to shorten the prompt, and the agent then calls the wrong tool, so keeping the full context and reloading it gets you the right tool call and a shorter wait at every step.

In Don’t Blink, Your Model May Miss It, a live camera sends a new frame every few seconds with the same instructions each time, and the images, video and documents are stored in an object bucket you can search in natural language. The first frame prefills the instructions, and time to first token drops on later frames because that KV cache is reloaded from Infinia. We ran two paths side by side, one with Infinia and one without, and the check is that the answer names a time inside the frame on screen, so that you can keep a vision pipeline current with the camera instead of describing a frame that’s already gone.

Welcome to the Loop AI Inference Lifecycle with DDN

What Did Nebul Measure on the Same Eight GPUs?

Nebul, a European sovereign AI cloud running on NVIDIA infrastructure, has now published results from its own testing of DDN Infinia as the store for KV cache reuse. The conditions matter as much as the multipliers. In Nebul’s words, “Nebul measured results on its own platforms, with the same eight GPUs, model and workloads on both sides.” The testing ran from short prompts to large-token contexts under concurrent demand.

Nebul lists “Nearly 10x faster time to first token with DDN Infinia than Nebul’s own cache on CPU and memory”, “2.5x more input tokens per second from the same GPUs” and “Nearly 60% lower cost per million input tokens, by reloading context instead of recomputing it.” Each of those figures has its own baseline, so the chart keeps them in separate panels instead of putting them on one scale. More input tokens per second from the same eight GPUs is the number a CFO cares about, because it means more users served per GPU before anyone buys another GPU.

Arnold Juffer, Nebul’s CEO, tied the speedup back to sovereignty in the same announcement.

“AI sovereignty should not come at the expense of performance,” said Arnold Juffer, CEO of Nebul. “With DDN Infinia, we can reuse model context across our GPU infrastructure instead of spending valuable GPU cycles recomputing it. That makes inference faster and more efficient while keeping the complete AI workload within the sovereign environment.”

Source, Nebul

Nebul’s post also covers the multi-tenant detail a sovereign cloud depends on. Tenant, subtenant and dataset scope remain part of the key namespace, so saved context stays isolated between tenants, and Infinia applies eviction policies at the dataset and tenant level as capacity fills, so inference keeps running while older cache data is removed.

Where Do Training and Checkpoints Fit in the Same Loop?

Infinia handles the inference half, and EXAScaler handles ingest, data preparation and training at frontier-model scale, including checkpoints. For training, the outcome we chase is GPUs that stay busy on data instead of waiting for it, and checkpoints fast enough to take often, so that a failure rolls a run back by minutes instead of hours. We build EXAScaler to scale performance and capacity together, so each expansion adds throughput in proportion to what it costs.

The two engines connect because the pipeline is a loop. Inference output, agent traces and labeled frames return as data for the next fine-tuning or training cycle. On the inference side, Infinia stores model weights, the retrieval corpus for RAG and the KV cache in one namespace, so documents are ingested and classified in the same place serving workers load models and read context from, so that you can move a new model or a refreshed corpus into production without standing up a separate storage silo for each.

Which Number Should Your Team Measure First?

I told the room I could quote GB/s all day and nobody would leave smarter. Each team judges inference by a different metric. The platform owner wants goodput, meaning tokens from requests that met the SLA. The CFO wants cost per million tokens and, more and more, tokens per watt. The infrastructure VP wants useful work per GPU-hour, the cache owner wants hit rate and recompute avoided, and the retrieval team wants recall@k within a latency budget.

The workloads split along the same lines. Chat and copilots are judged on p99 time to first token, agentic workflows on end-to-end time per task, multimodal on time to first token and end-to-end time, and RAG on recall within a latency budget. All of them run on the same cluster at the same time, which is why we tune against replays of real traces at scale and put little weight on synthetic benchmarks that no inference workload resembles.

If you want to test the argument on your own system, pick one workload with a repeated prefix, such as an agent’s tool definitions or a RAG system prompt. Replay a real trace at busy-hour concurrency twice, once recomputing and once reloading the KV cache from a shared tier, with GPU prefix caching off so the comparison is honest. Record p99 time to first token for the cold and warm calls and the eval result for both. If the warm call isn’t faster, or the answers get worse, you’ve learned something specific about your data path before you buy another GPU.

Join us at AI Infrastructure Field Day 6!

We are excited to be presenting at Tech Field Day AI Infrastructure Field Day 6 on November 5th, so make sure to tune in for the live event or watch the replay!

Thank you to the entire Tech Field Day organizing team and to the delegates who were a real pleasure to present to. Looking forward to more collaboration together!