Synthesizing Golden Traces for Realistic KV Cache Evaluation in Agentic Inference Workloads

Sizing inference clusters and storage tiers is challenging for many reasons. A leading reason is that conventionally, synthetic benchmarking tools are used but prove to be inadequate representations of real-world production workloads.   

The problem with most of today’s methods is the predicted GPU utilization, hit rates, and storage demand rarely, if ever, match what we actually observe when production agentic and RAG traffic runs at scale. The interleaved requests, input sequence length (ISL) distribution, and prefix cache behavior in live environments create variance that synthetic or uniform traces simply do not produce.

We end up carrying extra GPU headroom to absorb variance we cannot explain from the traces, and it becomes difficult to defend storage tiering decisions when the benchmarks used to evaluate them do not reflect the traffic patterns running in production. Direct comparisons between serving and storage configurations also lose credibility, because the evaluation workloads lack the interleaving and cache dynamics that we see in real enterprise and Neocloud environments.

To solve this long-standing issue, our team has developed internal tooling, grounded in traces collected from large-scale production environments, to generate repeatable workloads that better represent how agentic systems drive inference infrastructure. The intent is to move toward a set of reference traces that can be used more broadly to evaluate inference efficiency under realistic conditions.

The Gap Between Uniform Benchmarks and Production Agentic Traffic

Production traffic in agentic systems does not arrive as uniform sequences with homogeneous input sequence lengths. Prompts from tool-calling loops, multi-step planning, and parallel sub-agent execution exhibit high variance in both length and semantic overlap. This variance produces interleaved sequences of cache hits and misses that uniform ISL sweeps do not capture.

The consequence of interleaved, diverse AI workloads creates problems in tokenomics calculations. Models built on average-case hit rates derived from tidy benchmarks underestimate the periods of elevated storage traffic that occur when miss bursts coincide with longer context retrievals typical of RAG infrastructure. Storage therefore participates directly in the effective hit rate observed by the inference engines.

Characterizing Interleaved Access Patterns in Agentic Workflows and RAG Pipelines

Agentic workflows generate request sequences whose statistical properties differ from both simple chat and single-turn RAG. A single user query can trigger planning, multiple tool invocations, reflection steps, and retrieval of supporting passages whose embeddings were computed from heterogeneous sources.

The loop produces bursts of short tool-call prompts interleaved with longer context-augmented prompts. Prefix matches therefore appear and disappear in irregular patterns rather than at a steady average rate.

Trace Synthesis as the Foundation for Reproducible AI Inference Comparison

Golden traces address the reproducibility gap. Rather than relying on synthetic uniform ISL distributions, the synthesis process samples from observed ISL distributions and prefix-match statistics collected from representative agentic workloads. The resulting trace preserves the interleaving order and the correlation between request length and semantic overlap.

We’ve built our trace generator to expose tunable parameters that control sampling: the ISL distribution family, the prefix similarity threshold, and the arrival process that governs how quickly consecutive requests reach the serving stack. Because the trace is deterministic once parameters are fixed, any configuration (different cache tiers, different serving aggregation strategies, or different storage paths) can be evaluated against identical request ordering and identical prefix-match opportunities.

This approach aligns with the multivariate model. The observed hit rate under the trace becomes an input to the token-throughput and cost-per-token calculations rather than an assumed constant.

Experimental Configuration on DGX H100 with GPU Direct Storage

The evaluation used a single DGX H100 node running four inference server instances under aggregated serving. Each instance handled a partition of the synthesized trace while sharing the same storage backend. The node integrates Hopper GPUs connected by NVLink and high-bandwidth memory, with the storage path provided by a parallel file system supporting GPUDirect Storage.

 NVIDIA GPUDirect Storage changes the data path for KV blocks that miss in HBM-resident prefix cache. The mechanism bypasses CPU bounce buffers and system memory copies for eligible storage targets. Three cache hierarchy configurations were evaluated against the same agentic trace replay.

Table 1: Cache hierarchy configurations and data movement mechanisms

We map each configuration to its primary cache tier and the mechanism that handles misses, showing the expected interaction with hit-rate variance under interleaved agentic load.

This shows how each configuration interacts with the hit-rate and retrieval-latency variables from the multivariate model. The GDS path alters the retrieval-latency distribution specifically for the miss component that is prevalent during irregular prefix-match sequences.

Observed Outcomes Under Trace-Driven Load

Replay of the agentic trace produced hit-rate time series that fluctuated more sharply than the constant average assumed in ISL sweeps. Periods of clustered misses coincided with longer ISL requests typical of reflection or multi-document retrieval steps in agentic RAG.

The storage-backed configuration maintained higher sustained throughput during those periods because the direct data path removed CPU copy overhead. The net result appears in the token-throughput and GPU-utilization metrics that feed the tokenomics model: the effective tokens processed per GPU-second increased once the storage tier participated in the cache hierarchy under realistic interleaving.

The trace replay produced three measurable shifts relative to the multivariate model variables defined in our inference efficiency blog series:

• Hit-rate variance increased because prefix matches occurred in irregular clusters rather than at a steady average rate assumed by uniform ISL sweeps.
• Retrieval latency for miss traffic decreased when GPUDirect Storage replaced CPU-mediated copies during miss bursts.
• The breakeven context length at which storage tiering improves tokens per GPU-second moved lower under the agentic trace than under uniform ISL assumptions.

These are why storage path characteristics must be modeled explicitly when projecting token throughput and cost per token for production agentic workloads.

The breakeven point between HBM scaling and storage-tier offload can vary significantly when the workload exhibits the ISL skew and prefix-match interleaving characteristics of agentic workflows and RAG infrastructure.

Infrastructure sized against uniform benchmarks is at risk of under-provisioning across the storage path that is at risk of adding latency and bottlenecks when production traffic deviates from the benchmark assumptions.

Storage as an Active Tier in the KV Cache Hierarchy

In the configurations above, the parallel file system is much more than just a passive backing store. It supplies KV blocks at the throughput and latency required by the miss rate distribution of the trace. GPUDirect Storage removes the CPU from that path, which matters when miss bursts are both frequent and irregular (the exact pattern agentic workflows generate).

The same mechanism supports the checkpointing and weight-loading phases that occur alongside inference in production LLM infrastructure. The trace-driven methodology provides a single workload definition that exercises both the online serving path and the data-movement path that storage needs to sustain.

Path Toward Shared Golden Trace Corpora for Agentic AI

Training pipelines adopted golden datasets because they enabled reproducible comparison across model architectures and optimization techniques. Inference benchmarking requires an analogous artifact. A shared corpus of agentic traces, each annotated with its ISL distribution parameters and prefix-match statistics, would allow any vendor or customer to replay identical request sequences against alternative serving and storage stacks.

When a performance claim references a publicly described trace, reviewers can verify whether the reported improvement holds under identical interleaving patterns or whether it depends on assumptions that diverge from their production traffic. DDN contributes trace synthesis tooling that integrates with existing replay frameworks and the experimental methodology described here.

Uniform ISL sweeps remain useful for isolating single variables. They do not, however, exercise the joint distribution of request lengths, arrival order, and prefix overlap that determines realized cache hit rate in agentic workflows and RAG infrastructure. Golden traces close that gap by preserving the interleaving statistics observed in production.

When those traces drive evaluation on hardware that supports  GPUDirect Storage into a parallel file system, the storage tier becomes a measurable participant in the multivariate efficiency model. The resulting token-throughput and cost-per-token numbers reflect the workload that infrastructure teams actually run rather than the workload that benchmark generators find convenient to produce.