In Part 1 we established that inference efficiency is no longer a single-GPU tokens-per-second problem. Variable reasoning depth, prefill/decode disaggregation, KV cache hierarchies, compression, and mixed workloads turned it into a systems problem whose cost is the product of interacting terms.
In Part 2 we replaced synthetic warmup/query benchmarks with production-sourced interleaved traces as the foundation for measurement. Controlled baselines remain useful for ceilings and bounds, but only when interpreted against real contention, reuse patterns, and tiered retrieval costs.
In Part 3 we defined the core variables and the equations that connect them. The model produces an expected cost per successful answer, a retrieve-versus-recompute breakeven, and an effective tokens-per-dollar (or per-watt) figure that can be parameterized directly from traces.
This final post applies the model to the decisions that determine infrastructure cost and performance: accelerator selection, compression policy, the disaggregation boundary, and the design of the storage tier that holds KV state once it leaves HBM. We then walk through concrete sizing and cost-forecast examples grounded in the same workload parameters used earlier in the series.
Accelerator Selection and Heterogeneous Pools
Prefill and decode stress hardware differently. Prefill is compute-bound and benefits from high arithmetic intensity and large matrix engines. Decode is memory-bandwidth bound and benefits from high HBM capacity and bandwidth per token.
When the two phases run on the same homogeneous pool, the system is forced to size for the more demanding phase under the worst-case mix. When they are disaggregated, each pool can be sized and specialized independently. The model quantifies the trade-off by adding the disaggregation overhead term δ (KV transfer time plus serialization) to the cost of every request that crosses the boundary.
In the coding-agent example from Part 3 (R = 12, u = 0.90, typical prefix prefill cost 18 s), a pure homogeneous deployment must provision enough HBM and compute to absorb the full recompute penalty on every miss.
A disaggregated deployment can size the prefill pool for the observed arrival rate of new prefixes and the decode pool for the sustained generation rate. The network and storage path that carries the KV state becomes part of the critical path. If that path cannot deliver the state faster than the recompute cost, the specialization benefit disappears.
The practical consequence is that accelerator choice is no longer independent of the storage and network tier. A higher-bandwidth interconnect or a lower-latency external KV store improves the feasible region for disaggregation and therefore changes the optimal mix of prefill-optimized versus decode-optimized accelerators.
Compression Policy
Compression reduces the memory footprint of the KV cache by a factor κ. The immediate effect is higher effective capacity inside HBM and therefore a higher hit rate under the same working-set size. The secondary effects are an increase in arithmetic intensity (or an added decompression latency) and a potential quality impact that must be measured against the quality gate in the model.
The model captures the trade-off by adjusting both the hit-rate terms h_i and the per-token forward cost c_fwd. When κ is large enough that a previously evicted prefix now fits in HBM, the recompute penalty term drops. When the decompression cost exceeds the savings, the net effect is negative.
In practice the optimal κ is workload-dependent. High-reuse agentic loops with long shared prefixes benefit from aggressive compression because the amortized cost of decompression is paid only once per reuse window.
Low-reuse long-context RAG workloads see less benefit and can be hurt by the added latency. The model therefore requires the compression ratio to be swept against the measured reuse distribution rather than applied uniformly.
Disaggregation Boundary
The model evaluates the disaggregation boundary by comparing the end-to-end cost of a collocated path (δ = 0) against the cost of a disaggregated path that includes the transfer overhead δ plus any resulting change in hit rates from the larger shared cache.
When δ is small relative to the prefill cost of the typical prefix, specialization produces lower total cost. When δ grows (congested network, large KV tensors, high concurrent transfers), the collocated path becomes cheaper even if each GPU is less efficient at its assigned phase.
The same equation shows why storage performance matters. A shared parallel file system that can serve large sequential KV reads at low latency and high concurrent IOPS reduces the effective δ by making the external tier competitive with local DRAM or flash.
In the 100 k-token example referenced in Part 3, moving the retrieve path onto a high-performance parallel file system reduced median TTFT from approximately 42.9 s to 818 ms. That reduction is exactly the change in the retrieve-versus-recompute term.
Storage Tier Design
Once KV state leaves HBM, the system faces a pure retrieve-versus-recompute decision at every subsequent tier. The model ranks the tiers by the product of their hit rate under realistic churn and their retrieval latency L_i.
Under light load a local flash tier can dominate the useful hits. Under the interleaved mixed traffic measured in Part 2 (short prompts competing with long-context and agentic loops), the shared parallel tier that can absorb concurrent large reads becomes the primary contributor.
In the mixed-workload measurement of Part 1, pure HBM prefix caching achieved only about 7 % hit rate; the external tier supplied the majority of the useful hits.
The requirement we pull from this is that high concurrent IOPS and sustained sequential bandwidth for large tensors, together with a software path that can treat the KV state as a durable, shareable object rather than an ephemeral GPU-local buffer.
Parallel file systems originally built for HPC and large-scale training already expose these properties. When they are integrated into the inference serving path, the retrieve-versus-recompute breakeven moves in favor of retrieval for a wider range of context lengths and reuse rates.
Worked Sizing and Cost-Forecast Example
The following parameters are taken directly from the coding-agent example in Part 3 and are used to illustrate the sizing calculation:
- Average visible output 400 tokens
- Reasoning multiplier R = 12
- Prefix reuse u = 0.90 after the first turn
- HBM hit rate under load 0.25
- External tier hit rate 0.40
- Prefill cost for the typical prefix 18 s
- Retrieve latency from shared high-performance storage 0.9 s
Under the pure-recompute baseline the expected prefill-related cost per subsequent request is 18 s. With the measured hit rates and the external tier the expected cost falls to:
(1 − 0.25 − 0.40) × 18 s + 0.40 × 0.9 s ≈ 6.66 s
The absolute saving per request is therefore approximately 11.3 s of GPU time. Over a sustained rate of 50 concurrent agent sessions each generating 20 turns per hour, the daily GPU-hour saving is on the order of 110–120 hours relative to pure recompute, depending on the exact mix of new versus continued sessions.
Translating those hours into dollars requires only the unit cost of the chosen accelerator and the power cost. Adding the storage I/O cost of the external tier (measured in the trace) produces a complete C_answer figure that can be compared across alternative configurations. Repeating the same calculation while sweeping R, u, and the external-tier latency yields the sensitivity that should drive the final sizing decision.
Configurations that look attractive under a synthetic 95 % hit rate collapse under the measured 25 % HBM hit rate. Configurations that invest in a low-latency parallel retrieval path remain stable across the measured range of reuse and reasoning depth.
Measure (and Optmize) What Matters
The complexity of modern inference is causing many implementations to produce suboptimal results. Reasoning depth, disaggregation, cache hierarchy depth, and reuse patterns now determine cost as much as the model itself. Teams that continue to treat tokens-per-second on a warm GPU as the primary success metric will systematically mis-size their infrastructure and mis-forecast their spend. Consistency in measurement is the only reliable way to keep those decisions honest.
The measurement model developed across this series was meant to show how many competing levers we must pull to affect efficiency and performance depending on the workload pattern. Parameterize it with your own production traces, sweep the variables that produce the optimal results, and the same math that once produced only optimistic peaks now produces a forecast of cost per successful answer under realistic contention and tiered retrieval.
We have much more coming on how this performance is proven further up the stack at the application level, so make sure you bookmark our blog and watch for more!
It quantifies the cost of the disaggregation overhead δ against the specialization benefit of separate prefill and decode pools. When the KV transfer path is fast enough, heterogeneous pools become the lower-cost configuration.
When the increase in HBM hit rate under the measured reuse distribution outweighs the added decompression latency and any quality-gate impact. High-reuse agentic workloads benefit more than low-reuse long-context RAG.
Low retrieval latency for large sequential tensors and high concurrent IOPS under interleaved load. These properties move the retrieve-versus-recompute breakeven and raise the fraction of useful hits that come from outside HBM.
Replace the illustrative parameters with the statistics measured from their own production or staging traces, then sweep the dominant variables (R, u, external-tier latency, compression ratio) to produce a cost surface for the candidate configurations.
A parameterized model that converts production traces into an expected cost per successful answer and a set of policy thresholds (most importantly the retrieve-versus-recompute latency) that can be used to size GPU pools, configure cache hierarchies, and evaluate storage systems.