In Part 1 we established why the classic tokens-per-second view of inference efficiency no longer holds. Variable reasoning depth, prefill/decode disaggregation, KV cache hierarchies, compression, and mixed workloads turned a simple GPU metric into a systems problem.
In Part 2 we showed how to measure it properly. Production-sourced interleaved traces replace synthetic warmup/query runs as the foundation. Controlled baselines still matter for ceilings and bounds, but only when interpreted against real contention, reuse patterns, and tiered retrieval costs.
Now we build the model itself. This post defines the core variables, the equations that connect them, and why each term is required if the goal is predictable cost and performance under enterprise agentic and long-context workloads.
Why a Multivariate Model Is Required
A single-variable model (tokens/second on a warm GPU) fails the moment any of the following occurs:
- Hidden reasoning tokens multiply the compute cost of a query by 5–50× or more.
- Prefill and decode run on separate pools and the KV state must cross the network.
- A reusable prefix is evicted from HBM and the system must decide whether to recompute or retrieve.
- Compression or embedding strategies change the memory footprint and the arithmetic intensity of later layers.
- Multiple workload classes share the same scarce HBM and storage bandwidth.
The production cost of a successful answer is therefore the result of interacting terms across compute, memory, storage, network, and scheduling. The model must capture those interactions while remaining simple enough to use for sizing, policy decisions, and forecasting.
Core Variables
We group the variables into four categories.
Workload variables
- T_in : input (prefill) tokens
- T_out : visible output tokens
- T_hid : hidden reasoning tokens
- R : reasoning depth multiplier
- u : context reuse fraction (0–1) for a given prefix or agent loop
- M : mixture weights across workload classes (short chat, long RAG, agentic, etc.)
System and policy variables
- h_i : hit rate on tier i (HBM, CPU DRAM, local flash, shared storage)
- L_i : retrieval latency of tier i
- C_pre : compute cost (time or energy) of a full prefill for a given context length
- δ : disaggregation overhead (KV transfer time + serialization)
- κ : compression ratio applied to the KV cache (memory reduction factor)
- e : embedding strategy cost (per-layer vs. traditional, modality collapse, etc.)
Hardware and capacity variables
- B_HBM : effective HBM capacity available for KV state after model weights
- BW_net , BW_flash : network and storage bandwidth
- P : power draw and thermal constraints of the chosen accelerators
- Heterogeneous pool sizes (prefill GPUs vs. decode GPUs)
Economic variables
- $GPU , $power , $storage , $net : unit costs
- Quality gate success rate (fraction of outputs that satisfy the user need)
These are the minimum set that consistently appear when replaying production traces against different configurations.
Key Equations
Effective cost per successful answer
where Q is the quality-gate success rate.
Compute term (simplified)
c_fwd is the cost of one forward-pass token. The second term captures the recompute penalty when a reusable prefix misses in HBM.
Retrieve-versus-recompute decision
For any context length, retrieve is preferred when:
This breakeven point is the single most important policy threshold in a tiered cache hierarchy. In the 100 k-token example from Part 1, retrieving from an EXAScaler-backed tier reduced median TTFT from ~42.9 s to ~818 ms.
Effective tokens per dollar (or per watt)
This is the north-star efficiency metric. Raw tokens/second appears only as an intermediate quantity inside the compute term.
Cache hierarchy contribution
The expected cost of serving a prefix of size S is:
Higher tiers with lower latency and higher hit rates under realistic churn directly improve E.
How the Variables Interact
A few dominant interactions appear repeatedly in production traces which are important to note:
- Reasoning depth multiplies every other cost – A 20× hidden-token multiplier turns a modest prefill miss into a large absolute penalty.
- Disaggregation makes KV state a first-class distributed object – The network and storage tiers become part of the critical path.
- Hit rate under interleaved load is far lower than synthetic numbers – In the mixed workload of Part 1, pure HBM prefix caching achieved only ~7 % hit rate; the external tier contributed the majority of useful hits.
- Compression reduces B_HBM pressure – this raises effective hit rates, but it can increase arithmetic intensity or introduce decompression latency that must be measured.
- Workload mixture determines the infrastructure demand pattern – whether the system is prefill-bound, decode-bound, or storage-bound at any moment. Sensitivity sweeps across mixture weights are required for robust sizing.
These, and many of the production environment traces we use to test and validate our methodology, explain why no single “50× speedup” claim transfers across customers. The model must be parameterized with the actual trace statistics of the target environment.
A Minimal Workload Example
Consider a coding-agent workload with the following measured parameters (illustrative but grounded in the traces discussed earlier):
- Average visible output: 400 tokens
- Reasoning multiplier R = 12
- Prefix reuse u = 0.90 after the first turn
- HBM hit rate under load: 0.25
- External tier hit rate: 0.40
- Prefill cost for the typical prefix: 18 s
- Retrieve latency from shared high-performance storage: 0.9 s
The expected prefill-related cost per subsequent request becomes:
versus a pure-recompute baseline of 18 s. Over thousands of agent turns this difference compounds into substantial GPU-hour and power savings, exactly as observed in the 1.42× wall-clock improvement on the full replayed trace.
Extending the same structure with unit costs for GPU time, power, and storage I/O yields a direct forecast of C_answer and E.
Implications for AI Infrastructure
Once the model is in place, several infrastructure conclusions become quantitative rather than qualitative:
- HBM capacity alone is insufficient under realistic churn. Fast, parallel retrieval from an external tier is a first-order lever.
- Storage systems that can sustain high concurrent IOPS and low-latency sequential reads of large KV tensors directly improve the retrieve-versus-recompute breakeven.
- Disaggregated serving requires the storage and network layers to treat KV state as a durable, shareable object rather than an ephemeral GPU-local buffer.
- Sizing decisions (how many prefill GPUs, how large the external cache, what compression ratio to target) can be driven by sensitivity analysis on the measured variables instead of rules of thumb.
High-performance, intelligent data platforms that already solve parallel file-system access, tiering, and low-latency retrieval for traditional HPC and AI training workloads map naturally onto these requirements.
Next in the Series
Part 4 will move from equations to application with the representative workload traces to validate assumptions and results. We will examine concrete trade-offs across accelerator choices, compression techniques, disaggregation strategies, and storage tiers, then walk through sizing and cost-forecast examples for representative enterprise agentic workloads.
The model is only useful if it survives contact with real traces and real infrastructure constraints. With the variables and equations defined, we can now put it to work.
It is a framework that combines workload characteristics (reasoning depth, reuse), system behavior (tiered cache hits, disaggregation), hardware capacities, and unit costs to predict effective cost and performance under real production traffic.
They act as a multiplier on every forward-pass cost. A high reasoning depth can dominate total spend even when visible output is modest.
It appears as a conditional term: when retrieval latency plus transfer overhead is lower than full prefill cost, the model substitutes the cheaper path and updates the expected cost accordingly.
It determines the latency and bandwidth of the external cache tiers. Lower retrieval latency improves the breakeven point and raises effective hit rates under interleaved load.
Parameterize it with production traces, run sensitivity sweeps on the dominant variables, and use the resulting forecasts to size GPU pools, configure cache policies, and evaluate storage tiers.