The capacity buildout for AI workloads is soaring with no end in sight. IDC’s Quarterly AI Infrastructure Tracker shows spending more than doubled in 2025, to $318 billion, and is on track to surpass $1 trillion by 2029. So far, most of that outlay has gone to accelerated server hardware necessary to meet the parallel computing demands of AI training and inference workloads.
But, as enterprises and service providers race to deploy models and build AI pipelines and factories, one foundational decision that tends to receive less attention than it merits is data storage infrastructure – an oversight that can have significant consequences as AI workloads scale from test environments to production.
If storage systems cannot make data available at required performance levels, expensive GPU infrastructure sits idle, not only wasting investment dollars but also potentially delaying or diminishing the business value organizations hope to achieve with their AI initiatives.
IDC research shows growing recognition of the impact that subpar data infrastructure can have on AI results. In a 2025 IDC global survey of more than 1,300 AI decision-makers, 48% cited the difficulty of ensuring data quality, consistency, and governance as the top barrier in achieving their desired business outcomes. More than 40% also flagged data storage costs and efficient management, the complexity and time required for data cleaning and preparation, data ingestion speed, and handling large data volumes as key AI/ML pipeline challenges that are affecting business results.
Ultimately, infrastructure inefficiency has financial consequences. IDC’s Worldwide Digital Infrastructure Sentiment Survey (June 2026) found that 78% of enterprises view AI infrastructure as the single largest cost element underpinning their analysis of AI ROI and TCO, and 73% say infrastructure complexity is often a source of delay when moving AI workloads from pilot to production, deferring returns on AI investment. As tokens per dollar and time to first token become core performance metrics, storage efficiency directly impacts inference economics.
For many enterprises, AI-ready data infrastructure will represent a marked change from the traditional scale-up storage systems they have purchased over the years for their important business applications and databases. Designing a unified data architecture and selecting optimal infrastructure to ingest, process, and store information will place greater emphasis on scalability across hybrid cloud environments. A well-planned architecture can facilitate access and management of data in many different formats, regardless of its location, and promote consistent security and governance policies through a common control plane. The next pivotal step is ensuring the data storage infrastructure can handle the performance and capacity requirements of potentially massive AI datasets.
IDC’s 2026 Enterprise Infrastructure Pulse storage survey shows that enterprises use all types of storage – block, file, and object – across on-premises and public cloud sites with AI workloads. However, the SAN and NAS systems they use for mission-critical applications were not designed with AI requirements in mind. Modern software-driven systems architected to scale out performance and capacity are often better suited to data-intensive AI workloads. IDC research shows that AI workloads will be a major driver of the 21% compound annual growth rate for scale-out file- and object-based storage through 2030.
Each stage of an AI pipeline places fundamentally different demands on the data storage infrastructure, and a system that works well for one phase may be inappropriate or prohibitively expensive for another. For instance, a parallel file system is a common choice for active model training that requires extremely high simultaneous data throughput. However, parallel file systems are less suitable for data ingestion and preprocessing due to cost, complexity, and potential metadata overhead. Object storage makes more sense for that phase, with its ability to handle massive unstructured datasets economically, provide high durability, and scale to theoretically unlimited capacity.
The enterprise shift to AI inference ushers in a contrasting set of technical requirements for data infrastructure versus AI training, with an emphasis on small, random, latency-sensitive reads that require massive concurrency. Performance metrics that matter more than extreme bulk throughput are input/output operations per second (IOPS), latency, and time to first token, given the real-time response needs of model loading and context caching. As with AI training workloads, block, file, and object storage can play a role across the various phases of AI inference, but scale-out, software-driven systems tend to be a better fit than traditionally architected scale-up systems.
The largest storage consumer among the AI inference phases is the KV cache, which keeps the keys and values computed for every token during the context-processing prefill phase. Reusing contextual representations of past tokens, rather than expending precious resources to recompute them, is critical for maintaining performance and energy efficiency. Ideally, the KV cache would use only GPU high-bandwidth memory (HBM), system memory, or local flash storage, but capacity limitations and cost considerations generally necessitate an offload to external storage, whether a parallel or distributed file system, high-performance object store, or unified file-and-object key-value store equipped with NVMe SSDs, over a high-speed network. Moving forward, as context windows scale to millions of tokens with long-context and agentic workloads, GPU pioneer NVIDIA is advocating for a new memory tier and promoting a reference architecture to address the inference context bottleneck, in design collaboration with storage partners.
Searching for a single storage system that will cover every phase of an AI pipeline or factory may be an exercise in futility, but finding a vendor committed to investing in a data infrastructure portfolio capable of accommodating a broad range of AI performance requirements is well within reach. Ensuring the vendor fosters ecosystem partnerships with prominent AI players is another important consideration. With its GPUDirect Storage, Context Memory Storage Platform, and STX reference architecture to mitigate I/O bottlenecks, NVIDIA now preaches that storage cannot be an afterthought. Enterprises and service providers would be wise to heed that advice.