The capacity buildout for AI workloads is soaring with no end in sight. IDC’s Quarterly AI Infrastructure Tracker shows spending more than doubled in 2025, to $318 billion, and is on track to surpass $1 trillion in 2029. So far, most of that outlay has gone to the accelerated server hardware necessary to meet the parallel computing demands of AI training and inference workloads.
But, as enterprises and service providers race to deploy models and build AI pipelines and factories, one foundational decision that tends to receive less attention than it merits is data storage infrastructure —an oversight that can have significant consequences as AI workloads scale from test environments to production.
If storage systems cannot make data available at required performance levels, expensive GPU infrastructure sits idle, not only wasting investment dollars but also potentially delaying or diminishing the business value organizations hope to achieve with their AI initiatives.
IDC research shows growing recognition of the impact that subpar data infrastructure can have on AI results. In a 2025 IDC global survey of more than 1,300 AI decision-makers, 48% cited the difficulty of ensuring data quality, consistency, and governance as the top barrier in achieving their desired business outcomes. More than 40% also flagged data storage costs and efficient management, the complexity and time required for data cleaning and preparation, data ingestion speed, and handling large data volumes as key AI/ML pipeline challenges that are affecting business results.
Ultimately, infrastructure inefficiency has financial consequences. IDC’s Worldwide Digital Infrastructure Sentiment Survey (June 2026) found that 78% of enterprises view AI infrastructure as the single largest cost element underpinning their analysis of AI ROI and TCO, and 73% say infrastructure complexity is often a source of delay when moving AI workloads from pilot to production, deferring returns on AI investment. As tokens per dollar and time to first token become core performance metrics, storage efficiency directly impacts inference economics.
For many enterprises, AI-ready data infrastructure will represent a marked change from the traditional scale-up storage systems they have purchased over the years for their important business applications and databases. Designing a unified data architecture and selecting optimal infrastructure to ingest, process, and store information will place greater emphasis on scalability across hybrid cloud environments. A well-planned architecture can facilitate access and management of data in many different formats, regardless of its location, and promote consistent security and governance policies through a common control plane. The next pivotal step is ensuring the data storage infrastructure can handle the performance and capacity requirements of potentially massive AI datasets.
IDC’s 2026 Enterprise Infrastructure Pulse Storage Survey finds that enterprises use all types of storage —block, file, and object —across on-premises and public cloud sites with AI workloads. However, the SAN and NAS systems they use for mission-critical applications were not designed with AI requirements in mind. Modern software-driven systems architected to scale out performance and capacity are often better suited to data-intensive AI workloads. IDC research shows that AI workloads will be a major driver of the 21% compound annual growth rate for scale-out file- and object based storage spending through 2030.
Each stage of an AI pipeline can place different demands on the data storage infrastructure, and a system that works well for one phase may be inappropriate or prohibitively expensive for another. For instance, a parallel file system is a common choice for active model training that requires extremely high data throughput. However, parallel file systems may be less suitable for data ingestion and preprocessing due to cost, complexity, and potential metadata overhead. Object storage can make more sense for that phase, with its ability to handle massive unstructured datasets economically, support rich metadata, provide high durability, and scale to theoretically unlimited capacity. High performance object storage is also gaining momentum in other aspects of AI pipelines, including active model training, as organizations explore alternatives for massive data throughput.
The enterprise shift to AI inference ushers in a contrasting set of technical requirements for data infrastructure, with an emphasis on small, random, latency-sensitive reads with massive concurrency for certain phases. Performance metrics that matter more than extreme bulk throughput in some cases include input/output operations per second (IOPS), latency, and time to first token. As with AI training workloads, block, file, and object storage can play a role across the various phases of AI inference, but scale-out, software-driven systems tend to be a better fit than traditionally architected scale-up systems.
One of the fastest-growing storage consumers among the AI inference phases is the KV cache, which keeps the keys and values computed for every token during the context-processing prefill phase. Reusing contextual representations of past tokens, rather than expending precious resources to recompute them, is critical for maintaining performance and energy efficiency. Ideally, the KV cache would use only GPU high-bandwidth memory (HBM), system memory, or local flash storage, but capacity limitations and cost concerns generally necessitate an offload to external storage, whether a parallel or distributed file system, high-performance object store, or unified file-and-object key-value store equipped with NVMe SSDs, over a high-speed network.As context windows scale to millions of tokens with long-context and agentic workloads, GPU pioneer NVIDIA is advocating for a new memory tier and promoting a reference architecture to address the inference context bottleneck, in design collaboration with storage partners.
Searching for a single storage system that will cover every phase of an AI pipeline or factory may be an exercise in futility, but finding a vendor committed to investing in a data infrastructure portfolio capable of accommodating a broad range of AI performance requirements is well within reach. Ensuring the vendor fosters ecosystem partnerships with prominent AI players is another important consideration. With its GPUDirect Storage, CMX context memory storage platform, and STX reference architecture to mitigate I/O bottlenecks, NVIDIA now preaches that storage cannot be an afterthought. Enterprises and service providers would be wise to heed that advice.