AI and HPC teams increasingly run on the same GPUs, but their storage often stays split. Cold data sits outside the file system, and developers work in separate, slower environments. Each handoff means staging and copying while accelerators wait.
One Namespace From Start to Finish
Google Cloud Managed Lustre, powered by DDN EXAScaler, consolidates the AI and HPC lifecycle into a single namespace. The lower-cost Dynamic Tier keeps a team’s data in one place, with low latency for hot data. Interactive work such as cloning repositories and running notebooks gets a “local disk” feel on the same storage that holds the training set. Teams can start on Managed Lustre and scale with it as their use cases expand.
Managed by Google, Built on DDN EXAScaler
Google Cloud brings full-stack AI capabilities and infrastructure informed by its experience developing and operating AI-powered services. Google Cloud Managed Lustre, powered by DDN EXAScaler, delivers high-performance storage purpose-built for intensive workloads. Together, DDN and Google Cloud help customers run demanding AI and HPC workloads in the cloud, combining DDN’s storage expertise with Google Cloud’s AI infrastructure. Enterprises, national labs and other organizations that want the performance of a true parallel file system in the cloud can get it without the operational burden of running one.
Proven in Production
“Google Cloud Managed Lustre has become critical to our models for autonomous driving. Lustre’s high performance is complemented by high-speed transfer and integrated import/export options, allowing training data to be maintained in Cloud Storage and then served directly from Managed Lustre during training runs. We bulk import roughly 4.4 million objects into Managed Lustre in under an hour and then use Slurm jobs to write training artifacts back to Cloud Storage via Cloud Storage FUSE. This architecture combines the cost-efficiency of object storage with the high throughput of a parallel file system—without introducing operational complexity.”
Kazuki Odo, Staff Software Engineer, Turing Inc.
“The metrics speak for themselves. We are training models 1.5x faster than before. By saturating the hardware we already paid for, we estimate we have saved 42% in overall training cost compared to our previous storage solution.”
Avinash Gudagi, Software Engineering LMTS, Salesforce AI Research
The workloads differ, but the pattern holds. When the data layer keeps up with the whole lifecycle, the compute you already pay for does more of the work.
See It at SC26
DDN and Google Cloud will both have a major presence at SC26. Join Google’s session in our lightning talks at DDN Convergence to hear how teams are putting Managed Lustre to work across AI and HPC. [Add session time.]
Prefer a one-on-one conversation? Book a meeting at our booth.