Spoiler Alert: We’re basically introducing a new product category here (live @ SC’13).

Loyal readers, buckle up.

Fire up the flux capacitor and prepare to start jumping around both space and time. As your guide, I’m going to attempt to place you in the years 2010 and 2018 at the very same time.  While I cannot guarantee your safety, I can guarantee that YOU WILL LEARN a few things that just might save the life of your application agenda!

I’m about to open the kimono here in a way we’ve not done in the past. In order to do that, I need you to first understand that DDN enjoys a special perspective into the future that not every storage company can appreciate. As the company that supplies vastly more of the world’s largest systems than any other storage company, we get perspective.  This perspective springs from the partnerships we’ve made with the world’s biggest customers and smartest computing engineers in the world. Through these partnerships, we enjoy exclusive insights into the technologies, challenges and opportunities will be presented by the future of data-intensive computing.

OK, jump in the DeLorean, crank the power to 1.21GW. Set coordinates for 2010.

In 2010, DDN engaged a number of supercomputing technologists to leverage Flash technology in discussions toward reducing the amount of spinning disks in a cluster and saving money/power/space/cooling. The thinking was, “Disks aren’t getting faster, but clusters are. Hey look! There’s some Flash… how’s about we all jump back onto a Moore’s Law-governed technology curve.”  This, of course, all made great sense and DDN realized there may be very unique ways we could leverage parallel file systems in a way where a new system architecture did all the hard work.

NVRAM memory, at the time, was quickly becoming commoditized by the success of mobile smart devices that were bringing economies of scale to NAND fabrication.  I’m not writing anything revolutionary here to suggest that it was clear NVRAM, and hybrid storage architectures that combined NVRAM, and spinning high capacity disk had a lot of appeal as a means to drive down costs. Beyond Flash, research was pointing to advanced future technologies such as Memristor, Phase Change Memory and Spin Transfer Torque memory… all of which would usher in byte-addressable storage class memory that has the cost of flash, 10x the performance and far write superior endurance.

Around the same time, a very smart team at Los Alamos National Laboratory (now a new DDN customer!) happened to start publishing some really interesting research on the topic of the perils of massively parallel computing at exascale proportions (my favorite presentation can be found here).  The conclusion complemented our research and customer interest in the topic catalyzed our engineering efforts. It goes roughly as follows:

(Sorry, I have to do this with slides.  Too much text otherwise…)

Back in the DeLorean.  1.2GW.  #BurstForward. 2018.Because Moore’s Law no longer comes from CPU clock rate advances, clusters need to be built from a whole lot of stuff in the future.

We’re witnessing an explosion in cores right now.  Future architectures will have 4,000,000% more threading than they did in 2010.


Stuff breaks.  At exascale, clusters break all the time because they have so many components. Defensive I/O, application checkpointing, will be one approach to protecting system efficiency.

OK. 1 Billion Threads. Compute performance gains will all at the expense of file system efficiency.   Parallel file systems have their work cut out for them regardless of if your job is n:1, n:m, or n:n.  Concurrency is the killer… let’s consider a billion-way parallel write.

The hairy secret about parallel file systems is that the most truly parallel operations bring them to their knees. Shared writes are the worst, as metadata contention becomes unbearable when you get beyond a few 100K concurrent requests.

Still with me?  In case you blinked, let me summarize the opportunities and challenges.


  • NVRAM technology delivers 700% better performance than spinning disks
  • Hybrid storage architectures promise to cut storage costs, power, cooling by over 70%


  • In the future, clusters and web scale data centers will be littered with cores, nodes, etc.
  • In order to scale applications, everything will have to be parallelized; but,
  • As applications are parallelized, they become more susceptible to failure; so,
  • Defensive I/O and heavily threaded writes/reads, is going to Billion-way scale; and,
  • All this spells badness for file systems that have legacy locking semantics.
  • If nothing is done, I/O degradation will force application scaling to hit a punishing wall.
  • NVRAM is the way to address HW optimization, but doesn’t solve file system problems.
  • NVRAM also creates new questions such as node-local data protection, cost, jitter, etc.

Back in the DeLorean.  1.2GW.  2013.  Denver Convention Center. SC’13.

Hey, our first patent for hybrid storage architectures for exascale computing was granted this week!  What’s all this about?… Oh, a new product architecture!? … The timing is just toooo coincidental.

In true DDN fashion, we’re arriving at the annual conference of supercomputing masters in full force.  This week, we continue the time-honored tradition of taking the lid off of some of the most scalable solutions in the world…. but this year we’re doing something slightly different.

In 2012, to make whole our I/O and computing research program, DDN formed a group of engineers into a proper R&D organization.  This team is staffed by many of DDN’s long time storage experts as well as former customers who have come to DDN to build some really cool stuff.  The team does not have a commercialization charter, and is simply tasked with giving the engineering team good ideas, to identify and exploit technology disruptions.  This week, we’re taking the lid off our first skunkworks program; we call it the Infinite Memory Engine (the program formerly known affectionately as: Iron Monkey).

What is the Infinite Memory Engine (IME), you ask?  IME represents over three years of our effort around understanding I/O challenges at scale and how to side-step fundamental architectural hurdles in an effort to bring linear scalability to cluster environments.  It’s both a product architecture program and R&D effort.  We’re not announcing the product ETA yet, but we are bidding it already on some massive RFPs.

The future of DDN’s storage stack:

IME is:

  • A game-changing buffer cache that shatters limits of conventional file systems.
  • A distributed memory system designed to aggregate & present endless storage class memory to an application, without requiring any application modification.
  • A data protection layer that persists and protects data that can live either node-local or in DDN IME buffer cache appliances.
  • Fully integrated with leading-edge parallel file systems to seamlessly act as a buffer cache for concurrent and parallel file read and write I/O.
  • By eliminating the need to build disk-full, server-full storage architectures to support bursty I/O, IME is a revolutionary storage platform that dramatically reduces storage requirements.

IME is not:

  • A replacement parallel file systems; it simply and non-disruptively accelerates them.

Still with me? So, what’s so fancy about IME?  Here’s a quick summary of the advantages:

And, how does it work?

Does it scale? Hell yes!!!IME’s distributed algorithms are proving that they know no boundaries.

  • Unlimited Scaling? Check.
  • Fancy Features? Check.
  • Full Application Compatibility? Check.

Savings?  While the amount varies per use case, the business case is overwhelming.

Here’s one simple example we’ve modeled for a run-of-the-mill HPC system in 2018:

The $ amounts are so high due to the disproportionate amount of money people will be spending in the future to scale and protect their applications vs. the multi-core effect.  IME will help ameliorate this problem, big time, but the needle will move regardless.  The performance delta comes from the fact that we simply configured one memory-class-storage “disk” per compute node… and it turns out we beat our mark.

BTW – we’re showcasing similar HW/performance/power/datacenter savings gains even today. We simply use 2018 as our line in the sand at which point applications will not cross if they’re using POSIX file systems.

Our best secrets have yet to be revealed on IME.  I could write a lot more on the topic, but I think I’ve consumed enough electrons already. You may now step out of the Delorean, glad you made it back safely.

If you’re in Denver this week, please do come see the system at Booth #1307… the demo is really slick.  We’re showcasing 200,000% gains vs. parallel file systems live in the booth.  You can meet our R&D team and throw at them your toughest challenges.

It’s a lot of work, but you can only love working at a company like DDN.  Fifteen years of HPC leadership and expertise from working with the world’s leading computing organizations…and we’re not sitting idle at all.  This week we showcase how all of this learning is engineered into one innovative and revolutionary offering that will be woven into the fabric of tomorrow’s computers if we have any hope of continuing the scaling agenda.

Happy Scaling!

– jeff

  • DDN Storage
  • DDN Storage
  • Date: November 18, 2013