While Europe has been ratcheting up several large-scale supercomputing projects to make us all smarter, US-based government investment got hit hard this year by sequestration. The fact that China now has a 50 Petaflop computer almost 2x as fast as the US’s should prompt some re-thinking. While the debate rages on about the economic benefits of HPC innovation, no one is questioning the strategic importance of simulation and analysis as all of these governments look to out-compute each other.
That being said, the large systems race currently relies on scaling-out existing technology. While Tianhe-2 is a shining achievement and a big step forward for China to showcase it’s own HPC innovation, there is nothing revolutionary about the system from an architectural standpoint. The changes we are going to need to get from Tianhe-2 to Exascale (20x larger systems) were laid out better at ISC’13 than at any conference I have seen so far. This is probably because we have reached a point in the discussion about the major infrastructure pieces where the debate can now turn to how we’re going to implement not if. Memory systems, processors, interconnects, file systems – all of these will need to be completely re-worked if we’re to scale from today’s state of the art to something 20,000% larger.
Interestingly, one of the biggest things that is being proposed for re-design is how we actually measure system performance. While the topic of a new HPC benchmark is nothing new, when the father of Linpack makes the proposal – this one might actually have some legs. As a marketer – I can suggest we might want to work on the name… High Performance Conjugate Gradient doesn’t really roll off the tongue. What I love about this discussion is the parallels with an emerging discussion on the efficiency of various storage systems – and a move to real discussions around storage operating systems (listen to the HPCWire podcast here @ minute 5:30). As the company behind 1 million lines of HPC storage operating system software – we can’t wait to see this one develop.
The major infrastructure piece I am watching most keenly is the Lustre file system – today’s leading Petaflop file system technology, in use by over 60% of today’s Top100 HPC systems. As a member of the DDN HPC team – we have been a significant supporter of and participant in the Lustre community since its inception, is a founding member of OpenSFS, contributes to Lustre, sells more Lustre storage to supercomputing sites than any other vendor, and sells Lustre-based appliances.
I’m noticing that two general threads of conversation are beginning to gain momentum in Lustre dialogs: 1) community members are concerned about vendor influence over Lustre and, 2) there is mounting awareness of the need to address the increasing disconnect between the top power-users and the general user base about feature priority and features vs. usability.
Lustre is an open-source file system, Intel (Whamcloud) are currently the development gatekeepers, Xyratex owns the Lustre trademark and copyright. At the OpenSFS Breakfast last Wednesday, the early start time meant people were not awake enough to be anything but direct, and there were some pointed questions for both vendors.
For Intel the question is arguably simple: Why is a large commercial entity interested in gatekeeping this open source product? To me, the answer is pretty clear. The Whamcloud team continues to do the work they have always done – gatekeeping the community and underpinning the ‘canonical’ Lustre releases and they get paid to do it by OpenSFS. The effort is not inexpensive (as illustrated by Oak Ridge National Laboratory’s Galen Shipman’s 2011 Presentation at the Lustre User Group). Release content and quality have not deteriorated since the Intel acquisition of Whamcloud, and contributors have the same ability to influence content as before. In the end – gatekeeping will continue to be decided upon by the community (Lustre is “by the community, for the community”) and as this discussion evolves – we all have a job in ensuring that our gatekeepers continue to maintain and improve our standard of code integration and code quality. Intel’s internal support for Lustre does not end with the Whamcloud team either; even self-described ‘chip heads’ like Intel’s John Hengeveld happily fielded Lustre questions at the end of his talk. Also, Intel tends to like to foster technologies that support higher CPU counts.
The Xyratex acquisition of the Lustre trademark and copyright, however, has created some confusion. During the OpenSFS breakfast – there were several misconceptions around the subject of copyright ownership and code ownership. While Xyratex has dedicated significant podium time to reassuring folks they have ‘liberated’ the brand from Oracle, this has obscured the facts around code ownership and diverted the discussion from significant issues such as the plan for www.lustre.org and the community participation in the site. For Lustre to thrive, we need a safe and happy home for people to collaborate and contribute – and that is in everyone’s best interest.
So what’s the take away here? Vendors, when you have a hand in shaping an open source product that is crucial for its users, be as transparent as possible. Most Linux users mature (ahem!) enough to be using Lustre remember all the fun we had with Linux intellectual property challenges and other fun drama in the 90s. Now, they’re understandably gun shy.
Our second topic, while less likely to be adapted for daytime television, has more immediate relevance to the Exascale discussion – the need to do something about the growing disconnect between the top power-users and the general user base about feature priority and features vs. usability.
This is a story as old as software development – to be successful a technology must increase adoption. As adoption increases, user requirements homogenize and the need for stability slows the pace of innovation. This leaves the original users, who value new functionality (speed, scale, features) over the highest levels of stability, without the platform they need to achieve the same level of results as before.
So, why is this version of the story interesting? Well, Lustre is the de facto file system for high-end HPC. Therefore, logically it will be a part of organizations’ Exascale plans. If the community chooses to trade speed of innovation for increased stability, Lustre might gain more mid-range users, but this could leave the high-end sites without their IO foundation for Exaflop systems. And Lustre community contributors themselves have been debating the topic for some time. The difference now is that, even though Intel’s release team have optimized for both features and stability with a mix of feature and maintenance releases, we’re starting to see top10 and top20 sites unable, in some cases, to get the features they want within the lifetime of current systems deployed. This means the question is no longer academic and real choices are going to have to be made which will inevitably make Lustre slightly less attractive to one of its constituent types.
The same rationale applies to other levels of the storage stack. Today, the industry is actively drawing lines in the battle for node-local and network-local Flash storage – where a variety of architectures and approaches are being proposed to power file I/O at levels orders of magnitude than what is possible today. Next generation systems are being considered where the spectrum of deployment modes is broad – and smart decisions need to be made around broad market requirements today. We could devote a whole blog to this topic alone, but we’ll save that for some special announcements we’ll be making at SC’13 in Denver, CO.