Last week I returned from Salt Lake City, where I attended SC16. It was a fantastic week, full of exciting and stimulating conversations. Moreover, it was my first time seeing snow (no, I’m not kidding)! I took the picture below at the Salt Lake City Airport on my way home after the conference.
A lot happened during SC16, and you can read a general recap here. I thought I’d focus on SC16 from another perspective, which is how ecosystems like OpenStack and Hadoop can help us deliver better HPC. I was overwhelmed by all the amazing things universities and big labs are doing with Hadoop/OpenStack.
DDN Scaler Appliances: More than a Regular Parallel File System
On OpenStack Ecosystems
The apparent strength that an OpenStack ecosystem can bring to HPC is flexibility which, in theory, can make the job of a system administrator easier. For example, it can help a lot for single-node jobs that don’t require parallelism. We believe we are still in a virtual machine era. Containers are cool, but give end-users some time to get used to it.
Future of OpenStack in HPC
We imagine future HPC jobs will leverage multiple technologies. A state-of-the-art HPC job will leverage containers, bare metal, and storage (block, object) in a single job. How is that possible? Remember, OpenStack is API driven, which makes it possible!
On Hadoop Ecosystems
Let’s not fool ourselves: Hadoop and HPC can complement each other. Day after day, both ecosystems offer tools that help scientists deliver faster results. (Learn more about how to leverage in-place analytics at http://datadirect.wpengine.com/solutions/hadoop/). Hadoop is focused not only on performance, but also on the efficiency of your data pipeline.
External Collaborators and Unified Access
IMHO objects were introduced the wrong way. Sharing and giving access to external collaborators is a great example of a use case that can help you become familiar with objects. The idea is to share existing data on parallel file systems without the need of replicating data sources. One significant advantage of this feature is the ability to grant very limited permission to an external collaborator for one single tenant that would be accessible from a friendly protocol such as HTTP.