The Massachusetts Open Cloud – Redefining the Cloud Experience and Service in HPC Environments

by Jean-Luc Chatelain Friday, 25 April 2014

On Friday, the Governor of Massachusetts unveiled the Massachusetts Open Cloud (MOC) Project, an exciting collaboration at which DDN proudly held a front row seat. The MOC Project draws from all five MGHPCC universities, including overall project leadership from Boston University, operational leadership from Harvard University, development leadership from Northeastern University, community building from MIT, and related research by faculty from the universities (UMass, BU, Northeastern, Harvard, and MIT).  Together with our industry partners (Cisco, EMC, SGI, Red Hat, Juniper, Canonical, Dell, Intel, Mellanox, Brocade, Mathworks, Plexxi, Cambridge Computer Services, Enterprise DB, and Riverbed) we are contributing engineering and operational talent, equipment, financial support, and business guidance.

This is a huge step forward for the academic research community and validates the growing popularity of cloud computing and services in this critical field. Let’s take a moment to understand the market drivers behind the formation of the MOC Project, the significance of the MOC project, how it’s changing the computing equation and redefining the experience and service of shared computing infrastructures, as well as DDN’s role in it.

MOC, Mass Open Cloud,

With a mission-based focus on scientific research, universities and research institutes have long been at the forefront of technology innovation that embraces open standards and ease of collaboration. By nature, these types of organizations appreciate that open and transparent collaboration in a shared problem or purpose translates to benefits for everyone.  Such efforts result in accelerated innovation and the fueling of diverse solutions and approaches, which in turn ensures that good solutions persist while others can be tabled and improved upon later. The growth and popularity of cloud computing has added another dimension to collaboration, further connecting both researchers and industry. As such, it is quickly evolving into a premium model for scientific computation and big data that promises to change the face of high-performance computing in the future.

However, before the promise of cloud can be realized, there are pressing challenges that needed to be addressed, not the least of which is how to bring both sides of an open community together to discuss best practices that serve this community’s unique and very varied needs.  There are many service providers that offer both cloud computing and storage capabilities giving users a “pay as you go” model. Access and usage of these facilities is often provided through negotiated SLAs which allow users to reduce the need to establish their own facilities to accomplish the same tasks. Users are seldom aware of any details relating to the hardware and software utilized to provide these services. The vendors providing both hardware and software are not always aware of the actual usage models, so invariably, there is very little development effort or continuous process improvement other than the market-driven improvements of CPU speeds, storage capacities, and bus bandwidth. While these facilities generate revenue and are convenient for users they are not dedicated to improving overall cloud services.

The MOC Project changes this paradigm entirely by establishing a cloud compute and storage infrastructure as a completely open lab. Funded by the Commonwealth of Massachusetts with pledges of hardware and software support from the aforementioned industry leaders in the compute and storage markets, the MOC Project will use the cloud to study the feasibility of operating HPC applications in this environment. The Commonwealth of Massachusetts will also use the facility to distribute public materials to web based users.

What’s so significant about the MOC project unlike other cloud initiatives or forums is that this lab will be as useful to the vendors themselves, as it is to its users. Instead of implementing and testing design improvements in a test lab, equipment providers will be able to test scalability and performance in a real working environment with actual user feedback. This “closed loop” development process is exactly what the industry needs to accelerate the development and improvement of cloud-based technologies. Real-world collaboration will take place as vendors work together to solve user problems and to understand system bottlenecks for various applications and workloads. A quota and billing process managed by the MOC Project will also be part of this program to ensure full self-sustainability after the placement of the initial seed investment.

So what’s DDN’s role in the MOC Project? Well, we’re incredibly proud to be contributing our industry-leading Web Object Scaler (WOS) solution to enable a low latency object storage service. While the initial interface to the WOS cloud will be through Open Stack APIs, we expect to be able to test multiple interfaces and configurations at large scale. WOS has been designed to host 32 trillion objects and the MOC Project will give us a real-world deployment to further this use case and benchmark. We are also interested in data management in a federated environment to enable collaboration. The MOC Project has a long-term goal of federating with other data centers. Since DDN’s WOS platform has been designed to enable this service, we’ll be able to showcase this without involving servers or any external management devices.

We congratulate the Universities involved and the Commonwealth of Massachusetts for envisioning the first facility in the world to enable both the service and development function in one facility. DDN is looking forward to a long relationship in this and in future collaborative programs.