Storage Switzerland

12/17/2009

DataDirect Networks (DDN) is a leader in high performance computing (HPC) storage systems, counting 8 out of the top 10 HPC systems worldwide as customers. Their systems deliver remarkable performance, storage density and overall capacity: 10GB/s, 1M IOPs and up to 1200 disk drives (HDD or SSD) in two frames. With additional frames these numbers can grow significantly. They have customers with dozens of PBs in a single file system, getting over 240GB/s out of a single namespace.

DDN has taken this ‘extreme’ performance, density and capacity and added some new technologies to create the Web Object Scaler (WOS) intelligent, scalable, object storage platform. This modular, clustered topology features a global name space and policy driven distributed architecture that offers the ability to build a geographically dispersed storage system.

Storage modules are either 4U, 60-drive high density nodes or 2U, 12-drive standard density nodes, which collectively scale to 2.4 PetaBytes of usable capacity. The choice between SAS and SATA drives enables a differentiation between capacity-optimized nodes and performance-optimized nodes. The global file system supports over 200B files, and a multi-node cluster can support dozens of PBs. Nodes have built-in auto-balancing for simple, non-disruptive growth and a self-healing internal design to automatically correct disk, node and network issues.

The WOS architecture creates an object, or ‘container’ which is composed of the file, user created metadata and information WOS uses to manage the object within its global name space. Since nodes can be geographically dispersed, a WOS policy can enable both automated backup and DR process, allowing multiple copies of an object to be sent to different locations. The WOS client is a service running at user access points that manages the index of user filename to WOS Object ID (OID). The WOS system performs reads, writes and deletions and makes policy driven, intelligent routing decisions about where to store data. The WOS OID architecture is such that, a single disk operation is all that is required to retrieve any object.

The combination of in-memory lookups, a completely flat namespace and intelligent, network and load-aware clients enable single disk-seek retrievals for all objects anywhere in the cloud, giving the WOS system the ability to maintain performance as it scales. DDN claims that WOS outperforms traditional file system technologies which must conduct multiple disk operations (IOPS) in order to retrieve information. Growth is simple and granular, as the WOS clustered architecture enables non-disruptive expansion and a ‘pay as you grow’ format, essential for cloud applications.

Self-Aware, Self-Healing System

WOS has several layers of data protection and data recovery. Should an object be read and its embedded CRC (cyclic redundancy check) uncover an error, WOS immediately does two things to resolve the issue. First, it retrieves an accurate replica from within the WOS cluster and sends it to the user requesting the object. Second, it takes the accurate replica and replicates it again on another drive, per the user policy, thus ensuring the file is again, completely redundant. There is no delay to the user in retrieving the file and no user intervention is required to repair the damaged file.

Should a drive fail within the cluster, then WOS assumes all the replicas on that drive are lost and immediately creates new copies of all the objects that were on the failed drive, sending them to new locations within the WOS cluster. Once the damaged drive is replaced, WOS simply views this drive as a new member of the storage pool and uses it for new objects.

How WOS reacts to the loss of a node is similar in that replicas of the lost objects are duplicated as needed and the node is recreated, with one difference. Users can configure a delay when a lost node is detected, to allow for temporary network connectivity problems to resolve, before starting the rebuilding process. Similar to the loss of a node, the loss of a zone results in new object replication within the viable zone. Once the Zone returns, WOS takes the replicas it has made elsewhere, transfers them to the returned zone and deletes the excess replicas.

Storage Switzerland Take

ou could call it a ‘top-down’ evolution, as the technology this product sprang from was developed for extreme performance. Instead of pushing the envelope from the bottom up, as other manufacturers do, DataDirect Networks has taken the efficiency and performance advancements of their HPC technologies and applied them to a different problem: large, scalable, distributed file storage.

The WOS is a scale-out, cloud storage system with remarkable storage density, but solid, consistent performance as it grows. WOS matches the capability of the hardware with a distributed, object-based architecture that greatly reduces file system management complexity and overhead. The result is a system that supports thousands of concurrent users, making millions of operations to billions of files.

Read the Report