I just finished reading Simon Robinson’s article with great interest, and while a little tardy, I’m finally getting around to sharing some thoughts of my own. As a marketer, my first reaction was to respond to Simon’s plea for a better explanation of the benefits and use cases for object storage. However, I think there’s an even more interesting piece of the story to respond to. (That said, I’m always happy to discuss the “whys and wherefores” as they relate to object storage, so ping me on Twitter @tomme if you’d like more of a tutorial.)
The part I found particularly interesting in Simon’s article was about IT professionals being “concerned about introducing another storage silo into their environments.” While application integration and API standardization are the more popular topics of conversation when debating object storage adoption, I think the concern over creating additional storage silos is a very interesting one that is all too often overlooked. And perhaps most interestingly, it is also the ultimate paradox: after all, object storage was designed to solve the problem IT professionals have as they are scaling out their infrastructures. Now that we have a solution finally that allows users to scale their storage into multiple petabytes granularly and uniformly, without adding to management costs, users are worried about adding yet another tier to their infrastructure.
And they are right to be worried! Too many platforms have been (and still are) promising ease of scalability but have failed to deliver. Think of the prior-generation scale-out NAS and SAN products, which required large forklift upgrades to scale out.
So why is it so complex to scale out traditional (file-based) storage? File-based storage is a great concept: users can access the same resources through a corporate network. The file system takes care of permissions, access rights and avoids users overwriting each other’s data. File systems can even present data in a hierarchical “Directory” structure, which until now has been a very useful tool to keep data organized. The underlying software for such file systems contains a lot of “ingenuity,” which rapidly becomes “complexity” when scaling out the infrastructure.
Figure 1 – Large & Successful Cloud Services have been deploying object storage for 10 years
File systems require at least three layers of software constructs to execute any file operation. As they allow file amendment by multiple users, they must maintain complex lock structures with OPEN and CLOSE semantics. These lock structures must be distributed coherently to all of the servers used for access. The absurdity is that most of the complexities that come with file systems are not required: 80% or more – Simon can correct me on the numbers – of the data that is stored in file systems is never changed, the data is immutable.
Object storage systems are designed differently. Object storage does not have the concept of locking mechanisms: when a user changes a file – an object – a new object is created: as simple as that. In an object storage system, there is no “hierarchy” imposed on the data. This approach allows an object storage system to scale with both the requirements and size of the system, beyond where “file system” designs can.
The popular, and probably still conservative, growth prediction for digital data is 100% every 2 years. The growth numbers of IT staff are dramatically a whole lot less: >50% over the next decade. This means that the average storage operator will have to manage 15-20 times more storage a decade from now. It’s quite obvious that IT professionals, whether they like it or not, will be forced to add another storage tier to their infrastructure. If those professionals want their infrastructure to be manageable a decade from now, they will choose object storage. The hardest part will be selecting a platform that actually scales, but this is something we at DDN are happy to help with.
So Simon, we at DDN see it slightly differently: adding object storage to your storage infrastructure need not be the anathema of consolidation. Odd as it may sound, I believe object storage will eventually lead to consolidation by reducing the complexity involved in scaling file systems, thereby enabling homogeneous, massive pools of fast and cost-efficient storage for unstructured data.