My name is Donald Molaro, and I am the “Technical Architect” for WOS – meaning, I get to work with the product management, engineering, product marketing, sales teams and DDN customers to set the long term development path for our object storage products. I also have a few ideas of my own that I get to sneak into the mix – that’s the best part of my job. I’ve got a background in storage, software development and systems design.
I hope that these blogs encourage you to participate in a constructive dialog about how and why the storage industry is evolving the way it is.
I have the privilege of working with some very, very smart people and one thing we hear from time to time in the industry is the phrase “it’s just dumb storage.” The team I work on is tasked with the architecture for the DataDirect Networks Web Object Scaler – we call it “WOS.” One thing I am confident about is that WOS is definitely not “just dumb storage.”
WOS technology is different. Over the next weeks and months, I’m looking forward to highlighting some of the new, advanced and interesting things WOS does – and I might even be throw in some teasers about some things we are looking at “in the lab.” Frankly, I am looking forward to the opportunity of explaining just how “smart” storage can be.
The new release of WOS technology contains what some might see as a “minor” feature; I see it as revolutionary.
There are two things that I know for sure to be true – at least when talking about users, applications and their data. The amount of data involved is growing, and the time to make decisions about the data is getting shorter. In the next version of WOS DDN has developed the first version of a tool that will change the way that applications and users use and interact with their data…drum roll please… “Embedded Search.”
“Finding a needle in a haystack” is a euphuism for a frustrating and time consuming task; but is a real a problem that some and soon many more organizations will be facing. As with everything these days everything has gone digital – even our haystacks. We at DDN have, of course, been “part of the problem”, some of the largest, fastest, storage systems in the world are stamped with a DDN logo – a shameless corporate plug but they are paying the bills for this blog.
The new “Embedded Search” function of WOS storage turns the conventional thinking on the “needle in a haystack” problem on its head.
Our “old” WOS technology can store tens of billions; the “new” WOS hundreds of billions and in future versions, trillions of “data items.” The industry uses the term “object” for those data items; the information that describes data is called “meta-data.” When an application “puts” an object in WOS storage, it also can “tag” it with some “meta-data.” By way of example, a digital photo is more than just the picture – it is all the camera settings, the time of day it was taken, the GPS co-ordinates, orientation of the camera, and even the person who took the photo etc…. The photo itself is the “object,” all of the other “stuff” about the photo is the “meta-data.”
Finding one, a few or a few tens of thousands of objects that are of particular interest to a user and doing it in a few seconds from a pool of hundreds of millions, billions or more is a hard problem. The “brute force” method is easy – take all of the “meta-data” about the objects in question and put it in a single huge data base server and run queries against that.
Brute force, not scalable, dumb.
Embedded Search is different. WOS technology internally organizes objects into “collections” that have some relationships that matter when you get to scale. In a WOS instance with billions of objects there will be, at least, tens of thousands of these collections – the client systems don’t know about those collections, but internally in WOS, managing a few tens of thousands of things is easier and faster than managing a billion. Embedded search “attaches” a micro data-base to each of those collections. This is a really good idea, and I wish I had thought of it all by myself but by far the majority of the credit for it has to go to the team of very, very smart people that I get to work with.
So what does all this technical stuff mean?
It means that instead of a “slow” search against an extra external database, the WOS storage itself can perform the search against all of the databases “in-place,” and across the whole cluster simultaneously. The search can run on all the nodes of WOS, against all the collections reporting their results independently and with a high degree of parallelism. So, if I continue with the example of a massive multi-billion photo collection, it is now possible in a few seconds to locate a sub-set of photos by any of the searchable criteria. For example, if one wanted to find all the photos taken on a particular street corner last Tuesday – with WOS technology, now they can do that, very, very quickly.
Elegant, scalable, and definitely smart.