All hype aside, Big Data is real. Whether one looks at it as a basket of technology, a mindset change in decision-making or simply a description of the unstoppable output of a sea of sensors and hyper-social humans, Big Data is akin to a universe of possibilities.

Unfortunately, hidden by a wall of inflated expectations, this big data universe has a giant black hole in its core and this black hole has swallowed GRC (Governance, Regulation and Compliance), security and privacy.

Since the beginning of the information explosion, or roughly since the time when the knowledge worker was armed with a modern means of communications, the enterprise has been struggling with mundane functions like records management and archiving. In some markets such as financial services and life-sciences, this struggle has been made even more complex by compliance with government and industry regulations.

Much innovation has emerged while many dollars have been spent on addressing he issues of the GRC landscape, but unfortunately with somewhat doubtful results….. and this was on “small data”!

With Big Data two formidable challenges are looming which I would classify as “The era of Shadow Data” and “The plague of hyperscale”.

The Era of Shadow Data

Many of us are familiar with the CIO’s curse called “Shadow IT.” Simply put it is the “under the desk” rogue IT infrastructure that lines of business are assembling to run purpose-built applications since “Corporate IT” cannot produce solutions in a timely manner. These “Shadow IT” infrastructures are not only cost inefficient, they are ripe with security issues and management challenges.

The same issues will happen in the world of Big Data and (thanks to John Furier of SiliconAngle for coining the phrase) we are going to see the emergence of “Shadow Data”. We are going to witness a proliferation of data silos, some of them littered with personal information which unfortunately will not get subject to any form of governance, ultimately putting enterprises are risk; no ill intent from the lines of business but rather just plain ignorance of the new liabilities that Big Data may pose.

The Plague of Hyperscale

In the Big Data world we are now speaking of orders of magnitude more than information objects, new types of data that are even richer in personal information, and none of today’s GRC solutions are ready for that challenge. As I mentioned above, both technology innovators and end users have struggled with the scalability challenges of records management, indexing, e-Discovery and compliance archiving in a world where we were counting in millions of pieces of data. What will happen with Big Data where counts are in the billions if not the trillions? Add to this the new forms of information in play such as tweets or Facebook posts or machine generated data!

Of course just like the black plague, all this will get resolved and we will see the light  at the end of the tunnel but it will be hard, full of challenges, and of course, opportunities.

  • DDN Storage
  • DDN Storage
  • Date: June 19, 2013