Big Data—Big Challenges

You probably feel you have a good sense of the assets of your organization. You know how many employees you have because they are all in an HR database. You know the sales you made last month because they are all in a sales ledger database. You know how many items you have in stock because they are all in an ERP database. You know how many computers you have because they are all in an asset database. And at the end of the year, all these databases will be queried to produce the annual accounts.

However, what about the information you hold in documents, reports, drawings, photographs, videos, emails, blogs, and wikis, not to mention your websites? This information is also an asset of the organization, but do you know how much you have and where it is? Much of it will be in shared drives and in email servers. More will be in document management systems and records management systems. Then there is the customer management system and the visit reports in the service management system. Most organizations have no idea how much information they have stored away, because no one has bothered to track it down and count it. After all, you don't need to report it on a balance sheet of the annual report.

Insidiously, the volume of this information is growing very quickly indeed. You won't have noticed it because you have never measured it. Now it has a name, and the name is Big Data. According to IBM, Big Data spans three dimensions: variety, velocity, and volume. Big Data extends beyond structured data and includes unstructured data of all varieties: text, audio, video, click streams, log files, and more. Often time-sensitive, Big Data must be used as it is streaming into the enterprise in order to maximize its value to the business. And finally, Big Data comes in one size: large. Enterprises are awash with data, easily amassing terabytes and even petabytes of information.

In May 2011, the McKinsey Global Institute published a major report on the topic titled "Big Data: The Next Frontier for Innovation, Competition, and Productivity," and it is a must-read. For instance, if U.S. healthcare could use Big Data creatively and effectively to drive efficiency and quality, McKinsey estimates that the potential value from data in the sector could be more than $300 billion in value every year, two-thirds of which would be in the form of reducing U.S. national healthcare expenditures by about 8%. That would put a smile on President Obama's face! And on the other side of the Atlantic, McKinsey estimates that the EU government administration could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using Big Data.

Making effective use of Big Data is not just a technology problem. Much of the McKinsey report is about how companies and other organizations and policymakers need to address considerable challenges if they are to capture the full potential of Big Data. A shortage of the analytical and managerial talent necessary to make the most of Big Data is a significant and pressing challenge, and it's one that companies and policymakers can begin to address in the near term. The U.S. alone faces a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts to analyze Big Data and make decisions based on their findings.

Big Data is not just a problem for large companies, but for any organization where business-critical data needs to be processed and assimilated very quickly. The issue is the volume of data in proportion to the ability of the organization to manage it, not just the absolute values of variety, velocity, and volume.
Calling it Big Data tends to diminish its importance. We are talking about information management and not just data management, and the intersection of structured and unstructured information is the place to be at present. Business intelligence vendors are enhancing their search functionality, and search-based applications are emerging very rapidly. At least one major search vendor sees more growth from selling search solutions to other vendors on an OEM basis than in enterprise search itself.

No one is quite sure of the scale of the problems or of the opportunities of Big Data, but they are going to be big. Take the time now to learn about Big Data. It's coming to a server near you very soon. Maybe it has already arrived without you knowing!