The State of Big Data 2015

Article ImageBig Data has been a hot topic for some time, but it has yet to realize its full potential. The term refers to extremely large datasets that may be analyzed to reveal patterns, trends, and associations; they're often too large to analyze with conventional computing methods. "Our lives are becoming even more data-centric, and Big Data analytics is playing a much greater role within enterprises as they seek to better understand their customers, their behaviors and growth opportunities," explains Durjoy Patranabish, SVP of analytics at Blueocean Market Intelligence. "Statistics have always served as the backbone of analytics and data sciences. However, that trend is changing in this new age of analytics that is more reliant on artificial intelligence, machine learning algorithms, and unstructured data analytics."

According to Frank Acito, co-director of the Institute for Business Analytics in the Indiana University Kelley School of Business, one of the trends driving Big Data is the Internet of Things. "That is connecting every device, refrigerator, automobile, phone, house-you name it-together," he says. "It's a wonderful and interesting phenomenon, but it's going to create a tremendous amount of information-the size of information that we've never had before. Instead of thousands, we're talking about millions or billions of rows of data-and it's coming at very rapid speeds."

Recently, there's been a cross-functional shift in the use of Big Data within the enterprise, with marketers using its insights in an effort to improve customer experience. "While IT has been talking about Big Data for years, the term recently crept into the CMO's vernacular. This isn't a surprise given the marketing team's growing budget and use of technology," says Susan Ganeshan, CMO of Clarabridge. "We've watched some of our customers process as many as 50,000 customer experience records a day, totaling nearly 20 million per year. They are interested in year-over-year trends and gathering customer feedback from every possible listening post. In the world of voice of the customer feedback, marketers use Big Data to uncover trends in product usage, look for competitive intelligence or even learn ways to improve customer experience while saving money."


"Machine learning brings together two trends of 2014-real-time data collection and automation of business processes. The focus on machine-learning algorithms is increasing and enabling companies to handle tremendous data volumes, incorporate different sources of data, and implement sophisticated statistical algorithms," says Patranabish. "In 2014, there has been a significant increase in awareness about Big Data and its returns." As a result, more organizations are arming themselves with ample knowledge resources and highly skilled data scientists who are adept at not only building statistical models but also using machine-learning technologies.

The goal of Big Data has always been for companies to use it to make better business decisions. However, "data is only as powerful as the story it tells. In this past year, we watched marketers dig deep into the data to uncover the real story. One might think that you have to clear away unimportant data to find the true meaning, but in our world, we see the opposite. With customer feedback data, you keep adding more and more commentary until the picture becomes clear," says Ganeshan. "In the past year brands like Intuit, Verizon, and Best Buy started combining data from 40-plus different sources and bringing the customer feedback together with sales trends, industry buzz, and employee ideas to uncover opportunities for improvement. The magic really happens when this data is segmented and sliced and diced into digestible-but still statistically relevant-chunks and provided to the people in the business who can affect change."

On the technical front, John Onder (principal with CBIG Consulting) thinks the biggest advancements in Big Data in 2014 were around Hadoop, which IBM defines as an open source software project that enables the distributed processing of large datasets across clusters of commodity servers. "For example, tools around distributions like Impala from Cloudera. This will continue for many years," says Onder. This means in the near future, it won't be necessary to learn MapReduce, a programming paradigm that allows for massive scalability across hundreds or thousands of servers in a Hadoop cluster.


"Businesses are starting to view data as a strategic asset. With Big Data, organizations are beginning to understand their customers much better, how to price products and services better, how to monetize the data they have, and how [to] integrate new sources of data into their existing data sources to create entirely new products and services or operational efficiencies that were previously not realized through typical use cases," Onder explains.

"Key trends that will drive Big Data in 2015 include text mining and text classification of unstructured data, data security, and a push to develop more talent pools to support Big Data," says Patranabish. "More companies are slowly realizing the power of unstructured data, and there will be a conscious effort to tap all possible sources of data to generate a 360-degree view of business. Big Data is expanding from conventional industries (such as financial services, retail, and technology) to unconventional ones (such as sports, movies and entertainment, and tourism)."

"I like to call 2015 the year of predictive analytics. Now that we have a few years of data under our belt, and now that we've made changes based on the data, the next logical step is to use the combination of the data, the changes, and the results to predict and prescribe the next best action to business owners," says Ganeshan. "After that, the next area of nirvana will be when we can successfully mash up CRM [customer relationship management] systems with CEM [customer experience management] platforms. While most CRM applications are focused on sales and marketing, CEM platforms tell the sales and marketing team more about the customer, both at the aggregate and individual level. Bring­ing this data together will help sales and marketing better target, sell, and upsell customers based on their stated preferences as they interact with the brand."

Patranabish adds, "Moving for­ward, companies need to focus on investing in the infrastructure to sup­port Big Data, such as continued hardware improvements and soft­ware innovations, improving data security policies, and utilizing the enhanced infrastructure to support the ‘three Vs' of Big Data-volume, variety, and velocity."