The State of Big Data 2016

Article ImageWhen the University of California-Berkeley's (UC-Berkeley) School of Information asked 43 industry professionals for their definition of "Big Data" in 2014, it received 43 different answers. Most answers touched on the "three V" parameters of Big Data popularized by Gartner, Inc.: high-volume, high-velocity, and high-variety information assets requiring advanced forms of information processing to fully unlock their potential.

Andreas Weigend--former chief scientist for Amazon who teaches at UC-Berkeley and runs the school's Social Data Lab--has a different take: "Big Data is a mindset that allows you to turn mess into meaning." Weigend likens Big Data to electricity in a grid, both in its ubiquity and the infinite ways in which companies can choose to convert it into solutions for customers. "Fish in the water don't reflect on the water, and, in the same way, Big Data has dissolved into the classification," he says.

Weigend's characterization lines up with the 2015 edition of Gartner's conversation-provoking "Hype Cycle for Emerging Technologies." A year earlier, in the research firm's annual study of the lifecycle surrounding various emerging technologies, Big Data was poised to tumble into the "Trough of Disillusionment," when early interest wanes and vendors either step up and deliver or drop out. But with the 2015 report, Big Data was nowhere on the curve-not because it's over and done, but because it's gone mainstream.

"It's not special today," says Gartner analyst Nick Heudecker, co-author of "The Demise of Big Data, Its Lessons, and the State of Things to Come," a Gartner research article that explores the change in status. "It's just the way business is done. There are still challenges around volume, velocity, variety, and complexity, but they're not intractable."

The Year in Review

The adjective used in the term "Big Data"-characterizing the mounting corpus of digital data generated through and about human behavior in the 2010s-is understated to a fault. "There's always more data to collect," says Weigend. "Just think of how you spent the last hour." If you checked traffic on your phone, updated your Facebook status, took 150 steps while wearing your Fitbit, and summoned an Uber driver to take you to the airport, you've added another bucketful to the vast universe of 2.5 quintillion bytes of data that IBM estimates is created every day.

But Sharmila Mulligan-CEO and founder of ClearStory Data, which provides an integrated application and platform enabling business users to discover, analyze, and consume data at scale from different data sources-says the emphasis on the volume parameter of Big Data has been overplayed. After all, she says, "People have had hundreds of terabytes of data for decades." The more critical change is in the diversity of the data and in the speed of change. "It's driven by user-generated content, clickstream data, GPS data, and sensors. It's a goldmine to tap into, to learn about the consumer," says Mulligan.

Heudecker highlights the value of data coming from the growing Internet of Things (IoT). "IoT data is immutable," he says. "It's a constant stream of sensor data that can be monitored for rapid change, the ‘I'm on fire!' message, or for long-term historical data." As consumer demand for IoT-enabled devices expands, so does the potential for uncovering actionable insights within the data those devices and wearables generate.

The potential value of all that data points to one of the key drivers of the Big Data market in the past year: Users are all arriving at the party together. "Big Data hasn't had a typical early adopter lifecycle, where vendors offer a tech innovation and users come on afterward," says Mulligan. "In this case, the tech innovation is lagging behind the hunger for Big Data. Every industry and company size is looking to adopt this. And it's a global trend."

Heudecker agrees: "There's no one predominant industry. Every industry is investing in Big Data in some way-not necessarily by themselves, but by using ‘X'-as-a-service scenarios to conglomerate a solution."

A Look Ahead

There's Big Data, and then there's the ability to use it to solve problems. Don't confuse the two. "I don't want to be data-driven, but human-driven" is how Weigend explains the challenge in putting Big Data to work. Part of the problem is measuring ROI. "I call it practitioner's practicality," says Heudecker, pointing to a September 2015 Gartner survey analysis, "Practical Challenges Mount as Big Data Moves to Mainstream." This found that companies in the planning stage of Big Data are more likely to say they will track ROI than the ones who have actually made the investment already. "I hope when we get past the Big Data hype, we can get to more pragmatic questions about value creation," Weigend says.

One way to shrink the Big Data analysis gap is by having machines do more of the heavy lifting. "A lot of the new sources of data are very complex," says Mulligan. "We're applying machine learning to pull meaning out of the data." Distributed analysis will also make it easier to draw conclusions on-the-fly. "Data will be analyzed where it's generated," says Heudecker.

In part, the pressure to create tools to better automate how Big Data is handled relates to a stark hiring reality: Finding enough data scientists to go around remains a challenge. According to Heudecker, "It's the No. 2 challenge we see across industries every year-Who's going to do this stuff?" Weigend says that increasing data literacy is the reason he's teaching at UC-Berkeley: "I'm talking about understanding the power of data, because societies that do it well will set themselves apart."

And in this brave new Big Data world, data governance and security will continue to be top of mind. "A lot of organizations are unaccustomed to thinking about sensors as actual people, and there are concerns about ethical use of that data," Heudecker says. "The focus has traditionally been transactional security, but now it's about the analytical audit trail: the journey of the data, what was done with it, and by whom," Mulligan says.   

Related Articles

With several obvious trends standing out, determining the direction the digital marketing winds are blowing isn't exactly rocket science. But the work of actually marketing a product or service sure feels as if it is. Despite the light-speed pace of advancing technology, plotting the launch and trajectory of an emarketing missile targeted at consumers is increasingly complicated.
While the overarching goal of content analytics--to capture digital content and then apply business intelligence (BI) in order to glean actionable insights--has stayed relatively fixed in recent years, the continued proliferation in variety and volume of content means that both vendors and users of content analytics solutions must move at a brisk clip to even stay in place. In a 2015 industry survey called "Content Analytics: Automating Processes and Extracting Knowledge," conducted by AIIM (Association for Information and Image Management), the majority of the 238 respondents felt there was real business insight to be gained if they got content analytics right.