A Data Science Central Community
In the last week alone, it helped fight the Ebola epidemic, fought Parkinson’s Disease, and received 2.5 billion euro in funding from the EU. Okay we get it: big data is changing the face of your industry, no matter what your industry happens to be.
But is it, really? For every article describing how big data will save humanity, there is an accompanying skeptical report, like Wired’s Big Data: New Oil or Snake Oil? Which view is correct: Is big data our saving grace or just another passing fancy? From ROI to quality of insights, what can big data really offer your company?
Big Data and Your Bottom Line: What Can You Realistically Expect
Clearly, decisions relying more on data, and less on intuition, should pay off handsomely for innovation and your bottom line. The New York Times recently reported that, from academic research and the industry experience, you should expect to see “...gains of 1 percent to 5 percent from data-driven decision-making. Those seemingly modest gains tend to be in efficiency, cost savings and productivity.”
In some cases, that 1 to 5 percent will confer a sizable competitive advantage. The Times cited the commercial airline industry as an example where a “...1 percent gain in fuel efficiency would save roughly $3 billion a year worldwide.” Let’s assume that a 1-5% benefit is worth the effort to collect and sort big data in your organization. How can big data be utilized to provide you with actionable business insights?
The Bigger the Past Data, the Better the Prediction
Big data analysis is very good at detecting correlations that your brain would not otherwise pick up when looking at smaller data sets. The data must be big enough to have a basis for establishing trends and patterns. Considering that, every two days, the world produces the same amount of data created between 2003 and the beginning of time, access to large datasets gotten much easier.
Take a look at the entertainment industry, starting with Netflix, which has has big data dialed in. Gigaom noted back in 2012 (and we can safely assume the numbers have grown) that Netflix tracks 30 million “plays” a day, including when you pause, rewind and fast forward; four million ratings by Netflix subscribers; three million searches as well as the time of day when shows are watched and on which devices. Netflix uses this data to predict what other movies you will enjoy, but also to create new content that they already know will be a success. While developing House of Cards, Netflix relied on meta tags in selecting the director and cast. “The company knows how many people are watching Kevin Spacey and David Fincher movies and it knows how many viewers watch political thrillers,” writes Wired.
With access to large amounts of data and the proper tools, your company can predict events in the same way. For instance, Ecommerce shops use purchasing histories of similar customers, combined with average customer lifecycle, to better predict orders at key times.
Big Shows Correlations; Not Casualties
One critical caveat: Big data is effective at showing correlations, but it will not explain or prove causality. As an example, The New York Times noted an uptick in the sales of organic food and diagnoses of autism between 1998 to 2007. They skyrocketed together, but the correlation doesn’t have anything to do with causality. Likewise, the number of non-commercial space launches mirrors the number of sociology doctorates awarded, but nobody is suggesting a direct linkage between them.
Outlying data points are another potential issue. If your data is consistent and not extremely complex, big data will work wonders. But as the New York Times also points out, “Not every problem fits those criteria; unpredictability, complexity, and abrupt shifts over time can lead even the largest data astray.”
Just Because You Have Insights Doesn’t Guarantee You Have The Power To Act on Them
Big data can provide insights, but turning insights from numbers into competitive advantage may require changes that your business can’t afford, or simply doesn’t want to make. The Harvard Business Review explores a case study where through big data it was learned “that he could increase profits substantially by extending the time that items were on the floor before and after discounting. Implementing that change, however, would have required a complete redesign of the supply chain, which the retailer was reluctant to undertake.”
As databases grow, specialists continually find new ways to slice, dice, and create value from them. The big data movement has produced numerous successes visible in our everyday lives: traffic moves faster, medicines are more effective, ads are more personal.
Nevertheless, limits exist, and any new move to implement big data gathering and analytics should begin from a realistic assessment of the benefits. Analytic tools cannot show what will happen, but given the right inputs, they may tell you what is probable, or which what-if scenario is optimal for a situation.
Which means that they are not crystal balls, but in today’s world, they are still the next best thing.