Subscribe to our Newsletter

There has been a few people questioning the value of big data recently, and predicting that big data is going to get smaller in the future. While most of these would-be oracles are traditional statisticians working on small data and worried about their career, or practitioners in small countries (Canada and France in particular) who do not have access to big data, I was surprised to see Mike Jordan - a famous machine learning professor at Berkeley - going in the same direction. Obviously big data will be dead in 5 years: it will be replaced by titanic data.

The purpose of this article is to discuss whether or not data grew too big, to the point that it is better (for some companies) to not harvest big data and reverse back to a small-data world, or maybe even a data-free world, where executive decisions are made based on gut feelings.

Here I do not criticize the gut feeling approach, I actually use it a lot myself with great success, as a growth hacker, though it needs tremendous vision to get it to work. I just want to share some thoughts:

  • Big data has tremendous potential, including as an investment device
  • Big data is necessary for more and more companies in our highly competitive environment
  • Big data can be cheaply accessed and processed using vendors, rather than (or in addition to) home-made solutions
  • Intuition, gut feelings from a visionary data scientist, combined with big data, is the way to go
  • Collecting the right data, using the right KPI's, is critical

Part of the problem with big data is cultural. Americans want bigger stuff:

  • bigger cars, burning tons of gas and difficult to park in any city
  • bigger healthcare, costing more and delivering less
  • bigger stores, selling bad food
  • bigger breasts, to the point it is totally un-attractive
  • bigger houses with expensive mortgage
  • bigger universities teaching outdated material and charging exorbitant fees with no guarantee of positive return
  • bigger hamburgers that will send you straight to the hospital

But sometimes, bigger is better. A bigger army, if used properly, will yield significant benefits (unless it is too big and costs too much taxpayer money). And armies require big data to work properly - think about the NSA. Bigger universities, if well managed, are good: it allows each student to choose courses from a very large pool of professors - including highly specialized training that small universities can't afford to deliver.

Finally, big data can be a great investment for any company. Start collecting data now even if you don't use it: when you sell your company, your data might be one of your core assets. However you need to carefully chose the metrics and data that you want to invest in, and harvest. Just like any investment after all.

In our case, as a small company (zero employee, 7-digit yearly profit), we leverage big data from our vendors. The value is tremendous. It is used in our growth hacking strategy, and as an unfair competitive advantage. As an example, we use smart computational advertising

  • to operate the largest and fastest growing Twitter profile among all data scientists,
  • to efficiently advertise on AdWords using thousands of smartly selected keywords (updated regularly),
  • to discover how our competitors are growing their traffic
  • to optimize our content, as a digital publisher

Related Article: Definition of big data

DSC Resources

Additional Reading

Follow us on Twitter: @DataScienceCtrl | @AnalyticBridge

Views: 530

Comment by Richard Ordowich on November 3, 2014 at 8:15am

It seems that whenever the term big data is used companies like Google, Facebook or Twitter are cited as examples. However most companies don’t sell advertising or sell data as their primary product.

Other cited industries such as financial services are data factories. They consume and create data and attempt to exploit data as their primary source of revenue. Unlike Google, Facebook etc. their product is not advertising but arbitrage.

Bigger is not as “big” an advantage as many claim. Historically a bigger army may have been an advantage but when fighting today’s battles, agility and self-sufficiency are important. An army does require data. The NSA is not an army. The NSA is a data analytics agency and their effectiveness in predicting future events is questionable, like the data analytics departments in many companies.

Just because you can manipulate data in a spreadsheets or create elaborate models doesn’t make the outcome “real” any more or less than gut feel. Knowing how to use a calculator doesn’t make you a mathematician.

Many of the office jobs in industry are data jobs. But the value creation in these jobs is questionable. Many of these jobs involve pushing data from one database or spreadsheet to another and it is these “data pushers” I expect will no longer be required as their jobs can be automated.

There are some novel uses of data such as self-driving cars but these devices rely on sensor data not data from humans. Data from humans is inherently error prone and inexact and those using this data for purposes other than advertising should be suspect.

Big data is potentially harmful as it can lead to data obesity. What is more concerning is data illiteracy. Many of those using data are satisfied with correlations as being “real” and “trustworthy”. A self-driving car knows that its sensors are compromised in a snowstorm and must adjust or shut down when encountering this situation. However humans keep relying on the data without question. Whether they are suffering from data obesity or data hubris, the result is the same; an inevitable data crash. Just ask Wall Street.    

Comment

You need to be a member of BigDataNews to add comments!

Join BigDataNews

On Data Science Central

© 2019   BigDataNews.com is a subsidiary of DataScienceCentral LLC and not affiliated with Systap   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service