Subscribe to our Newsletter

Data silos are unavoidable, and maybe desirable

One of the reasons why ROI on big data is lower than it should be is data silos (or click here for another reference on data silos).

IStock_000006846616XSmall different teams

Having a management consultant review your enterprise analytics spending (internal and external) can help improve the big picture. Or having a in-house high level, business-savvy statistician interact with all the departments in your company, can help.

In my case, in 2002 at CNET (back then it was a small company with 1,600 employees), I was employed in that capacity: supporting sales (including inventory forecasting, price elasticity study, sales forecasts), finance (preparation of quarterly Wall Street reports including choice of KPI's), IT (acting as bridge between data warehouse people and business analysts / marketing people), product development, web analytics, fraud detection, competitive intelligence (based on 3rd party research and data such as Nielsen), marketing mix optimization etc.

However, I believe that for bigger companies, it is not only more difficult to have one data scientist who knows everything about the business to connect independent silos together (and work with other data scientists across multiple teams), but it might not be a desirable solution.

Such an employee (with a considerable amount of insider information about your company - sometimes more than the CEO) would be so critical to the company that if she gets sick, leaves the company for a competitor or disappears for whatever reason, it would create a potential high risk for the company, and it would be impossible to replace and train a new one. So in some sense, data silos, while bad, might not be as bad as the fix.

Do you see a solution to this problem? Maybe making the data more structured and integrated into a highly centralized system, with ad-hoc access privileges depending on the user? Or having a CEO great at sales, with a sharp analytic mind (possibly after some training?)

Or maybe this is truly a case where 1 + 1 < 1: having one data scientist assigned to finance, and another one assigned to marketing, is not as good as one single data scientist taking care of both and understanding the bigger picture about the business. In short, having a director assigned to two departments rather than two individual contributors, each assigned to one department, and reporting to different managers. To put differently, in this case, talent + talent < vision, or $110,000 salary + $110,000 salary < $150,000 salary (in terms of total return). What do you think?

Related articles

Views: 1690

Comment by patrick anderson on July 9, 2013 at 3:41pm

I work for a state public health department and data silos are a big issue. Different programs with their own data sharing security procedures make linking data a challenging task. This makes it much more difficult to look at comorbity in a population because a researcher may not be able to access the necessary data from different programs which weakens the analysis and may lead to inaccurate results. To get around this we develop program to program MOU's so that we can get as much data as we can without violating disclosure laws. But these are only good for the data agreed upon and if more data is needed the agreement must be updated before the data can be released. Also if a new project is started with different data needs, a new agreement must be written. The only other way is to develop security protocols that allow for linkage. But again some of these are mandated by the federal government and in those cases we may not be able to link the desired data sets at all.

Comment by Scott Raspa on July 18, 2013 at 6:36am

Good post. I actually mentioned this post in a recent blog post "Breaking down data silos" ( 


You need to be a member of BigDataNews to add comments!

Join BigDataNews

Sponsored By

On Data Science Central

© 2021   TechTarget, Inc.   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service