What do Facebook, Google, Twitter, LinkedIn and Etsy have in common? 

They were all founded between 2004 and 2008. Ninety percent of the world’s data has been created over just the last two years alone as a result of this new breed of “big data” companies that store every activity their customers perform on their sites to feed collaborative filtering algorithms. These big data pioneers recognize that sharing and re-enforcing preferences is what makes successful collaborations work: Data analysis and machine intelligence can now be seen as a form of collaboration between humans and machines — we generate the content; they analyze it for us — so we don’t really need our own brains anymore.

This sounds like an extremely dystopian vision, especially for those people who still enjoy thinking. But the good news is that it’s not going to happen as soon as you think. In fact, I’d venture to say that it will never happen — because data analysis by itself isn’t really useful for much beyond early-stage problem detection. We need our human brains for real-world problem solving and those collaborative filtering algorithms cannot do this job nearly as well as a human brain simply because they lack semantic understanding of what they see and recommend.

What humans can uniquely offer is the ability to understand patterns in the raw data establish their meaning and take meaningful actions based on these insights. Without us, big data becomes nothing more than a clock that no one has winding up or knows how to stop.

Unfortunately, we’re nowhere near being able to fully harness the power of all this data. One problem is that the algorithms poorly map the real-world problems they are supposed to solve; another is that existing tools are not designed for what I call “semantic data analysis”, which requires finding relationships between data points that are meaningful in a specific business context.

Most analytics tools rely on what you already know to look for correlations or patterns in your big data. But if your hypotheses about the cause and effect relationship between two variables aren’t correct, even an enormous volume of seemingly informative data won’t help you find whatever it is you’re looking for. It would be like standing in Times Square with a metal detector trying to find precious metals while simultaneously being blindfolded — lots of things will make a lot of noise, but you still won’t get very far.

To get the most out of big data we need to set up our analytical processes in ways that allow us to find insights we don’t already know we’re looking for — and this is where semantic analysis can help: 

• It analyzes raw data on its own terms and establishes relationships between the entities and concepts it identifies without relying on outside knowledge. The result is more reliable information that can be fed back into collaborative filtering algorithms so they improve their outputs over time. This symbiosis has improved collaboration efficiency by several orders of magnitude in many applications such as micro-targeted ad placement, digital music streaming services and machine translation systems. However, there’s still a long way to go before large enterprises can become fully collaborative.
• What makes an intelligent system truly smart is its capacity to understand the entities it sees and hears, to determine relationships between them and take actions based on what it has learned. Semantic technologies are now making this possible by analyzing raw data directly, allowing us to focus our human intelligence on business-critical priorities rather than wasting time filtering noise. The future will be all about getting more done with less effort — so start training your brain today for a collaborative future driven by big data analysis where quality insights replace long hours of manual work as the new competitive advantage! – Joseph Sirosh, Corporate Vice President at Microsoft Corp.

Conclusion: 

The future is collaborative.

It may take several years before computers fully understand the meaning behind data, but that’s okay because we won’t stop improving our own abilities either. The goal isn’t to replace humans with machines or vice versa, it’s to make sure both can do their best work together—that means working smarter instead of harder. 

The real problem is how to learn faster and not “slower” as many are conditioned to believe after being taught by traditional education systems for so long. We need smart crowds who know how to think critically, debate ideas collaboratively, and then act decisively upon these insights—this is the only way that big data will truly become useful beyond mere early-stage problem detection. This kind of thinking requires a deeper understanding of how we can unlock our own potential, and learn to collaborate with others in a way that accelerates the rate at which we solve problems together.