Use Big Data to Make a Science of Observation

As we saw last week, people just aren’t good at noticing things! I hope you had fun with Richards Wiseman’s entertaining and amazing ‘magic’ trick, and are now suitably skeptical of any effort to collect actionable information by human eye. This is what Big Data is for: to make a Science of Observation.

Remember what Big Data is:

Big Data is detailed, real world data, in large quantities, which can be used to answer important questions that have not been asked yet.


How do we build a system to answer unknown questions?

Collect Automatically

Collect the data automatically. If you are still unsure whether this is really worth the trouble, go watch Richard Wiseman’s card trick again!

Who Where When What

Put it in context. You don’t know what problems your data will have to solve, but you do know what context these problems will occur in. You know what the important measures are for your business, and you know the key factors which are likely to influence any outcome. If in doubt, check that you can answer the following questions about each piece of data you collect:

  • What is it?
  • When was it collected?
  • What was happening at the time?
  • To Whom or What?
  • By Whom?
  • Where?

Data ScientistRemember that People are not Data Scientists! The vast majority of people are poorly equipped to analyze data, and are uncomfortable spending time searching through data for an answer that might or might not be there. Being caught back at your desk “browsing” through your computer when the problem is out there in the real world! — on the production floor or in the field or on the Customer’s site — is a career limiting choice that people rightly avoid. So you make it easy for them, by “pre-analysing” the data as far as possible. Again, you know the key measures and inputs for you business, and the formulas to get from inputs to measures: the problem may change, but these will not.

Now you have INFORMATION, not just data, or even, heaven forbid, educated guesses!