The saying “the Hedgehog knows one thing, the Fox knows many things“ from Isaiah Berlin’s essay is a good summary of why useful Big Data systems are so difficult to deliver: the sheer breadth and variety of knowledge required to build the right system and make it work is overwhelming.
I have been involved in, led, and delivered many Big Data projects over the years. Usually, the Hedgehog recognizes the need for the Big Data system first and drives it: “If only we were collecting all this data, then we could solve all these problems.“ And they pursue that goal single-mindedly. With luck, they end up with a system that gives them… a big pile of data.
With superhuman effort and great ingenuity, this pile of data is usually forced to yield some new insights about the business. However, the effort is too great, and the learning too uncertain to justify further investment. The work is then abandoned but stands as a roadblock to further Big Data projects: “We spent $X million last time and you really didn’t use the data.“
This is where the Fox is needed. In my experience the Fox does not typically have the focus to make people take action as the Hedgehog can. Seeing the bigger picture, the Fox tells too complex a story and, unfortunately, often loses their colleagues in the process. The Fox knows that it is not sufficient to simply collect data. The Fox knows that a “Big Data” project is really a “Big Information” project. To be successful, the Fox will need to understand how to collect the data, how to turn it into Information that is useful, and how that Information will be used by people trying to make decisions. The Fox knows that people won’t wait if this information is not available instantaneously, and that the business’ needs will change over time. Therefore, the system must be capable of changing as well.
Big Data projects need a Hedgehog and a Fox. A Hedgehog to get the organization to take action, and a Fox to understand the subtleties and ensure that what is delivered really fulfills what is desired, which may be unspoken.
You truly need a Hedgehog and a Fox who work together and respect each others’ complimentary skills. It’s all about people in the end!