I recently stumbled upon one of The Martin Agency's hilarious Geico caveman ads and wondered, rather geekily, why they didn't do one about data analysis. I think if a caveman suddenly arrived in the 2010s he or she would see parallels between his life and the activities of today's knowledge worker.
When I thought it through, it seemed obvious that knowledge workers need to be more like farmers and less like hunter/gatherers if they want to achieve the full potential of business intelligence.
For hunter/gatherers, food is there for the taking, but the taking can be tough. Lucky tribes live peacefully among abundant small game and plants bearing nuts, berries, and fruit. Others follow herds of antelope or buffalo and hunt in teams, and so on. Despite our rosy imagination, primitive life features risks of disease and extreme weather. Even so, surely many in these societies took great satisfaction in their lives and valued the skills that enabled them to survive and sometimes prosper.
Today's knowledge worker collecting stove-piped data resembles a hunter/gatherer. Lucky analysts work surrounded by abundant sources of compatible operational data meeting day-to-day needs. Others seek out scarce data from obscure internal or external sources in response to urgent requests. Despite our rosy imagination, life as a knowledge worker features wrestling with massive spreadsheets and hand-correction of incompatible data sources. Even so, analysts operating in such organizations pride themselves on the research abilities, MS Excel skills, and capacity for hard work that makes them so important to success of the organization.
Between 7,000 and 10,000 years ago agricultural and herding societies emerged. People gathered in communities and maintained animal herds or planted grains, vegetables, and fruit. Communities coalesced around farming and ranching centers. While hunter/gatherers sought food every waking hour, farmers could take time to assess challenges and generate novel ideas like irrigation, bronze, commerce, and more. Maybe rugged hunter/gatherers ridiculed farmers and herders as "soft" or "settled", but the social and technological advantages of settled society speak for themselves from our vantage point today.
Since the data warehousing movement began to take hold in the 1990s, some organizations have gathered data into integrated stores - whether virtual or logical - scrubbed of the incompatibilities that result from stove-piped operations. These organizations support a community of reporting and analysis practice among knowledge workers. The rise of data management standards and values have reduced cost of routine analysis and resulted in unforeseen insights. Early on one leading innovator built a credit card empire by understanding how to lend to sub-prime borrowers. In the years since, retail, supply chain management, and many other entire industries transformed as companies gained unforeseen efficiencies and opportunities from managed data.
The Present: Removing Barriers to Integration
Thirty years after James Martin, Bill Inmon, and others developed the foundations of data management, some organizations have implemented effective organization-wide capabilities, but data hunting/gathering remains dominant. The evidence is all around: multiple, inconsistent sources of the same business-critical data, important business data missing, key operational reports sourced from a single spreadsheet maintained by a single analyst, and so on. Maybe the root of the problem is that we see this as an IT challenge.
If you think about it, IT is part of the environment: business knowledge workers are the ones seeking nourishment here. As long as analysts continue working as isolated actors pursuing short term goals they will continue to hand-integrate inconsistent data. When business stakeholders choose collective, settled data management rather than valuing hand-integration will win and enable vast improvements that we can't imagine today.