Summing Up: Will Access To Big Data Further Enable Fact-based Decision Making Or … Analysis Paralysis?
"To T. S. Eliot's prescient words 'Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?' we might now add, 'Where is the information we have lost in data?'"
That's Paul Nicholas' reaction after reading most of the responses to this month's column. It's not a bad "sense of the meeting," in which many contributors offered suggestions to managers wishing to get the most out of so-called big data and avoid paying the price of paralysis in the process.
The challenges Big Data pose for managers include "identifying which data are relevant" (Subrata Chakraborty) and "seeing through the woods to know what to use and what not" (Pieter J de Beer). Scott Waller expressed the fear that "the age of Big Data seems to be arriving at the time of death for the Big Thinker." Gerald Nanninga cautioned us that "the big risk is that it gives executives a false sense of comfort." Clifford Francis Baker added, "My concern is primarily focused on the possibility of complacency … data derived from data analytics must … be handled with care." Mok Tuck Sung commented that "technology and knowledge advancement have again developed faster than the managers' and business owners' capabilities to leverage its usefulness in their decision making process." Philippe Gouamba reminded us that "perfection is but an illusion … A successful outcome is almost never the result of perfect information."
Among the suggestions for managers were these:
- Help "data analyzers consume and translate the data" (Scott Kemme, who also suggested an alternative title for the column, which I instead used above),
- Know "what not to look at." (Phillip Clark)
- Work to reduce turnover among business analysts who "tend to be lower level employees and have a high turnover," creating a "losing battle" through the loss of "institutional data knowledge." (Kim Kraemer)
- Avoid the belief that "whatever is new will solve their problems," concentrating on the appropriate application of Big Data. (Seena Sharp)
- Avoid allowing Big Data to remain the "purview of the select few" only for use for one-off and one-time decisions." (Jonathan Spier)
- Maintain the attitude that "fast is better than perfect." (Mike Flanagan)
- Avoid the temptation of harvesting just the "'low hanging fruit' rather than waiting for the analysts to gain understanding (presumably of the decisions to be made)." (Sean O'Riordain)
- Rather than concentrating on the data, focus on "being able to formulate the right questions to ask at precisely the right moment." (Edward Hare)
That's quite a list.
Tom Dolembo drew a picture of direct process control in which decisions have to be made "in the flow" in situations where there isn't time for conventional analysis (at least by humans). Is this a glimpse into the future of decision-making (without analysis paralysis) based on big data for a widening range of decisions or is it confined to a special set of conditions? Will access to Big Data further enable fact-based decision making or … analysis paralysis? What do you think?
Original Article
Ideas and trends converge from time to time in a way that suggests the possible shape of the future. Sometimes I think I can comprehend what they may mean. But other times I know I need help. This is one of those times.
Just two decades ago, we didn't have Google and other information sources; storage constraints would not have permitted Google to provide everyday access to the "world's information." If we had had the information, we couldn't have accessed it effectively anyway. Email systems were not widely available, let alone mobile devices with capacity to access the data. Now the capacity to store and access information through cloud computing is so great that we are entering a post-Google era in which new organizations like Factual (founded by a former Google employee) have set as their goal that of providing access to all of the world's facts. Presumably this means data such as the location of every factory in the world, data that has not already been massaged and spun. Some facts have to be acquired and organized. Other facts are generated by so-called digital sensors operating worldwide in industrial equipment, autos, and the like. By linking the sensors, an "industrial Internet" can be created.
These trends appear to have "opportunity" written all over them, particularly for those who are training now for jobs in data analytics. In addition to less wasteful marketing efforts (we should be able to know, for example, "which half" of advertising is effective, thereby making an old marketing saw obsolete), they should produce more effective business strategies and inject added certainty into the appraisal of opportunities for new business startups. Furthermore, analytics (not the data) should be a source of continuing competitive advantage. In his new book, Charles Duhigg describes how the retailer Target uses data on consumption patterns to discern and address promotions to pregnant customers, perhaps even before they've announced their pregnancy to friends (and Target competitors). This is particularly important because pregnancy is one of those life events associated with significant shifts in consumption habits.
A problem is that the shortage of experts in data analytics (some call them "data whisperers") is so acute that it may be years before a sufficient supply can be trained. The McKinsey Global Institute estimates that up to 190,000 are needed now in the US, along with 1.5 million managers capable of using their work. The shortage appears to be growing along with the potential for competitive advantage associated with data analytics.
This all raises many questions. Will the age of big data eliminate most or all uncertainty from business decisions for those most able to make effective use of "all the facts in the world?" Will it fuel the next "gold rush" for talent in a quest for competitive advantage? Will analytics, as well as the supply of analytics-savvy managers, so badly lag "big data" that it will only lead to confusion and misguided decisions? Or is this just the latest management fad? How, if at all, should this affect education for management? What do you think?
To Read More:
Charles Duhigg, The Power of Habit: Why We Do What We Do and How to Change It , Random House, 2012.
Quentin Hardy, Just the Facts. Yes, All of Them, The New York Times, March 25, 2012.
Steve Lohr, The Age of Big Data, The New York Times, February 12, 2012.
The challenge—and opportunity—of 'big data', McKinsey Global Institute, May 2011.
Beyond the need for analytical resources is also the need for tools to help consume and translate big data into understandable and, more importantly, actionable analyses that can drive change into the organization. The sheer size of Big Data necessitates that kind of assistance.
An additional benefit of such tools are they may enable a broader population of data analyzers than might otherwise be possible by the way such data is presented to the end user.
If you help the data analyzers consume and translate the data, management can then assume the expected role of decision making rather than analytical support.
I suppose the great opportunity question is how this rolling sea changes us as customers? What we found is that every process is really unique, mass production is a nineteenth century mythology (imprecise fits, imprecise data). It never existed. If we can have unique products that change as we do, be the color and do things we want them to do, before we want them to do them, be platforms for data transformation, not just appliances, we're closer. Can a car be a coffee maker and a toaster and a toothbrush and a sensory experience, data transforming transportation into something else? We have developed markets for discrete products that only do discrete things, just go to any garage sale, obsolete needs populate our trash. Can we own a Zip Clip that drives us to work or entertains us keeps us warm and feeds us and clothes us when and where and how we want? Not really that hard to do. Think inside the flow of the data, the data itself is a distraction.
We have entered the age where the smart person will be the one who knows what not to look at. You cannot make deadlines or good decisions looking at everything.
Analytics is not an end in itself, but rather a means to some result or goal that the organization has. By assessing analytic projects in terms of their contribution to corporate goals, we can be assured that we are using this new resource in the most optimal way.
The problem with new ideas/concepts/phrases is that companies are so anxious to succeed that they believe whatever is new will solve their problems and bring the growth they're seeking.
Like re-engineering, Big Data will be erroneously applied in many companies and will result in poor decisions.
Data is very useful, but it must be recognized as measuring the past. It provides an alert to changes, but because of the time lag and today's constant changes, Big Data may be disinformation.
The popular management saying, "If it can't be measured, it can't be managed," fails to recognize that not everything fits into metrics.
That's the enormous hidden value of market intelligence. It reveals what's changing and emerging - before data can be gathered. That's the source of opportunities, threats, and growth.
Seena Sharp
Sharp Market Intelligence
Author, Competitive Intelligence Advantage
But to impact management the big challenge is: How can big data become more than just more, well, data at a time when we're all already drowning in data?
For big data to impact management means that it is tied into practical, every day uses. After all, although the long term order of magnitude changes are exciting, what most of us are looking for are near-term ways to make the existing things we do X% more efficient.
In other words, as long as big data really requires armies of "data whisperers", it will remain the purview of the select few for one-off and one-time decisions. When it will get more exciting is when we can bring the fire to the masses by embedding better analytics into many of the things we do in business every day.
Can technology fix that problem?
That said, rejecting the irrelevant needs focus. We are in a real world of business. Our resouces - time, energy and money - are scarce and have to used very carefully. Once we are serious on this the need for such a large backlog of expert data analytics and the user-managers may not be there. It is neverthelss a hard fact that skills of all sorts are in very short supply; this applies to all work ares and the cases in question are no exception.
As other studies have shown, managers will draw their own conclusions and then get analysts to find the data to back up the conclusions... confirmatory bias...
The big risk is that it gives executives a false sense of comfort. It breeds the idea that "the computers have it all figured out; it is based on science and math, so what can go wrong?" This lets the executives off the hook on having to think deeply or creatively...or so they believe. So we get lazy in our strategic approach (and pay the price).
I'd rather have a few creative geniuses than a roomful of data whisperers. Some of the advice I've seen from data whisperers is simply awful, because they are so out-of-touch with real people doing real things.
I wrote more about the limits of analytics here: http://planninga-from-nanninga.blogspot.com/2009/05/strategic-planning-analogy-259-quant.html.
My advice.....find those in the organization who can separate the wheat from the chaff. They may be lurking anywhere and could be those you'd least suspect. Spend the time to identify them. Even if it will take more of your time and is a lot more tricky to do than subscribing to the latest fad.
Fast forward to today, with the advancement in data management technology and analytical skill in the past 13 years, they should have a new version of their Profit Pattern.
My observation, is a repetition, where technology and knowledge advancement has again developed faster than the managers and business owners capabilities to leverage it's usefulness in their decision making process.
It is exciting to dream about making perfect business decisions as we hold "all the facts in the world" in our hands but perfection is but an illusion and one that we need to be very caution with.
Lets take a step back and away from this illusion of perfection and think for a second about three men: Bill Gates, Mark Zuckerberg and Steve Jobs. We all know who they are and what they have accomplished. If you do not know them, please google them when you have a free moment. Here is my point: If they had "all the facts in the world" in their grasp, would they have accomplished what they did? I think NOT. All the facts in the world include the fact that most start-ups fail in the first year of existence. It includes the fact that most ".coms" did not survive the .com bubble which exploded famously. If EVERYONE had perfect information and perfect decision-making skills would EVERYONE be a Bill Gates, a Mark Zuckerberg or a Steve Jobs? NO! If President Bush had "all the facts in the world", would he have kept us out of the conflict in Iraq? No! He was driven by things other than the facts.
If there was ever a perfect business decision it did not come about because some manager or executive had "all the facts in the world". It came about because someone did some research, committed some resources to a worthwhile idea and made a decision and stood by it. A successful outcome is almost never the result of perfect information. It is the result of thoughtful leadership, of risk-taking, of creativity and of taking full advantage of a window of opportunity that may or may not ever come again.
The age of big data may be here but even it has not and cannot change the nature of man, nor the fact that leadership and decision-making still require human interaction in the form of gut-checks, risk-taking, intuition, memory, experience, timing, creativity and leadership among many other contributing human in-puts.
Thank you, Professor Heskett for your questions which are always though-provoking.
All these managers are sometimes right and sometimes wrong whether the decisions are based on data analysis or not based on data analysis. Hence, the objective of scientific data analysis is to reduce the chance of making the wrong decision ideally to 1% or 5%. If a decision is made with no data analysis there is no basis to verify how likely the outcome could be incorrect. If a decision is made with data analysis there is a basis and verification but it still depends on assumptions made especially that historical data could forecast the future outcomes, data size (big or small), data quality ("cleaning" data is as important as analysis, understanding and interpretation), analytical techniques (techniques that apply to small data sets not necessarily apply to big data sets and vice versa) and data analyst's (business analyst's) ability to understand and interpret the data! Data analyst (business analyst) may provide the evidence and the assumptions of the basis but
the manger could be a nonbeliever of data analysis or would guide the data analyst to back his/her management decision already inked. On one hand, the greatest harm a data analyst (business analyst) can do is not to speak up his/her reservations on how the basis is used for decision making when it is incorrect and on the other hand a warning is timely. Speaking up your reservations may bring the analyst into disfavor with his/her supervisors, bosses or internal clients who may have done the wrong basis for so long and they may set you up and get you fired for a flimsy reason (or make you throw in the towel and resign from the company) to bury their ills from their superiors!
at the future is ripe for engineers who are pioneers and are able to extend the concept of firewalling for individual organisations - tailor-made services which would save a large percentage of valuable employee hours discarding and deleting irrelevant and unecessary "information" and "news" at the same time adding to that organisation's competitveness in their own marketplace. We have to escape the current concept that" too much ain't ever enough" and return to individualised data management including informatics and analytics but on a bespoke scale which could be engineered even for separate hierarchical levels in a company whether it is public, voluntary or private sector . Training pioneers is virtually impossible likewise entrepreneurs who have the ability and aptitude to "see" opportunities therfore it is left to Business Schools to devise relvant open programmes where the criteria for assessment is perhaps taken from the indivi
dual student rather than imposed on by a risk-averse mechanical process so often found in places of "learning" .
The early 21st century obsession with data and its mining for information perhaps mirrors the obsession of late19th Century physics with a quest for exactitude - nothing new was expected more than that everything was to be measured to ever greater degrees of accuracy - until early 20th Century physics found the world is full of uncertainty.
To T S Elliot's prescient words "Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?" we might now add "Where is the information we have lost in data?"
My concern is primarily focused on the possibility of complacency developing on the part of corporate board members and officers toward the basic need for request due diligence necessary in any decision making process, as they depend too exclusively on the data formulated and presented.
As anyone who is even vaguely familiar with accounting procedure and methodology knows the ease by which accounting data can be manipulated, I believe all and any data can an most likely will also be manipulated.
The recent negative and near disastrous effect on global markets caused in part by investor dependence on the data and conclusions of many analysts of contemporary market behaviors, I consider to be exemplum primi est.
As with handling a loaded gun, data derived from data analytics must also be handled with care.
Is it relevant? Is it current? Is it from reliable sources. What does it mean? What should we do about it?
1. The destination or goal
2. How to navigate the way there
3. The way in which to swim
If he can build the biggest company in the world by just focus and simplicity then Big Data is nothing but analysis-paralysis!! More data means more information, and more information means more ways of analysis it, and more ways of analysing it means more conclusions you can reach!
New possibilities prompt the need for new ways of management. Systemic thinking with a flair for analyzing and synthesizing data would be some of the traits of big data management professionals.
Current data managers will need to categorize their applications to determine when to use and not use the possible resources for information. They will need to be able to determine the need for filtering and distilling the data to derive value of the information.
A whole new type of data products reinventing industries is possible with big data with the convergence of several technologies such as artificial intelligence, bioinformatics, sensors and networks to name a few. Big data product managers will be foreseeing the product road maps of such products. Big data application managers seem to be like those senior editors who work with data journalists in the newspaper / publishing industry.
Their skill set needs to match their nature of work, a fundamental shift from the more established general, process, product, project, program managers in any industry. The data revolution would be big with immense opportunities and the growth could be exponential.
In fact, I think it's the wrong question whether we will find (or have) a shortage of analytics saavy managers. Certainly a manager with some level of depth into analytics is helpful, but they should not need to have a background in data science to manage their department. Some of this responsibility should be shouldered by vendors that are producing intelligence tools for managers. Not in the sense of sourcing and recruiting professionals, but making business analytics more human so even a manager without a deep data background can effectively use intelligence tools for informed decision making.
I believe with the advent of mobile offerings we are starting to see the beginnings of some of these platforms that will only continue to expand into the workforce.
better analysis....
best to integrate....test...and continue to tweak..
Evolution of data is in real-time. Must keep that in mind, when analyzing. Then push the data further, by not getting caught up in detail. Think holistically, with a systematic process in mind.