The mainstream financial media is built around having experts appear on screen to give the comforting impression that they know what is going on.
Whether the stock market has gone up or down, they always have an explanation. We've been conditioned to expect this from them.
It is human nature to make coherent stories out of what we see happening around us. We crave the simple narrative of cause-and-effect.
So when the market drops on the same day that high unemployment figures are released, we link the two things together. We don't, however, make the same connection when the market goes up on the day that even worse unemployment figures are put out a month later. We instead attribute this to something else.
The need for order
The author of The Black Swan: The Impact of the Highly Improbable, Nassim Taleb, calls this the narrative fallacy. It is our natural impulse to assign reasons for everything that happens, even when the evidence is flimsy.
It is one of the ways in which we make sense of the world. Since we are constantly faced with so much information, we have to put it in some sort of order to be able to process it. The easiest way to do that is to see a cause for every effect.
In simple matters, this is important and often critical. If your foot is burning because you have put it too close to the fire, then it is worthwhile being able to understand that.
In complex matters, however, the accuracy of this approach is far less perfect. This is because there is often far more going on that we can see on a superficial level, and random chance plays a much bigger role in the world than we like to admit.
Understanding the market
This is certainly the case with stock markets, which are highly complex systems. There are always many competing forces at play. Even when prices are falling sharply, for instance, there still have to be two parties for any trade to take place.
The people who are desperate to sell their shares have to be selling to somebody. Logically, this must mean that any one-dimensional rationale for anything that happens in the market is an over-simplification.
The human mind also revolts against the idea that chance and luck play a huge part in how the world works. Take Warren Buffett for example. He is widely viewed as one of the most successful investors of the modern era. Many very intelligent people can make extremely persuasive arguments about why this is the case — how his brilliant investment process explains his ultimate success.
However, a sober assessment of Buffett's career would reveal that his approach was not particularly remarkable. It certainly wasn't revolutionary. Buffett himself has said that you don't have to be a genius to be a good investor. Mostly he relied on the ideas of others that came before him.
Without diminishing Buffett's achievements, there were also any number of things that might have disrupted his investment. There were any number of times that bad luck could have derailed his path, but didn't.
There is no question that Buffett is extremely astute, but he has also been exceptionally lucky. Consider that if there is a normal distribution of investing success, it is inevitable that someone would have to be as successful as Buffett.
The important question, however, is whether you could have known when he started out that that individual was going to be him?
It's an illusion
The pioneering behavioral scientist Daniel Kahneman writes that this is really the key. When looking back over Buffett's career with the advantage of hindsight, it might appear that his success was inevitable. However, Kahneman writes that:
“The ultimate test of an explanation is whether it would have made the event predictable in advance.”
Nobody, including Buffett himself, could have foreseen 40 years ago, that he would be who he is today. The simple fact that his investment process is public knowledge and many people have followed it without anywhere near the same level of success emphasizes how true this is.
This, notes Kahneman, highlights the 'illusion of understanding' – our predisposition to build stories from the information available to us. And the better the story is, the more likely we are to believe it.
Buffett's career makes for a great story. But it becomes a lot less appealing if we emphasize his luck, rather than his skill.
Looking back, looking ahead
The real danger of the "illusion of understanding", however, is that it doesn't just shape our view of the past. As Kahneman writes:
“The core of the illusion is that we believe we understand the past, which implies that the future should also be knowable.”
In other words, because we believe that we are able to assign reasons to everything that happened in the markets yesterday and the day before, we also believe we are able to predict what they will do tomorrow. In other words, we overestimate our ability, and the ability of "experts" to make accurate forecasts.
However, as Kahneman reminds us, the world is unpredictable. Many studies have shown that the ability of even leading experts to make accurate forecasts about anything is extremely poor.
This is worth remembering the next time any expert insists that a market is going to perform a certain way, or there is a particular investment outcome is certain. Even if they have a good story to go with their assessment, there is really no way of knowing if they are going to be right. And there's about an even chance that they won't be.