Daniel Kahneman: Financial Advisers Aren’t Immune from the “Illusion of Skill”

A few weeks ago, Daniel Kahneman, the Nobel prize-winning psychologist and pioneer of behavioral economics, wrote a fascinating article in The New York Times Magazine that examines the illusion of skill as it pertains to the stock-picking prowess of financial advisers.

In the article, which was adapted from Kahneman’s new book, Thinking, Fast and Slow, he relates the story of a time he had been invited to speak to a group of investment advisers in a firm that provided financial advice and other services to ultra-wealthy clients. To prepare for the talk, Kahneman requested and was given a spreadsheet summarizing the investment results of 25 of the firm’s wealth advisers over eight consecutive years. The advisers’ scores for each year were the main factor in determining their bonuses.

Kahneman — who will be speaking at the 65th CFA Institute Annual Conference in Chicago in May — sought answers to two questions: Did the same advisers consistently achieve better returns for their clients year after year? And did some advisers consistently display more skill than their peers?

To find the answers, Kahneman calculated the correlations between the rankings of advisers in different years, comparing year one with year two, year one with year three, and so on through a comparison of year seven with year eight. This exercise generated 28 correlations, one for each year.

Kahneman was taken aback by what he discovered. “While I was prepared to find little year-to-year consistency,” he writes, “I was still surprised to find that the average of the 28 correlations was .01. In other words, zero. The stability that would indicate differences in skill was not to be found.”

To which the behavioral theorist offered this kicker: “The results resembled what you would expect from a dice-rolling contest, not a game of skill.”

How, then, do you figure the directors of the investment advisory firm reacted when Kahneman told them, as he puts it, that when it came to portfolio construction “the firm was rewarding luck as if it were skill”?

Where they stunned? Horrified? Puzzled? To the contrary: They were completely unfazed.

“This should have been shocking news to them,” Kahneman writes,  “but it was not…. I am quite sure that both our findings and their implications were quickly swept under the rug and that life in the firm went on just as before.” In fact, Kahneman adds, while en route to the airport he was told by one of the firm’s executives, somewhat defensively, that he had “done very well for the firm and no one can take that away from me.”

One of the conclusions Kahneman draws is that “the illusion of skill is not only an individual aberration; it is deeply ingrained in the culture of the industry.”

Which brings us to what is perhaps the central conundrum in all of this — and something that private clients and their advisers wrestle with constantly:  who can you trust?

“We often interact with professionals who exercise their judgment with evident confidence, sometimes priding themselves on the power of their intuition,” Kahneman writes. “In a world rife with illusions of validity and skill, can we trust them? How do we distinguish the justified confidence of experts from the sincere overconfidence of professionals who do not know they are out of their depth?”

Kahneman suggests that “true intuitive expertise” is learned “from prolonged experience with good feedback on mistakes.” This is a point that reminds me of a similar sentiment that J.P. Morgan Asset Management CEO Mary Callahan Erdoes expressed to the Financial Times back when she was chief executive of the firm’s private bank: “I think that if you’re going to be a successful money manager, you have to have lost money…. The smartest people have to go through up markets, down markets, sideways markets and you have to feel the pain of losing money.”

When it comes to knowing whether you can trust a particular intuitive judgment, Kahneman says that you have to ask two questions: First, “Is the environment in which the judgment is made sufficiently regular to enable predictions from available evidence? The answer is yes for diagnosticians, no for stock pickers.” And second: “Do the professionals have an adequate opportunity to learn the cues and the regularities? The answer here depends on the professionals’ experience and on the quality and speed with which they discover their mistakes.”

So what is one to make of all of this? In general, says Kahneman, “you should not take assertive and confident people at their own evaluation unless you have independent reason to believe that they know what they are talking about.”

That may be easier said than done. And Kahneman concedes that this advice is hard to follow: “Overconfident professionals sincerely believe they have expertise, act as experts and look like experts. You will have to struggle to remind yourself that they may be in the grip of an illusion.”

For more information on the work of Daniel Kahneman, read Michael Lewis’ December 2011 profile in Vanity Fair or this CFA Magazine roundtable discussion from 2005 featuring five Nobel laureates, including Kahneman, whose work has profoundly influenced investment theory and practice.