Making Sense of Big Data: Nate Silver on the Signal and the Noise
Thanks to the constant barrage of data associated with the information age, many claim we are living in the “era of big data.”
Nate Silver, author of The Signal and The Noise and founder and editor-in-chief of FiveThirtyEight, explains how to differentiate between “signal” and “noise,” and discusses the danger of false signals, in a recent Take 15 interview with Lauren Foster.
“I think the challenge people don’t realize is that when you have more and more data, in some ways that makes it harder,” Silver says. “Now you have to choose which data you’re looking at, what you took to be the signal versus the noise. What I think we need is a series of best practices . . . We need more experience working with this data.”
Silver notes the difficulty of defining data. “Everything is data in a sense — anytime you’re trying to go from an anecdotal example to generalize, then you at some point have to use a common language,” he says. Given this broad definition, Silver cites the importance of classifying data before utilizing it for forecasting. “Data sets that are unstructured are very raw,” he says. “Before you get into analysis and prediction, organization is important.”
As for why signals and noise are easily confused, Silver looks to brain development. “The world’s evolved a lot since when our brain chemistry and our common sense were developed, and we’re bombarded now by a lot more data than we were used to,” he says. “As this becomes more complicated, there are things we might not understand, or there are things that are caused by random chance that we mistake for a signal.”
Given this difficulty, Silver identifies two important components to forecasting: thinking in terms of probabilities or uncertainties, and starting out from a position of skepticism. “I used to play poker,” he says. “Poker’s a game where there’s a lot of luck, a lot of skill. You learn how easy it is to confuse the two, and you have your ups and downs. I think that mentality helps (in forecasting).”
If you liked this post, don’t forget to subscribe to the Enterprising Investor.
All posts are the opinion of the author. As such, they should not be construed as investment advice, nor do the opinions expressed necessarily reflect the views of CFA Institute or the author’s employer.