https://ritholtz.com/2019/02/the-plural-of-anecdote-is-data/
One of the key lessons of behavioral economics is the danger of not examining your own beliefs. Failing to consider the reasons that underlie our decision-making increases the risk of error. Although we can operate in our daily life on autopilot, other situations are not so forgiving. When we put capital at risk, the danger of not understanding what influences our decisions can be monetary losses, career risk or worse.
In this context, let's consider the aphorism, "the plural of anecdote is not data." This is the idea that a single example should never be used to extrapolate a broader rule about, well, anything. This applies to stocks, the economy, politics, Brexit — just about any situation where a compelling narrative might influence your views in spite of a dearth of evidence.
The truth of this statement is so self-evident that there seems to be little reason to have to investigate whether the quote is accurate or not. We tend to use it almost reflexively, usually in an attempt to refute a conclusion that cites a data point. Anecdotal evidence is not mathematically or scientifically sound. When our sample set consists of a single example (N = 1), our conclusion will have a margin of error of plus or minus 100 percent. In other words, as the fine print states, it is statistically insignificant.
We have learned the problem with extrapolating from single examples thanks to the work of Amos Tversky and Daniel Kahneman.1 In 1973, they were studying various mental shortcuts that people rely on when making decisions with incomplete information. How easily an example might come to mind — including anecdotes or random examples — might not be representative of the real world. Thus, the "Availability bias" was discovered.
Perhaps the best real-life example of the availability bias are shark attacks. Most of the time, interactions between humans and sharks occur without harm to the humans, but when they are to someone's detriment, lots of media coverage tends to follow.2
But the reality of the danger is very different. More people were killed by mosquitoes last year than have been killed by sharks in the past 100 years. Indeed, annual deaths from selfies exceed yearly shark fatalities. In reality, you are much more likely to die from a medical error, which is the third largest cause of deaths in the U.S., than from a shark attack. What else is more deadly than sharks? Try armed toddlers — young children who happen to get their hands on a firearm. But shark attacks are more memorable and dramatic, and therefore readily and easily recalled.
Which brings us back to anecdotes: As it turns out, the original quote about anecdotes had a very different context, and a much more nuanced meaning. It is attributed to Ray Wolfinger, who was a political scientist at the University of California-Berkeley.
Wolfinger's original statement was quite literally the very opposite of what we all have been using. He had actually said "the plural of anecdote is data." This should might affect the way we think about and use data.
The earliest discussion I could track down of the original quote was via the American Dialect Society. Fred Shapiro, former editor of the Yale Dictionary of Quotations, had an email exchange with the professor about the statement's origins. Wolfinger recalled responding to a student's dismissal of a factual statement as a mere anecdote, told Shapiro: "It was meant to suggest that data does not have an immaculate birth, and that anecdotes lead to deeper research and then data."
The professor's take was not a warning against extrapolation or anecdotal evidence. If anything, he was encouraging data scientists to delve deeper into their experiences to discover fertile new areas of research and exploration. To the alert observer, a compelling anecdote should start the process of digging into the data to determine if something is merely an intriguing one-off or emblematic of a broader trend. A good story should be considered preliminary evidence, the start of a more serious inquiry.
In other words, the plural of anecdote, to be more precise, might be valid data leading to a potentially significant conclusion. For that reason, when an unusual anecdote capture one's attention, it shouldn't be casually dismissed, lest a deeper truth be missed.
Consider the ramifications of this for how analysts, economists, fund managers do their jobs. Algorithms are increasingly replacing repetitive tasks, and for people who work in finance, this is potentially an existential risk to their careers. The ability to identify something via an anecdotal observation, then use data to discover a new idea or concept might be relatively immune from the machines coming to replace you. So take heart: It might be a while — if ever — before AI and big data are sophisticated enough to do just that.
_______
1. Availability: A Heuristic for Judging Frequency and Probability, The Hebrew University of Jerusalem and the Oregon Research Institute, Cognitive Psychology 5, 207-232 (1973)
-- via my feedly newsfeed
No comments:
Post a Comment