The Simpsons’ Predictive Powers Demonstrate the Limits of Forecasting

by
Dan Crawford
My articles My booksMy videos
Follow on:Twitter

by Jeff Soplop (Dan here…not so heavy post)

The Simpsons’ Predictive Powers Demonstrate the Limits of Forecasting

The long-running TV show The Simpsons is attracting much attention for some of its uncannily accurate predictions. The United States’ curling team’s defeat of Sweden in the Olympics – something The Simpsons predicted in 2010 – being the most recent example. And, of course, there is the notorious episode from 2000, which referenced a Trump presidency.

Given these examples and other predictions that have come true, I’m left wondering whether The Simpsons’ writers are good folks to ask for stock tips. Theories about the show’s prescience abound. They include those who believe its creator, Matt Groening, is a Freemason with supernatural occult powers, as well as more prosaic explanations, such as its many Harvard-educated writers have imbued the show with above-average intelligence.

Understanding The Simpsons’ touted predictive power requires exploring the nature of forecasting and the limits of forming accurate predictions.

One of the most sophisticated projects on humans’ ability to make accurate predictions, The Good Judgment Project (GJP), was launched in 2011 as a program within the US government’s Intelligence Advanced Research Projects Activity. (Full disclosure: I was a participant in this project for a year soon after it launched).

The GJP recruited participants to function as forecasters in a two-year tournament that involved submitting probability estimates for a variety of geopolitical events. Each participant was required to have at least a bachelor’s degree and had to pass a number of psychological and political-awareness tests. Because the GJP was interested in finding the best drivers of predictive accuracy, they tested the impact of several forecasting methods: training, teaming, and tracking.

Training was provided through several learning modules to a subset of participants. It included scenario training, which teaches how to overcome biases, or probability training, which teaches how to formulate and assign meaningful probabilities to events. Teaming involved splitting participants into subgroups of independent actors, teams, or a crowd-belief group, in which individuals made their own decisions but had access to information on what others had chosen. Tracking involved taking the top 60 forecasters from the first year of the tournament to create five new teams of “superforecasters” for Year 2, who would compete against the general population.

Some of the results from the tournament were in line with expectations (dare I say, predictable?): those with training outperformed those without, and crowd-belief forecasters and those working in teams outperformed individual forecasters. But the most interesting outcome of the experiment was the margin by which the superforecasters outperformed the rest of the field, which is summarized in the chart below (accuracy was measured with a metric known as the Brier score; a lower score indicates higher accuracy and consistency):

Not only did the superforecasters crush the competition, but their accuracy improved remarkably during the second year of the competition. The researchers summarized these findings, a bit drily, as:

“These superforecasters outperformed all other groups by a wide margin. There was no evidence of Year 2 regression to the mean; political forecasting appeared to be at least somewhat skill based, and the acquisition of skill accelerated when top performers worked together.”

Naturally, the emergence of an elite cast of superforecasters led to a fair amount of media hype, and one of the tournament’s founders, Philip Tetlock, wrote a book. In his writings and interviews, he notes that characteristics of superforecasters include high levels of fluid intelligence, open-mindedness, being self-critical, analytical, and focused on constantly updating their forecasts as new information becomes available.

Superforecasters are real. But when a team of superforecasters isn’t available, prediction markets such as PredictIt and Almanis, which allow participants to gamble on the probability of a future event happening based on the value of shares that fluctuate with each transaction, are another often-discussed mode of divining future events by harnessing the wisdom of crowds. These markets have grown with the spread of the internet, and one study showed they outperformed polls 74% of the time in predicting the outcome of presidential elections. But prediction markets have their limits, too, and overwhelmingly failed to predict the outcome of the 2016 election.

So are the writers of The Simpsons a select group of superforecasters or, perhaps, using a secret prediction market to generate their storylines? It’s hard to say, and they’ve certainly gotten plenty of things wrong over the years, too. I can recall one episode where dolphins leapt from the seas to successfully challenge mankind for dominion of the land. (Although, at the rate we continue to pollute the oceans, it’s best not to rule that one out yet.)

Nonetheless, the more likely explanation for The Simpsons’ success is that the writers are just really good at capturing – and mocking – the deeply irrational underpinnings of our cultural and political trends. Matt Groening admitted as much in an interview with The Guardian in 2016 when he said, “We predicted that he would be president back in 2000 – but [Trump] was of course the most absurd placeholder joke name that we could think of at the time, and that’s still true. It’s beyond satire.”

The work of e GJP and prediction markets show that some folks are better than others at forecasting the future within an environment focused on choosing between near-term, plausible events. But venture outside the confines of these experimental designs, and the reality of human behavior erodes the predictive power of these methods.

While The Simpsons’ writers might appear to possess uncanny forecasting abilities, as Groening’s quote demonstrates, they aren’t really making predictions at all. Instead they’re using seemingly implausible storylines to poke fun at American society. That some of these stories happen to come true just shows some things really are too absurd to predict.