Bayesian Statistics and Political Science: Where Polling Goes Wrong
A favorite demonstration in statistics classes is to show a coin and ask what is the probability it comes up heads when flipped. Students will correctly reply 1/2. You then flip the coin high into the air, catch it, slap it on your wrist, look at it, and cover it up again with your hand. Now what is the probability of heads? It’s 1 or 0 to you, but it’s still 1/2 to the students.
This is a post-election post in the sense that it takes place after the voting has done, but it’s a mid-election post in the sense that not all the votes have been counted.
I’ll do some probability calculations based on our forecast, but first let’s go over the big things.
Our pre-election forecast gave Biden a win probability beginning in the 80-90% range and concluding at 97%. Right now it looks like Biden is favored to eke it out in the electoral college, but in any case we can project the outcome near the lower end of our predicted 95% interval, which was [259, 415]. Such things happen, but you don’t go into a forecast anticipating such an unlikely event, so it pushes us to question our assumptions. A couple days ago I gave three good reasons not to believe our numbers, and we should return to that.
Our final pre-election forecast gave Biden a 99%+ probability of winning the most votes, with a 95% predictive interval of [51.6%, 57.1%]. It’s safe to say that Biden didn’t win anything close to 57.1% of the popular vote. 51.6% is closer to the mark. We can’t be sure at this point, but we can guess that his popular vote share was, again, at the low end of our 95% forecast interval.
Our forecast was based on state polls, national polls, and fundamentals, all of which favored Biden. None of these pieces of information is perfect; indeed, the reason our forecast interval of [259, 515] was so wide was that it included for the possibility of systematic errors in all three of these sources.
I don’t think we need to talk too much about the fundamentals right now, except to say that with political polarization we could argue that the fundamentals-based prediction should’ve been more closely anchored at the results of recent national elections (Democrats with 51% to 52% of the national two-party vote), giving less weight to economic factors.
How were the national and state polls off by so much? I can’t say much right now given the information available to me. My first guess would be a combination of differential nonresponse (Trump supporters being less likely to respond to polls and Biden supporters being more likely) and differential turnout (Trump supporters being more likely to go out and vote and Biden supporters being less likely). Other possible factors include differential rates of ballot rejection and last-minute changes in opinion among undecided voters. I’m not sure what to think about the differential nonresponse explanation, as some of the polls did adjust for party identification as well as demographics, but there still can be bias even after making these adjustments.
If we could go back and fix our model of the polls, what would we do? For the reasons we and others have discussed, that 96% forecast looked overconfident even at the time. We’ve written about incentives for over- and under-confidence in forecasts, but in this case what was driving the high probability was not so much any social incentives as much as the logic of the statistical model. But if the result is so extreme this suggests a problem with the model—somewhere.
Speaking in general terms, I can see four ways the model could’ve been modified ahead of time to produce forecast probabilities closer to 50%:
1. Wider uncertainties. Our model already accounted for the possibility of large polling errors at the national and state levels. We could’ve made these intervals even wider, on the theory that future errors can be more extreme than the past. This is kind of a boring answer, but increasing the uncertainties is a start, and our model had so many moving parts that we perhaps didn’t fully understand how uncertainties in survey response and voter turnout would interact.
2. A systematic shift toward the Republican party, based on the theory that the polls were off in 2016 and this election will be similar. We were assuming the problems from 2016 would’ve been addressed, but maybe that was a mistake.
3. Incorporation of additional information such as new-voter registration numbers that favored the Republicans in several key swing states. Any model is conditional only on what it includes, and in retrospect if turnout was an issue than voter registration data could be relevant, even if in some past elections such numbers were not so predictive.
4. A directional shift in the model, not toward the Republicans but toward 50/50 in the electoral college or maybe 51/49 in the popular vote based on the idea that (a) with political polarization, votes are stable, and (b) it’s appropriate to be uncertain about the predictions of interest. This takes us back to the fundamentals model, but we could also think of this “shift toward 50/50” as an external factor applied after all the modeling is done: If the forecast says 95%, we should believe 80%; if the forecast say 80%, we should believe 60%; etc. I’m not so sure how to think about this. For example, suppose we were applying this model in 1984, the day before Ronald Reagan was reelected? Would we really only want to give this an 80% probability? Maybe.
There’s also the option of saying that our forecast was fine and we just had bad luck. I don’t buy that, though. What happened was that we were whacked by a big-ass error term, but “error terms” come from somewhere. It wasn’t some god rolling dice. The real question is how could we have better anticipated this. Lots to chew on. I will say that there is a perverse benefit to making a strong prediction that didn’t turn out well—again, you don’t want to be at the edge of that 95% interval if you can avoid it—and that’s that it pushes us to really think hard about what went wrong with the model.
Conditional probabilities
As I’m sure you’ve heard, poll-based forecasts, including ours, were off in several key states. Given the information that’s readily available to me, it’s hard to be sure exactly how large the errors are. For example, our point forecast for Biden’s share of the two-party vote in Florida was 51.7% with a 95% interval of [47.8%, 55.5%]) the general picture is here but I’m actually working off the simulations that are linked near the bottom of that page:
> sims_economist <- read.csv("electoral_college_simulations.csv") > florida <- state_sims[,"FL"] > mean(florida) [1] 0.5166 > quantile(florida, c(0.025, 0.975)) 2.5% 97.5% 0.4779 0.5552
But in the newspaper Biden’s Florida vote share is currently reported as 48.2%, so if that particular number holds up we’re already talking about an unexpected event. Not outside the 95% predictive interval but certainly a surprise.
We can do some analyses conditioning on Biden only getting 48.2% of the Florida vote. (We could simply condition on him losing Florida, but that’s not quite right, if we have a sense as here of how much he lost by.) This can be done directly using the simulations. For the purpose of computational stability, we can condition on simulations that are within a narrow band of the reported outcome:
subset <- abs(state_sims[,"FL"] - 0.482) < 0.005
We can then compute Biden’s expected national vote share conditional on this result, along with the probability of him winning the electoral college:
> popvote <- sims_economist[,"natl_pop_vote"] > evote <- sims_economist[,"dem_ev"] > mean(popvote[subset]) [1] 0.5248 > mean(evote[subset] >= 269) [1] 0.7787
He only needs to win 269 because it seems that he won that contested electoral vote in Nebraska. So an estimated 52.5% share of the popular vote (down from the 54.4% in our forecast!) and an estimated 78% of winning the electoral college (down from our notorious 96%, which seems to have crept up to 97% on election day).
OK, now let’s do all the states conditioning on Biden getting 48.2% in Florida:
> biden_wins <- state_sims > 0.5 > conditional_biden_win_prob <- apply(biden_wins[subset,], 2, mean) > round(conditional_biden_win_prob, 2) AK AL AR AZ CA CO CT DC DE FL GA HI IA ID IL IN KS KY LA MA MD ME MI 0.01 0.00 0.00 0.27 1.00 0.99 1.00 1.00 1.00 0.00 0.30 1.00 0.11 0.00 1.00 0.00 0.00 0.00 0.00 1.00 1.00 1.00 0.90 MN MO MS MT NC ND NE NH NJ NM NV NY OH OK OR PA RI SC SD TN TX UT VA 0.94 0.00 0.00 0.00 0.33 0.00 0.00 0.92 1.00 0.99 0.76 1.00 0.06 0.00 1.00 0.70 1.00 0.00 0.00 0.00 0.11 0.00 0.98 VT WA WI WV WY 1.00 1.00 0.89 0.00 0.00
Let’s just look at the states where either candidate has a reasonable chance:
> close <- conditional_biden_win_prob > 0.05 & conditional_biden_win_prob < 0.95 > round(conditional_biden_win_prob[close], 2) AZ GA IA MI MN NC NH NV OH PA TX WI 0.27 0.30 0.11 0.90 0.94 0.33 0.92 0.76 0.06 0.70 0.11 0.89
We know more than this, of course, but not a lot more. Of the above states, we know that Trump won Iowa, Ohio, and Texas; Biden won Minnesota and New Hampshire, and the others are too close to call. We can condition on those outcomes:
subset <- abs(florida - 0.482) < 0.005 & (state_sims[,"IA"] < 0.49) & (state_sims[,"OH"] < 0.49) & (state_sims[,"TX"] < 0.49) & (state_sims[,"MN"] > 0.51) & (state_sims[,"NH"] > 0.51)
They’re still waiting for mail ballots so I don’t feel comfortable conditioning on the reported totals, but I’m conditioning on the winners’ share being more than 49% in each state, on the theory that if it could be closer than that, the news organizations wouldn’t have felt confident enough to make the call.
What happens conditional on the results now?
> round(mean(popvote[subset]), 3) [1] 0.522 > round(mean(evote[subset] >= 269), 2) [1] 0.79
Biden’s then predicted to win 52.2% of the national vote and, according to the model, has a 79% chance of wining the electoral college.
And here are the close states again:
> round(conditional_biden_win_prob[close], 2) AZ GA IA MI MN NC NH NV OH PA TX WI 0.17 0.22 0.00 0.96 1.00 0.22 1.00 0.68 0.00 0.74 0.00 0.96
OK, that’s all based on a model that, as we’ve discussed, had problems. I’m not intending the above numbers to be forward-looking predictions (Biden with a 96% chance of winning Wisconsin, etc.), just working thru what the model is saying.