In Defying the Odds, we talk about the social and economic divides that enabled Trump to enter the White House. In Divided We Stand, we discuss how these divides played out in 2020. In both elections, however, polls tended to overstate Democratic margins.
AAPOR report on polling problems in 2020.
Several proposed explanations can be ruled out as primary sources of polling error in 2020. Our analyses suggest the following.
- Polling error was not caused by late-deciding voters voting for Republican candidates. More voters voted prior to Election Day in 2020 than ever before and the number of undecided voters was relatively small. Only 4% of poll respondents, on average, gave a response other than “Biden” or “Trump” when asked by state-level presidential polls conducted in the final two weeks. Unlike in 2016, respondents deciding in the last week were as likely to support Biden as Trump, according to the National Election Pool exit polls.
- Polling error was not caused by a failure to weight by education. A suspected factor in 2016 polling error was the failure to weight by education (Kennedy et al. 2016). In the final two weeks of the 2020 election, 317 state-level presidential polls (representing 72% of all polls conducted during this period) provided information on the statistical adjustments accounting for coverage and nonresponse issues; of these 317 polls, 92% accounted for education level in the final results.
- Polling error was not primarily caused by incorrect assumptions about the composition of the electorate in terms of age, race, ethnicity, gender, or education level. There is no evidence that polling error was caused by the underrepresentation or overrepresentation of particular demographics. Reweighting survey data to match the actual outcome reveals only minor changes to demographic-based weights.
- Polling error was not primarily caused by respondents’ reluctance to tell interviewers they supported Trump. 7 The overstatement of Democratic support occurred regardless of mode and the overstatement of Democratic support was larger in races that did not involve Trump (i.e., senatorial and gubernatorial contests).
- Polling error cannot be explained by error in estimating whether Democratic and Republican respondents voted. Trump supporters and Biden supporters were equally likely to vote after saying they would. This conclusion is based on validating the vote of registration-based samples shared with the Task Force by some AAPOR Transparency Initiative members.
- Polling error was not caused by the polls having too few Election Day voters or too many early voters. Among the 23 state-level presidential polls conducted in the final two weeks that reported how respondents said they would vote, the proportion of Election Day voters closely matched the percentage of certified votes cast on Election Day
The report reaches no definitive conclusions on what went wrong, but offers some plausible hypotheses:
- At least some of the polling error in 2020 was caused by unit nonresponse. The overstatement of Democratic support could be attributed to unit nonresponse in several ways: between-party nonresponse, that is, too many Democrats and too few Republicans responding to the polls; within-party nonresponse, that is, differences in the Republicans and Democrats who did and did not respond to polls; or issues related to new voters and unaffiliated voters in terms of size (too many or too few) or representativeness (for example, were the new voters who responded to polls more likely to support Biden than new voters who did not respond to the polls?). Any of these unit nonresponse factors could have contributed to the observed polling error. Without knowing how nonrespondents compare to respondents we cannot conclusively identify the primary source of polling error.
- Factors that worked well in correcting for nonresponse in previous elections (including demographic composition, partisanship, or 2016 vote) did not render accurate vote estimates for the 2020 election. Poll data provided by some AAPOR Transparency Initiative members were reweighted to match the 2020 certified outcome. It was necessary to increase the percentage of Republicans (or 2016 Trump voters) and decrease the percentage of Democrats (or 2016 Clinton supporters) in the outcome-reweighted sample. In contrast, there are only slight differences between the originally weighted poll data and the outcome-reweighted data in terms of standard demographic categories.
- Weighting to a reasonable target for partisanship and past 2016 vote does not fully correct the polling error. Reweighting the polls to reproduce the 2020 outcome requires a much larger margin for Trump in 2016 than actually occurred among respondents who report voting in 2016. The larger 2016 margin for Trump among those who reported voting for Trump in 2016 could be caused by the following: an issue with the weighting targets, i.e., the implied vote share among 2016 voters who voted in 2020 was different from the 2016 actual outcome; or differences in opinion within groups that responded, e.g., the 2016 Trump supporters who responded to polls were more likely to vote for Democrats than those who did not. It is impossible to know which caused the larger 2016 margin.
- It is possible that 2020 pre-election polls were not successful in correctly accounting for new voters who participated in the 2020 election. There were many new voters in 2020 and it is unclear whether the proportion of new voters in the polls matched the proportion of actual new voters. It is also unclear whether the new voters who responded to polls had similar opinions to those who did not respond. Given the relative proportion and self-reported voting behavior of these new voters in the data available to the Task Force, this group of voters pushed the overall polling margins in the Democratic direction. Error in polling this group could have produced the observed polling error.
