Recent polls suggest that President Biden and former President Donald Trump are running neck and neck in a 2024 rematch. But can such polls be taken seriously?
Many Americans may suspect otherwise. It is widely perceived that the presidential polls failed miserably in 2016 and, just as importantly, performed no better in 2020. Late that year, The New York Times ran an article titled “2016 Dealt a Blow to Polling. Did 2020 Kill It?” while former Secretary of State James Baker declared that “this time, we were promised that the pollsters would get it right. They didn’t.” The prestigious American Association of Public Opinion Research was more critical of the 2020 polls than those in 2016 even though, unlike polls in the earlier year, those in 2020 correctly identified the winner of the election.
But these assessments of the 2020 polls may be too harsh.
The state-by-state polls can be construed as raw materials for a sophisticated aggregator like Nate Silver’s FiveThirtyEight, which, aware of the biases and frailties of individual surveys, adjusts and skillfully combines them in pursuit of an accurate forecast. If the aggregator succeeds, then the polls on which it performed its surgery should not be viewed as hopeless failures.
Holding that viewpoint, we decided to review the performance of FiveThirtyEight in the 2020 presidential race. We found that, buttressed by its elegant use of the laws of probability, FiveThirtyEight did an excellent job, yielding state-by-state predictions about victory that were nearly always correct. However, there was one blemish: estimates of Trump’s vote share were too low in 46 of the 50 U.S. states (as well as Washington, D.C.), by an average of 2 percentage points. (The exceptions were Alaska, Colorado, Louisiana and Maryland.) But the overall level of success was impressive, especially given that the COVID-19 pandemic caused changes in 2020 voting procedures and uncertainty about voter turnout.
We therefore prepared to write a paper lauding FiveThirtyEight for its achievements in 2020. But then, something strange happened.
We thought that, to summarize our findings, it would be illuminating to quantify the extent to which FiveThirtyEight’s accuracy exceeded that of the potpourri of local polls that it straightened out. As a first step, we decided to compare the website’s forecasts of Trump’s state-by-state vote shares to forecasts based on a simple average of local polls taken shortly before the 2020 election. We focused on the individual toss-up states where the election would be decided, rather than states like California and Arkansas where the outcome was a foregone conclusion.
To our considerable surprise, we found that the simple average did just as well as FiveThirtyEight, underestimating Trump’s vote shares by the same 2 percentage points.
Then we looked beyond the simple-average forecasts to those arising from the original local polls that had been averaged together. Here we experienced even greater surprise: The 2020 election-day forecasts based on local polls were only off on average by about 2.5 percentage points, in one direction or the other. And this happened among polls subject to the traditional 4-point margin of sampling error based on their sample sizes.
To be sure, the local polls were more often too low than too high, and they underestimated as a group by about 2 percentage points. But that matches what FiveThirtyEight had achieved.
In short, the local pre-election polls were not clumsy and unreliable in 2020, dependent on a savior like FiveThirtyEight to make sense of them. They did just about as well as FiveThirtyEight. The two of us had fallen victim to groupthink, taking it as a given that local polls performed poorly and that the only issue was whether anything could be salvaged. Our myopia is of little importance in itself; what matters is the conclusion we reached almost by accident.
What does that conclusion suggest about the 2024 election? To put it simply, believe the polls, and pay particular attention to the local polls in toss-up states. Pollsters that were broadly successful in the last presidential election are unlikely to fail colossally in the next one.
Of course, polls taken well before the election portray the state of the race at that time. Candidates, campaigns and voters unnerved by what the polls say can try to change matters. And they can be confident that any success they achieve will be reflected in future polls.
Arnold Barnett is George Eastman Professor of Management Science and Professor of Statistics at MIT Sloan School of Management. Arnaud Sarfati was a student in the Master of Business Analytics program at MIT.