Markos Moulitsas: The fault in our polls

We live in an age of political empiricism, with traditional punditry increasingly being elbowed aside in favor of data-driven prognostication. Daily Kos, The Huffington Post, Nate Silver, Princeton’s Sam Wang, The New York Times and The Washington Post are all enjoying success with data-driven forecast models. 

But what happens when the underlying data is suspect? 

{mosads}All these models depend, to some degree, on public polling to fuel their calculations. For the models to be accurate, polling has to be accurate. And there’s increasing cause to be concerned about the validity of public polling data. 

This wasn’t much of a problem in past cycles. Sure, people can laugh at Gallup’s prediction that Mitt Romney would win the 2012 presidential election 49 percent to 48 percent, but the polling aggregate was not far off. In The Huffington Post’s aggregation, President Obama led 48.2 percent to 46.7 percent, while the final results were Obama 51.1/47.2. State-by-state presidential returns were also pretty good: In battleground states, the polling aggregate was generally within 1 to 4 points of the final results.

But even in 2012, the efficacy of polling was threatened by two major trends: the dramatic increase in cellphone-only households, which are harder and more expensive for pollsters to reach, and plummeting response rates. And those trends have only accelerated. 

In May 2012, Pew Research reported that 62 percent of polling calls were even answered, much less responded to — and just 14 percent of those called answered questions. Those numbers had been 90 percent and 43 percent, respectively, in 1997. In just the two last years, the number of cellphone-only households has risen from 38 percent to 41 percent, according to the Centers for Disease Control and Prevention. And those who keep landlines don’t seem keen on answering them, particularly around election time in heavily contested states, where campaign and advocacy organizations bombard voters with persuasion calls, robo calls and get-out-the-vote calls.

This collapse in response rates has forced pollsters to get creative, employing new statistical modeling and turning to the Internet. There is nothing inherently wrong with Internet sampling — 4 of the 7 most accurate pollsters in the 2012 presidential election used Internet samples, according to Silver (as did some of the worst … the source of a sample can’t save terrible pollsters like John Zogby or Rasmussen). But it does mean that the status quo is being upended, and these new approaches toward sampling must prove themselves over several cycles.

Finally, the flood of public numbers from partisan-minded pollsters, whose goal is to shape “narratives” with data (rather than provide true snapshots of public opinion), has the potential to further confound forecast models. 

So what does it mean for this year’s election? It means polls are all over the place. For example, over the last three weeks in Arkansas, we’ve seen polls showing Democratic incumbent Sen. Mark Pryor trailing by as much as 7 points and leading by as much as 3 points. In Alaska, we have polls showing Democratic Sen. Mark Begich leading by as much as 5 points and losing by as much as 5 points. 

There’s no doubt that pollsters will figure out how to survey in this changed landscape. But until we have better insight into today’s best polling methodology, forecast models are at the mercy of noisy data. Which is to say, I wouldn’t bet money on anyone’s model.

Moulitsas is the founder and publisher of Daily Kos.