Mellman: When it comes to polls, mode matters

With fewer people willing to be interviewed for surveys, pollsters fret about our samples no longer being representative. And if you don’t have a representative sample, there’s no reason to believe you’ve got an accurate poll.

Pew Research found response rates plummeting from a none-too-high 36 percent in 1997, to a mere 9 percent in 2017.

Intuitively, low response rates sound like a problem. However, what matters is not how many people respond to a survey, but how representative respondents are of the public one is polling.

Low response rates affect accuracy, only if the people who do participate are different from those who don’t.

For many years, even as response rates fell, careful studies found little systematic difference between those who responded and those who didn’t.

Indeed, Pew’s website still sports a 2017 article which concludes, in part, there is “strong evidence that decisions to participate in telephone surveys are not strongly related to political, social or religious attitudes. So even at low response rates, telephone surveys that include interviews via landlines and cellphones, and that are adjusted to match the demographic profile of the U.S., can produce accurate estimates for political attitudes.”

In short, by ensuring you reach mobile phones, and with lots of demographic weighting, you can make it turn out right.

Those of us sampling from voter files possess even greater ability to weight, as we know so much more about the political habits of our respondents, and of the actual universe of voters. 

Until recently, what physicians call “heroic measures” — high risk therapies, undertaken as a last resort, with the understanding that anything less would surely result in failure — were sufficient to produce accurate polls.

That’s changing though as low response rates now combine with radically different “telephone” habits. 

First, was the rise of cellphones and the decline of landlines, tracked regularly by National Health Interview Survey which last year found only 43 percent of households even have landline telephones. Seventy-two percent of households are either wireless only or mostly. (Since most robo polls can only dial landlines, be very wary of their results.) 

But unsurprisingly, differences by age are dramatic. Just 31 percent of those 65 and older live in wireless-only households, compared to 76 percent of those 25-34.

Add another layer: Texting is now ubiquitous, and many people rarely answer calls on their phones, (especially from numbers they don’t know), though they do respond to texts. Seventy-five percent of millennials prefer texting to talking. Older individuals are more likely to text than talk.

Our firm adapted to these changes by combining three different modes of interviewing, landline, cellphone and text-to-online.

The first two use live operators and are generally familiar to those fluent with polling practice. 

The third is less so. We text mobile phones, asking respondents to click a link in order to complete a survey. 

It turns out that different kinds of people prefer different modes of interviewing and, more importantly, they answer key questions quite differently.

Take the poll of Nevada Democratic caucusgoers we just completed for Jon Ralston of the The Nevada Independent.

Former Vice President Joe Biden cleans up among those who took the survey via landline, garnering 41 percent of the vote, to 17 percent for Sen. Elizabeth Warren (D-Mass.) and 8 percent for Sen. Bernie Sanders (I-Vt.). 

Among those reached on cellphones, responses were quite different: Biden still leads, but with a lesser 29 percent, to much larger 26 percent for Sanders and 16 percent for Warren. 

Our text-to-online respondents put Warren first with 24 percent, Sanders close behind with 22 percent and Biden in third place with 18 percent.

Radically different results depending on the mode of interview. Biden’s vote varies by 23 points with mode, Sanders’s by 18 and Warren’s by 8 points.

Ignoring any one of the three modes would have produced strikingly different, and, I dare say, less accurate, results.

Very few, if any, public polls combine interviewing modes this way. Keep that in mind as you evaluate results they publish.

Mellman is President of The Mellman Group and has helped elect 30 U.S. Senators, 12 Governors and dozens of House Members. Mellman served as pollster to Senate Democratic Leaders for over 20 years and as President of the American Association of Political Consultants.