Robo-polls and human error

Political pollsters’ conceit is that, unlike traditional market researchers, Election Day provides a report card on the quality of our work that reveals unequivocally whether we are right or wrong, accurate or inaccurate.

By that standard, IVR, or robo-polls, that use recorded voices, not live interviewers, and ask respondents to push buttons on the phone keypad to record responses, have a pretty good record. Several studies demonstrate they are no less “right” just before elections than traditional polls.

{mosads}This fact infuriates traditionalists because, for a host of methodological reasons, it should not be the case — nonetheless, so far, it is. Of course, some individual pollsters do have better records than others, but, on average, robo-polls are not systematically worse than other polls in the days just before votes are cast.

However, being correct on Election Day is not the same thing as being right two weeks, or two months, in advance of the election. Unfortunately, there is no objective standard for being “right” prior to Election Day.

As Mark Blumenthal of Pollster.com pointed out, two polls were “spot-on” just before Scott Brown’s victory — ours, done with live interviewers, and robo-poll PPP. However, 10 days earlier these two polls diverged dramatically. PPP had Brown one point ahead at nearly the same time The Boston Globe showed Coakley with a 17-point margin. Our survey reported a 14-point lead. While we were both right just before the election, we could not both have been right earlier.

Our polling, and the Globe’s, suggest Coakley was ahead at the beginning of January and then, in the last week, the race changed dramatically.

Given the timing of ads and the feel on the ground, our story strikes me as more plausible.

This is hardly the first time IVR polls tell a less realistic story, even when they are right at the end.

Take Sen. Maria Cantwell’s (D-Wash.) 2006 reelection battle. Our polling tells a sensible tale. In June, against a relatively unknown challenger, Cantwell held a 21-point lead, which diminished slightly to 15-18 points as Election Day neared and advertising on both sides brought our opponent’s name recognition up to 75 percent. In the end, Sen. Cantwell prevailed by 16 points.

The IVR polling offers quite a different narrative. A June lede in the Seattle Post-Intelligencer screamed out, “Dwindling voter support for U.S. Sen. Maria Cantwell’s reelection bid has put her in a statistical toss-up with her Republican opponent,” as Rasmussen showed the Democratic incumbent clinging to a tenuous four-point lead. In January, the same poll had given the incumbent a 15-point advantage, which apparently evaporated.

Which is it — a big initial lead that narrowed somewhat as the campaign engaged, or the bottom suddenly falls out for Cantwell for no discernible reason, but she recovers her advantage after both sides hit the airwaves?

I’m biased, but our story seems a more reasonable depiction of reality.

Rasmussen presented an initially exciting story about Ned Lamont’s unsuccessful challenge to, by then, Independent Joe Lieberman (a race in which I was not involved). Tied in July, Lieberman and Lamont were still neck-and-neck in September. By October, Rasmussen acknowledged Lieberman had pulled into an 11-point lead that he maintained, posting a 10-point win.

The arc of the race looked quite different to Quinnipiac’s pollsters. When Rasmussen called the race even, Quinnipiac gave the incumbent a 24-point edge. September found the challenger closing the gap to 10 points, a lead Lieberman held through Election Day.

{mosads}Both polls were correct as the final Day of Judgment approached, but told very different stories about how the candidates got there.

The lessons of this brief inventory: Election Day accuracy is not an unerring measure of pre-election accuracy, and while most polls reveal something, evaluating the stories they tell is no simple task.

Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982. Current clients include the majority leaders of both the House and Senate.