What you see not always what they got

In examining poll results, you assume the data has probably been subjected to some demographic weighting — a perfectly legitimate statistical procedure — but you also trust that the numbers you see are a reasonable construction of the numbers the pollster got.

Last cycle, though, I uncovered two egregious examples of cooking the numbers that shook that faith.

{mosads}The first incident involves a Republican firm that shall go nameless because I respect them deeply. Polling for a Wisconsin business coalition a couple of weeks before the election, they conducted a survey purporting to show Gov. Jim Doyle leading Republican Congressman Mark Green by a mere four percentage points — and Doyle was only garnering 45 percent. If undecideds followed the typical pattern and broke to the challenger, Green could eke out a victory.

The toplines circulated by the pollsters, as well as their briefing, only reported that four-point margin, with no hint of what lay beneath. However, buried deep in the crosstabs, under a heading indecipherable to the uninitiated — “UNWGT” — was a clear indication that the results the pollsters actually obtained were quite different.

The “unweighted” column revealed that the raw data collected by the firm had Gov. Doyle ahead not by four points, but rather by 11, with 49 percent in a three-way race. Those data told a rather different story.

How could a landslide 11-point margin come out as a nerve-wracking four-point lead? The answer was probably in the party weighting. Again, there is nothing wrong in principle with weighting data — though weighting by party is particularly dangerous, especially when, as in this case, the pollsters turned an eight-point Democratic advantage in party identification into a two-point deficit. In an era when partisanship is so tightly bound to vote, weighting party is tantamount to weighting the vote.

Was there justification for this statistical back-flip? In 2002, there were no exit polls, but in 2000, Democrats enjoyed a five-point advantage in partisan identification. However, in 2004, Republicans came out ahead by three points. Was there an argument for party weighting this extensive? Perhaps, but it renders the data a pollster collects, and the story he or she tells, almost completely subservient to weighting decisions.

By the way, the 2006 exit poll showed a five-point Democratic advantage, and our own polling at the same time as the GOP poll had Gov. Doyle ahead by seven, in a race he won by eight points.

Michigan’s governor’s race presented another example of the problem. In a Nov. 5 story headlined “Granholm Surges,” The Detroit Free Press reported on its poll by the always-careful Ann Selzer, which showed a 13-point margin for the Democratic incumbent.

Good news, to be sure (for us). The only problem: Those were not the actual numbers in the full poll. It seems that Selzer’s poll of 800 showed Granholm ahead by 16 points. But that was radically different from other public polls.

EPIC/MRA, polling for another major paper, claimed the margin was just seven points, and its executives were publicly castigating the Iowa-based Selzer for not knowing Michigan. The ever-enigmatic Strategic Vision weighed in with Granholm only four points ahead.

Fearing its own pollster might be too wrong, the paper apparently removed from the sample the best night of calling for Granholm, reporting instead based on the incredibly odd sample size of 532.

It was a lucky guess — Granholm won by 14.

Sadly, though, we may have reached the point where poll readers simply cannot be certain that the results they see bear any resemblance to the numbers the pollster got — and that threatens to call the entire enterprise into question.

Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982. Current clients include the majority leaders of both the House and Senate.