Mellman: When surveys are really bad

We dissect the failings of political pollsters at length — and there’s much to criticize. 

However, those of us in politics face a test each Election Day. We’re either right or wrong, and because some of our surveys become public, folks know it.

{mosads}Most political pollsters take methodology at least somewhat seriously. Interviewers are carefully trained and regularly monitored, while questionnaires are intensively reviewed (OK, many pollsters would say too intensively). 

Commercial market researchers face no similar test. Take, for example, my experience the other night as a respondent to a ride-booking survey undertaken for Uber, Lyft, Car2Go, Zipcar or Enterprise CarShare. I list the names to make clear these are large companies with significant budgets, not fly-by-night operations.

My interviewer proved engaging and solicitous. However, her accent was thick and her English was certainly not native. After some probing I learned that she was calling from Germany.

As a non-native reader and speaker, she skipped around question texts leaving me to interpret them, along with the answer categories. Not understanding exactly what the answer categories were, I would offer a response, and she would tell me which category she thought it fit into and ask whether that seemed right. It was akin to me saying, I “like” a political figure and the interviewer responding with, “Well, that sounds like somewhat favorable, right?” 

As nice as the interviewer was, that’s just completely unacceptable. It means responses aren’t comparable. 

However, it wasn’t just the absence of fluidity with questions and answers. The questionnaire itself was poorly designed.

I may be one of the most persnickety survey respondents around, but there is a reason for that. I know that if questions aren’t clear, then neither are the answers; if questions are open to multiple interpretation, then so are the answers. 

When I was asked how many times a day I listen to the radio, my first thought was “rarely,” because I walk to work, so I don’t use a car radio. 

However, I listen to Pandora often. Should that have counted?  

I certainly sympathize with the desire to keep media usage questions on the same scale so researchers can determine where to find me most often, but one of my greatest difficulties came when the interviewer inquired “how many times a day” I watch television.

Befuddled, I posed my own question. “Well, if I watched TV for six straight hours would that count as ‘once a day’?” “Yes,” came the interviewer’s confident reply. So “once a day” was my response (I don’t watch for six hours, but I usually do my viewing in one sitting). 

Of course, if someone had watched a half-hour sitcom, taken a half-hour break and watched another half-hour sitcom, the right answer for them, according to my interviewer, would be twice a day. 

The end result: Someone who watches one hour of TV is recorded as watching twice as much television as someone who watches six straight hours. 

It’s data. But it’s useless, misleading data.

Then came “message” questions.

“When you decide which of these companies to use, how important is it to you that the vehicle is able to carry more than one passenger?”

Do you anticipate my problem? 

“Well, if I’m traveling alone, it’s not important at all, but if I’m with a colleague, it’s extremely important.” 

My interlocutor had no idea how to cope. “Just in general,” she prompted, trying to be helpful. Well, on average I guess I have half a colleague, so if there’s a little extra room it’s OK.  

No business should trust the “data” emerging from this poll, but there’s little doubt they will. The client will make advertising and investment decisions based on the results. They may do well or poorly, but they’ll probably never realize just how awful their market research was.

Damn few political pollsters would make these basic mistakes. 

Mellman is president of The Mellman Group and has worked for Democratic candidates and causes since 1982. Current clients include the minority leader of the Senate and the Democratic whip in the House.