AAPOR updates poll standards

AAPOR, the American Association for Public Opinion Research, wrapped up its 65th annual conference in Chicago last week. A meeting highlight was the announcement of the membership’s approval of a revised Code of Professional Ethics and Practices. Even if you have never heard of AAPOR and would never consider joining or wearing one of the “AAPOR: Freqs and Geeks” T-shirts being sold at the conference registration desk, this updated code is something you should know and care about.

The revision that most affects inside-the-Beltway types springs from the Transparency Initiative AAPOR launched after its study of 2008 pre-election polls. You may recall that one polling firm, the Georgia-based Strategic Vision, was reporting suspect numbers. AAPOR asked Strategic Vision, along with 20 other firms, to disclose their methods in polling presidential primaries. The Georgians balked, forcing AAPOR to issue a 2009 statement that “nondisclosure by Strategic Vision LLC was inconsistent with the association’s Code of Professional Ethics and Practices and contrary to basic principles of scientific research.” This snub was evidently a knockout punch, as Strategic Vision seems to have left the “public polling” community. 

{mosads}It would be a mistake, though, to see the AAPOR code revisions as merely a dust-up over pre-election polling. There is so much more out there that needs and deserves transparency. AAPOR is responding to the explosion of new research and sampling methodologies that are coming onto the scene. We’ve got a lot more to worry about than screening questions for primary polls. That’s why the heart of AAPOR’s action is new poll disclosure guidelines. For years, the gold standard of disclosure was the National Council of Public Polls (NCPP) minimalist guidelines: sponsorship of the survey, interview method, dates of interviewing, sampling method, population sampled, sample size and margin of error. Scan most any poll release and you’ll find those essential elements. Less frequently, there is also adherence to the secondary principle of revealing the precise wording of questions and disclosure of the exact order in which questions were asked. But as data collection methods and sampling frames have become more exotic, including robo-calls and online panel surveys, new standards are clearly indicated.

 One particularly noteworthy change is that poll sponsors are being asked to reveal the names of sample suppliers. This is interesting information. For phone surveys, I use Aristotle samples mostly, but occasionally my sampling supervisor tells me that an alternative provider may have a more recently updated sample for a particular state or locality, or that another sampling firm’s data has a higher phone-match rate. We look into it. Little details may make important differences.

The principal reason for the new focus on samples, though, springs from the rise of online surveys with their opt-in panels. Researchers are being asked to reveal methods used to recruit these panels and the nature of incentives or compensation for panel solicitations. This may open up some eyes to potential problems in over-reliance on Web surveys.

In addition, AAPOR is asking that polls released to the public routinely include some items not always found in past releases, including the response rate (or sample disposition data so that response rates can be computed) and data weighting methodologies. I suspect these details are overkill for the “quick and dirty” releases you typically see regarding most political polls. I don’t even see them in most public poll releases. The response rate, of course, is typically so low that it undermines the whole notion of random sampling and the attendant “margin of error.” Low response rates would theoretically nullify claims of error not exceeding plus or minus four percentage points.

I’m glad that AAPOR is keeping an eye on quality and the standards of the survey industry. As the code says, I can “point with pride” to my membership.

David Hill has been a Republican pollster since 1984. This cycle he is polling for gubernatorial campaigns in four states.