By Curt Anderson January 27, 2020 6:21 pm Category: Education (5.0 from 1 vote)
Rules of the Post
I usually defend polling organizations. Polls, when properly done, are the best tool we have to measure public opinion on a particular issue or candidate. However, not all polls are created equal.
Open-access polls vs. scientific polls
We have all seen online polls in which we are asked to opine on various topics. These are known as open-access polls. The participants are self-selected, and anything but a random cross-section of voters or of any other identified group. They are fun, but not very meaningful.
For polls to be considered meaningful they must be scientific polls. The most glaring difference between an open-access poll and a scientific poll is that scientific polls randomly select their samples. Scientific polls use statistical weights to make them demographically representative of the target population.
People who are critical of polls (often when they don't like the results) sometimes ask how can a relatively small number of polled participants, often around 1000 people or so, be representative of a state or even all of America. There is a soup making analogy that answers that question. If you are making a large amount of a cream soup and want to know if it is properly seasoned, you don't need to consume the entire cauldron of soup. If you stir the soup sufficiently for minute or so, you can get a pretty good idea how much pepper, salt or other spices are needed by tasting a single spoonful. In other words, that spoonful you tasted represents a randomized sampling of all the soup in that cauldron. That is what a scientific poll endeavors to accomplish.
One of the polls that caught my attention is the Emerson poll. You might think I am picking on Emerson. My criticism might be considered unfair since I have not scrutinized other polls in the same way. I am less than confident in how Emerson achieves their results, and I will explain why.
In their defense, I will note that Emerson College Polling is well-regarded. Emerson polls are cited by major news organizations. FiveThirtyEight.com gives Emerson an A- grade. That said, being naturally skeptical, I tend to look at the methodology of polls. Here is how Emerson says they do their polling:
The National Emerson College Poll was conducted January 21-23, 2020 under the supervision of Assistant Professor Spencer Kimball. The sample consisted of registered voters, n= 1,128, with a Credibility Interval (CI) similar to a poll's margin of error (MOE) of +/- 2.8 percentage points. The data was weighted based on 2016 voter model of gender, education, age, ethnicity, and region. It is important to remember that subsets based on gender, age, party breakdown, ethnicity, and region carry with them higher margins of error, as the sample size is reduced. Data was collected using both an Interactive Voice Response (IVR) system of landlines only (n= 741) and an online panel provided by Amazon Turk (n= 387).
The terminology used in Emerson's description of their methodology caused me to search the Internet for definitions. Here is my glossary of the bold type terms above.
Credibility Interval (CI)
AAPOR [American Association for Public Opinion Research] urges caution in the interpretation of a new quantity that is appearing with some nonprobability opt-in, online polling results – the credibility interval. The credibility interval is not the margin of sampling error (MOE) that the public has come to understand as the statistical uncertainty of probability based scientific polls. Instead, credibility intervals are being used when reporting on some nonprobability polls (typically opt-in online polling), or for model-based inferences such as arise in small-area estimation.
Margin of error
The margin of error is a statistic expressing the amount of random sampling error in a survey's results.
Interactive Voice Response
Interactive voice response (IVR) is a technology that allows a computer to interact with humans through the use of voice and DTMF tones input via a keypad.
Online panel provided by Amazon Turk
This term in particular had me flummoxed. I went to the website that explains it. Here is what it says:
Make money in your spare time
Get paid for completing simple tasks
Want to make money in your spare time? Become an Amazon Mechanical Turk (MTurk) Worker and start applying your skills to the thousands of available tasks in the MTurk marketplace.
What type of work is available on Amazon Mechanical Turk?
Workers help companies collect and edit information by:
Filling out market research or survey data on a variety of topics.
IVR and Amazon Turk respondents may not be like the rest of us
I am suspicious of Emerson's methodology not their sincerity. I suspect that only a certain sort of person has a landline and the time and patience to answer the questions of a robotic voice. I would be excluded from that group. I don't answer the phone unless I recognize the caller's number. Maybe if I were lonely I would pickup the phone whenever it rings. Maybe if I had more time in my life I'd be willing to "talk" with a robot.
I also doubt that Amazon Turk workers who are paid to respond to pollsters are representative of all registered voters. I assume that they are low paid workers without many benefits. I can't say if the IVR or the Amazon Turk respondents are any more liberal or any more conservative than the entire voting population. It certainly seems that those groups would have characteristics that not all of us share.
The views and claims expressed by contributors are their own and do not necessarily reflect the views and beliefs of SelectSmart.com. Not every statement made here can be assumed to be a fact.
Comments on "Can we trust what polls are telling us?":
Be the first to comment on this article.
Comment on: "Can we trust what polls are telling us?"
I usually defend polling organizations. Polls, when properly done, are the best tool we have to measure public opinion on a particular issue or candidate. However, not all polls are created equal.
