Comments posted organically Homepage
Can we trust what polls are telling us?

By Curt Anderson
January 27, 2020 6:21 pm
Category: Education

(5.0 from 1 vote)

Rules of
the Post

Rate this article
5 Stars
4 Stars
3 Stars
2 Stars
1 Star
0 Stars
(5=best, 0=poor)

I usually defend polling organizations. Polls, when properly done, are the best tool we have to measure public opinion on a particular issue or candidate. However, not all polls are created equal.

Open-access polls vs. scientific polls
We have all seen online polls in which we are asked to opine on various topics. These are known as open-access polls. The participants are self-selected, and anything but a random cross-section of voters or of any other identified group. They are fun, but not very meaningful.

For polls to be considered meaningful they must be scientific polls. The most glaring difference between an open-access poll and a scientific poll is that scientific polls randomly select their samples. Scientific polls use statistical weights to make them demographically representative of the target population.

People who are critical of polls (often when they don't like the results) sometimes ask how can a relatively small number of polled participants, often around 1000 people or so, be representative of a state or even all of America. There is a soup making analogy that answers that question. If you are making a large amount of a cream soup and want to know if it is properly seasoned, you don't need to consume the entire cauldron of soup. If you stir the soup sufficiently for minute or so, you can get a pretty good idea how much pepper, salt or other spices are needed by tasting a single spoonful. In other words, that spoonful you tasted represents a randomized sampling of all the soup in that cauldron. That is what a scientific poll endeavors to accomplish.

One of the polls that caught my attention is the Emerson poll. You might think I am picking on Emerson. My criticism might be considered unfair since I have not scrutinized other polls in the same way. I am less than confident in how Emerson achieves their results, and I will explain why.

In their defense, I will note that Emerson College Polling is well-regarded. Emerson polls are cited by major news organizations. gives Emerson an A- grade. That said, being naturally skeptical, I tend to look at the methodology of polls. Here is how Emerson says they do their polling:

The National Emerson College Poll was conducted January 21-23, 2020 under the supervision of Assistant Professor Spencer Kimball. The sample consisted of registered voters, n= 1,128, with a Credibility Interval (CI) similar to a poll's margin of error (MOE) of +/- 2.8 percentage points. The data was weighted based on 2016 voter model of gender, education, age, ethnicity, and region. It is important to remember that subsets based on gender, age, party breakdown, ethnicity, and region carry with them higher margins of error, as the sample size is reduced. Data was collected using both an Interactive Voice Response (IVR) system of landlines only (n= 741) and an online panel provided by Amazon Turk (n= 387).

The terminology used in Emerson's description of their methodology caused me to search the Internet for definitions. Here is my glossary of the bold type terms above.

Credibility Interval (CI)
AAPOR [American Association for Public Opinion Research] urges caution in the interpretation of a new quantity that is appearing with some nonprobability opt-in, online polling results – the credibility interval. The credibility interval is not the margin of sampling error (MOE) that the public has come to understand as the statistical uncertainty of probability based scientific polls. Instead, credibility intervals are being used when reporting on some nonprobability polls (typically opt-in online polling), or for model-based inferences such as arise in small-area estimation.

Margin of error
The margin of error is a statistic expressing the amount of random sampling error in a survey's results.

Interactive Voice Response
Interactive voice response (IVR) is a technology that allows a computer to interact with humans through the use of voice and DTMF tones input via a keypad.

Online panel provided by Amazon Turk
This term in particular had me flummoxed. I went to the website that explains it. Here is what it says:

Make money in your spare time
Get paid for completing simple tasks
Want to make money in your spare time? Become an Amazon Mechanical Turk (MTurk) Worker and start applying your skills to the thousands of available tasks in the MTurk marketplace.

What type of work is available on Amazon Mechanical Turk?
Workers help companies collect and edit information by:
Filling out market research or survey data on a variety of topics.

IVR and Amazon Turk respondents may not be like the rest of us
I am suspicious of Emerson's methodology not their sincerity. I suspect that only a certain sort of person has a landline and the time and patience to answer the questions of a robotic voice. I would be excluded from that group. I don't answer the phone unless I recognize the caller's number. Maybe if I were lonely I would pickup the phone whenever it rings. Maybe if I had more time in my life I'd be willing to "talk" with a robot.

I also doubt that Amazon Turk workers who are paid to respond to pollsters are representative of all registered voters. I assume that they are low paid workers without many benefits. I can't say if the IVR or the Amazon Turk respondents are any more liberal or any more conservative than the entire voting population. It certainly seems that those groups would have characteristics that not all of us share.

Cited and related links:


Comments Start Below

The views and claims expressed by contributors are their own and do not necessarily reflect the views and beliefs of Not every statement made here can be assumed to be a fact.
Comments on "Can we trust what polls are telling us?":

Be the first to comment on this article.

Comment on: "Can we trust what polls are telling us?"

* Anonymous comments are subject to approval before they appear. Cookies Consent Policy & Privacy Statement. All Rights Reserved. SelectSmart® is a registered trademark. | Contact | Advertise on | This site is for sale!

Find old posts & articles

Articles by category:
Report spam & abuse home page

From our contributors:
Display Order:

USA ties England at nil in World Cup!!!!!
Sports by HatetheSwamp     November 25, 2022 12:59 pm (Rating: 0.0) Last comment by: Curt_Anderson (4 comments) [127 views]

Before the Innocence Project there was Erle Stanley Gardner's Court of Last Resort
History by Curt_Anderson     November 26, 2022 10:53 am (Rating: 0.0) Last comment by: (0 comments) [19 views]

Ye & Trump 2024? Kanye West announces presidential bid
President by Curt_Anderson     November 25, 2022 10:58 pm (Rating: 0.0) Last comment by: Donna (1 comments) [78 views]

Mike Lindell Weighing Run Against GOP's Ronna McDaniel 'If God Willing'
Politics by HatetheSwamp     November 25, 2022 1:29 pm (Rating: 0.0) Last comment by: Curt_Anderson (1 comments) [114 views]

China's "Zero-COVID" Policy and the Riots that have followed
News by Donna     November 25, 2022 7:08 am (Rating: 0.0) Last comment by: HatetheSwamp (3 comments) [114 views]

Will Elon Musk release Twitter internal discussions to censor NY Post’s Hunter Biden laptop story before US elections? Here is what he said
Media by HatetheSwamp     November 25, 2022 4:01 am (Rating: 0.0) Last comment by: (0 comments) [33 views]

The Hill: Trump may not make it to the primaries
Politics by HatetheSwamp     November 22, 2022 6:03 am (Rating: 0.0) Last comment by: Curt_Anderson (24 comments) [349 views]

Apparently, Larry Saboto was cheating off pb in predicting the election
Politics by HatetheSwamp     November 7, 2022 3:52 pm (Rating: 0.0) Last comment by: Curt_Anderson (7 comments) [144 views]

"Russia’s invasion is now an ongoing operation of mass murder"
Crime by Donna     November 22, 2022 1:32 pm (Rating: 0.0) Last comment by: oldedude (7 comments) [182 views]

Good news regarding Trump!
Religion by Donna     November 22, 2022 9:14 am (Rating: 0.0) Last comment by: Donna (32 comments) [574 views]

Education selectors, pages, etc.