News & comments posted organically
Report spam & abuse home page

Can we trust what polls are telling us?

By Curt Anderson
January 27, 2020 6:21 pm
Category: Education
(5.0 from 1 vote)

The posted rules of the Post
Rate this article
(5=best, 0=poor)
5 Stars
4 Stars
3 Stars
2 Stars
1 Star
0 Stars

I usually defend polling organizations. Polls, when properly done, are the best tool we have to measure public opinion on a particular issue or candidate. However, not all polls are created equal.

Open-access polls vs. scientific polls
We have all seen online polls in which we are asked to opine on various topics. These are known as open-access polls. The participants are self-selected, and anything but a random cross-section of voters or of any other identified group. They are fun, but not very meaningful.

For polls to be considered meaningful they must be scientific polls. The most glaring difference between an open-access poll and a scientific poll is that scientific polls randomly select their samples. Scientific polls use statistical weights to make them demographically representative of the target population.

People who are critical of polls (often when they don't like the results) sometimes ask how can a relatively small number of polled participants, often around 1000 people or so, be representative of a state or even all of America. There is a soup making analogy that answers that question. If you are making a large amount of a cream soup and want to know if it is properly seasoned, you don't need to consume the entire cauldron of soup. If you stir the soup sufficiently for minute or so, you can get a pretty good idea how much pepper, salt or other spices are needed by tasting a single spoonful. In other words, that spoonful you tasted represents a randomized sampling of all the soup in that cauldron. That is what a scientific poll endeavors to accomplish.

One of the polls that caught my attention is the Emerson poll. You might think I am picking on Emerson. My criticism might be considered unfair since I have not scrutinized other polls in the same way. I am less than confident in how Emerson achieves their results, and I will explain why.

In their defense, I will note that Emerson College Polling is well-regarded. Emerson polls are cited by major news organizations. gives Emerson an A- grade. That said, being naturally skeptical, I tend to look at the methodology of polls. Here is how Emerson says they do their polling:

The National Emerson College Poll was conducted January 21-23, 2020 under the supervision of Assistant Professor Spencer Kimball. The sample consisted of registered voters, n= 1,128, with a Credibility Interval (CI) similar to a poll's margin of error (MOE) of +/- 2.8 percentage points. The data was weighted based on 2016 voter model of gender, education, age, ethnicity, and region. It is important to remember that subsets based on gender, age, party breakdown, ethnicity, and region carry with them higher margins of error, as the sample size is reduced. Data was collected using both an Interactive Voice Response (IVR) system of landlines only (n= 741) and an online panel provided by Amazon Turk (n= 387).

The terminology used in Emerson's description of their methodology caused me to search the Internet for definitions. Here is my glossary of the bold type terms above.

Credibility Interval (CI)
AAPOR [American Association for Public Opinion Research] urges caution in the interpretation of a new quantity that is appearing with some nonprobability opt-in, online polling results – the credibility interval. The credibility interval is not the margin of sampling error (MOE) that the public has come to understand as the statistical uncertainty of probability based scientific polls. Instead, credibility intervals are being used when reporting on some nonprobability polls (typically opt-in online polling), or for model-based inferences such as arise in small-area estimation.

Margin of error
The margin of error is a statistic expressing the amount of random sampling error in a survey's results.

Interactive Voice Response
Interactive voice response (IVR) is a technology that allows a computer to interact with humans through the use of voice and DTMF tones input via a keypad.

Online panel provided by Amazon Turk
This term in particular had me flummoxed. I went to the website that explains it. Here is what it says:

Make money in your spare time
Get paid for completing simple tasks
Want to make money in your spare time? Become an Amazon Mechanical Turk (MTurk) Worker and start applying your skills to the thousands of available tasks in the MTurk marketplace.

What type of work is available on Amazon Mechanical Turk?
Workers help companies collect and edit information by:
Filling out market research or survey data on a variety of topics.

IVR and Amazon Turk respondents may not be like the rest of us
I am suspicious of Emerson's methodology not their sincerity. I suspect that only a certain sort of person has a landline and the time and patience to answer the questions of a robotic voice. I would be excluded from that group. I don't answer the phone unless I recognize the caller's number. Maybe if I were lonely I would pickup the phone whenever it rings. Maybe if I had more time in my life I'd be willing to "talk" with a robot.

I also doubt that Amazon Turk workers who are paid to respond to pollsters are representative of all registered voters. I assume that they are low paid workers without many benefits. I can't say if the IVR or the Amazon Turk respondents are any more liberal or any more conservative than the entire voting population. It certainly seems that those groups would have characteristics that not all of us share.

Cited and related links:

The views and claims expressed by contributors are their own and do not necessarily reflect the views and beliefs of Not every statement made here can be assumed to be a fact.

Be the first to comment on this article.

Cookies Consent Policy & Privacy Statement. All Rights Reserved. SelectSmart® is a registered trademark. | Contact | Advertise on | This site is for sale!

From our contributors:
Display Order:
Trump's Supreme Court nomination may have just hurt his chances with the voters he needs.
by Curt Anderson     September 27, 2020 10:40 pm (Rating: 5.0)
Last comment by (0 comments) [2088 views]

RBG's death has sparked an eyepopping flurry of contributions to Democrats.
by Curt Anderson     September 19, 2020 3:04 pm (Rating: 5.0)
Last comment by (0 comments) [8022 views]

What was Bob Woodward's responsibility in reporting on what Trump said about COVID?
by Curt Anderson     September 14, 2020 7:32 pm (Rating: 5.0)
Last comment by (0 comments) [4891 views]

Trump's Nobel Peace Prize nomination means less than you think.
by Curt Anderson     September 9, 2020 1:29 pm (Rating: 3.5)
Last comment by (0 comments) [6347 views]

Who actually accomplished progressive change in America? A case for Joe Biden.
by Curt Anderson     September 6, 2020 11:25 am (Rating: 5.0)
Last comment by (0 comments) [3275 views]

First they came for the peaceful Portland protesters...
by Curt Anderson     July 25, 2020 3:27 pm (Rating: 4.2)
Last comment by Anonymous (3 comments) [11591 views]

Hurricane damages Confederate monument after officials voted to keep it
by Curt Anderson     August 27, 2020 11:04 pm (Rating: 5.0)
Last comment by (0 comments) [1029 views]

Why the Republican Party Loathes and Despises Democracy
by Ponderer     August 11, 2020 8:33 am (Rating: 5.0)
Last comment by Anonymous (2 comments) [8487 views]

Trump is unintentionally sabotaging his own and Republican candidates' campaigns.
by Curt Anderson     August 15, 2020 5:52 pm (Rating: 5.0)
Last comment by (0 comments) [12010 views]

Trump is behind in the polls. Does he really have an enthusiasm edge over Biden?
by Curt Anderson     August 10, 2020 10:59 am (Rating: 0.0)
Last comment by (0 comments) [677 views]

He Predicted Trump's Win in 2016. Now He's Ready to Call 2020.
by Curt Anderson     August 5, 2020 7:32 pm (Rating: 5.0)
Last comment by (0 comments) [3864 views]

Real progressives know what to do in 2020
by Curt Anderson     August 1, 2020 1:55 pm (Rating: 5.0)
Last comment by (0 comments) [4757 views]

Why Snoop Dogg Changed His Name to Snoop Lion
by natjones     July 28, 2020 9:44 am (Rating: 5.0)
Last comment by (0 comments) [4193 views]

America is not so great, compared to the rest of the world, when it comes to COVID-19.
by Curt Anderson     July 17, 2020 1:56 pm (Rating: 5.0)
Last comment by (0 comments) [7749 views]

New ad uses Bible, ‘Amazing Grace,’ Jesus to call on Republican Christians to vote against Trump
by Curt Anderson     July 14, 2020 1:14 pm (Rating: 5.0)
Last comment by (0 comments) [3666 views]

Google Trends indicates which political attacks on Biden and Trump are working.
by Curt Anderson     July 5, 2020 11:09 pm (Rating: 5.0)
Last comment by (0 comments) [7940 views]

A new declaration of independence for these times.
by Curt Anderson     July 3, 2020 6:50 pm (Rating: 5.0)
Last comment by (0 comments) [2229 views]

Great People? Trump Retweets Video of Supporters Yelling White Power
by Curt Anderson     June 29, 2020 12:01 pm (Rating: 0.0)
Last comment by (0 comments) [3340 views]

Why did the South secede from the Union? The Confederate States Vice President explains.
by Curt Anderson     June 22, 2020 4:19 pm (Rating: 0.0)
Last comment by (0 comments) [5395 views]

Trump spends 16 minutes explaining his mincing walk down the West Point ramp and why he needed two hands to lift a glass of water.
by Curt Anderson     June 21, 2020 2:15 pm (Rating: 5.0)
Last comment by (0 comments) [711 views]

Trump Crowd Size In Tulsa Underwhelms, Campaign Blames Protesters
by Curt Anderson     June 20, 2020 8:09 pm (Rating: 5.0)
Last comment by (0 comments) [610 views]

Should we view statues as the glorification of imperfect people of the past?
by Curt Anderson     June 16, 2020 10:44 am (Rating: 5.0)
Last comment by (0 comments) [3809 views]

Peel's Nine Principles of Policing: Pertinent now more than ever.
by Curt Anderson     June 8, 2020 11:57 am (Rating: 5.0)
Last comment by (0 comments) [8685 views]

The past is prologue: A 1932 demonstration in Washington DC echos today
by Curt Anderson     June 2, 2020 11:15 pm (Rating: 5.0)
Last comment by (0 comments) [5425 views]

Find old posts & articles

Recently rated items