News & comments posted organically
SelectSmart.com
Report spam & abuse

SelectSmart.com home page

Can we trust what polls are telling us?

By Curt Anderson
January 27, 2020 6:21 pm
Category: Education
(5.0 from 1 vote)
SelectSmart.com    SelectSmart.com    SelectSmart.com

Share
The posted rules of the Post
Rate this article
(5=best, 0=poor)
5 Stars
4 Stars
3 Stars
2 Stars
1 Star
0 Stars

I usually defend polling organizations. Polls, when properly done, are the best tool we have to measure public opinion on a particular issue or candidate. However, not all polls are created equal.

Open-access polls vs. scientific polls
We have all seen online polls in which we are asked to opine on various topics. These are known as open-access polls. The participants are self-selected, and anything but a random cross-section of voters or of any other identified group. They are fun, but not very meaningful.

For polls to be considered meaningful they must be scientific polls. The most glaring difference between an open-access poll and a scientific poll is that scientific polls randomly select their samples. Scientific polls use statistical weights to make them demographically representative of the target population.

People who are critical of polls (often when they don't like the results) sometimes ask how can a relatively small number of polled participants, often around 1000 people or so, be representative of a state or even all of America. There is a soup making analogy that answers that question. If you are making a large amount of a cream soup and want to know if it is properly seasoned, you don't need to consume the entire cauldron of soup. If you stir the soup sufficiently for minute or so, you can get a pretty good idea how much pepper, salt or other spices are needed by tasting a single spoonful. In other words, that spoonful you tasted represents a randomized sampling of all the soup in that cauldron. That is what a scientific poll endeavors to accomplish.

One of the polls that caught my attention is the Emerson poll. You might think I am picking on Emerson. My criticism might be considered unfair since I have not scrutinized other polls in the same way. I am less than confident in how Emerson achieves their results, and I will explain why.

In their defense, I will note that Emerson College Polling is well-regarded. Emerson polls are cited by major news organizations. FiveThirtyEight.com gives Emerson an A- grade. That said, being naturally skeptical, I tend to look at the methodology of polls. Here is how Emerson says they do their polling:


The National Emerson College Poll was conducted January 21-23, 2020 under the supervision of Assistant Professor Spencer Kimball. The sample consisted of registered voters, n= 1,128, with a Credibility Interval (CI) similar to a poll's margin of error (MOE) of +/- 2.8 percentage points. The data was weighted based on 2016 voter model of gender, education, age, ethnicity, and region. It is important to remember that subsets based on gender, age, party breakdown, ethnicity, and region carry with them higher margins of error, as the sample size is reduced. Data was collected using both an Interactive Voice Response (IVR) system of landlines only (n= 741) and an online panel provided by Amazon Turk (n= 387).


The terminology used in Emerson's description of their methodology caused me to search the Internet for definitions. Here is my glossary of the bold type terms above.

Credibility Interval (CI)
AAPOR [American Association for Public Opinion Research] urges caution in the interpretation of a new quantity that is appearing with some nonprobability opt-in, online polling results the credibility interval. The credibility interval is not the margin of sampling error (MOE) that the public has come to understand as the statistical uncertainty of probability based scientific polls. Instead, credibility intervals are being used when reporting on some nonprobability polls (typically opt-in online polling), or for model-based inferences such as arise in small-area estimation.

Margin of error
The margin of error is a statistic expressing the amount of random sampling error in a survey's results.

Interactive Voice Response
Interactive voice response (IVR) is a technology that allows a computer to interact with humans through the use of voice and DTMF tones input via a keypad.

Online panel provided by Amazon Turk
This term in particular had me flummoxed. I went to the website that explains it. Here is what it says:

Make money in your spare time
Get paid for completing simple tasks
Want to make money in your spare time? Become an Amazon Mechanical Turk (MTurk) Worker and start applying your skills to the thousands of available tasks in the MTurk marketplace.

What type of work is available on Amazon Mechanical Turk?
Workers help companies collect and edit information by:
Filling out market research or survey data on a variety of topics.


IVR and Amazon Turk respondents may not be like the rest of us
I am suspicious of Emerson's methodology not their sincerity. I suspect that only a certain sort of person has a landline and the time and patience to answer the questions of a robotic voice. I would be excluded from that group. I don't answer the phone unless I recognize the caller's number. Maybe if I were lonely I would pickup the phone whenever it rings. Maybe if I had more time in my life I'd be willing to "talk" with a robot.

I also doubt that Amazon Turk workers who are paid to respond to pollsters are representative of all registered voters. I assume that they are low paid workers without many benefits. I can't say if the IVR or the Amazon Turk respondents are any more liberal or any more conservative than the entire voting population. It certainly seems that those groups would have characteristics that not all of us share.

Cited and related links:

  1. emersonpolling.reportablenews.com
  2. aapor.org
  3. en.wikipedia.org
  4. mturk.com
The views and claims expressed by contributors are their own and do not necessarily reflect the views and beliefs of SelectSmart.com. Not every statement made here can be assumed to be a fact.
Comments:

Be the first to comment on this article.




Cookies Consent Policy & Privacy Statement. All Rights Reserved. SelectSmart® is a registered trademark. | Contact SelectSmart.com | Advertise on SelectSmart.com | This site is for sale!


From our correspondents:
Display Order:
Meet the swampy recipients of Trump's pardons and commutations!
by Curt Anderson     February 19, 2020 1:50 pm (Rating: 5.0)
Last comment by (0 comments) [223 views]


Fair warning: The Democratic primary race in 2020 is looking a lot like 2016.
by Curt Anderson     February 18, 2020 6:43 pm (Rating: 5.0)
Last comment by (0 comments) [98 views]


Don't pester me for political donations when you refuse $2700 and less from billionaires!
by Curt Anderson     February 15, 2020 12:51 pm (Rating: 5.0)
Last comment by (0 comments) [250 views]


Bernie Sanders wrote and voiced this documentary on his hero, the socialist Eugene V. Debs.
by Curt Anderson     February 13, 2020 11:17 pm (Rating: 5.0)
Last comment by (0 comments) [134 views]


A Pete Buttigieg Candidacy Would Have Been Unthinkable Not Too Long Ago.
by Curt Anderson     February 13, 2020 12:37 pm (Rating: 5.0)
Last comment by (0 comments) [75 views]


Salacious, slightly obscene slogans of past presidential campaigns
by Curt Anderson     February 11, 2020 12:33 am (Rating: 5.0)
Last comment by (0 comments) [172 views]


Apparently undeterred by coronavirus, cruise ship ad sings about pills and delirium
by Curt Anderson     February 9, 2020 10:29 pm (Rating: 5.0)
Last comment by (0 comments) [91 views]


Damian Lillard and the Blazers got screwed. There is a simple solution so it won't happen again
by Curt Anderson     February 8, 2020 7:19 pm (Rating: 5.0)
Last comment by Curt Anderson (1 comments) [118 views]


Was Mike Bloomberg the real winner in Iowa?
by Curt Anderson     February 5, 2020 5:33 pm (Rating: 5.0)
Last comment by Curt Anderson (2 comments) [117 views]


Was Pelosi correct in saying that Trump has ripped up the Constitution?
by Curt Anderson     February 6, 2020 12:36 pm (Rating: 2.5)
Last comment by (0 comments) [125 views]


Nancy Pelosi tears a new page in State of the Union history...
by Curt Anderson     February 5, 2020 2:04 pm (Rating: 5.0)
Last comment by (0 comments) [19 views]


Iowa's caucus system is an unDemocratic and undemocratic way to nominate a president
by Curt Anderson     February 4, 2020 3:21 pm (Rating: 5.0)
Last comment by (0 comments) [117 views]


Rush Limbaugh announces he has advanced lung cancer
by Curt Anderson     February 3, 2020 12:50 pm (Rating: 5.0)
Last comment by (0 comments) [113 views]


Will conspiracy-minded Trump supporters enter en masse into Darwin Award competition?
by Curt Anderson     February 1, 2020 6:16 pm (Rating: 5.0)
Last comment by (0 comments) [146 views]


Republicans will likely be able to block impeachment witnesses. Good, I am glad!
by Curt Anderson     January 31, 2020 1:52 pm (Rating: 5.0)
Last comment by Curt Anderson (1 comments) [90 views]


Gusty winds blew over a portion of Trump's border wall into Mexico
by Curt Anderson     January 30, 2020 11:13 am (Rating: 5.0)
Last comment by (0 comments) [95 views]


Dershowitz detours into crazy town: Trump believes his re-election is in the national interest, therefore a legitimate quid pro quo.
by Curt Anderson     January 29, 2020 1:04 pm (Rating: 5.0)
Last comment by (0 comments) [93 views]


Not much chance that Bolton will be allowed to testify at the impeachment trial now.
by Curt Anderson     January 26, 2020 4:20 pm (Rating: 5.0)
Last comment by Curt Anderson (2 comments) [325 views]


Can we trust what polls are telling us?
by Curt Anderson     January 27, 2020 6:21 pm (Rating: 5.0)
Last comment by (0 comments) [81 views]


Kobe Bryant Dead At 41.
by Curt Anderson     January 26, 2020 12:36 pm (Rating: 5.0)
Last comment by (0 comments) [35 views]


Why was Hunter Biden paid big bucks by Burisma?
by Curt Anderson     January 21, 2020 10:02 pm (Rating: 5.0)
Last comment by Curt Anderson (2 comments) [131 views]


Trump, like a mob boss: Get her out tomorrow. I don't care. Take her out. OK? Do it.
by Curt Anderson     January 24, 2020 1:09 pm (Rating: 5.0)
Last comment by (0 comments) [47 views]


Tulsi's $50 million lawsuit against Hillary is a cynical ploy to bilk supporters
by Curt Anderson     January 22, 2020 11:54 pm (Rating: 5.0)
Last comment by (0 comments) [39 views]


NY Times' Krugman: The Sanders campaign has flat-out lied about Biden
by Curt Anderson     January 21, 2020 1:12 pm (Rating: 5.0)
Last comment by (0 comments) [38 views]


Find old posts & articles

Recently rated items