Al-anwar Newspaper English news in Australia
This past weekend Marquette University Law School Professor Charles Franklin published an interesting analysis of President Donald Trump’s approval rating. He noted that Trump tends to do better in polls in which there isn’t a live interviewer present. The Washington Post’s Philip Bump (who put together some really good charts) pointed out that this difference could arise out of people, especially Democrats, being afraid to admit support for Trump because it is socially undesirable to say you’re for Trump.
It’s an argument I’ve heard before, and, while it certainly cannot be dismissed, I am quite skeptical that respondents are lying to pollsters. Live interview surveys (such as CNN’s) are probably accurately gauging Trump’s support. The difference in results between the different types of polls probably has a lot to do with how different pollsters are obtaining their samples.
CNN typically doesn’t report most non-live interview surveys (i.e. internet surveys or robopolls, which are almost always supplemented with an internet sample) because in most internet polls respondents aren’t chosen at random. Respondents voluntarily sign up, which means that they may, in some way, be unreflective of the population at large even after the overall sample is adjusted to match the population.
Still, there are some internet surveys in which respondents are chosen at random (i.e. probability based) from the entire population. This allows us to test whether live interview surveys are less favorable to the President because A. people don’t want to say they back the President or B. most internet samples are not probability based.
Respondents in these probability based online surveys were not more favorable to the President than in live interview polls. If anything, they were less favorable, which indicates that Americans aren’t lying to live interview pollsters.
To my knowledge, there have been three pollsters who have asked about Trump’s approval rating and who have internet panels that have been randomly selected: National Opinion Research Center, Pew Research Center (which also does live interview polls) and the USC-LA Times poll. National Opinion Research Center had conducted five surveys about Trump’s approval rating, Pew (via its internet panel) had done three and the USC-LA Times had done two polls on Trump’s approval rating since he came President.
For each of these surveys, I took an average of the net approval rating (approval rating minus disapproval rating) of live interview polls with an end date between three days before and three days after the end date of the probability based online poll. If pollsters were lying to live interviewers, we’d see the average Trump net approval rating in live interview polls be lower on average than the probability based internet polls. Instead, we see the opposite.
In all five of the National Opinion Research Center surveys, the live interview polls had an equal (once) or lower net approval rating (four times) for the President than the National Opinion Research Center polls. The average National Opinion Research Center poll actually featured a net approval rating of 10 points below the net approval rating of the live interview polls conducted during the same period.
One USC-LA Times poll showed Trump with a net approval rating of 2 points above the average of live interview polls done during the same period, while one showed it 4 points below an average of the live interview polls done over the same period. The average USC-LA Times poll had a net approval rating of 1 point below live interview polls done during the same time frame.
Finally, the same holds true for Pew. In all three Pew surveys, the average net approval rating was either equal (once) or lower (twice) than the net approval rating of live interview polls taken during the same period. The average Pew internet probability based sample had a lower net approval rating than the live interview polls by 4 points.
Interestingly, Pew’s seven telephone polls over the course of Trump’s presidency featured a net approval rating that averaged 3 points below other live interview polls during the same period. In other words, the Pew internet and Pew live telephone polls had net approval ratings that were about the same in relation to other surveys. In fact, a Pew report comparing one of their live interview polls with an internet survey of theirs conducted at the same time found no difference in opinions toward Trump between the two styles of interview.
Now, the sample size of probability based online surveys (10) is small enough that we cannot say that telephone polls are actually overestimating Trump. But the fact that we see no systematic improvement in his approval ratings in probability based online polls compared with live interview telephone polls strongly suggests that telephone polls aren’t underestimating his strength because people are lying to pollsters.
It is, of course, possible that pollsters overall are missing some segment of the Trump population, even if people aren’t lying about their intentions. Pollsters may not be weighting properly or some people may not be answering surveys for some reason. Polls underestimated Republican candidates across the board in the 2016 election, though it wasn’t Trump specific (i.e. people weren’t lying about Trump).
Just remember, though, that polling errors don’t favor one side consistently from year to year. In elections during 2017, polls didn’t underestimate Republicans. We shouldn’t be expecting the polls to be underestimating Republicans now.
One final element is that all the surveys show similar trends. So even if live operator polls have Trump consistently higher or lower, when he goes down he usually goes down in all the live operator polls and online surveys. When he goes up, he goes up in all of them. So directionally they tell a similar story.
