How polls are used to manipulate public opinion
By Ron Tapscott, and Richard Newhauser | December 22, 2023
In his recent State of the City speech, Mayor Corey Woods referenced the ETC Institute’s Tempe Community Survey which reported that 8 out of 10 Tempe residents say they’re satisfied with city services. Sorry Mr. Mayor, but it ain’t necessarily so reports Dave Wells, a prominent Arizona policy analyst, who examined the data and method of this year’s ETC survey. What he found was yet another push poll.
Before the Coyotes and the Tempe City Council went public with the proposal for a hockey arena, some Tempe residents received phone calls asking their opinion. If you received one of those calls, you may recall that the pollster wasn’t really interested in your thoughts. This “push poll” was designed to get answers that the sponsors of the poll were looking for. Why go to all the trouble? Partially, push polls try to create partisan opinions by the way their questions are phrased, but they are also used to produce favorable results to show support for those opinions. If you can claim that the majority of people polled were in favor of something, perhaps that thing will seem inevitable.
The push poll supporting the arena project is an extreme example, but it’s not the only one. How polls and questionnaires are designed can all too often influence the resulting outcomes. And those outcomes can be used to create the illusion of public support. In what follows, Dave Wells has provided some cautionary analysis making it clear that the Council’s latest Community Survey has important design flaws. As we have seen from other actions by the current Tempe City Council, this is not the first time that a survey or a proposal that Council members have backed is less representative of our community than it seems to be.
Tempe Community Survey 2023: Methodological Analysis
The ETC Institute indicated that it mailed the survey to an unknown number of randomly selected households in Tempe and followed up with emails and text messages to encourage responses. The ETC Institute does not indicate what the return rate from this sample was. In addition, the City of Tempe advertised the surveys and encouraged responses to a web site via utility bills, emails to resident distribution lists, and social media platforms such as Facebook, Twitter and LinkedIn–where presumably one would need to be following the City. A link was provided. This latter part is convenience sampling.
A total of 1,032 households completed the survey, even though in many cases individuals, and not households, were targeted, meaning that more than one respondent from a household could complete the survey. The ETC Institute claims this provides a margin of error of 3% with 95% level of confidence, which means 19 times out of 20, one would get the same results within 3% of the true number in the City of Tempe. If one plugs 1,032 as the sample size in a margin of error calculator, the 3% is properly calculated.
The methodology has a number of shortcomings:
- 1) The survey respondents DO NOT match a representative sample of City of Tempe households. It skews high on education, high on income, high on age, high on home ownership, and, as has been the case consistently in the last few years, skews high on length of time living in Tempe (20+ years). This is not surprising, given the mixed methods used to accumulate respondents and the failure to weight respondents by actual demographics for the City of Tempe.
- a) That means that the margin of error is incorrect. Unless the survey has worked toward being representative of its population (households in Tempe) and is not a convenience sample (folks who are motivated from their utility bills or have access to links from social media), then respondent results are biased. This matters if those left out have different views than those who completed the survey. You can pull out results by group which partially helps with this issue–though that is a challenging process–at the ETC website.
- b) It’s common for survey respondents to not match the demographics of the intended population, in which case weights are used. This appendix from a recent survey of hospitality workers at Phoenix Sky Harbor illustrates this principle. Weights were not used for overall results in the Tempe Community Survey.
- c) To illustrate, one example of how respondents are skewed take results on age as reported in the survey with results for age for adults only from the American Community Survey from the Bureau of the Census for Tempe.
One can debate how to handle the ASU population for purposes of this survey. However, the relative relationships of the other numbers should be consistent. Instead, it’s highly evident that the Tempe Community Survey over-samples those 65-74. This is a smaller group than 35-44, 45-54 and 55-64, but in the Community Survey sample it is the largest of those groups. GCI did not do further analysis to illustrate other means that the sample is not representative by education, income and home ownership, as well as length of time living in Tempe. The standard correction is to weight the responses to correct for this error, as illustrated in the appendix cited above. However, due to the unknown level of convenience sampling done here, even that correction may not be adequate. No weighting was done..
- 2) The survey itself has methodological issues. It has over 200 questions packed into it. One should always ask what a survey is measuring. For example, one question asked what services that respondents had used. For instance, 8% had used city swimming pools in the last year, but 70% of respondents rated their satisfaction with the quality of City swimming pools as very satisfied or satisfied and only 3% dissatisfied. But exactly what were 92% of respondents answering to? Actually results don’t indicate skipped questions, but this, like many questions, is largely measuring perceptions more than experience or measuring experience from years or decades ago, most likely. That doesn’t help the city identify whether or not city pools are really satisfactory.
The City should reconsider what it’s trying to achieve with the survey and to the degree the city is trying to measure actual experiences of Tempe residents to develop an instrument that is more representative of Tempe residents and has far fewer questions that focus on the most relevant areas of interest.
Dave Wells lived in Tempe for 15 years, and now resides just outside its boundaries in Mesa. He holds a doctorate in political economy and public policy and is often sought out for his public policy expertise. He can be reached at MakeDemocracyWork@gmail.com.