Election polling results are interesting because we like to use them to predict an election. However, polls represent a snapshot in time so extrapolating results to a future event may lead to faulty predictions. With respect to the Regional Chair race poll, we released this week the need for caution is even greater, as this was a single poll in a low turnout election with a response rate of 0.2%. Three issues warrant particular consideration.
1) The Survey Could be a 1 in 20 Outcome
Public opinion surveys are an attempt to ascertain what the population believes about a topic by asking a small group of people. When reporting a single poll a margin of error is typically given at the 95% confidence level, indicating a range of plus or minus a few percentage points within which the population’s belief actually falls. So, for example, in the case of the poll for Regional Chair, 36.5% of respondents were undecided with a range of plus or minus 4.3%. Meaning statistically the poll predicts somewhere between 32.2% and 40.8% of voters were undecided 19 times out of 20. This caution of 19 times out of 20 is an important one to note as statistically speaking, even if the poll is a perfectly random sample 1 in 20 times the actual result is expected to fall outside of the margin of error (i.e. 1 time in 20 the poll will simply be wrong).
The best defence against this problem is multiple polls on the same topic by multiple polling firms. A single poll showing a result may be an outlier, multiple polls showing the same result is unlikely. Multiple polling firms tackling the same topic also decreases the likelihood that a bias may be built into the sampling procedure (i.e. how people are selected to participate in the poll). Yet, even when multiple poll results show the same result, the polls may not be predictive of future events. The most recent presidential election in the United States provides a case study in the need for caution when extrapolating from polls to a future election.
2) The Undecided Voters
The survey revealed that 36.5% of voters have yet to make up their mind. These individuals were first asked who they would support if the election were held today, then asked if they were leaning towards a particular candidate. Asking vote preference twice is considered best practice and tends to capture even soft supporters for a candidate. The 36.5% in this survey who were undecided, are therefore individuals who are likely quite open to being persuaded by at least two of the candidates. An additional 8.7% said they would prefer not to share their preferred candidate, meaning we have no idea how they will vote. Finally, of those who indicated a preference, 8.0% were only leaning towards a candidate (i.e. when first asked who they would support they said they did not know). These results indicate that over 50% of voters preferences are either unknown or open to changing. Undecided voters could therefore dramatically change the election results. How dramatically? Extrapolating from these numbers, and assuming the poll results are accurate and not an outlier, Karen Redman’s support on Election Day could fall anywhere between 27.6% and 89.3%.
3) The Sample May Not Be Accurate
It is also possible that the results of the poll may be off because the sampling method introduced an unknown error into the process. The sample of landlines was created by purchasing an electronic list of listed landlines from www.telephonelists.biz. Landline numbers were then randomly sampled from this list. In addition, a list of likely cellphones and unlisted landlines was added into the sample. This list was created using data published at www.cnac.ca on the area code and exchange (NPA & NXX). The data from the Canadian Number Administrator can be used to ascertain the first six digits of phone numbers that when originally activated belonged to someone who activated a phone within Waterloo Region. However, with number portability, it is possible to keep your phone number when moving into or out of Waterloo Region. The survey revealed 6% of respondents no longer live in Waterloo Region. These individuals were excluded from the final results. However, there was no correction made to introduce people with cell phones originally activated outside of Waterloo Region. It is impossible to know what percentage of phone numbers were excluded from the survey because they were not included in our sample and it is possible that these individuals support different candidates than those included in the survey.
The second problem with respect to the sample is those who choose to participate in the survey. The response rate was less than one-quarter of a percent. It is possible that the 99.8% of the population that did not complete the survey are different than those who did participate. A total of 86.5% of people called did not answer the phone. It is not possible to know if these people are somehow different. Perhaps the reason they were all busy on a Tuesday and Wednesday evening makes them predisposed to a particular candidate. There is no way to know. Of the 13.5% of people who did answer the phone when called; only 1.5% completed the entire survey. Again, it is not possible to know if these people are somehow different. Perhaps these people exhibited a Shy Tory effect and were unwilling to participate because they did not want to admit their preferences to a (liberal) Conestoga College professor.