Menu Close

Evaluating the 2021 Federal Election Projections

With the final seat tally now official, it is possible to evaluate the prediction results. For this analysis, CBC Canada Votes 2021 at 8:00 am on September 25, 2021, is being used. The analysis examines six websites that aggregate polls from the polling firms and then create a model to predict the election are examined (338Canada, Calculated Politics, CBC Poll Tracker, Lean Toss Up, Too Close to Call, and The Signal), three projection websites using their own models (Election Prediction Project, Election Atlas, and Advanced Symbolics) and four polling firms which use their own data to predict the election (Ekos, Forum, Innovative Research, and Mainstreet).

The results show the Liberals with 159 seats, the Conservatives with 119, the NDP with 25, the Bloc with 33 and the Greens with two. Most projections underestimated Liberal support. Mainstreet was predicted exactly 159 seats. The average estimates were missed by five seats. The Conservative support was very close with a mean of 118. Six projections overestimated Conservative support, seven underestimated Conservative support, and the CBC Poll Tracker had exactly 119 seats. The average estimates were very close only missing by one seat. NDP support was overestimated by every projection by an average of seven seats. The Bloc seats were overestimated in ten projections and underestimated in four. The mean of Bloc support was only off by two seats. Seven projections accurately predicted the Greens would win two seats and the average projection was two seats. Five projections predicted the Greens would win three seats and two predicted one seat. Three projections incorrectly predicted the PPC would win one seat.

To evaluate the overall accuracy of the different projections the projection for each party is subtracted from the actual seats won. The absolute value is then taken (i.e. if it is negative it is turned positive) and these values are added together. This gives a value of the error of each projection. (A more accurate method would be to look at each individual riding and see what percentage of the ridings were accurately predicted. This information is not readily available for each of the projections).

The average projection error was 24. The most accurate projection was by Mainstreet who only missed by 12. Mainstreet also had the most accurate projection in 2019. The poll aggregates as a group were the most accurate. Five poll aggregator projections (Calculated Politics, CBC Poll Tracker, Lean Toss Up, Too Close to Call, and @politicstacan) had an error of 14. The most accurate other projection method was Election Atlas with a projection error of 16. Ekos also had an error of 16.

Part of the accuracy of the models used to predict the vote share of each party in the election is due to the accuracy of the projection of the vote share of each party. The next comparison examines how accurate the vote share was projected. The average error is calculated in the same way as the previous analysis, with the absolute value of the difference taken for each party.

The average error was 10.1 percentage points. The most accurate projection was from The Signal with an error of 6.2 percentage points. Too Close to Call was just behind at 6.5 percentage points off. All but one of the aggregators had better projections than each polling firm. Calculated politics had the same tracking error as Mainstreet at 10.7 percentage points. Mainstreet was the most accurate polling firm once again. On average, the projections underestimated the Liberal, Conservative, and Bloc party support were underestimated. The NDP, Green, and PPC support was overestimated by the projections.

Every projection accurately predicted a Liberal minority government with the Conservative party in second place. Seat projections for the Bloc, Greens, and PPC were also reasonably accurate. The NDP seat projections were the biggest miss with every projection overestimating the projected seat total with an average error of seven seats. The Liberal party support on average was overestimated by five seats on average. A post-election analysis of why the NDP was down in the projected vote and seat count is warranted to improve future projections. Overall, the projections were very accurate, and the projection modellers should be satisfied with their predictions.

2021 Federal Election Projections

With federal election day having arrived the campaigns are turning their attention to getting out the vote. The last poll results have been reported by the big survey firms and the final election results will not start showing up until the evening. Ultimately, it’s a slower day for those who love politics waiting for the election results. A day where many will spend it speculating about the results and making predictions on who will win.

While many of us like to predict what will happen on election day based on intuition, poll aggregating websites and individual polling firms use models to attempt to project which party will win every seat in Canada. The results from multiple Canadian prognosticators can provide a snapshot of what is predicted to happen tonight.

Results are shared here of seven websites that aggregate polls from the polling firms and then create a model to predict the election are examined (338Canada, Calculated Politics, CBC Poll Tracker, Lean Toss Up, Too Close to Call, and The Signal, @politicscan), three projection websites using their own models (Election Prediction Project, Election Atlas, and Advanced Symbolics) and four polling firms which use their own data to predict the election (Ekos, Forum, Innovative Research, and Mainstreet).

Every single projection predicts a Liberal minority government. However, the highest projection has the Liberals within a couple of seats of a majority. The average projection has the Liberal Party dropping a single seat to 154 from the numbers at dissolution. The Conservative Party projections have them on average at 120 seats, up one from dissolution. The NDP are predicted to make the most gains moving up eight seats. The Bloc are projected to remain the same . The Green Party is expected to have 2 seats. Note, one seat was vacant at dissolution.

Some of the differences in projections may be due to differences in how the poll aggregator websites averaged the polls. Similarly, the differences in the polling firms’ projections are due to both their models and what the firms’ final polls projected as the vote share of each party.

The table above provides an overview of the predicted vote share Canada-wide for the poll aggregators and the polling firms. Using the poll aggregators average the Liberal Party is expected to drop 1%, the Conservative Party 3%, the NDP are up 3%, the block down 1%, the Green Party is down 3%, and the PPC is up 5%.

After the election, a second post will provide a basic examination of how well the projections did at predicting the actual election results.

*Updated at 10:46 am to reflect changes to 338Canada‘s projections

*Updated at 2:30 pm to reflect changes to Ekos and The Signal‘s projections and to add @politicscan projections.

Evaluating the 2019 Election Projections

With the votes mostly counted and almost all seats decided, it is now possible to evaluate the election projections and predictions. For this analysis, I have used data from the CBC Canada Votes at 11:00 am on October 22, 2019.

This analysis will examine eleven poll aggregators, Four analysis-based projections, and three polling firm models. 338 CanadaCalculated PoliticsCanadian Election WatchCBC Poll Tracker, Lean Toss UpLISPOP, Too Close to all, Visualized Politics@EarlWashburn@repdonsman456, and @politikstcan each aggregate multiple polls from different polling firms to predict the election. Election Prediction Project and Teddy on Politics makes predictions using an analysis process, which examines data from the public alongside general trends. Teddy on Politics and @kylejhutton made similar ‘gut-based projections’ of the election. EkosForum, and Mainstreet are polling firms who use their data to model the election results.

Liberals won 157 seats. Overall, we can see that all the poll aggregators except for Lean Toss Up underestimated support for the Liberals. All the analysis-based projections also underestimated the Liberal support. However, two of three polling firms Ekos and Mainstreet overestimated support for the Liberals. Ekos was the closest to the Liberal projections missing by one seat at 158.

The Conservatives won 121 seats. Every poll aggregator projection, except for Lean Toss Up and @politicstacan, overestimated support for the Conservatives. All three polling firms underestimated Conservative support. Calculated Politics was the closest Conservative projection missing by one seat at 122.

The NDP won 24 seats. Every projection overestimated NDP support. Lean Toss Up and Mainstreet were the closest to the NDP at 25. The Bloc won 32 seats and every projection except for Ekos and Mainstreet estimated Bloc support. Calculated politics were closest to the Bloc at 34.

The Green party won three seats. Seven projections accurately projected this result. The People’s Party of Canada won zero seats. Seven projections accurately projected this result as well. One independent won, fifteen projections predicted this result.

To evaluate the overall accuracy of the different projections two methods are being used. First, the projection for each party is subtracted from the actual seats won. The absolute value is then taken (i.e. if it is negative it is turned positive) and these values are added together. The second method involves taking the Sum of Squares difference. This is done by subtracting the difference between projection and result and squaring this value then adding these together. This value is then divided by 7 (for the 7 projections) and then the square root is taken to show the average difference between each individual projection. The results of the two approaches are relatively similar.

Mainstreet was the most accurate projection by polling firm with Ekos close behind. Lean Toss Up was the most accurate poll aggregator, and third most accurate. The Elections Prediction Project was the most accurate analysis-based projection and fourth overall.

Part of the accuracy is determined as a result of the models the polling firms and aggregators use to project the election. Some of the differences in accuracy can also be attributed to how the polling aggregators model the polls of the election.

When comparing how the poll aggregators did at projecting the election Lean Toss Up once again was the most accurate only missing the actual results by an average of one percentage point. It should be noted that some of the poll aggregators did not share the projections for the PPC of Independents.

While individual models varied slightly in their performance. Overall, the aggregators, analysis-based projections, and polling firms should be satisfied with how they performed at projecting this election. All models predicted a minority government and most projected the Liberals would win the most seats.

*Updated at 1:00 pm to include the final Mainstreet seat projections