OPINION POLLS ARE one of the main talking points of the recent local and European Parliament elections. Tánaiste Micheál Martin and Jack Chambers, Fianna Fáil Director of Elections, publicly criticised pollsters using online panels.
Fianna Fáil has good reason to be frustrated. General election polls have significantly underestimated support for Fianna Fáil and vastly exaggerated support for Sinn Féin. For them, this creates a mistaken media narrative about popular support for Sinn Féin.
But does this underestimation of support for FF in general election polls mean that we should completely ignore opinion polls? Do they still have value for politicians, journalists and voters? How can the media, politicians and voters engage more constructively with public opinion polls? The starting point is to recognise that general election polls do not measure support for parties in local elections. Irish general election polls have always been a poor predictor of local elections, but are good predictors of general election results.
One criticism raised by politicians and commentators after these elections relates to the huge difference between support in current opinion polls for the upcoming general election and the local election results.
Let’s untangle local and general election polls. Are opinion polls for general elections accurate? This is the first question I will address.
The Irish Polling Indicator project, which Tom Louwerse (Associate Professor, Leiden University) and I started 10 years ago, aggregates all opinion polls into a single estimate of public support.
We have collected all opinion polls published since the mid-1980s, resulting in a sample of 717 surveys.
This tedious exercise involved going through newspaper archives, checking various websites and contacting academics. The resulting dataset not only allows us to present aggregated estimates but also helps us understand how well polls predict elections.
To assess the accuracy of polls, I extracted the Polling Indicator estimate on the day of the election and compared the figures to the actual first preference vote shares. Next, I calculate the absolute mean error, which is the difference between the first-preference vote share at the election and the aggregated estimate. If a party obtained 25 per cent of first-preference votes, but the poll of polls estimated support at 21 per cent, the absolute error for this party amounts to 4 percentage points. This analysis is performed for all parties to calculate the average absolute difference.
The graph below reveals that the absolute mean error results to be just 1.32 percentage points across the eight general elections between 1989 and 2020. In other words, on average, the poll of polls is just 1.3 points off the election result. This is very accurate.
From a political science perspective, it is also very encouraging: most estimates fall within the margin of error. There are, of course, exceptions to the rule. In 2002, the polls overestimated support for Fianna Fáil by 4.6 percentage points but underestimated the party’s support by 4.4 points in 2007 and 3.4 points in 2016.
In 2020, the aggregated polls overestimated support for Fianna Fáil by 2.4 points. In 2016, the polls overestimated support for Sinn Féin and Fine Gael by around 3 points each. Yet, overall, polls have done a pretty good job in predicting party support in Irish general elections over the past 25 years.
What about Local Elections? Pollsters received a lot of criticism for underestimating Fianna Fáil support and vastly overestimating the Sinn Féin vote last week. Yet, opinion polls do not intend to predict local election results.
Looking at the second figure here supports this argument. The absolute mean differences range from 2.46 (2004) to 3.77 (1991) points. Last weekend, the mean difference was 3.18 points. At almost every election, the estimates for at least one party are completely different to the outcome.
Why are the polls so far off? Irish local elections are unique. Incumbency plays a huge role in Local Elections. Our research shows that even incumbents who got elected on the last count face a huge advantage in rerunning and getting re-elected compared to the candidate who lost on the last count. Canvassing, name recognition and community ties are highly relevant. Door-to-door campaigning is a unique feature of Irish politics and it is a crucial condition for getting elected.
An Ireland Thinks poll fielded on the day of the local and European Parliament elections mirrors these claims. 75 per cent of Fianna Fáil voters cast their first preference for a candidate rather than the party the candidate represented. 66 per cent of Fine Gael voters ranked the party over the candidate.
The focus on candidates is lower for smaller parties – possibly because they ran fewer incumbents and their candidates were simply not as well known.
Fianna Fáil and Fine Gael also canvassed very successfully. 35 per cent of respondents said they met a Fianna Fáil candidate at their doorstep. 34 per cent of respondents had direct contact with a Fine Gael candidate. Only 18 per cent met a Sinn Féin representative. This might be part of the explanation why Sinn Féin performed even more poorly than anticipated: they are not only losing support in opinion polls, but they had few incumbents and their candidates were simply not as active on the ground as Fianna Fáil and Fine Gael. To sum up, incumbency and local ties are crucial, and Fianna Fáil and Fine Gael outperformed Sinn Féin in this hugely important aspect of campaigning.
We need to keep in mind that the electorate is becoming more volatile. According to Exit Polls from the past 25 years, around half of the voters make up their minds about the first preference vote in the last month before an election. Since the financial crash, around four in 10 voters have switched between parties from one general election to the next. Campaigns and canvassing are crucial in general elections and can reshuffle the Irish political landscape.
The debate between the efficacy of online and face-to-face polls continues to be contentious. While some, like Fianna Fáil’s Micheál Martin and Jack Chambers, argue that online polls tend to underestimate their support, face-to-face polls also present challenges, such as low response rates and social desirability biases. Red C Research, despite being criticised at the moment, accurately predicted Fianna Fáil’s performance in the 2020 general election.
Ideally, we rely on a mix of methodological approaches to better understand Irish party politics. Telephone polls, face-to-face surveys, and online surveys all have advantages and shortcomings. Michael A. Bailey, Professor at Georgetown University, describes these issues in an excellent recent book “Polling at a Crossroads”. Having an open mind towards public opinion will ultimately improve our understanding of public opinion.
What can we learn from this debate on opinion polls and how can we move forward the debate? I have four recommendations:
1. Media outlets should adjust their reporting. We must consider structural differences between polling companies, focus on trends rather than individual polls and not overplay small “changes” lying within the margin of error as “bounces”, “rebounds”, and “recoveries”. Sampling variation drives most of these “changes”, which have usually been within the margin of error of ±3 percentage points. Political scientists Erik Gahner Larsen and Zoltán Fazekas show that Danish news reports frequently interpret deviations within the margin of error as substantive changes. The authors conclude that “a horserace coverage of polls about change can rest on a foundation of stability.” Irish newspapers often apply the same rhetoric. A move away from horserace coverage is necessary.
2. In addition to changing the language, polling companies and newspapers could present party support as within windows (i.e., the 95 confidence interval). Reporting single numbers projects a false sense of precision. Other countries are slowly adopting this approach. For example, Italian media reported the results from the exit poll for the European Parliament elections in intervals. Rather than focusing on a single number – and suggesting a wrong sense of certainty – their headline finding was that “Meloni’s FdI to get 26-30% in EU elections – exit poll Centre-left Democratic Party to come second with 21-25%”. Irish media should follow suit.
3. We should focus more on trends. For example, the polling trends clearly showed the loss of support for Sinn Féin in general election polls – across all companies. While pollsters deviate systematically in their survey mode and baseline results, these developments are unambiguous. Voters should approach poll results critically, considering the broader context and acknowledging uncertainty.
4. We need to keep in mind that polls provide a single snapshot and that using polls to predict elections is extremely difficult. Paul Perry, back then Vice Chairman of Gallup Polling, focused on these problems in a Public Opinion Quarterly paper published 45 years ago:
All one has to do is use a properly drawn sample of the electorate large enough to minimise random sampling error, get honest answers from everyone, do the questioning close enough to the time of voting to minimise changes in voting intentions, anticipate how the undecided will vote, and, finally, distinguish between voters and non-voters in the electorate – (Perry, 1979: page 312).
Pollsters should need to be even more transparent in their reporting methods. While most outlets now list the poll’s margin of error and time when the survey was in the field, the exact approach of selecting households for face-to-face polls, selecting respondents from the online panel, non-response rates, and the weighting of respondents is still a black box. This is understandable because polling is highly complex and relies on cutting-edge statistical methods. Media outlets could add a few sentences about the sample selection and weighting.
Finally, it is important to note that polls are extremely useful for understanding voter preferences and behaviour. Journalists and politicians need polls to understand the electorate. I believe the current debate on the usefulness of polls is constructive and ought to lead to important reflections among all actors involved. I hope that mentioning methods and uncertainty will increase trust in public opinion. Ultimately, these changes would help us to focus on policy issues and the solutions that parties offer.
Stefan Müller is Assistant Professor in the School of Politics and International Relations at University College Dublin. He maintains the Irish Polling Indicator and the Irish Politics Data project.
To embed this post, copy the code below on your site
have your say