Long Read: A Plea for Plausible Polling

TL;DR Version of this piece: You may have seen a poll this weekend from FindOutNow. You may also have noticed Ballot Box Scotland has not covered it, nor did I cover the previous one. That is deliberate. In short, I think it’s the biggest pile of rubbish I have ever seen come out of a poll in seven years running this project.

It is obviously not the case that there is a statistical four-way tie for second place. It is obviously not the case Alba are surging whilst the SNP are stabilising, without their central figure and after failing to do so whilst the SNP were in freefall.

Scotland doesn’t get enough polling for us to be able to afford bad data being put into the mix. Stop commissioning bad polling. You might think you’re getting a bargain but all you’re doing is muddying the waters.

The Monkey's Paw

Apparently, I am the owner of a monkey’s paw. I’m not sure where it is, and I didn’t come across it when I was moving house the other month, but the proof of ownership is overwhelming. I keep making wishes related to Ballot Box Scotland, and they keep coming true in the most cursed ways.

Although I’ve not yet done a full writeup, see the proposal from Boundaries Scotland to reduce the South Scotland region to 8 constituencies, and thus 15 MSPs overall (down from 9 and 16). When they previously suggested sticking East Kilbride and Hamilton, Larkhall and Stonehouse in there, I myself said they should consider the special geographic circumstances of rurality and not do that. My reasoning was sound, they clearly agreed, and so out went those two seats… and out remained East Lothian. A monkey’s finger curled.

Redfield and Wilton Precedent

An earlier curling of the paw was when I had wished for more diversity in Scottish polling. That was duly delivered by Redfield and Wilton, who started doing monthly tracker poll in 2023. Yet when their smaller party (Green and Lib Dem) polling especially proved incredibly bouncy, ricocheting between highs and lows. Over two summer months (three polls), the Greens for example went from 13% to 8% to 14%; far too much movement for such short periods over the quiet recess period to be credible. The Lib Dems too rocketed to record results when everyone else had them languishing.

Their Senedd polling was similarly bouncy for smaller parties; going from 4% to 9% to 6% for the Greens over one run of three, or from 10% to 12% to 5% for the Lib Dems over another run . Other pollsters were bemused by some London findings suggesting that the Conservatives were more popular in Inner London than Outer, and Labour the reverse, despite the first page of London Politics for Dummies telling you the opposite. Not trusting the accuracy, and with a monthly contribution being too dominant (more on that later), I excluded them from my reporting and tracking, the first time I’d ever made such a decision.

Challenger Approaching

Having not learned my lesson, I have been wishing for more Scottish polling diversity since the UK General Election. Yet another finger curled on the paw, as rather than getting some tried-and-tested (though never perfect, by the nature of polling) YouGov, Ipsos or Savanta, FindOutNow were commissioned for their first polling beyond Independence. Their December outing was such clear rubbish I immediately discarded it. Published this weekend, their January poll is hands-down the biggest pile of nonsense I have ever seen a pollster produce in Scotland.

It cannot be exaggerated how bad things have to be for me to writing up a full piece explaining my concerns. Scotland simply doesn’t get enough regular polling for our political sphere to be able to afford data this bad.

FindOutNow, All Fantasy?

FindOutNow are relatively new pollsters, certainly to Holyrood and Westminster polling in Scotland. They have previously done Independence polling which, although it has been notably Yes-leaning, I had previously assumed was generally fine and included in trackers because it wasn’t all that different to findings by e.g. Ipsos. Needless to say, given I’m about to go into some detail about why I think their Holyrood polling is nonsense, they won’t be included going forward.

It’s also worth nothing that FindOutNow had one of the biggest misses ahead of July’s UK General Election, which given that was a remarkably badly polled election (in England – Scotland was almost spot on, on average) is really saying something. Their final poll prior to the vote had the Conservatives on just 15% (behind Reform on 17%), the lowest of any pollster’s final call poll, and one of only two alongside “PeoplePolling” to think Reform would beat the Conservatives. The actual final shares were around 24% Conservative and 15% Reform. As such, there’s already prior reason to disregard their polling.

FindOutNow Regional Poll, January 2025

Almost everything about this is complete and utter bollocks. 25% for the SNP would be a record low for them this term, and do you know who the only other pollster in the tracker to have had them that low is? Two Redfield and Wiltons from the period before I blacklisted them. A low share alone isn’t fatal to the poll’s credibility, but it runs counter to both Norstat and Survation findings recently, which have found modest recoveries in SNP shares. Both have found the SNP up 4% from low points in the summer; from 28% in August to 32% in December for Norstat, and from 27% in August to 31% in January for Survation.

Labour have been doing badly lately, and I wouldn’t actually be shocked by a 15% figure in general… if it wasn’t paired with the SNP doing so poorly, and what is in statistical terms effectively a tie for second place with the Conservatives, Greens and Lib Dems who are all on 13%. Those are much higher figures for the two smaller parties than anyone else has found lately. Although they aren’t unprecedented in this term, I struggle to see both being at that level in the same poll in the current political atmosphere. I also simply don’t buy that we’ve got five parties running that close in support to one another, or that Reform are the smallest of those right now.

Most notably however, and this has some dedicated takedown to follow, Alba on 7% is the best they have ever polled. This especially is so obviously absurd that it should have led to FindOutNow apologising, refunding the Herald, and never publishing the findings. In addition, look at that 4% for “others”. What others? What other parties meaningfully exist in Scotland beyond these seven right now? Legally, of course, there are plenty, but who is actually voting for them? 4% “others” is the kind of share I expect to see from pollsters that haven’t got a separate Alba line. It’s extremely dodgy when Alba are actually broken out.

Although I’ve focused on the Regional figures, the other figures have their own bizarre bits. Apparently, the Greens are on 10% of the constituency vote; a figure that does, admittedly, track with a 13% list score based on relative performances in 2021, but which is far higher than anyone else has ever found for that vote. Reform polling 17% of the Westminster but only 11% of the list similarly seems odd.

An Absurdly Astronomical Assessment of Alba

To see just how out of kilter these findings are with everything else, we can compare Alba’s polling since 2021 across three measures. First, the 46 polls I have in the BBS tracker from other pollsters that gave Alba figures (i.e. not counting those that didn’t as 0%, which would lower the average). Secondly, the 12 polls by Norstat (or their prior name, Panelbase) that likewise gave Alba figures. Finally, the 3 FindOutNow polls I am aware of, two of which are very recent.

Alba Polling Average
Alba Peak Share

The differences here are huge in relative terms. The bulk of Scottish polling since 2021, by firms including Savanta, Ipsos, YouGov, Opinium and Survation, has averaged out at just 2.1% for Alba, and indeed none of them have found anything higher than 3%. When you consider that margin of error for a standard 1000 sample poll is about 1% at the 3% mark, in statistical terms most pollsters don’t think Alba have advanced at all since 2021, and never have in even a single on of their polls.

Norstat’s average is twice that at 4%, and they have also peaked at 5% in a few polls. Until we started getting these FindOutNow polls, that made Norstat the only pollster to put Alba above 3%. Given that (as Panelbase) they had most overestimated Alba in 2021, I’ve consistently made the point I view this as nothing more than a house effect unique to them as a pollster. In general, such house effects aren’t the worst thing or a reason not to include a pollster. In a period where all we’ve been getting is Norstat and Survation polls however, that was becoming problematic, as there wasn’t enough other polling to smooth those spikes out.

FindOutNow have blown that totally out of the water. Their average at 5.7% is almost three times what the other pollsters are finding, and their peak at 7% in the most recent poll is more than double their top score with everyone else bar Norstat. Just by simple comparison to other pollsters, and the knowledge that the pollster finding the (until now) strongest Alba shares pre-2021 got it wrong, alarm bells should be ringing.

You might fairly point out that the FindOutNow polls are more recent, whereas the fact I’ve been complaining we haven’t heard enough from other pollsters obviously means that’s older data. However, Norstat have plenty of data both pre-and-post GE24, including two 5% polls beforehand. It’s also important in polling to look at trends in movement as well as actual vote shares. Had Survation polls shown growth, even if it was only to Norstat levels, it would have suggested Alba were indeed on the up. It didn’t. They continued to show no real gains on 2021. The balance of probability therefore remains that high-Alba pollsters are outliers.

Why an Alba Surge Seems Suspect

Alba are Leaderless

This is a bit difficult to talk about because it feels a little indelicate. Nonetheless, it’s a matter of simple fact that Alex Salmond died in October last year. Although Salmond was never elected under the Alba banner (he didn’t come close, but nobody has) it was absolutely his vehicle. He may have been one of the least popular politicians in the country, but he was also an expert attention grabber and headline generator. Whatever your personal views on the man, he was clearly a talented politician, and the only figure lending Alba more credibility than other Indy-Fundamentalist micro-parties. His party has remained leaderless since his passing. It is currently engaged in a leadership contest between two candidates who lack anything approaching Salmond’s talent and ability.

In one corner it has a former Cabinet Secretary, who left Holyrood in 2016 and is so far removed from public relevance and popularity that he placed behind an Alba defector running as an Independent last July. In the other corner it has the widely-considered joke candidate from the SNP’s chaotic 2023 leadership election, who has so little impact at Holyrood she was nonsensically claiming this week to have a budget deal on the basis of things that were actually in the Green deal (school meals) or confirmed by the government before it even lodged the draft budget in December (universal winter energy payment).

This factor alone makes me extremely sceptical of a sudden Alba surge. Bereft of its central figure, controversial and offputting as he was in his latter years, it stretches credibility to suggest that what was functionally the Alex Salmond Party is now on a winning track. However, this isn’t the only factor.

The SNP have Stabilised

The moment that Nicola Sturgeon announced her resignation in February 2023, SNP support started to slide. It spent about 18 months crashing, through an embarrassing leadership contest; a Yousaf leadership that reeled from crisis to crisis before blowing itself up out of sheer stupidity; and a feeling that it was time for new government at both levels. It culminated in a shattering loss of seats in last year’s UK General Election.

All throughout that period where the SNP were haemorrhaging support Alba did not benefit. All you need to do is refer back to the previous section about polling to see that. If there was ever a time for dissatisfied SNP voters to swing behind Alba, it was over those 18 months. They didn’t.

Since then, the SNP have steadied the ship. They’ve gone from lagging Labour in the polls to often quite significant leads, as Labour suffer a double-whammy of a rocky start to government and a challenge from an insurgent Reform UK. Their actual polling figures haven’t recovered enormously, but they’ve stopped falling. Again, it completely stretches credibility to suggest that nobody was interested in Alba when voters were deserting the SNP en masse, but suddenly Alba are doing better than ever when those desertions have stopped.

Barely Bothering with By-Elections

One straw some might nonetheless grasp at is by-elections. At the end of last year, Alba managed a run of what I would describe as “modest-yet-creditable” vote shares at by-elections; in Perth City North; Inverclyde West; and Maryhill. For the first two in particular Alba were using them to argue they were on track for Holyrood wins, as those shares would be sufficient for a seat if replicated region-wide. That’s an “if” so big you could comfortably fit every fantasist who has ever assumed they were going to waltz easily into Holyrood inside.

In reality, Alba were only able to pretend they’d do well outside those by-elections because they simply didn’t stand elsewhere, ensuring there was no opposing evidence. Of the 39 wards that had by-elections in 2024, Alba contested 15 of them in 2022. They contested just 6 of those by-elections, not even half as many. That does not speak to a party on track for Holyrood success. Either they lack the infrastructure to find candidates for totally cost-free by-elections, in which case they have a fatal lack of capacity, or they are avoiding standing more widely precisely because they know that’ll show up their lack of support, instead cherry-picking a small number of places to focus on where they can work the low turnout of by-elections to their advantage.

I said as much in my preview for the Inverclyde West by-election. That by-election was held on the same day as three in Aberdeenshire. Two of those were in wards that Alba had seen councillors defect to them prior to and then re-stand in 2022. Fraserburgh and District’s Brian Topping was considered by the party their best shot at winning, as they indignantly told me in 2022 when I (a one-person project who had the cheek not to know the ins and outs of all ~1200 sitting councillors and ~2400 candidates standing across Scotland) suggested Inverclyde West was. The third by-election was in Salmond’s own Central Buchan ward, and nominations closed shortly before his death.

Alba contested none of those Aberdeenshire by-elections, but had stood in all in 2022. If they aren’t even standing candidates in their then-leader’s ward (within their second-strongest 2021 constituency) or what they thought was their best 2022 shot (within their strongest 2021 constituency), they simply aren’t a serious party with a serious chance at winning Holyrood seats. Compare with Reform UK, polling well at the moment, who are making a mark in loads of by-elections, very nearly beating Labour in Bannockburn. Compare with the Lib Dems, who feel their long years in the wilderness are ending, and are standing in wards they’ve never previously contested like Kilpatrick or Kilmarnock North.

"You're Only Unhappy Because You're a GREEN!"

Congratulations, Sherlock Holmes

People often point out my historic affiliation to the Greens (I stood for them in 2017 and 2019) as if it’s some deep dark secret I’m desperately trying to hide from the world whilst falsely presenting myself as a neutral figure. It’s on my About page. I said when I stood in 2019. I wrote about having stood in 2017 when looking at what wards might flip in Glasgow in 2022. I tweeted about it when Lana won the seat that had eluded me. I shared an anecdote about having stood in 2017 as part of my preview of last year’s Hillhead by-election.

If I’m trying to hide my past affiliation, I’m doing a very, very poor job of it. As ever, there are two very important points here for folk dubious about those affiliations. First is that I haven’t had any role or involvement in the Greens since before the 2021 election. I completely burnt myself out and couldn’t go back even if I wanted to. Which I don’t. I’ve chosen to focus on Ballot Box Scotland for what time I have available for political activity now.

Secondly, anyone who is into elections enough to run a website about them is going to have personal opinions on the outcome. Taking the existence of personal views as an inherent disqualifier to reporting even-handedly is just silly. If we required all commentators and/or journalists to be a blank slate with no views, we wouldn’t have any commentators or journalists.

Surely I'd Want Those Juicy 13%ers?

More to the point though, the two most recent FindOutNows have had the Greens on 13%. The last time a poll had the Greens that high was an October 2023 Savanta. If I include these polls in the average, the Green share jumps from 9.2% to 10.6%. A 1.4 percentage point difference isn’t a lot in absolute terms, but in relative terms it’s enormous, representing support 15% higher. In 2021 terms it would be equivalent to around 38,000 additional votes compared to what I’m currently estimating. It would take the Greens to their best average in my tracker since November 2023.

If BBS was some kind of deliberate front operation to boost the Greens, by rights I should be gleefully taking a poll that says the Greens are part of a four-way scrap for second place at Holyrood. I should be taking numbers that would put them on a nearly 18-month high. I should be taking the possibility of at least 5 extra Green MSPs.

And yet, I’m not. Because the polls in question are obvious rubbish, and I want things to be accurate, not fit (what you perceive to be) my partisan preference. I couldn’t possibly make a success of this project by acting as a shill for one party, and as much as the most tiresome, idiotic partisans might think that is what I am doing, that claim is obviously laughable when I’m discarding strong Green polls.

Reform are Also Doing Undeniably Well

In addition, if you thought my problem was merely a party you would assume I dislike is doing well, how would you square that with my reporting on Reform results and polling? Whilst I have cautioned that it may be as high as it is right now as part of the inter-election lull where protest voting is most appealing, I’ve also said you’d be a fool to take that as a given. Further, I’ve noted that the difference is unlikely to be “no versus some seats”, but instead “some versus lots”. Reform UK appear to have finally broken a seal in Scotland, and there’s no going back to the old era of us sitting separate from political shocks sweeping England.

The politics of the Greens and Reform UK could not be further apart. Again, if my driving force was a Green bias and an unwillingness to believe other parties are growing, surely I’d be dismissing recent Reform success, rather than suggesting that the only possible dispute is about the scale of that success going forward.

Bad Polling is Everyone's Problem

Hardly Anyone Will See This

I invest a lot of time and effort in Ballot Box Scotland and consider myself to be relatively well known and respected these days, but I have no illusions about just how prominent I am, or how many people will see this. Social media engagement and reach has always been much higher than clickthroughs to the website. I also obviously lack the reach of established media platforms.

I can write comprehensive, well-argued pieces as much as I like, but they’ll never reach as many people as the original headlines. A combination of shy reluctance to put myself out there and pitch ideas to papers, and disinclination to work with parts of a media landscape that have spread or excused lies and hatred against LGBTI+ people like me and my loved ones also means my opportunities to appear in that same media are limited.

And that is a core problem in this case. Many more people are going to see the absolutely nonsense polling than this takedown of it. Even some of those who do see this piece will, for their own nakedly partisan reasons, discount it. Journalists and columnists in papers constantly bleeding cash will get a lot more clicks and attention for taking dramatic polling at face value than you will for finding something less titillating to write about. In Scottish polling as in so much about politics, the space for evidence and reason has been worn away to almost nothing.

Poisoning the Polling Well

We shouldn’t stand for this though. Polling like this, which provides junk data for junk narratives, is poisoning the well of our political discourse. Polling is supposed to illuminate, to give us a sense of what people think, to help prove or disprove theories about who is gaining and losing and why. It’s meant to improve, not worsen, our understanding. 

Scotland already suffers from a dearth of polling. That’s exactly why I find myself so frequently wishing I was getting more from more sources. Even allowing for the fact my tracker doesn’t have every poll in it (due to the exclusion of Redfield and Wilton and now FindOutNow, plus following YouGov’s own advice not to compare their SCOOP polls with normal efforts), we’ve still only averaged about two polls a month since the 2021 election. Recent UK-level polling averaged about five polls a week in January alone.

This creates problems in and of itself. The more frequent UK polling allows a range of averaging options. You can do a rolling monthly average. You can do an average from the most recent poll by different pollsters. You can even do clever weighting system for how much you let different pollsters influence the average. In Scotland, we can’t do any of that. Two polls a month – more around elections, fewer at other times – doesn’t allow for monthly averaging. Averaging between pollsters doesn’t work if it’s been six months to a year since you last heard from some of them. You can’t cleverly weight what doesn’t exist.

That’s why on Ballot Box Scotland most of the time I run with an incredibly crude “last 5 polls” average. It’s the only vaguely workable thing, and it’s deeply imperfect. When elections approach and the frequency of polling picks up, I can and do swing to averaging between pollsters and over periods. That isn’t where we are now though, and it isn’t where we spend most of the term of parliament.

The upshot of this is our polling arena is much more vulnerable to bad data. If it’s the only poll we’ve had in weeks or even months, it sets the entire tone for our national discourse. If we’re not hearing from many pollsters, the house effects of those we are distort the average more without the balancing from opposing effects. Whereas a weird UK-level poll may be counterbalanced by one saying something completely different on the same day or next, in Scotland all we have in such cases is often the weird one.

Stop Commissioning Bad Polling

If you’re a paper or other outlet that commissions polling, I am begging you: stop commissioning bad polling. Presumably, those commissioning FindOutNow think they are getting a bargain. I assume you are. Unfortunately what that’s buying you is cheap rubbish that is damaging to our politics, damaging to your credibility, and damaging to my work here on Ballot Box Scotland.

There are established, seasoned, credible pollsters we haven’t heard a peep from since the election last July. As much as I would love more frequent polling, what I want most is quality polling. If it costs you a little more to pay a proper pollster, then that is a price worth paying. If you want a poll every month but can’t afford that, do one every two months, or quarter.

I hate doing this. I hate having to pop my head above a parapet and go “actually, this is complete and utter bollocks”, because I don’t like being shouted at by nasty partisans who assume bad faith on the part of anyone saying anything they disagree with. I hate having to write pre-rebuttal sections to pushback I’ll get. I hate that instead of just getting to report on polling numbers I’m having to say why I’m not reporting on it. But I’m not willing to let this kind of nonsense pass unchallenged.

If you find this or other Ballot Box Scotland output useful and/or interesting, and you can afford to do so, please consider donating to support my work. I love doing this, but it’s a one-man project and takes a lot of time and effort. All donations, no matter how small, are greatly appreciated and extremely helpful.
(About Donations)