One of my main pieces of content here on Ballot Box Scotland, and indeed a unique selling point of the project, is my coverage of Council By-Elections. Using the wealth of information available as a result of machine counts, I go into depths that other polling and election accounts don’t have the time or expertise for. As time has gone on, I’ve perfected my process and added various bells and whistles, and it seemed like it might now be useful to put together an explanation for the whole deal.

The Voting System

Council elections in Scotland are conducted via the Single Transferable Vote (STV). This is a form of (partially) proportional representation, first introduced for elections in 2007 as a result of a deal the Liberal Democrats did with Labour to form a coalition Scottish Executive in 2003.

As first introduced, every Council in Scotland was divided into wards electing 3 or 4 councillors. The Islands Act has since allowed wards containing inhabited islands to have just one or two councillors, whilst another Electoral Reform Act has expanded the range for mainland wards to between 2 and 5 councillors. Councils with inhabited islands will see changes to their wards under these new rules in time for the 2022 elections, whereas for the rest of the country changes won’t come until at least 2027.

To elect those councillors, voters simply mark the candidates 1, 2, 3… etc in order of preference. Candidates have to achieve a “quota” of votes cast in order to be elected, and lower-ranked candidates will be eliminated and their votes transferred until all seats are filled. Candidates who are elected with more votes than they need will also have a portion of those votes transferred. A more detailed explanation of the maths is on the Councils page.

By-elections work on exactly the same basis as a full election, but just for fewer councillors. This is almost always just one councillor. Rarely, there will be a by-election for more than one councillor in a ward. There were two such by-elections in 2015, and one in 2019, which are thus far the only examples of this happening. A single councillor election is functionally equivalent to the Alternative Vote.

There are two things that are important to bear in mind about STV. The first is that your later preferences have no effect on your earlier preferences. A false claim is sometimes made that you should only mark one candidate or one party, or you lower their chances. This isn’t true, as your vote will only transfer and count for someone else if that candidate wins or drops out. The other is that as this is a transferable voting system, it isn’t possible to “split the vote”. If anyone is telling you otherwise, they either don’t understand the system or are deliberately misleading you.

How Council By-Election Previews Work

My preview posts include a huge amount of data about the ward(s) facing a by-election. This is conveyed via maps and charts. Charts are provided in both an interactive Google Chart form which gives the most detail, as well as with a link to an image form which you’ll see posted on social media.

Ward and Polling District Map

These maps show the outline of the ward in question, as well as previous boundaries if there have been changes since 2007. These maps are intended to be clean and simple, using data on buildings and roads to show the key areas in the ward and how they are connected, without the clutter of a full geographic map. 

A small inset map shows the location of the by-election ward relative to the whole council, tinted to the colour of the vacating party. The main map shows the ward divided into polling districts, with each district tinted to the colour of the party that won the most first preference votes there. This tint relates only to in-person votes, as postal votes are not broken down by district, and are instead shown as their own little bubble.

(Note that Lib Dem tint for vacating party couldn’t be real in this ward as they don’t have a councillor – I don’t think there’s a ward where all of the Holyrood 5 led in at least one district, so this is the best I could represent all parties but with otherwise real data!)

Maps including polling districts first started appearing as a feature in October 2020. Maps before then simply showed the ward boundaries, and were tinted to the colour of the vacating party. For by-elections in late 2020, you may notice preview posts have both a basic boundary map and a polling district map. Those posts effectively sit in a transitional period where I’d done a lot of the preparatory work already, and it was too time consuming to create a merged map.

2017 First Preferences

First preferences are nice and easy. These charts simply show the total number of first preference votes by party at the previous election – again, this is either the full election or a more recent by-election in the ward.

Before October 2020, these appeared purely as image versions.

First Preferences by Polling District

This chart shows the percentage vote share for each party (or major Independent candidate) in the most recent full election for the ward – or, where there has since been a by-election which was machine counted, at that by-election. This information is available thanks to those machine counts, which generally count each ballot box individually, allowing them to be paired up with polling districts.

Note however that boxes with relatively small numbers of voters are required to be merged with another box to help preserve the secrecy of the ballot. This can effectively mean that the data for two or more districts has to merge entirely as well, which may disguise different voting patterns across those districts. This is especially true if the previous election was a by-election, as those generally have lower turnout overall.

Starting in February 2021, these appear as interactive Google Charts which allow you to see both the proportion and absolute number of votes per district. For the intervening period between introducing these October 2020 and then, they exist purely in image for, and only showing proportions.

Second Preferences by First Preference Party

This chart shows the breakdown of second preferences based on first preference party. So for example, the first column shows Conservative first preferences, of which 1,167 (37%) went to the Liberal Democrats, whereas 907 (28.8%) voters didn’t mark a preference beyond Conservative candidates. This is data we don’t get just from the actual election transfers, because as soon as the first candidate drops out, we can’t directly see how many second preferences they had from other parties.

These are useful for giving a relative indication of party and candidate favourability between parties. They also clearly burst the assumption made by so many that voters move neatly between parties according to “obvious” patterns, for example that the SNP and Greens form a unified pro-Independence bloc and the Conservatives, Labour and Lib Dems a pro-Union bloc, with votes moving within but not between blocs. Similarly, voters don’t all move obviously along the left-right spectrum.

Starting in February 2021, these appear as interactive Google Charts which allow you to see both the proportion and absolute number of votes per district. Before then, these mostly only appeared in results posts rather than previews, and in entirely image form.

2017 Re-calculation

Charts for transfers are much more complicated, though the same basic format of having an interactive and an expandable image version suitable for social media applies.

As STV at a full election results in multiple councillors, we can’t directly compare results in the traditional “gain-hold-loss” manner people are used to with First Past the Post. A party that won 20% and elected a councillor probably won’t win a by-election if that councillor resigns and there’s another party that won 40%, so though they’ve lost the seat in real terms, it’s not a loss (or gain for the other party) in terms of the electoral maths.

Additionally, because votes transfer, there’s no guarantee the party that had the most first preference votes would win a by-election for a single seat, but we can’t tell that using the transfer flows from a three or four councillor election. However, because these elections are machine counted, there is a detailed record of every ballot and the transfers on them. This can be used to re-calculate the previous election for one (or two) seats with perfect accuracy.

That’s not the same as saying that’s definitely how that election would have turned out with just one councillor, as voter behaviour might have been slightly different. But it is an absolutely spot-on measure of how the election would have been according to the votes actually cast.

Initially I was doing these re-calculations by hand, but in early 2020 I finally cracked a way to automate it, which has reduced the work from two or three hours to about five minutes. At the same time, I also began to eliminate lower ranked candidates from parties that stood multiples at the start of the count process to best approximate by-election conditions. That doesn’t change the end result, just some of the intermediate values.

Prediction Call

From the re-calculation, I report the total number of votes the top two candidates had at the last competitive round of the count, as well as the number of votes that had “exhausted” and didn’t transfer. I then combine that outcome with how current polling is looking, the results of any other elections since or recently, and just a pinch of gut feel to make a call as to how I think it will go. There are four calls that I can make:

  • Tossup – I’m expecting the election to be close or unpredictable, and do not feel comfortable calling it for any one of the parties identified as possible winners.
  • Leans – I’m expecting the election to be close, but reckon that one party has a particular edge. I will not be surprised if another party identified as competitive wins.
  • Likely – I’m expecting the party in question to be the clear frontrunners. I will be surprised if another party identified as competitive wins, but not entirely.
  • Win – I’m expecting the party in question to win the by-election hands down. I will be extremely surprised if another party wins, and will have to pre-heat the oven for the crow I’ll need to eat.

By-Election Results

For the most part, coverage of by-election results are basically the same format as the preview. I provide charts of first preferences and transfers. However, as by-elections are allowed to be hand counted instead of machine counted, my level of analysis beyond that will vary.

If it has been another machine count, and the data has been made available in good time, I’ll usually do all the same analysis as for a preview. If it’s a hand count, you’ll only get the first preferences and transfers, as that’s the only information available to anyone.