Algorithm bias behind UK exam mess

There’s a nasty example of the algorithm bias can do from the UK.

New Zealand has an algorithm charter which could protect us from similar problems. Although that’s not certain, read on.

Thanks to the Covid-19 pandemic, schools in England and Wales were closed during this year’s exam season. The British school year ends in July and the main exams are held in June.

Students couldn’t sit exams the normal way. Instead the exam authorities set up an assessment system. Like other things these days, this meant going digital and using an algorithm.

The tyranny of a normal statistical spread

The exam regulators made a point of using a system that would give a normal statistical spread of grades. That way they could avoid grade inflation.

It’s important for another reason. In the UK there is stiff competition for the best university places. They go to the students with the best exam results. The entry conditions for certain courses can be strict and tough.

To get exam results, the regulators used an algorithm that combined grades given by teachers with a student’s past performance and the past performance of their school as a whole.

In many cases, as many as 40 percent of the total, the qualifications authorities marked students down, below the grades recommended by teachers.

Take from the poor, give to the rich

There was one huge problem with the exercise. It was skewed towards giving students from the ‘better’ schools a shift up and those from the underperforming schools a penalty.

In the UK the best schools are all in the richer areas. People pay a huge premium to buy a house in a better school zone. Which means the exam results rewarded students from better off families.

The bias was huge. The Guardian newspaper described the algorithms used as “a sociological sorting process which entrenches class divides in the state system”.

’…by building in a criterion of past school performance to this year’s A-level and GCSE results, Ofqual has tied the fortunes of individual students to pre-existing inequalities of outcome.”

Algorithm bias means talent misses out

At first, many less well-off students who expected places at Oxford or Cambridge or, say, medical school missed out.

A-levels are important in the UK, to a degree they determine the next decade of a students’ life. They are more important than New Zealand’s NCEA exams in that sense.

This week the authorities backed down and went back to grading students based on teacher assessments. Which may fix matters, but after a huge amount of stress and upset plans.

New Zealand’s algorithm charter might not stop a similar abuse here but it could help. That’s because it makes algorithm decisions and the logic behind them transparent. The problem with the UK algorithm was less a lack of transparency and more a set of assumptions that are neither fair nor just.

You can hear me talking about this on RNZ Nine-to-Noon with Kathryn Ryan: Exam algo bias, fighting back against the boss snooping on you | RNZ