Data science gone bad

I am reading, and enjoying, Weapons of Math Destruction by Cathy O’Neil. It is on the problems created for society by the use of algorithms and data. Data analysis is key to science and engineering, and so lies behind new medicines, faster computers etc, but like most powerful tools it is not guaranteed to always lead to good. It can create problems or make existing problems worse if used incompetently and/or in a way that benefits a few while hurting many.

O’Neil’s first example is the financial crash of eight years ago, a familiar example. But as someone working at a university, I was struck by her second example: university league tables.

The book has a particularly striking example of how university league tables can be gamed. You can see the result here, which shows the 2015 world league tables for mathematics departments. King Abdulaziz University in Saudi Arabia has steamed past somewhat more established universities such as Oxford and Cambridge to be officially the 6th best university in the world. This is only three years after it started offering math PhDs.

At first sight this is a bit puzzling as King Abdulaziz University  is not even the best known university in Saudi Arabia, that honour belongs to King Abdullah University of Science and Technology. Should Britain’s smartest maths whizzes cancel their applications to Oxford or Cambridge, and see if they can study in Saudi Arabia?

Probably not, what the King Abdulaziz University has done is simple. They realised that a big factor in league table positions is citations to papers by academics at the university. The league tables are largely based on research reputation and one way to measure this is to look at how many times work from a department is cited by other researchers.

Then they went off and bought citations. The rules for affiliations for authors on papers allow a second affiliation on top of an academic’s primary institution. So, they just offered prominent mathematicians in other universities $72,000 a year to put King Abdulaziz University as their second institution (and spend 3 weeks at the university). The letter and more details are here. The result is a more-or-less instant large flow of citations, which has caused the mathematics department to zoom up the ratings.

This is all entertaining stuff, and in some sense it does not show anything new. League tables are based on metrics, metrics can always be gamed, and gaming them is easy if you have money to spend. This is obvious. But in the USA and the UK, millions of prospective students and their parents are basing life-changing decisions on these tables, and the education of students at university is being affected by the need of universities to perform well in these league tables. Better data analysis would help these students.