Every year The Guardian publishes a new league table of universities. This year St Andrews is deemed the best university in the UK. There are so many problems with these league tables I am not sure where to begin*, and prospective students and their parents use them to inform life-changing decisions, so this matters. As an ex-admissions tutor, this bothers me, so most years when the new league table comes out, I write a polite letter to The Guardian.
The methodology is so bad I can complain about something different every year. This year I wrote to the readers’ editor and as I was worried a bit about a technical complaint being hard for the readers’ editor (who is unlikely to be a scientist), for my email I went for the unfairness of excluding some institutions from the ranking. The Guardian‘s ranking excludes specialist institutes. They do state this but only in a paragraph about 40 paragraphs down in their method description. I don’t think this is best practice, few will read the methods webpage and fewer still will go past paragraph 40.
This year the readers’ editor wrote back, which is nice, they must get a lot of correspondence. Perhaps rashly, they even suggested that if I had technical queries, that I could write back. So I did. I think the odds of this changing minds is low, so to stop my efforts being completely wasted, the letter is below:
Dear Kathryn (and those who compile The Guardian university league tables),
Thank you for your reply. I appreciate it. Regarding the absence of Hartpury, I think explaining this in a separate methodology is not very transparent and so not best practice but OK you and I will have to agree to differ here, I think.
You also ask me to “raise any technical queries”. Thank you for that opportunity. I think the short answer here is that The Guardian should compare its methodology to that described in any textbook or other resource on quantitative data analysis, and as taught in universities from Aberbeen to Sydney, and note the many many differences.
But when I teach this material to students I typically use a simple example as I think this is the clearest way to explain what can be hard to follow. So I will take you through a simple example – which to keep it simple is not that used by The Guardian league tables.
Let us say, for example, you are a student choosing whether to study Applied Psychology, and you are torn between the universities of Gloucestershire and Chichester. The NSS has data (https://www.officeforstudents.org.uk/data-and-analysis/national-student-survey-data/download-the-nss-data/) on both these courses. As an example let us look at the first question from the 2023 survey, this question is: “Q01: How good are teaching staff at explaining things?”. The 2023 NSS data is
U Gloucestershire: 54 responses from students, “positivity measure” = 93% with standard deviation 4%
U Chichester: 13 responses from students, “positivity measure” = 89% with standard deviation 7%
The Guardian league tables ignore uncertainty estimates, here the standard deviations reported by the NSS in the data The Guardian uses. Data analysis 101 is to use uncertainty estimates as almost all numbers (eg those from the NSS) have uncertainties. So it is simply an egregious error by The Guardian to drop the uncertainty estimates provided with the NSS data. If you don’t believe me please look at any reputable reference on data analysis, which will emphasise the importance of uncertainty estimation when making decisions on the basis of quantitative data.
It is an egregious error because it results in erroneous judgements. If the uncertainty estimates are neglected then a league table based on this question will rank the University of Gloucestershire’s course higher than that of Cheshire (93% vs 89%), even though the uncertainties overlap. This is not right.
What the NSS data is telling us, is that the percentage of students positive at Gloucestershire is very likely in the range of values 93-7=86% to 93+7=100%, i.e., is likely to be in the range of 86% to 100%. While at Chichester, the percentage positive is likely to be in the range 82 to 96%. These two ranges (86 to 100 and 82 to 92%) overlap, which means (in accordance with basic data analysis) that you cannot conclude with any confidence that students on the Gloucestershire course are more positive about their course then those at Cheshire.
To recap: The Guardian’s methodology ignores uncertainties in the NSS data, this misleads prospective students by spuriously differentiating between courses. This misleading of prospective students is due to The Guardian ignoring the basic rules of data analysis.
Three final points:
1. If anyone at The Guardian is tempted to make a reply along the lines of “we use a number of different metrics …” then before they reply I would very much appreciate them looking into what the science of data analysis tells us about what happens when multiple highly-correlated metrics are combined.
2. The Guardian recently published an article by Giorgio Parisi (https://www.theguardian.com/commentisfree/2023/sep/25/tiktok-global-crisis-world-trust-scientists-online-attack) which looked at problems caused by lack of trust in scientists. I am a scientist, perhaps you should trust me when I explain basic principles of data analysis to you.
3. Analysing data competently is hard. And estimating uncertainties is the hardest part. I teach physics undergraduates and most struggle with it. Those compiling the league tables are welcome to either look at books, web resources on data analysis – please just ask and I can provide recommendations – or enquire of a university of The Guardian’s choice if it will provide a bespoke continuing professional development (CPD) course.
Thanks again for the correspondence, Richard
letter to the readers’ editor of The Guardian, 30th October 2023
I hope this simple example makes clear one of the problems with subject league tables in particular. As is very clear from the NSS (National Student Survey) data, due to the small number of responses there are large uncertainties in how the cohort of students on a course, feel about the course**, and in many cases this means that ranking one subject in one university above another is not possible, with any reliability.
This is, as far as the science of data analysis goes, the truth. And I trust the well established methods that lead to this conclusion. But unless influential media organisations such as The Guardian, abandon cargo-cult science for the real thing, prospective students and their parents, will continue to be misled.
* The (too numerous for a single blog post) problems are why I think it is fair to call league tables cargo cult science. On the surface they do the things that scientists do, gather data and put them in spreadsheets*** and analyse them to derive results. Indeed the methods webpage is very detailed****. But the analysis is just not done competently, so the result is analysis that looks scientific at a first glance but when you look at the details, falls apart.
** These uncertainties are a very standard thing in surveys, see the Wikipedia page, and, for example, the ONS does this (as it should, unlike The Guardian the ONS does analysis competently).
*** Good data scientists use R, Python Jupyter notebooks etc not spreadsheets. But is (sadly) true that many scientists do use Excel spreadsheets to analyse data.
**** The complexity of the methodology, and the fact that it changes from year to year, are yet more terrible features of the analysis. Good science data analysis should be as simple as possible, whereas in cargo cult science complexity can be useful and as it can act to hide the terrible underlying assumptions.
Chance & Necessity 
1 Comment