The government has introduced the Teaching Excellence Framework (TEF) which purports to assess the excellence or otherwise of teaching in English universities. Surrey was awarded the highest score, a gold, in 2017*. But measuring teaching is hard, it is subjective, and so mostly what the TEF measures is statistics about a university, plus a text summary. There is no actual observation, let alone direct assessment of, teaching, in the TEF. The august body, the Royal Statistics Society (RSS), has just issued a critique of the TEF. The critique reads like the feedback on a piece of statistics coursework submitted by an unusually weak student.
I particularly like the part where the RSS very helpfully point out that Wikipedia has pages TEF compilers can read to learn about the stats errors they are making:
“Wikipedia has entries on this ‘look-elsewhere’ effect and some examples of where this
misuse has caused erroneous scientific conclusions. The ‘Data dredging’ page should
also be consulted, which refers to this kind of incorrect analysis as “the misuse of data
analysis to find patterns in the data that can be presented as statistically significant when in fact there is no real underlying effect”. ”
This continues for six pages. As an academic I do a lot of marking, and unfortunately although many students do excellent stats analysis on their data, many make some of the errors the RSS identifies in the TEF. So I enjoyed reading the critique. I recognised a lot of what the RSS is trying to do. It is clearly identifying serious problems, saying why they are problems, and giving clear feedback on how to do better. I can learn from the RSS on how to give strong feedback.
But there is a serious point here. It is one thing for a student to submit a report with basic stats errors, for which they will be marked down, it is another thing for a government body to make these same mistakes. In 2018 over 500,000 prospective students applied to UK universities, and some may have relied on TEF scores. It is clear that TEF needs significant improvements in methodology, to avoid potentially misleading about 500,000 people a year.
* Of course this post is just my personal opinion, not that of my employer, and I was not involved in TEF submission or assessment.
Chance & Necessity 