School league tables (or, as the Department for Education calls them, School Performance Tables) were published last month, to much complaint from some private schools. Schools such as Eton, Harrow and Winchester scored 0% because they no longer enter their students for GCSEs, preferring International GCSEs, which are thought to be more challenging and therefore better suited to more able pupils.
I’ve long been troubled by league tables, though for a different reason: an A grade is not the same as an A grade.
League tables are based on grades obtained in public examinations. I’m simplifying here, but basically you add up the number of A grades obtained in each school, divide by the number of students in that school and you get an average. Good schools have high averages, bad schools hve low averages. Pretty uncontroversial, surely?
I argue that pretty much every step of the process is flawed. For example, adding up the number of A grades. This is only meaningful if all A grades are the same. Does an A grade in Maths mean the same thing as an A grade in Theatre Studies? (If so, what does it mean?)
Let’s try an easier question. Does an A grade in Maths mean the same thing as an A grade in Maths? That's a ridiculous question, surely? Well, no. Different schools use different exam boards for their maths exams. Can we be certain that A grades given by different exam boards mean the same thing? (OCR's exams have always struck me as a much harder that Edexcel’s, for example.)
Let’s narrow it down further. Does an A grade in Edexcel’s Maths mean the same thing as an A grade in Edexcel’s Maths? Not necessarily. Students can choose which modules they sit. Are the exams on the Statistics modules directly comparable to those on the Mechanics modules? (Edexcel’s module in Decision Maths is often seen as markedly easier than the other modules, despite counting equally for the final grade.)
Hmm. OK, then. Does an A grade in Edexcel’s Maths mean the same thing as an A grade in Edexcel’s Maths where the modules taken are the same? Not if they’re not taken at the same time. Can we be certain that the C3 exam and the marks obtained in it by candidates are consistent from one exam sitting to another? (Edexcel’s C3 exam in June 2013 was an internet sensation within hours of the end of the exam because it was considered unusually difficult. Edexcel responded by dropping the grade boundaries quite markedly. How precise is that process? How precise could it possibly be?)
Surely I’ll concede that if two students sit the same maths modules set by the same exam board at the same time and both get A grades, then those two A grades are equal?
Nope. An A grade requires an average of 80 marks per module. Or more. One candidate could have got an average of 80. The second could have got an average of 90.
Which makes the second one better? Not necessarily. Maybe the second one got very high marks on the easier modules which boosted his average. C1 is the simplest module, but it counts equally. Very high marks in C1 can make up for low marks in, say, C3. Maybe the second student was sitting some of the modules for the second time, having tried them a year earlier and not done so well.
I think it’s perfectly possible that a student with a B grade is meaningfully better at maths than a student with an A grade. Yet the A grade student will be off to a top-ranked university, and the B grade student will have to settle for his second choice.
But now I have fallen into the league table trap. Top-ranked university. What does that mean? If you can’t even compare A grades in the same subject and be sure that you’re making a meaningful, consistent judgement, how can you compare entire universities and say that some are ‘better’ than others?
I'll bet there are some lecturers at London Metropolitan (ranked bottom of the Guardian’s table) who are better than some lecturers at Cambridge. (Uh oh. Better. What does that mean?) Stephen Hawking was a professor at Cambridge: that didn’t mean you’d be certain to be taught by him or even that you’d ever see him at all. And just because he’s incredibly clever doesn’t mean he’s an incredible teacher. I know I’m not the only person who gave up on A Brief History of Time well before the final chapter.
Malcolm Gladwell agrees with me. He wrote an excellent piece for the New Yorker on the subject of ranking colleges in the USA.