TEF and the Controlled Reputational Range

One core function of the Teaching Excellence Framework (TEF) is to celebrate teaching in our universities and other providers.  The publication of the results of the second round have been widely discussed in the sector, with some inevitable media coverage of what it means to be given the ‘silver’ or ‘bronze’ levels, particularly whether those providers are second or third rate.  But, for all my concerns about TEF, I wonder if there is a message that the sector should take away from this process – that it confirms the continued existence of the Controlled Reputational Range.

One of the concepts that Sir David Watson explored in his writing was that of the Controlled Reputational Range.  This was the notion that in UK Higher Education, although there could not be exact parity between all universities, there was a framework that meant that no institution was able to behave so badly that it damaged the overall reputation of the sector.  He described it in 2002, but returned to it several times.

Watson not only described the range, but he did much to bring it into effect.  He was closely involved in the merger of the two HE sectors in 1992, working on the funding arrangements for the new sector and more prominently on the quality assurance of the new sector.  The 1992 merger, particularly the awarding of university title to all the polytechnics and two colleges, caused much alarm.  Quality assurance was more systematic on the PCFC side; a function of both CNAA and HMI regimes.  The autonomous universities on the UFC side had a better public perception of their quality, but not necessarily their quality assurance.   The regime of Teaching Quality Assurance – which later became Subject Review – was vital in calming down nerves about a unified sector which suddenly looked as if it had jumped from an elite to a mass system.  The former UFC universities did better than might have been expected, generally achieving a higher number of ‘excellent’ judgements, but the PCFC side held up well.  By 2001 it was clear that the mass inspection system had played its role, and it was dropped in favour of Teaching Quality Information (TQI).  You could argue TEF is the apotheosis of TQI.

TEF accentuates the difference between providers with the Gold, Silver and Bronze awards, but the analysis done by both WonkHe and THE shows that these judgements are not based on the metrics alone.  Some Silver providers have ‘better’ outcomes on the benchmarked metrics than those awarded Gold.  Ranking using z-scores accentuates the difference between providers, but the benchmarking closes the gap in performance overall.  I think this shows that the controlled reputational range has worked; clearly universities recruiting highly prepared students have better absolute outcomes, but universities have exceeeded expectations across the reputational range.

Given that higher education has always been wider than those institutions with university title or access to government funding, the TEF does show how the new registration regime with OfS may help to maintain the reputational range.  The write-up for Moorlands College achieving a provisional grade might be slightly over the top, but being in the TEF will be a new marker of quality in the sector.  Those alternative providers with sufficient data to be graded achieved each of the three levels.

The funded part of the HE sector was subject to some very considerable, if sometimes invisible, regulation through their data.  HEFCE rules on, say, completion drove institutional behaviour.  The TEF may have a similar effect.  That is why the review must look at the data chosen for the exercise as it will drive performance – probably even for universities who you might have thought could shrug off a ‘bronze’ award.  Of course, if we could just drop the pretence that the TEF was actually about teaching excellence and abandon the gimmicky grading system (and the further rankings coming from it), then it could be a light touch system of ensuring that providers stay within that controlled reputational range.



TEF: The ultimate degree classification

We are in the season where boards of examiners are meeting to decide the classifications of students. Data on the performance of students is carefully brought together, and a set of rules applied to render three or four years of work into a single classification. There is a parallel with the TEF – a single classification for the teaching – at least the undergraduate teaching – of a whole university. Sadly, many of the same problems apply.

One of the complaints about the degree classification system is that it presents a cliff-edge between grades.  Even ministers have backed the elaborate mechanisms put in place to present the HEAR and GPA alongside the honours classification.    TEF will also have its extra information, but the first focus will be on the three grades: gold, silver and bronze.

TEF Bronze

If any import is attached to these grades then the cliff-edge will really matter. It’s very helpful that ministers have explicitly ruled them out of bounds in the visa system but other uses apart from fee setting will be found in the future.  As with a degree system, the difference between the very lowest of the silver and the very highest of the bronze will be tiny.  Here the sorities paradox must come into play (see my discussion of that in terms of visa rules here).  Universities who excitedly claim they’ve moved 10 places in a league table know that they are only fractionally different from the peers they moved past – but that may take them over a threshold.

The good news (and simultaneously the bad news) is that the TEF will be published with the metrics data, the submission and the panel’s reasons for the judgement.  In my degree classification analogy; not only do we get the degree class, but a transcript (the metrics) and the outcome of the viva (the submission).  When the data is released to both universities and the media, there will be a frantic scramble to discover who has a ‘good’ silver, or who only scraped their gold. More fun, we’ll surely be able to tell whose written submission pulled them up, left them where they were – or conceivably pulled them down (viva voce exams tend not to have the scope of reducing your grade – but the statement might do that).

The TEF gets described as a new ranking – there’s no way the ranking business will let the sector get away with three grades.  All those data will get puled apart to create a GPA for the sector (no doubt with new means of stretching the data).

Now that the fee-level jeopardy has been pushed into the future, some of the jeopardy has been reduced, but we have already seen signs of universities reacting to TEF by changing the things that contribute to the sets of proxy data, not by being better at teaching. It’s easier to get students who will do better in the TEF, than get better outcomes for them –  that’s certainly the context of some comments about Manchester’s redundancy round.  If so, we’ll have the most awful consequences of accepting proxy data for assessing teaching excellence.

I should say that I write this from a Rawlsian veil of ignorance. I don’t know the metrics data of any university, what that might have indicated their award would be, or what they tried to say in their submission.  I know that no rational university ought to be pleased with the situation we will find ourselves in when TEF is published.  Even if they’ve just done enough to get a silver…