Higher Education and Research Bill

Lord Willetts Excerpts

Committee: 5th sitting (Hansard): House of Lords
Monday 23rd January 2017 Lords Chamber

The NSS in the TEF is using—or rather, abusing—statistics for a purpose for which the NSS was never designed. My amendments are designed to reduce that risk for good colleges with good teaching that are in danger of falling foul of a statistical lottery. I beg to move.

[Lord Willetts (Con)]

My Lords, I rise to briefly comment on the interesting and important observations we have just heard from the noble Lord, Lord Lipsey. I completely support his commitment to using statistics with integrity. There are issues about the NSS. I would argue—as I did in Committee last week—that the NSS itself is changing and increasingly has genuine questions about student engagement and academic experience. For example, I know from speaking to many vice-chancellors that how their university does on the metric of academic feedback is something they pay a lot of attention to; it reflects genuine concern among students sometimes when they do not get essays back in time and they do not get prompt feedback.

I would like, however, briefly to comment on the noble Lord’s specific point as to whether the use of the NSS, as proposed in the TEF, meets the required standards. He briefly gave a quote from the ONS on its views, saying that it would not be right to use the raw NSS data. I would like to assure him that, to my understanding, the TEF does not use raw NSS data. Using raw data simply means taking all the universities and seeing how they stand. Instead, the way in which the TEF is being constructed is to benchmark universities against similar universities. Using his own example of students from ethnic minorities, it would be possible to compare groups of universities that all have roughly similar proportions of students from ethnic minorities, so the data that will be used are not raw data. Universities will find themselves being assessed and compared with a peer group. That itself, interestingly, raises a new set of questions, but at least it means that the TEF is not exposed to the charge which the noble Lord, Lord Lipsey, has levelled this afternoon.

[Baroness Blackstone (Lab)]

My Lords, I find myself in agreement with the noble Lord. There is a slight danger that this will become a club of former higher education Ministers. However, as a vice-chancellor and former Minister, I found that the National Student Survey was a rather useful device—in a rather broad-brush way, admittedly—for telling us something about what students perceive about their own experience as undergraduates. It is not done for graduate students. I am somewhat at loggerheads with my noble friend Lord Lipsey, and I am sorry about this because normally we agree on many things. I would say that a 70% response rate that—if I understand correctly—my noble friend was quoting to be unacceptable, is a rather high response rate in most surveys of this kind. It is sometimes possible to do deep dives and find out a bit more about the group that had not responded to see whether they are in any way different in their views or backgrounds. I had not read the critique that he quotes by the ONS and the RSS. It is important that the Minister comes back and tells us whether the Government have looked at those criticisms. If not, why not, and will they in future?

I have a lot of concerns about the TEF and how it should be done. The Government are taking on a very difficult and complex task. I am not sure whether they realise how difficult it is to get reliability and validity in the responses provided. I look forward to hearing what Professor Chris Husbands, who has a lot of expertise on this, will say. I would also like to hear his response to the criticisms and comments of the ONS and RSS.

We cannot entirely take out and ignore what the NSS tells us about students’ experience. There is only a small number of questions about teaching, but there are some. There are many other questions about things that are relevant to the successful completion of their courses, including how they are assessed and examined. I hope we can look at this in a bit more depth and not completely rule out the contribution that a rethought NSS can make to any assessments of how our universities, and departments within them, are teaching, and whether it meets the kind of quality that we expect it to meet.

--- Later in debate ---

[Baroness Wolf of Dulwich]

My Lords, I support the comments of the noble Lord, Lord Lipsey, on the National Student Survey, and will speak to Amendments 194 and 201 standing in my name. Before doing so I would like to underline that we are talking about the use of measures to give ratings. With respect to the comments made by the noble Baroness, Lady Blackstone, I think that there is a huge difference between what is useful internally and what is suitable for a high-profile, high-stakes national rating system. In my first amendment I have suggested, or requested, that any measures used should be criteria-referenced, and therefore provide a substantive rating and indication of attainment or degree of attainment. I am slightly alarmed that this is even at issue, and take issue with the noble Lord, Lord Willetts, when he suggests that benchmarking is the way forward.

I have an example from the rail regulator. We can be told what proportion of trains are late, which is a substantive measure: we can have a target—which in fact it has—which says that it is reasonable that there should be X per cent, and then you fall this far short. We can be told whether a given rail company is doing better or worse than the others. This year it is really pretty easy for everybody to do better than Southern, but does that mean that they are all doing well? I do not think that you can conclude that.

If you have benchmarked or relative measures, the problem is that all that you are being told is how people stand relative to each other. We might have a system in which the quality of teaching was excellent across the board, yet in which half the institutions would by definition be below average; or we could have a system in which all the institutions were doing rather poor-quality teaching, yet in which half of them would be above average. That is not the sort of system that we wish to use. We would not wish to imply to students that that gave them helpful information. A measure that is bad does not become good by being made relative; and a measure that is good is good in its own right, not simply by being turned into something in which you rank people on the curve. That is an important aspect of how the Office for Students approaches the sorts of ratings that it gives and the way in which it conceives of them.

[Lord Willetts]

Does the noble Baroness accept that her objection is the opposite of the one raised by the noble Lord, Lord Lipsey? His objection was that these are raw data that cannot be trusted. As a result of that concern, they are being benchmarked, and that indeed raises the valid questions which she has raised.