Data Driven Regulation

by ukcivilservant

I am pleased to be able to publish this guest blog by Paul Hazell.

Regulation tends to only get noticed when it fails. Examples of that are appalling care in a hospital; a plane crashes; a run on a bank; or a poor student experience. But good regulation and effective oversight should mean none of this happens.

Regulation, it has to be said, can be dry and technical. The regulation of higher education, particularly in England, is no different. It’s about conditions, thresholds, risk and data.

How then to make regulation interesting and engaging? This is important if the student experience matters to us and we’re concerned about value for money and quality.

Well, within the higher education sector, regulators have looked at how others do regulation and what we might learn from their experience.1  Research of this sort is typically used to develop policy and expert thinking. But has this brought regulation to life, so that we engage and take notice? Probably not.

Higher education is not just about research, though. It’s about people. And great things tend to happen when bright people come together to learn, develop and share new knowledge. Talking – particularly with people who have different experience and expertise – can bring a dry topic to life. It can help personalise a topic so that we can see how it affects our lives. Talking can help us see something in a new, fresh way.

So that’s what I did. I spoke to two people who really know about regulation, risk and data. Their key message is this: data driven regulation works well if its limitations are recognised and mitigated.

Andrew Dilnott

Andrew was the chairman of the UK Statistics Authority and the founding presenter of Radio 4’s ‘More or less’. He’s now the Warden of Nuffield College, Oxford.

The key risk of data driven regimes, Andrew said, is the gaming of metrics and data.

To mitigate this risk, pick five pieces of data at random but don’t tell the regulated what they are. Then report on their performance against that data set. Next, after a few years, change the audited data but don’t tell your stakeholders what the new data are. This approach can help mitigate some of the risks – such as Goodhart’s law – associated with targets.

Related to this point, data driven regimes need high quality data. This is vital. Big and prohibitive penalties for poor data can help assure the quality of data regulators receive.

While there are no magic bullets with data driven regulation, when done right there are real benefits. These include accountability; identifying and improving weak institutions; public information to inform choice; and improving quality across the board.

Martin Stanley

Martin Stanley was a senior civil servant and former Chief Executive of the Competition Commission and the Better Regulation Executive.  He now edits Understanding Government.

Martin said that data is an excellent regulatory tool, but we need to be aware of and work with its limitations. A good understanding of risks, and a fair degree of regulatory wisdom, can help isolate the signal from the noise in the data.

Regulators also need investigatory powers. It’s important to check, verify and audit what’s been said or reported.

Regulators should also keep their ear to the ground, listen to what’s being said and what their industry’s chatter is. This should be a key part of the regulatory mix: this sort of intelligence can provide early warnings about non-compliance in providers, or wider sectoral issues.

And going native is a danger for any regulator. Warning signs – often used in co-regulatory regimes like higher education – include over long and technical consultation processes and giving notice of inspection visits.

Summing up

Used with care and thought, data and regulation work very well together. But, to quote the Quality Assurance Agency’s research, data ‘can distort if not applied carefully; education is a complex activity which places some limitations on the use of such technologies.’[1]

In this context, checks and balances in the regulatory framework for higher education can help mitigate the risks associated with wholly data driven regimes. These include:

  • the Office for Student’s data assurance regime[2]
  • a breach of the regulatory framework for poor quality data returns[3] and initial and ongoing conditions of registration[4]
  • powers of entry and search
  • data supporting (not replacing) qualitative assessment in the Teaching Excellence Framework.[5]

Data is a fantastic resource when used and interpreted with care. But behind the data and the regulatory theory there are students and their many and varied experiences. This is important. Because it’s students – not numbers – that count.

Paul Hazell
Email  –




[3] See para 54