Geoff Wake, Professor of Mathematics, University of Nottingham
In the last few years, I’ve been in more than a handful of meetings where randomised controlled trials (RCTs) have been referred to as “the gold standard” of research. I am not going to argue that here. I believe that such a claim is flawed. Having said that, much of the research I have led for a number of years now has been exactly of that nature. “Why is this?” you might ask.
First, let’s consider why, and how, this claim of RCTs as being “gold standard” came about in the context of education.
RCTs are primarily associated with medical research where they are used extensively to explore the efficacy of drugs, vaccines and such medical interventions. It is often assumed that this is where they originated. In fact, the first medical trial of 1946 was predated by trials in education that were carried out almost half a century earlier (Torgersen and Torgersen, 2001). However, the prevalence of the use RCTs in educational research declined at the same time as their use in medical research increased.
Eventually a call for the greater use of RCTs in educational research came from a medical researcher, Dr Ben Goldacre. He was co-author of a report ‘published by the Behavioural Insights Team that argued that the impact of government policies should be assessed by RCTs. (Reported in the Guardian here, and the report here) in 2012. This call for the use of RCTs in educational research was taken up by the newly formed Education Endowment Foundation that, with a large founding grant from the government, put in place a programme of trials, and other evaluations, that substantially used RCTs to research interventions that sought to improve student learning. This provided much impetus to increase the use of RCTs once more in educational research – in England.
RCTs came to much prominence with the public in the first year of the Covid-19 pandemic in the context of vaccine development and trials. At that time our research team in maths education was working as research partner in the Centres for Excellence in Mathematics (CfEM) programme. We were working to investigate the potential of interventions using four different approaches to teaching GCSE resits: mathematics focused on motivation and engagement, using context, using technology and a mastery approach. Eventually we combined aspects of each to design an intervention that focused on mastery approaches, that we researched using an RCT. You can find out more about the parallels between this RCT and drug trials that I wrote at the time here.
Let’s return to the “Gold standard” claim. I was in a seminar discussing the design of RCTs in education some years back now. As the discussion carried on it turned out that there were medical researchers in the room and they asked why we would carry out a RCT in an educational setting. They pointed to the complexity of such situations and given that they will be affected by so many factors such as teacher variability, lengths of lessons, composition of classes, resources and so on…They pointed out that in medicine RCTs focus primarily on the use of drugs and vaccines not on surgery which is much more analogous to classroom teaching: situations where many other factors can impact on outcomes. Of course, it is the randomisation process that seeks to equalise the two or three arms in any trial. The assumption that is made is that the one difference between the different messy populations is the planned intervention. That’s why we need to research with substantial numbers of teachers, students, and their different settings.
Our own trial, exploring a particular approach to teaching GCSE Maths points to why RCTs whilst important aren’t a “gold standard”. Perhaps quite unusually our trial was three-armed. Some teachers who signed up were assigned to take part in the complete intervention, being involved in professional development (PD) sessions, working with exemplar lessons, and taking part in five “lesson studies”, some were involved in everything bar the lesson studies and some went about their teaching as usual. We found that the students of teachers in the full intervention did better than those in the other groups. You can find out the full details here. Indeed, the students from the most deprived backgrounds taught by these teachers achieved marks in their GCSE resit equivalent to having been taught for an extra two months.
Here is the crux of the gold standard issue. The RCT provides some evidence of what makes a difference – and this is worth exploring further. Ideally the outcome can be confirmed at scale. This is a next step. Our group has funding from the Education Foundation for a further RCT – an effectiveness trial to add to the evidence base. The evaluation will also provide parallel research that explores the implementation of the approach giving an opportunity to investigate why the intervention was most effective with the students from the most deprived backgrounds.
Although an RCT can provide us with some sense of the effectiveness of an intervention that is carried out in a particular way, at a particular time their importance lies in the partial evidence that they bring to provide us insights into what may work. There is no one “gold standard” in terms of methodology in educational research. To have evidence of how to improve learning outcomes we need parallel research that explores the what, how and why of teaching and learning in classrooms.
RCTs might be considered necessary but not sufficient. Our own next step is to explore if the Mastering Maths success can be replicated at scale. If you can, help us, and potentially your students, by signing up to be part of the biggest RCT that has ever been conducted in the context GCSE resits. Find out how to do this here.
________
Torgerson, C., & Torgerson, D. (2001). The need for randomised controlled trials in educational research. British Journal of Educational Studies, 49(3), 316 – 328
Geoff Wake is a Professor of Mathematics Education based in the Observatory for Mathematical Education at the University of Nottingham. He leads the Mastering Maths research study and is more widely concerned with ensuring more students connect with more mathematics more successfully.
GCSE Resits Hub Project 