I discussed the attempt to move RCTs from health research into education in a technical paper 3 years ago, which focused chiefly on reading research. Since then, we have had one excellent study by Professor Debra Myhill on the value of teaching grammar to young secondary pupils, an a further attempt to introduce the health service model to education from Dr Ben Goldacre, an epidemiologist and columnist for the Guardian (Bad Science). Dr Goldacre's paper, with replies from Professors Mary James and Geoff Whitty, is here.
Dr Goldacre says there is
a huge prize waiting to be claimed by teachers. By collecting better evidence about what works best, and establishing a culture where this evidence is used as a matter of routine, we can improve outcomes for children, and increase professional independence.
He then says
it's only by conducting randomised trials - fair tests - that we've been able to find out what works best.
Well, sometimes. There is also a long and important history in medicine of clinical investigation based on a doctor deciding what to do in individual cases. The papers published in The Lancet by Joseph Lister in 1867 represent one of the greatest advances in medical history, indeed in human history, and were not the result of controlled trials. Clinical experiment remains a key source of evidence in surgery; to suggest otherwise is to present part of medical research as the whole.
Dr Goldacre continues:
Where they are feasible, randomised trials are generally the most reliable tool we have for finding out which of two interventions work best.
This is an important limitation. The trial tests one intervention against another. The other may be doing nothing at all, or providing a different type of teaching. What if we have more than one choice? What if a slight alteration in approach might produce a different answer in the taught control group? What if we can't find a suitable comparison group? And what if our trial is not "blind" as it is with a drugs trial? Teachers need to know what they're doing or they can't do it. The idea of a partially sighted trial is not known to science.
But Dr Goldacre does not discuss these limitations, jumping instead to an example from microfinance. He also says, in a passage intended to debunk myths about trials, that
there are some situations where trials aren’t appropriate - and where
we need to be cautious in interpreting the results
So, where are RCTs appropriate? Dr Goldacre tells us:
Randomisation, in a trial, adds one simple extra chink to this existing
variation: we need a group of schools, teachers, pupils, or parents, who are able
to honestly say: “we don’t know which of these two strategies is best, so we don’t
mind which we use. We want to find out which is best, and we know it won’t
harm us.”
In the case of a drug, this is the right approach. Expensive and promising drugs have been refused licences because they have failed their trials, and others have been shown to be unexpectedlly effective. But teaching a child is not the same as adminstering a drug. There is more than one choice most of the time, and infinite variation in the ways things can be done.
Professor Debra Myhill's important study of contextualised grammar vs no grammar was favourable to contextualised grammar, but did not show us what should be taught, at what age, or why higher-attaining pupils benefited more than others, or evenwhether contextualised grammar was better than decontextualised grammar. This is not Professor Myhill's fault, but a limitation of the research technique - it lets us decide between two clear alternatives, but only that.
Dr Goldacre closes with this:
Now we recognise that being a good doctor, or teacher, or manager, isn’t
about robotically following the numerical output of randomised trials; nor is it
about ignoring the evidence, and following your hunches and personal
experiences instead. We do best, by using the right combination of skills to get
the best job done.
And once again, I agree. But we are not, in education, dealing with a straight choice between statistical evidence and personal experience. Experience in education is moderated by other people's experience, notably that of HMI, just as the personal experience of a surgeon is informed by clinical practice. It is also informed by smaller scale research, that need not be randomised or expensive (Professor Myhill's trial cost £750k), but that can provide indications that can be interpeted using professional judgement and experience. A culture of research in education is vital. RCTs can and should be part of it. They are not the only source of knowledge about what "works best".