Some Principles of Small-Scale Research Design.

The weaknesses of randomised controlled trials are described in several postings, beginning here. These large-scale studies are outside the scope of teachers, and indeed of individual university researchers, even at doctoral level. Thses notes, which take account of the experience of reading researchers over several decades, are intended to help colleagues plan practical and effective approaches to research in their own context I welcome correspondence, and am happy to help pro bono. I am a Fellow of the Chartered College of Teaching. 

1. The aim is not to establish absolute truth using the methods of the physical sciences, but to apply the principles of John Locke to establish the best available evidence, and where possible to answer the classic teacher's question "What can I do on Monday morning?" though on a longer time-scale. The closest medical parallel is not the randomised controlled trial, but the development of surgery, which has been based on individual observations, including the papers published in The Lancet by Joseph Lister in 1867. It is also impossible to have a blind trial in an educational context, as the teacher needs to know what he or she is doing. There is no scientific basis for a partially-sighted trial, and many recent oncology trials are open, rather than blind. The idea of a single, gold-standard research method in education is supported by assertion rather than evidence, and is an illusion.
2. The limitations of each study need to be identified as part of the planning, in something akin to a risk assessment. Some issues, such as follow-up to ensure that results are sustained and not washed out, need to be considered in all studies. Others reflect the methodology and goals of individual studies. There is no single correct answer.
3. Sampling may be:

• A class taught by a teacher, compared with a similar class taught by the same teacher, either in the same or a previous year. Any differences between the classes should be noted and taken into account in evaluation. If there is a need to compare classes taught by different teachers, this should be carefully considered, as other factors than the issue under investigation may be very important, and indeed invalidate results.
• A year group, compared with earlier year groups. Again, differences in teaching staff should be noted.
• A small group, compared with a similar small group given a different form of additional provision, and a third given no additional provision. Differences in teachers should be noted, and more than one experimental group used if possible.
• A case study. Comparison in this case may be to pupils with similar patterns of learning, or to the same pupil's earlier work.

4. Timing needs to be long enough to allow teaching to have the desired effect. I suggest an interim review after half a term, and a term and a year as likely to give a useful result.  Follow-up should be planned after one, two and five years.  This allows us to detect both short-term washout, and long-term benefits.
5. Assessment needs a clear measurement of starting and finishing points. It may take any form appropriate to the issue investigated. Standardised tests are not available in the field of languages, and existing frameworks may need to be defined more precisely for research purposes. It is therefore essential that assessment should be a key part of research planning, and that it should be described and explained as part of the research report. Assessment needs to be carried out in "clean" conditions - pupils must not have access to extraneous material from which they might copy or which might give them hints that would not be available to other pupils in the study.

An approach to assessing writing that I have used is to have pupils write all they can, from memory, words, phrases or sentences, in five minutes in a clean room, before and after. The work can then be scored on the basis, of words recognisable, words accurate, meaningful groups and sentences.

For listening, the system used in the British Picture Vocabulary Scale could be applied directly in French. The pupil has to pick  the correct picture from four, and the test stops after six errors. It may be possible to use the original pictures, or we might need new ones.

Speaking could involve responses to set, open questions, presented in context and related to the work the pupils have been doing. I have not yet tried to assess this.

Reading assessment is perhaps better established, and could involve picking correct responses in English (or the new language - discuss!) from a range.

Note: Problems that have arisen in the field of reading research include:

• short-term gains that are not sustained
• no follow-up, or inconsistent follow-up, of results beyond an intervention period
• unfair comparisons (eg by comparing the effect of extra provision solely against the work of pupils who have had no extra help)
• biased samples and poorly-matched control or comparison groups (eg gender-imbalance, unequal starting points)
• intervention periods that are too short to show any results
• samples that are too small to show statistical gains, but lack the detail to show qualitative ones ( Very small samples require an unrealistically large effect size to reach statistical significance.)
• assessment systems that do not test gains fairly, or that over-generalise benefits
• statistical manipulation in such samples that disguises the effect of the results of one or two pupils
• descriptions of teaching methods that are too vague to be used as evidence
• grouping together the results of large numbers of studies, in the hope that this compensates for flaws rather than accumulating them, then presenting the results as authoritative
• variations in the quality of teaching, either in areas studied, or in other areas (eg class management) that are not accounted for
• statements of principle that are not substantiated by evidence and are not capable of investigation
• leaving negative findings unpublished
• bias in editorial panels and peer review

In foreign language research it is also unfair to compare

• pupils learning English as a second language with that from learners of other languages, as English is so widely available to the first group, who also have stronger economic motives
• adult learners with children

and

• evidence from people learning a second language as residents in the country speaking it with evidence from those learning a language in school only.

Contact.

[email protected] 

References will be added prior to publication.