Summary
- Policy organisations play a large role informing the public debate and sharping the policy agenda. While producing timely and informative research can be insightful, the focus on speed and impact can come at the expense of quality. This is especially true in policy organisation as this research is unlikely to have undergone the same scientific scrutiny as research published in academic journals
- The lack of substantive scrutiny can lead to policy vrecommendations that are entirely driven by poor research methods
- To demonstrate this, my recently published paper (Fullard, J. 2023) shows that the results in an influential meta-analysis (Fletcher-Wood, H. and J. Zuccollo 2020), performed by a leading policy organisation (Education Policy Institute), is entirely driven by poor research methods
Methodology
I attempt to replicate an influential piece of policy research. The research selected is a recent metaanalysis investigating the effect of teacher professional development on pupil outcomes.
This work is cited as evidence for the effectiveness of teacher professional development (PD) on pupil outcomes in government reports (DfE 2020, Ofsted 2023) and academic papers.
It is used to inform a cost-benefit analysis that finds an increase in PD will increase each student’s lifetime earnings by over £6,000 and a net societal benefit of over £61bn (Van den Brande, J. and Zuccollo, J. 2021).
Key findings
I demonstrate that the originally found positive effect is entirely driven by poor research methods:
- First, of the 49 estimates (from 42 studies) included only 14 (from 10 studies) are valid estimates of the effect of professional development on pupil outcomes. Many of the studies, such as a Randomised Control Trial (RCT) without a control group, should be easily identified as invalid
- Second, the authors selection criteria ‘when multiple outcome measure were supplied, we chose the first listed’ introduces upward bias into the meta-analysis as the positive, statistically significant effects, are more likely to be reported first
- Third, the authors, without justification, exclude a valid study from the meta-analysis
- When the invalid studies are removed, the bias in the selection criteria adjusted and valid study included the effect size falls from 0.09 to −0.008
Author’s main message
My research highlights the importance of maintaining research standards by demonstrating that the claims in an influential meta-analysis are entirely driven by poor research methods and the subsequent policy conclusions recommendations (i.e., an increase in PD will increase each student’s lifetime earnings by over £6,000 and a net societal benefit of over £61bn over ten years) are unlikely to be true.
In a time when departmental budgets are becoming increasingly stretched, it is essential that the research community maintains research standards to ensure that policy recommendations are driven by robust research.
This is particularly true in policy organisations as it is not clear if the research in policy settings have undergone the same scientific scrutiny as research published in academic journals. With that in mind the research community needs to have a serious debate about what measures can be put in place to maintain research standards in these organisations. A requirement for policy organisations to make their data and code publicly available for replication purposes as well as having a substantive independent peer review system in place to check research, before it influences the policy debate, seem like a good place to start.
Another conclusion from my paper is that a general improvement in empirical methods in education research is necessary to help researchers design more robust experiments. Roughly 25 percent of the RCTs reviewed in Fletcher-Wood, H. and J. Zuccollo (2020) have statistically significant differences on key covariates – this means that the control group cannot be used to estimate what would have happened to the treatment group, had they not received the treatment. This suggests poor design. In a large RCT, where data in schools, teachers and students is collected before the intervention(s) takes place, researchers should check if there are any differences between groups at baseline and, if necessary, rerandomise (Bruhn, M. and D. McKenzie 2009).
Additionally, experimenter demand effects (EDE) could also bias the estimates in this context and the studies reviewed do very little to mitigate these potential sources of bias (e.g., a placebo treatment arm). For example, teachers know they are taking part in a study, and this might change their behaviour. This is problematic because EDE are likely to be stronger in the treatment group (those who are more actively involved in the study) than the control group (business as usual). In this setting EDE are likely to exist and produce upward bias. Without an appropriately designed experiment it is challenging to identify this.
A general improvement in empirical methods in education research is necessary to help researchers a) design more robust experiments b) evaluate the quality of existing experiments.
Dr Joshua Fullard
Assistant Professor, Warwick Business School, University of Warwick, and Research Associate at the Research Centre on Micro-Social Change
joshua.fullard@wbs.ac.uk
References
Bruhn, M. and D. Mckenzie (2009): ‘In Pursuit of Balance: Randomization in Practice in Development Field Experiments’, American Economic Journal: Applied Economics, 1, 200-232
DFE (2020): National Professional Qualification (NPQ): Leading Teaching Framework
Fletcher-Wood, H. and J. Zuccollo (2020): The Effects of High-Quality Professional Development on Teachers and
Students: A Rapid Review and Meta-Analysis, Education Policy Institute
Fullard, J. (2023): ‘Invalid Estimates and Biased Means. A Replication of a Recent Meta-Analysis Investigating
the Effect of Teacher Professional Development on Pupil Outcomes”, Social Sciences & Humanities Open, 8, 100605
Ofsted (2023): Independent Review of Teachers’ Professional Development in Schools: Phase 1 Findings
Van Den Brande, J. and J. Zuccollo (2021): The Effects of High-Quality Professional Development on Teachers and
Students: A Cost-Benefit Analysis, Education Policy Institute
© MiSoC November 2023
DOI: 10.5526/misoc-2023-008
Explainer Download