Last week, one of our senior consultants, Jo, attended a seminar at the University of Plymouth hosted by the Royal Statistical Society’s South West Local Group. The talk, given by ZhiMin Xiao, of the University of Exeter, discussed the small average effect sizes obtained in randomised controlled trials (RCTs) in education, such as those funded by the Education Endowment Foundation (EEF) in the UK and National Center for Educational Evaluation and Regional Assistance (NCEE) in the US.
The EEF is dedicated to breaking the link between disadvantage and educational achievement, and as such require their evaluators to analyse trial results for the subset of pupils who are eligible for free school meals. In many situations subgroup analyses are viewed as bad practice, as they are prone to being under-powered. Though EEF evaluators are now required to present power calculations for the analysis of this pre-specified subset of pupils.
ZhiMin proposed a new approach using a Pupil Advantage Index to estimate the outcomes for all pupils in the dataset. This approach involves building a model which takes account of all available background information, including free school meal eligibility, along with any interactions. This exploits the heterogeneity of the data, recognising that pupils and education are multifaceted and complex, rather than considering background characteristics individually in subgroup analyses. As an alternative to subgroup analyses, rather than answering the question “does this intervention work, on average, for this subset of pupils?”, the Pupil Advantage Index asks “for what kinds of pupils does this intervention work?”. As well as knowing which pupils benefited most from a particular intervention in a trial, it can also help future decisions by answering the question “which intervention(s) are best suited to my particular set of pupils?”
“This was a really interesting seminar”, said Jo. “It was great to learn of the latest developments in the analysis of education data and to hear about further analyses that are being conducted on the wealth of data collected by the EEF”.