Novel Educational Research using Longitudinal Survey Data

A recent article in the Guardian newspaper reports on a study published last month that looks at parents’ responses to news from the Ofsted inspection of their child’s school. Aside from the eye-catching headline, we found a number of aspects of this study interesting.

The research, published by the Institute for Social and Economic Research used longitudinal survey data collected in Understanding Society, the UK Household Longitudinal Study (UKHLS), which asked parents how often they helped their child with their homework. The study selected data from households where the child’s school had been inspected by Ofsted in the same year that the survey took place. The researchers exploited a natural feature of the survey and of Ofsted inspections; that either could occur at any point throughout the year. Households were divided into two groups; one group where the Ofsted inspection occurred before the survey interview and the other where the Ofsted inspection occurred after the survey interview.

Researchers exploited a natural feature of the survey… [for] random assignment. In education, often randomisation is not possible, is impractical, or sometimes considered unethical.

This meant that one group of parents knew the outcome of the Ofsted inspection while the other group did not. The authors say that “This is as good as random assignment.” Random assignment is rare in studies which analyse education data, as well as in many other areas of social science. Most studies are based on observational data, often because randomisation is not possible, is impractical, or sometimes considered unethical. Random assignment is preferred over observational data as it avoids underlying differences between the groups, in both observed and unobserved characteristics, which might bias the results.

The researchers linked data published by Ofsted with data published by the DfE and used a statistical model to predict the likely outcome of Ofsted inspections given schools’ performance data and background characteristics. The results of this model were used to identify households where the Ofsted inspection was better than would be expected, and parents received good news, and other households where the Ofsted inspection was worse than expected. At Select, we regularly combine our clients’ data with publicly available data in this way. For example, we recently incorporated school demographic data into our analysis of PG Online’s teaching materials.

A difference in differences model, a technique often used in the field of econometrics, was applied to explore whether the amount of help parents gave their child with homework had changed from the previous survey wave, while taking into account a number of associated factors. An alternative approach, from the more traditional statistical toolkit, may have been to apply a mixed effects model to the data.

Most attention is focussed at the school level… this study explores potential changes in parents’ behaviour and the support a child receives a home.

While controlling for differences in child, parent, household and school characteristics the research specifically focussed on whether any change was the same or different for parents who knew the outcome of the Ofsted inspection compared to parents who didn’t know the inspection outcome, and whether the parents who knew the outcome had received positive or negative news. The study found that parents who receive positive news about the Ofsted inspection of their child’s school tended to reduce the amount of time they help their child with homework. While it is widely known that there are often unintended consequences of accountability systems, including testing and inspections in schools, most attention is focussed at the school level. This study is unusual in that it explores potential changes in parents’ behaviour and the support a child receives at home.

There are many national surveys, conducted by a variety of organisations who make data available for research via services like the UK Data Service. Select analysed one such survey, a longitudinal survey of household finances, using a statistical model to explore the relationship between the level of cash savings and the avoidance of problem debt in our project for debt charity StepChange. While such national surveys and administrative data collections play a role in producing national statistics, population estimates and for monitoring trends, their value is truly realised when they are further used in novel research such as the above.