Calculating Accurate Estimates of Sports Provision and Participation in Welsh Schools

Select provided survey weightings for both the School Sport Survey in 2013 and the School Sport Survey 2015 on behalf of Sport Wales. Since the publication of this case study we were again asked to provide support for the School Sport Survey 2018. These weights are used by Sport Wales to ensure that their reporting of the results of the survey is representative of the population of school pupils in Wales at various geographic levels. To see some headline results of the survey visit: www.schoolsportsurvey.org.uk.

The statistical rationale behind the use of survey weights for estimation from population samples is well established and is an essential step in the analysis of any survey of this sort. Appropriate weights are applied to adjust the sample data after it has been collected to derive meaningful estimates. It is seldom possible to obtain responses to a survey from all members of the population and even with high response rates by chance there may be differences between those who completed the questionnaire and those who did not which should be accounted for.

The Challenge

Sport Wales is the national organisation responsible for developing and promoting sport in Wales and they are responsible for producing the biennial School Sport Survey. The survey is used strategically by Sport Wales and the Welsh Government to monitor and shape sports policy, and by local authorities, governing bodies and schools to inform and plan local delivery. The survey comprises two components: a survey of sports participation among school pupils aged 7-16 which has been run biennially for more than a decade; and a survey of the provision of physical education (PE) by schools which is completed by the Heads of PE and PE coordinators on behalf of their schools. All schools within Wales with pupils in Years 3 to 11 are encouraged to take part in the survey. However, participation in the survey is voluntary, and reliant to a large extent on the goodwill of schools and partners towards the survey.

In 2015, the survey was also extended to include participation data from Year 12 and 13 pupils in sixth forms incorporated in secondary schools, and from pupils in separate further education colleges – the Further Education Sport Survey 2015.

We helped Sport Wales by producing the sampling weights that are used to account for differences between the sample of responses collected and the population of schools and pupils in Wales. Such differences are an inevitable consequence of the fact that we cannot question every single individual in the population and so we have to rely on sample data instead. By weighting the sample data, we can account for these differences and ensure that they do not lead to biases or errors in the published results. These differences can arise, for example, because of non-response which might lead to an imbalance in the proportion of schools and pupils completing the survey in different local authorities, or because of incomplete coverage of the population meaning that the distribution of key factors affecting sports participation might not properly represent the population. It is always good practice to calculate appropriate sample weights to ensure that robust and reliable results are obtained.

The Approach

Sampling weights compensate for differences between the sampled data and the population and can be used to produce accurate estimates of population characteristics of interest together with their associated sampling errors. The Pupil Level Annual School Census (PLASC) data collated by the Welsh government provide us with the population distribution of schools and pupils in Wales which we match against the survey data to produce the sampling weights.

For example, in the Census a particular Local Authority might have approximately 8,000 primary school pupils, however in the survey we might receive only 5,000 responses from primary school pupils in that Local Authority. This Local Authority would therefore be under-represented within the general population which would mean that its individual characteristics would not be fully reflected in any national-level results. The sampling weights are used to account for this under-representation, by giving this Local Authority greater weight than it would otherwise receive based solely upon the sample results.

We may also find, for example, that in the Census 14% of the primary school pupils in a particular Local Authority are Year 3 boys, whereas in the survey only 12% of the primary school pupil responses from that Local Authority are from Year 3 boys. As before, Year 3 boys in this Local Authority would be under-represented which would lead to biases in the un-weighted results at both the Local Authority and the national level.

The Value

The weighted sample of survey responses is used to produce accurate estimates of population characteristics of interest and their associated sampling errors to ensure that the reporting of the results of the survey is representative of the population of school pupils in Wales. This ensures that schools, governing bodies, local authorities, Sport Wales and the Welsh Government all have accurate data on sports provision and participation and can make informed plans for the future of sports opportunities in Wales.

We were impressed with the clarity and advice given in all of the communication we had with Select Statistics throughout the project. All timescales were met, with regular updates provided either by telephone or email. Written communication was produced in a style that made a complex topic accessible. Sport Wales received a comprehensive technical report of the method used, which will allow this weighting protocol to be applied to future surveys. It was a pleasure to work with Select Statistics on this project and we would recommend their services to others looking for expertise in this field.Becca Mattingley – Senior Research Officer, Sport Wales

Services