Judging at the Big Bang Fair

The Challenge

The Big Bang South West is an annual science fair aimed at inspiring and enthusing young scientists about careers in Science, Technology, Engineering and Maths (STEM) subjects. During the fair, the students get the opportunity to exhibit their science projects which are judged throughout the day by volunteers from the local business community.

To ensure the smooth running of the day, the organisers must allocate the judging of each of 120 projects both to a pair of judges and a time slot. This allocation has to take into consideration the following constraints: each project must be judged twice to check for consistency, there are 6 time slots during the day when judging takes place, and there are a total of 50 judging pairs. It’s inevitable that certain biases will arise within the judging process. For example, one potential source of bias is the judges themselves. With a fairly subjective marking scheme, some judges will have a natural tendency to mark high whilst others might mark low. Timing is also important. Big-Bang-logoMost judges are new to the event and the first project that is judged is bound to be somewhat of a learning experience. So, projects judged first are likely to receive a more conservative score than those judged later in the day. The aim, therefore, is to design a judging process that ensures that each project is given a fair score that takes these potentially biasing factors into account.

The Approach

The judging process is very similar to that of a standard statistical survey and the statistical techniques we use for those can be used here to adjust the raw scores to take account of any potential biases. From a statistical perspective, in order to estimate the judging effect, the ideal solution would be for every judge to see every project and this would be a so-called balanced design. This would enable us to see how each judge marks each project differently and to identify the habitually high and low markers. Similarly, in order to get a feel for how the time slot affects the marks, it would be best to have as many judges as possible visit every project at every time slot. This would help us see if there is any pattern to the marks given in each time slot and whether, for example, projects judged at particular times of day tend to have higher or lower marks than at others. Of course, neither of these are viable options because of the logistical constraints, but standard statistical designs can be used to assign projects to different combinations of judge and time slot that still gives us the information we need whilst not overloading the students or requiring that judges be in two or more places at once!

The best solution is to adopt a so-called balanced incomplete blocks design which randomly assigns judges to pairs (referred to as blocks) and then assigns each judging pair to a specific pattern of projects. The judge/project combinations are then randomly assigned to time slots (so-called randomisation) and the design is complete. Once the judges have recorded their marks, they are collected together and the judge and time slot effects can be estimated. The raw scores can then be adjusted to predict, for example, what score would have been obtained for each project had they been marked by one specific judge at a specific time slot. This then gives us a fair basis for comparison and we can decide which projects should receive the various awards on offer.

The Value

A formal design for project allocation not only meets the organisational constraints, but also maximises the information gained during the judging process and allows the effects of any biases to be estimated. Once these effects have been estimated, it’s possible to use them to adjust the raw scores to create a fair score for each project that can be compared with all the others and to rank them within the relevant categories to decide who wins each award.