The post Data in Devon 2019 appeared first on Select Statistical Consultants.

]]>Select were pleased to attend the conference and our Senior Consultant and Acting MD, Sarah, was delighted to be asked to present at the event, holding a session on ‘What is R and Why Should You Use It?’.

In her talk, Sarah discussed the value of using R for data analysis. R is a programming language, specifically designed for statistical computing and graphics and is both free and open source.

R has been developed with visualisation in mind – there’s a huge range of different types of charts, graphs and plots available. Analyses conducted in R can also be turned into dynamic and interactive web applications using the package Shiny, which Sarah demonstrated during her talk.

With R, all the different processes required to turn data to insights – from data processing and manipulation to complex statistical analyses to visualisations and creating interactive dashboards – can be accommodated within a single platform.

We really enjoyed attending the event and it was a great opportunity to promote the benefits of R to other data professionals in the South West!

The post Data in Devon 2019 appeared first on Select Statistical Consultants.

]]>The post Select Works with PG Online to Explore the Effectiveness of its Teaching Materials appeared first on Select Statistical Consultants.

]]>We were asked to analyse the 2018 school-level GCSE and A level results, published by the Department for Education (DfE), for schools that had purchased PG Online’s materials, comparing these with similar schools that were not PG Online customers. In addition to exam result data, we used other publicly available school demographic data, so that our comparison was to schools that were similar in as many aspects as possible.

“Select have been meticulous in their attention to detail.”

We found that the average GCSE point score of schools that had purchased PG Online’s materials for the new 9-1 GCSE syllabus was, on average, 0.12 points higher, (with a 95% confidence interval of (0.05, 0.20)) than other schools, taking into account differences in schools’ background characteristics, such as average GCSE Computer Science score the previous year, school type, region and the proportion of students with levels of special educational needs (SEN) and those eligible for free school meals (FSM; a proxy for deprivation). This can also be expressed approximately as 3 students out of every 25 being expected to achieve one grade higher at GCSE.

There was less data available for the A level model than for GCSE, owing to the DfE suppressing the results for schools that were based on fewer than 5 students. From the data available, we again found that the average A Level point score of schools that had purchased PG Online’s A Level materials was, on average, 0.23 points higher (with a 95% confidence interval of (0.07, 0.39)) than other schools, again taking into account differences in schools’ background characteristics (specifically average A Level Computer Science score the previous year and the proportion of students who were eligible for FSM). This is equivalent to just less than 6 students out of every 25 being expected to achieve one grade higher at GCSE.

PG Online produced the following infographic, illustrating our findings:

Robert Heathcote, Director at PG Online, said of the work,

*“Select have been meticulous in their attention to detail and care in the output of their results, factoring in many external variables that we hadn’t previously thought possible. Their analysis was accurate, in-depth and on time, and has enabled us to test a key part of our mission which informs our future development.”*

Further details of the analyses and the results can be found in the full report published on PG Online’s website.

The post Select Works with PG Online to Explore the Effectiveness of its Teaching Materials appeared first on Select Statistical Consultants.

]]>The post Restaurant Ratings Model: Visualising the Key Findings appeared first on Select Statistical Consultants.

]]>We often use a logistic regression model to answer questions about the strength of factors associated with a binary outcome; for example, “What are the biggest factors associated with stroke occurrence?”. The logistic regression model can be tricky to interpret as the results are often presented as relative probabilities. For example, we might say that having a medical history of strokes increases the odds of having a stroke by 50% relative to someone who doesn’t have a medical history of strokes. The clever part about this visualisation is that we see the results as absolute probabilities – for example that the probability of having a stroke for those with a medical history of strokes is 40%. This probability statement is less convoluted and can be derived from exactly the same model.

To test the concept’s suitability to questions outside of political science, we found some open source data about restaurants – their characteristics and customer ratings – and we started by asking the question: “What are the biggest factors associated with restaurant ratings?”. We used a logistic regression model to predict the probability that a restaurant will receive a positive rating in a customer review. The model estimates how the expected probability of receiving a positive rating is related to restaurant characteristics. For example, are restaurants that have on-site parking rated more highly on average? Do we expect restaurants in different locations to attract different ratings?

Here is our version of The Economist’s graph:

It turned out that, on average, a restaurant had a 42% chance of receiving a positive rating. The graph is centred on 42% and the bars show the differences from the average; the longer the bar, the bigger the deviation. The bars indicate the estimated probability of a positive rating for a restaurant within a group of restaurants which all have the feature labelled at the end of the bar, but otherwise have the average characteristics of the sample. So, for example, the average restaurant within a group of restaurants which have formal dress codes, but otherwise reflect the sample distribution of characteristics, is predicted to have a 93% chance of receiving a positive customer rating. Whereas the average restaurant within a group of restaurants which have permissive smoking policies, but have all other characteristics at the average rates, is predicted to have a 32% chance of receiving a positive customer rating.

The graph works so well because the green and red bars allow viewers to quickly assess which restaurant features are important. We can see that dress code, smoking policy, and the availability of other services are clearly important factors associated with restaurant rating, but parking, alcohol selection, and ambience are clearly not. We can also compare across attributes. For example, a formal dress code is associated with a probability of a positive rating about 40 percentage points higher than the probability associated with smoking confined to the bar area.

The tools available for us to create flexible, eye-catching graphics have never been better. We can create graphs such as this using packages in R, our standard statistical analysis software. In fact, we have used the same software to produce this graph that the BBC use to create the visualisations that accompany their stories (called ggplot2). Of course, the statistical details are an integral part of any analysis and should be considered when interpreting the results; but as a gateway to the key findings, striking visualisations are very useful for us and our clients.

The post Restaurant Ratings Model: Visualising the Key Findings appeared first on Select Statistical Consultants.

]]>The post Congratulations to HiLo on Global Safety Award appeared first on Select Statistical Consultants.

]]>Select were delighted to provide statistical support to HiLo, carrying out an independent review of their risk modelling and providing statistical advice on the approach. We look forward to hopefully working with them again on future projects.

The post Congratulations to HiLo on Global Safety Award appeared first on Select Statistical Consultants.

]]>The post Jo Joins Women in Statistics and Data Science SIG appeared first on Select Statistical Consultants.

]]>The SIG aims to raise the profile of women working in statistics and data science, to advocate for opportunities for and to support women working in these fields and to share experiences.

One of our senior consultants, Jo, was pleased to be accepted as a volunteer on the committee.

Speaking of her involvement, Jo said, “My secondary school encouraged me and others, particularly young women, to pursue science and technology, and my mother encouraged me more generally not to restrict choices and thinking based on stereotypes. Not all young women have those influences and it would be good to provide positive role models and to contribute to showcasing careers that involve statistics, to encourage those interested in STEM subjects”, adding that, “while from my experience, the field of statistics has more gender balance than in some other STEM areas (there were proportionally more women in my statistics Masters class than in my school technical drawing class) I am keen to see women represented at all levels of the profession.”

The post Jo Joins Women in Statistics and Data Science SIG appeared first on Select Statistical Consultants.

]]>The post EU Freedom of Movement & the Migrant Workforce: How have the numbers changed? appeared first on Select Statistical Consultants.

]]>The latest release (in February 2019) from the ONS for the UK labour market shows that the number of EU nationals working in the UK is 2.27 million as of October to December 2018, out of a total 32.60 million people aged over 16 in work (7.0%). While this is a slight increase on what was observed in January to March 2016 (at 2.14 million), plotting the data over time (see right-hand plot in Figure 1) there’s an apparent leveling-off of the number of EU nationals employed in the UK. This follows a steady increase over the last 10 years, as noted in our previous blog on migrants in the UK workforce. A similar pattern is also observed for EU born workers (see left-hand plot in Figure 1).

The ONS also notes in their February 2019 release of the October to December 2018 figures, that compared to the same period in the year before (2017), there has been a decrease of 61 thousand EU nationals working in the UK, whereas UK nationals and non-EU nationals have both seen increases of 372 thousand and 130 thousand, respectively. We should however note that though we may see a year-on-year change, since the referendum (indicated by the red vertical line in Figure 1), the number of EU migrants in the workforce has remained broadly consistent compared with the previously increasing trend over the past decade.

The latest figures available from the ONS, as presented above, are for the last quarter of 2018. It’ll be interesting to see if and how the picture changes when the data for 2019 become available and as we reach the deadline of Article 50.

More than two years on from our post on EU freedom of movement numbers, Figure 2 shows how migration to and from the UK has changed between 2015 and 2017. The bar chart shows the estimated total number of UK migrants in EU and non-EU countries (green bars) for 2017 and 2015, together with the number of migrants in the UK from EU and non-EU counties (mustard bars) for those years. These figures are taken from the United Nations Population Division from mid-2015 and mid-2017 population estimates of migrant stock, where a migrant is defined as “a person who is living in a country other than his or her country of birth”.

We can see that the largest change has been the decrease in the number of UK migrants outside the EU, while the number of migrants from the UK in the EU has remained at a similar level. In terms of immigration to the UK, we see the opposite: non-EU migrants have reduced in number, whereas the number of EU migrants in the UK has increased. Despite this, we still see that the majority of migrants living in the UK are from non-EU countries, though between 2015 and 2017 the gap has slightly narrowed.

Due to the time it can take to gather and process data before it becomes available for public use, we do not have the full picture as to the current UK migration statistics and how these may have changed since the EU referendum over two years ago. However, there have clearly been changes in the interim – in particular, it’s interesting to see that the number of EU born workers and EU nationals working in the UK has not continued to increase at the same steady rate as over the previous decade. In an upcoming post, we will look at updating the figures from some of our other blogs in our EU series, including looking at the UK’s trading partners.

The post EU Freedom of Movement & the Migrant Workforce: How have the numbers changed? appeared first on Select Statistical Consultants.

]]>The post An Alternative Approach to Evaluating Interventions in Education appeared first on Select Statistical Consultants.

]]>The EEF is dedicated to breaking the link between disadvantage and educational achievement, and as such require their evaluators to analyse trial results for the subset of pupils who are eligible for free school meals. In many situations subgroup analyses are viewed as bad practice, as they are prone to being under-powered. Though EEF evaluators are now required to present power calculations for the analysis of this pre-specified subset of pupils.

ZhiMin proposed a new approach using a Pupil Advantage Index to estimate the outcomes for all pupils in the dataset. This approach involves building a model which takes account of all available background information, including free school meal eligibility, along with any interactions. This exploits the heterogeneity of the data, recognising that pupils and education are multifaceted and complex, rather than considering background characteristics individually in subgroup analyses. As an alternative to subgroup analyses, rather than answering the question *“*d*oes this intervention work, on average, for this subset of pupils?”*, the Pupil Advantage Index asks *“for what kinds of pupils does this intervention work?”*. As well as knowing which pupils benefited most from a particular intervention in a trial, it can also help future decisions by answering the question *“which intervention(s) are best suited to my particular set of pupils?”*

“This was a really interesting seminar”, said Jo. “It was great to learn of the latest developments in the analysis of education data and to hear about further analyses that are being conducted on the wealth of data collected by the EEF”.

The post An Alternative Approach to Evaluating Interventions in Education appeared first on Select Statistical Consultants.

]]>The post Meta-analysis: Reducing Salmonella in Animal Feed appeared first on Select Statistical Consultants.

]]>*Salmonella Typhimurium* is a pathogen which can cause gastroenteritis in many mammals, including humans. Select were asked by Anitox to combine the results from 90 individual studies which each estimated the mean percentage reduction of *Salmonella Typhimurium* in animal feed following the application of a pathogen-control product (Berge, 2015).

The studies were conducted for different types of animal feed, and under different conditions including: recontamination or not, different concentrations of pathogen-controlling additive, and different sample sizes, i.e., numbers of culture plates tested. As a result, each study yielded a different effect size for the additive in reducing the presence of *Salmonella Typhimurium.*

The aim of the project was to collate the results from the individual studies to give overall estimates of the effects of the additive for different feed types and doses, allowing for other differences in the study conditions.

When dealing with multiple studies estimating similar effects, a meta-analysis can be used to collate the results, while accounting for how much evidence is provided by each study.

The studies were split into groups based on the type of feed (poultry feed, pet food or protein meal) considered, the additive dosage used, and whether recontamination had occurred or not. This stratification ensured that only those studies designed to estimate similar effects were combined in each case.

However, within each of these groups, examining the treatment effects from the different studies, it was clear that they varied more than we might have expected by chance. In order to account for the heterogeneity in observed treatment effect between the different studies, a random-effects statistical model was fitted to the data, following a similar approach as described in our previous case study: ‘Meta-analysis: Combining Results from Multiple Studies’. This approach allows for there to be real differences in the treatment effect due to, for example, differences in the location, batch of animal feed or study protocol (such as length of time over which the product was applied) used in each study.

As well as producing an estimate of the overall average effect, when performing a meta-analysis, we can also use forest plots as a means of visualising the analysis. These plots provide a way of visually comparing and combining the studies by representing several individual studies, alongside the estimated overall average effect (and the associated confidence intervals), on the same axis. Figure 1 below shows an example of such a forest plot.

By using a meta-analysis, we were able to condense the information from 90 studies conducted under different conditions into a set of clear and consistent results for each separate treatment effect of interest. This helped Anitox to better understand the effects of their pathogen-control product, and to optimise treatment conditions and application technology accordingly.

By using a meta-analysis to aggregate the results of different studies, we can draw together many individual, smaller trials, gaining a higher statistical power overall. This means our final estimates are more robust compared to each study individually, and allowing us to make best use of the individual trial results.

The post Meta-analysis: Reducing Salmonella in Animal Feed appeared first on Select Statistical Consultants.

]]>The post Presenting the Results of a Multinomial Logistic Regression Model: Odds or Probabilities? appeared first on Select Statistical Consultants.

]]>When fitting a multinomial logistic regression model, one generally wants to understand what motivates the choice made by the student (e.g. what are the key drivers) and how those drivers affect the choice made. For instance, in our example, we might be interested in understanding how the choice of programme varies for students with different maths scores.

As we saw in the model coefficient table in our previous blog, we get two coefficients, one of -0.11 for the comparison between a general and an academic programme and one of -0.14 for the comparison between a vocational and an academic programme. As they are both negative, this tells us that as maths score increases the log-odds of choosing either a general or a vocational programme (instead of an academic programme) decrease. But what does this actually mean? And which programme is actually the most popular choice?

To help understand the relationship between maths score and programme choice, we can plot the predictions from the models for a hypothetical student, assuming all other drivers in the model are fixed (i.e. assuming the student is from the middle socio-economic status (SES) group, attends a public school, and has average prior reading and science scores).

In Figure 1, we use the model coefficients to look at the linear relationship (on the log-odds scale) between programme choice and maths score. We can see that, as the maths score increases, the log odds of choosing a vocational vs. academic course are decreasing faster than the log odds of choosing a general vs. academic course, reflecting the difference of size in their model coefficients.

What is interesting from this plot is that for maths scores below 45, both sets of log odds are positive indicating that choosing either of the two programmes is more likely than choosing an academic programme.

We can also see that for a maths score of just above 60 the two lines cross over, i.e. the odds of a general vs. academic choice are now higher than the odds of a vocational vs. academic course. But what actually are those odds?

To obtain the odds of either set of choices, we can take the exponential of the predicted log odds above and again plot them against maths score in Figure 2.

Figure 2 highlights the non-linear nature of the relationships between maths score and the odds of either choices. The difference between the odds is more pronounced for lower values of maths scores with the gap narrowing as maths scores increase. As in the case of a logistic regression, the odds are a measure of the relative association between maths score and programme choice. For example, for a maths score of 40, the odds of choosing a general versus academic programme is 2.1, while the odds of choosing a vocational versus an academic course is 4.4. This means that a student with such a maths score is 2.1 times more likely to choose a general course compared to an academic course, and 4.4 times more likely to choose a vocational course over an academic one.

In addition to what we saw in Figure 1, for maths scores from about 55 upwards both sets of odds are actually quite small, getting close to 0, indicating that both general and vocational courses are very unlikely to be chosen.

Both the odds and log odds plots are useful and accurate representations of the model coefficients. However, they only provide relative measures of the association between maths score and programme choice, using academic programme as the reference for the comparisons. These are thus useful to gain an understanding of the student relative preferences between programmes.

At the same time, the model coefficients cannot directly be used to assess which course is most likely to be chosen by an average student attending public school from a middle socio-economic background, as a function of the maths score. To get that information, the odds above needs to be converted to the predicted probability of each outcome (see Figure 3).

It is only when the results of the model are converted to probabilities that it becomes obvious that for students with a maths score up to just over 50, a vocational course is more likely to be chosen over any of the other two, while for students with scores higher than that, an academic course is the most likely chosen programme. The other interesting point is that across the whole range of observed maths scores, an average student of middle socio-economic status from a public school is not likely to choose a general course. So regardless of the maths score, middle socio-economic students from public schools with average reading and science scores are unlikely to ever choose a general programme.

As we saw above, the coefficients obtained directly from the model provides an immediate indication as to the direction of the relationships, with the conversion to odds ratio giving an estimate of the relative changes in the odds of choosing one alternative option versus the reference option. However, it is only when converting those odds back to probabilities (as in Figure 3) that one can really see the relationships between the explanatory variables and the likelihood of each outcome.

The output from a multinomial logistic regression model may appear complicated at first and converting the coefficients back to probabilities does make it easier to interpret the model and thus gain useful and actionable insights from it.

In most practical cases, as in the example given here, one is often more interested in how likely each outcome actually is, rather than the relative chance (or odds) of observing one outcome versus another. For instance, a business with a new pricing policy might want to know how sensitive online, in-store or phone customers are to different prices, and might not be so interested in the changes in the balance between online vs. in-store customers, and phone vs. in-store customers as price varies.

The post Presenting the Results of a Multinomial Logistic Regression Model: Odds or Probabilities? appeared first on Select Statistical Consultants.

]]>The post Will it be Turkey this Christmas? appeared first on Select Statistical Consultants.

]]>Today, it seems like everyone has turkey at Christmas, but what about the rest of the year? If we look at the data available from the Department for Environment, Food and Rural Affairs for the number of turkeys slaughtered per year, we can see a definite spike around December.

From Figure 1 we can see that over the last 20 or so years, nearly 1 million more turkeys are killed around December each year compared to May. The British people tend to enjoy turkey as a seasonal meat – we can see that turkey numbers increase throughout September, October and November before peaking in December. Think about all the Christmas related food sold around this season – Turkey and stuffing sandwiches are not only sold during December!

Since turkeys were originally an American import, how does the UK annual pattern compare to that of the US? Americans have less of a specific tradition around Christmas dinner – “Turkey day” for them refers to Thanksgiving, which takes place on the fourth Thursday of November.

Looking at data available from the United States Department of Agriculture we can see these trends reflected. Figure 2 below shows the UK and US turkey production in pounds, estimated for the UK by using the average turkey weight 14 pounds, and scaled by the population in each country, so that the graph tells us how many pounds of turkey is produced each month per person.

There are two interesting things to note from this plot. Firstly, the smoothed average line shows an increase in the USA’s production of turkeys in October and November, in the lead up to Thanksgiving, compared to the rest of the year. This is interestingly followed by a drop in December; the opposite to what is happening in the UK. Secondly, there is much less variation in turkey production over the whole year compared to the UK, implying that Americans tend to eat turkey at a reasonable steady rate all year long, and not just as a special treat during the holidays.

We can also look at whether turkey’s consumption has changed by plotting the annual total of turkeys slaughtered in pounds for both countries over the last 24 years (Figure 3 below). It is noticeable from the two previous graphs that there is a lot of variation across years, most particularly so in the UK.

Since the beginning of the century, it appears that UK turkey consumption has been steadily decreasing, and nearly halved between 1995 and 2007, while in the US there has been a slow increase. Could this be due to turkey being replaced by other meats, fish or vegetarian alternatives at Christmas? And you, what are you having for Christmas dinner?

The post Will it be Turkey this Christmas? appeared first on Select Statistical Consultants.

]]>The post Select’s Star-tistical Christmas Social appeared first on Select Statistical Consultants.

]]>

Nestled in a cosy restaurant in a picturesque village, we were treated to an expert workshop to create fairy-light-studded, willow star decorations. You could almost hear the cogs whirring as Select’s brains switched gear from the statistical to the star-tistical. The cake-fuelled tranquillity was occasionally pierced by flailing willow branches – but thankfully all survived unharmed.

We all had a fantastically relaxed afternoon, aided in no small part by generous hosting and delicious food. The delightful results hang proudly in the homes of Select, and are a lovely reminder of an afternoon very merrily spent.

We would like to heartily thank Victoria Westaway for showing us her considerable skills and kindly complementing our sub-professional attempts; and Vitamin Sea Restaurant for their tasty food and charming hosting. We enthusiastically recommend both to anyone in the Exeter area.

The post Select’s Star-tistical Christmas Social appeared first on Select Statistical Consultants.

]]>The post Making the Most of Budgets for Public Health Interventions appeared first on Select Statistical Consultants.

]]>The National Institute for Health and Care Excellence (NICE) is responsible for assessing new medicines, medical technologies and diagnostics to identify the most clinically- and cost-effective treatments available. This helps to ensure that those products which offer the best value for patients are adopted for use by the NHS and in public health programmes implemented by local government. NICE therefore needs to evaluate the trade-off between how well a new treatment works and how much it will cost.

One method of working out how effective a treatment is, is to look at the average change in life expectancy for the person to whom it was given. However, this does not give the full picture; the quality of life of the patient should also to be taken into account, i.e. factoring-in their ability to carry out daily activities, freedom from pain and mental anguish. All of this information can be combined into one metric, frequently used in health economics, called a quality-adjusted life year (QALY). This is a measure of the life expectancy of a patient, weighted by a quality of life score (on a scale from 0 to 1) over each year. For example, one QALY could represent either 12 months at ‘perfect health’ (a quality of life score of 1), or 24 months at ‘50% health’ (a quality of life score of 0.5). These scores are routinely calculated from questionnaire responses where patients are asked to rate aspects of their quality of life including mobility, ability to self-care and anxiety/depression.

Generally, QALYs are calculated using a naive method. This involves plotting quality of life scores over time and measuring the area under the curve (AUC) individually for each subject included in the study. This fragmented approach does not apply statistical modelling to bring the data together and make inferences across the whole dataset. This leads to limitations in two main areas:

- Bias being introduced where there are missing data for a patient, as there is no natural means of accounting for missing values with this naive approach. This can give rise to inaccurate AUC estimates.
- Inefficient use of the available data. For each patient in a QALY study, we have information about their expected survival and what their quality of life is like. Using the simplistic AUC calculations does not allow us to take account the association between these outcomes; intuitively we expect the quality of someone’s life to be linked with their life expectancy.

These limitations associated with the standard approach to calculating QALYs motivate the use of a more rigorous statistical approach which will enable us to calculate more accurate estimates, more efficiently.

Joint longitudinal-survival modelling can be applied to data including life expectancy and quality of life information to help obtain improved QALY estimates. Joint modelling is a new statistical approach that has recently been developed. It essentially combines two established types of model: mixed effects models and survival models. Mixed effects models are appropriate for analysing longitudinal data, accounting for repeated observations collected for the same subject over time. Survival models are designed to analyse time-to-event data, such as survival times. They account for censoring where, for example, we may only know that the time to the event is greater than the current number of days of follow-up. A joint longitudinal-survival model is designed to analyse datasets that include both of these types of data, allowing inferences to be made about the survival and the quality of life over time trends from a single model.

Data collected to calculate QALYs fit this scenario. For each patient, we have survival times with censoring information, and repeated measurements over time recording the quality of life scores the patients gave. We can combine both types of data into one joint model which can examine the association between patients’ survival and how good their quality of life is, as well as looking at overall trends in survival and quality of life and factors that can affect these two things separately. The joint model also reduces bias due to missing data, by sharing information across all of the subjects included in the study, rather than considering each patient individually (as with the AUC method).

The final fitted model can then be used to estimate QALYs under different scenarios through simulation and taking expectations. For example, they might be used to compare the average QALY for a patient who was taking a new, experimental treatment, with someone taking the standard, currently available treatment. Combining this information with estimates of the costs of each treatment, these results can then be used to assess their cost effectiveness and decide whether it is worth investing in a new treatment.

By applying a more rigorous statistical analysis, joint models can increase both the accuracy and the efficiency of calculating QALY estimates. This in turn can help to improve the service provided to the public by NICE and the NHS in two major ways.

Firstly, by accounting for the correlation between quality of life and life expectancy, joint modelling allows more efficient use of the available data to be made. This may help to reduce the sample sizes required in studies contributing to life expectancy and quality of life estimates, thereby reducing costs and potentially shortening timelines. This will allow decisions to be made more quickly and cost-effectively.

Secondly, by reducing biases and therefore obtaining more accurate QALY estimates, NICE and the NHS can make better informed choices as to which treatments and technologies it would be most cost-effective to spend taxpayers’ money on. This in turn will help improve overall resource management, maximising the health benefits provided for a given cost and leading to an improved service for the public.

The post Making the Most of Budgets for Public Health Interventions appeared first on Select Statistical Consultants.

]]>The post Sample size calculation for complex designs appeared first on Select Statistical Consultants.

]]>Experimental controlled trials are an essential part of the development of new production methods or treatments. Calculating the appropriate sample size to demonstrate the efficacy of a new production method or a new treatment is not always straightforward, as the practical aspects of the trial need to be considered.

For example, in an agricultural context, a trial might be set up in a production setting to measure performance indicators of farm animals (such as feed conversion ratio or daily weight gain) in order to compare a new formulation (which we will refer to as the treatment) with a control feed. These types of trials will often be constrained by both animal welfare considerations and ensuring that the trial conditions are as similar to real-life conditions as possible (e.g. minimum and maximum flock or herd sizes, limited numbers of pens within a barn, all-in all-out production systems, etc.).

These logistical constraints mean that it is generally unlikely that a standard experimental design, such as a fully randomised block design, can be used. Therefore, though estimates of the expected effects, and sources and sizes of variation can be obtained from the literature or previous smaller scale studies, there is generally no off-the-shelf sample size calculator or formula that can be applied.

Calculating the appropriate sample size for a trial is about getting the right balance between having a sufficient number of subjects to be statistically confident that the minimum desired effect size can be detected and ensuring that the trial is logistically feasible (i.e. it does not need to be run for an extensively long time, or with far too many subjects).

In the absence of a “standard” sample size calculator, we can use a simulation-based approach. This approach consists of simulating a large number of datasets where the number of animals (or individual plants or crops), the number of blocks or other higher-level groupings (such as pens, herds, farms or fields) are varied across a range of values corresponding with likely practical designs.

As with any sample size calculation (or equivalently power analysis), this requires the use of sensible estimates of the expected effect size of the treatment compared to the control group as well as any other important contributors to the trial (e.g., the estimated variability between individual animals, or between groups of animals at the different levels of the design). These estimates can usually be obtained from previous studies, the literature or from expert panels.

The result is a number of simulations that each correspond to a specific sample size. For each simulation, we can fit a statistical model that, combined with repeated runs of the simulation, provides an estimate of the associated power to detect a statistically significant effect of the treatment of interest. The results of the simulations can then be visualised in the form of power curves where the estimated power to detect the desired effect is plotted against the range of sample sizes tested (an example of this is given in Figure 1).

This type of simulation approach gives us added flexibility for it also allows us to test a number of different scenarios. For example, for a range of possible sample sizes, we can investigate how varying the expected size of the treatment effect affects the estimated power to detect a significant effect (demonstrated by the different coloured lines in Figure 1).

The results of the simulations in Figure 1 indicate that as the sample size increases, so too does the estimated power to detect an effect. However, the power also varies considerably across the different effect sizes. For example, it is clear that whilst a sample size of 400 individual animals is more than adequate to detect a large effect size if it existed (in fact this effect could be detected with a power of over 80% with a sample size of only 120), this sample size would not be sufficient to detect a small effect size. For a medium effect size, we see that a sample size of 400 individual animals results in a power of 65%, which is equivalent to a 35% chance of failing to detect a statistically significant improvement if one were to exist. Therefore, even for a medium effect, it is likely that a larger sample size would be needed.

Note that this approach could also be carried out on other key variables (not just different possible effect sizes) such as the impact of different amounts of variability between individual subjects (or between groups of subjects) or any other confounding factor that should be controlled for during the analysis of the trial data.

Whichever final design you choose for your experiment, a simulation-based approach to calculating your sample size ensures that you will have the necessary power to detect a meaningful effect of your treatment with the level of confidence you require, whilst meeting the practical constraints of the trial.

Without having to run a number of different and potentially expensive trials, you can explore a range of scenarios, changing trial conditions or likely treatment effects, to understand their impact and thus inform the final design choice. This further enables you to be confident that your chosen design, including the sample size, will be appropriate to demonstrate and support the aims of your study. While we illustrated the approach in an agricultural experiment context, this type of simulation approach is well suited to any study with a complex design, including nested classifications, such as customer surveys or new stock management processes in the retail or leisure industry, or intervention studies in the education sector.

An additional advantage is that when using a simulation-based approach to calculate the sample size for your experiment, the statistical model that will be used for the data analysis needs to be specified. This also means that once the data has been collected at the end of the trial, the analysis can be done more quickly as the statistical model that will fit the data best has already been developed.

The post Sample size calculation for complex designs appeared first on Select Statistical Consultants.

]]>The post Measuring Human Behaviours and Traits with Item Response Theory appeared first on Select Statistical Consultants.

]]>Item response theory (IRT) is a statistical modelling technique used in the field of measurement. IRT assumes that there is a single underlying trait, which is latent and that this latent trait influences how people respond to questions. As IRT is often used in education the latent trait is often referred to as ability. Item response theory models the probability of a person correctly answering a test question (an item), given their ability. After running an IRT model we can obtain estimates for each person’s ability, and also for each item’s difficulty (some items are easier or harder than others). We would expect someone with a lower ability measure to get the easier items correct but get fewer or none of the more difficult items correct. And we would expect someone with higher ability to get more of the difficult items correct. While we tend to still talk in terms of ability and item difficulty, IRT can be used to measure other character traits, not just academic ability, with items or questions that measure these.

As such, item response theory can be used to analyse surveys since a commonly used survey question format is the Likert scale, where respondents are asked how much they agree to a series of statements, with response options ranging from ‘strongly disagree’ to ‘strongly agree’.

An example are the statements below, which were asked of Year 8 to Year 10 pupils in the NHS’s National Study of Health and Wellbeing Survey of children and young people in 2010.

Questions of this kind ask respondents about a number of components that are closely related and which assess different aspects of a single underlying trait, in this case resilience. People with high levels of resilience will tend to agree and strongly agree with these statements, while people with low levels of resilience will tend to disagree.

Using IRT we can measure respondents’ resilience and also their propensity to endorse statements like those above.

IRT puts the thresholds (the boundaries between “strongly agree” and “agree”, and between “agree” and “neither”, etc) for all of the statements onto the same scale, i.e., in terms of the respondents’ levels of resilience in this example. The figure below shows the thresholds for the 5 statements above. The coloured bands show at what point on the scale each response (from “strong disagree” to “strongly agree”) is most likely, depending on someone’s level of resilience. You can see that these don’t all occur in the same place; that the point at which ‘agree’ is more likely than ‘neither’ for “*I try to stay positive*” is lower down the scale than for “*I am good at solving problems in my life*“.

While a chart like the diverging stacked bar chart is useful for comparing levels of agreement and disagreement (as demonstrated in this previous blog: Analysing Categorical Survey Data), it positions neutrality in the same place and symmetrically for all statements. By positioning the statements on the same scale, i.e. in terms of resilience in this case, we can compare how easy each statement is to endorse. Of the five statements, pupils who responded to this survey found “*I am a very determined person*” one of the easiest statements to endorse. The threshold between agree and strongly agree is 0.7. Whereas the equivalent threshold for the statement “*I am good at solving problems in my life*” is 1.9. This second statement is more difficult to endorse; pupils need a higher degree of resilience (scores higher than 1.9) before “strongly agree” becomes the most likely response.

To illustrate this further, take two example pupils; one with a resilience score of -2.8, the other with a resilience score of 1.3. These are marked on the diagram below with arrows.

Each pupil’s most likely response is indicated by the region in which their arrow lies. The pupil with the lower resilience score of -2.8 is most likely to “disagree” with the first and last statements, “*I can usually think of lots of ways to solve a problem*” and “*I am good a solving problems in my life*“; they are most likely to answer “neither” to “*I try to stay positive*” and “*I am a very determined person*“; and they are most likely to “strongly disagree” with the fourth statement, “*I really believe in myself*“.

The pupil with the higher resilience score is most likely to “agree” with the first and last statements, “*I can usually think of lots of ways to solve a problem*” and “*I am good a solving problems in my life*“; and most likely to “strongly agree” with the other three statements.

While using pupils’ IRT scores to understand their most likely responses to the statements is a useful exercise, it is mostly illustrative (it can be useful for reporting, for example). IRT is itself a statistical model (also known as a confirmatory factor analysis) and provided this model fits the response data well, the resulting IRT scores provide robust, continuous measures that can be used in further analyses.

IRT scores could be used to compare latent traits like ability or resilience between, say, groups of pupils. This would allow comparisons to be made between those who have received an intervention and those who haven’t, to assess its effectiveness. These scores could also be used in further statistical modelling to, for example, explore relationships between these attributes and other characteristics or outcomes.

The post Measuring Human Behaviours and Traits with Item Response Theory appeared first on Select Statistical Consultants.

]]>The post Sally is Awarded Graduate Statistician Status appeared first on Select Statistical Consultants.

]]>We’re pleased to announce that Sally has gained the designation of __Graduate Statistician__ (GradStat). GradStat status, granted by the __Royal Statistical Society__ (RSS), is a professional award formally recognising a member’s statistical qualifications. Sally recently achieved a distinction in her MSc in Statistics from Lancaster University, a degree accredited by the RSS. Accreditation signifies that the course meets the Society’s academic standards, measured against their qualifications framework, ensuring that an appropriate breadth and level of statistical knowledge and skills has been demonstrated at a Graduate and Masters level.

As a GradStat member, Sally is required to abide by the Society’s __code of conduct__ and to adhere to their __comprehensive CPD policy__. On obtaining GradStat, Sally is also now eligible to work towards becoming a __Chartered Statistician__, the highest professional award for a statistician which is also granted by the RSS. This award requires at least five years’ post-graduate experience as a professional statistician, in addition to an approved degree and training.

The post Sally is Awarded Graduate Statistician Status appeared first on Select Statistical Consultants.

]]>The post Women in Maths appeared first on Select Statistical Consultants.

]]>Lynsey was pleased to be invited to give a presentation on her current role and her career to date following her PhD. The afternoon session where Lynsey spoke consisted of three speakers who had all undertaken a PhD in Mathematics or Statistics, but had chosen different career paths. The other speakers had studied Pure and Applied Maths and one was now a Lecturer at the University and the other an Analyst at GCHQ.

“I was really pleased to be asked to present at this event as I’m always keen to promote and inspire mathematics and statistics, particularly to young females early in their career,” said Lynsey. “It was a fantastically organised event with a great attendance. I particularly enjoyed listening to the other speakers to hear about the different interesting and vibrant careers that they had embarked on following their PhDs. It’s clear just how wide and varied the opportunities are after studying mathematics and statistics!”

The post Women in Maths appeared first on Select Statistical Consultants.

]]>The post Visualising Refugee and Asylum-Seeker Data appeared first on Select Statistical Consultants.

]]>Looking at the most recent mid-2017 estimates, we find that the population of refugees in the UK at that time consisted of 112,698 refugees from 114 different countries of origin. (To calculate these numbers, we first removed two origin categories from the data – those refugees that have been categorised as ‘Stateless’ (1,841) and those with ‘Various/Unknown’ origin (6,775)). We then ranked the population of refugees in terms of their country of origin and plotted the top 20 countries in the map below (Figure 1), where the colour of the country reflects the percentage of the total number refugees currently residing in the UK to have come from that country. We have also provided a table of the countries of origin and the number/percentage of refugees. This map highlights that the largest proportion of refugees are from the Middle East, but there are also a relatively high proportion from Africa (particularly the country Eritrea).* [Note that we did not map all 114 countries given that the top 20 account for almost 90% of all refugees currently in the UK, meaning mapping all countries on one scale masked some of the more interesting patterns.]*

Country of Origin | Number/ Proportion of Refugees | Country of Origin | Number/ Proportion of Refugees |

Iran | 15,438 (13.7%) | Albania | 2,350 (2.1%) |

Eritrea | 13,886 (12.3%) | Nigeria | 2,090 (1.9%) |

Afghanistan | 9,938 (8.8%) | China | 1,676 (1.5%) |

Syria | 8,758 (7.8%) | Libya | 1,546 (1.4%) |

Zimbabwe | 8,513 (7.6%) | Gambia | 1,320 (1.2%) |

Sudan | 7,885 (7.0%) | Ethiopia | 1,297 (1.2%) |

Pakistan | 7,003 (6.2%) | Dem. Rep. of Congo | 1,205 (1.1%) |

Sri Lanka | 5,829 (5.2%) | Uganda | 1,143 (1.0%) |

Somalia | 5,328 (4.7%) | Bangladesh | 1,105 (1.0%) |

Iraq | 3,947 (3.5%) | Myanmar | 896 (0.8%) |

We can examine how these numbers compare to different countries in Europe. For example, we have plotted the same map, but for the number of refugees in Germany in Figure 2. Germany has over seven times as many refugees compared to the UK, with 831,264 refugees originating from 132 different countries. Whilst Germany also has a large proportion from the Middle East (which is unsurprising due to the recent conflicts in these areas), there are fewer African countries in the top 20, and Russia and some of the Eastern European countries feature. Whilst Syria makes up 7.8% of all refugees in the UK, Syrian refugees account for over 50% of all refugees in Germany which may be a reflection of a difference in immigration and refugee policy between the two countries since the Syrian conflict.

Country of Origin | Number/ Proportion of Refugees | Country of Origin | Number/ Proportion of Refugees |

Syria | 458,871 (55.2%) | Sri Lanka | 3,854 (0.5%) |

Iraq | 118,497 (14.3%) | Ethiopia | 3,573 (0.4%) |

Afghanistan | 82,233 (9.9%) | Azerbaijan | 2,864 (0.3%) |

Eritrea | 41,254 (5.0%) | Nigeria | 2,098 (0.3%) |

Iran | 32,920 (4.0%) | Armenia | 1,663 (0.2%) |

Turkey | 19,378 (2.3%) | China | 1,527 (0.2%) |

Somalia | 14,980 (1.8%) | Dem. Rep. of Congo | 1,493 (0.2%) |

Serbia and Kosovo | 9,235 (1.1%) | Egypt | 1,493 (0.2%) |

Russia | 5,995 (0.7%) | Bosnia and Herzegovina | 1,453 (0.2%) |

Pakistan | 5,773 (0.7%) | Vietnam | 1,287 (0.2%) |

As well as looking at a snapshot in time, we can also look at how the numbers and spatial patterns of refugees have changed over time since annual data are available from 1988. Below we have produced an animation of the top 20 countries of origin for refugees in the UK since 1988 where the colour of the country reflects the total number refugees currently residing in the UK to have come from that country.

This animation highlights how the numbers of refugees in the UK has changed over time. It seems that there was a steady increase until around 2005 followed by a reduction to the present number. We can confirm this pattern by plotting a time series of the overall number of refugees to have entered the UK in Figure 3 (note that this time series is plotted from 1951; we have total refugee population numbers between 1951 and 1987, but these are not broken down by country of origin). The longer time series also highlights that the number of refugees in the UK steadily decreased from the 1950s until the 1980s before rising once more. This rise in number and the geographic pattern of where refugees come from are likely to coincide with a number of recent conflicts such as those in Bosnia, Iraq, Afghanistan and Syria. We can see that, for example, from the early 2000s, the three countries where large numbers of refugees originate from are Iran, Afghanistan and Somalia. Other historical events may be reflected in these numbers; for example from 1992 to 1997 the Russian Federation is in the top-20 countries (the Soviet Union broke up in 1991).

In addition to looking at refugees, we have also analysed the number of asylum seekers present in the UK. At the start of 2017, the UK had 43,597 asylum-seekers from 109 different countries. Plotting the same maps as those we created for the refugee population in the UK, we found similar spatial patterns. A potentially more interesting set of data are the information that the UNHCR provides on the number of decisions made during the first half of 2017. The UK processed 45,191 asylum applications by mid-2017 of which 30% were recognised, 57% were rejected and the remaining were otherwise closed. In Figure 4 we map the proportion of decisions made by country and in Figure 5 we provide the success rate in each country. We can see in Figure 4 that the largest numbers of decisions were made for asylum seekers that originated from the Middle East, India, China and North East Africa. Countries with particularly high success rates include Syria and Tajikistan (both 70% and above). Countries with low success rates particularly given the number of asylum seekers present in the UK include India and China.

Finally, we have also produced a circular plot of the flow of asylum-seekers in 2017, also known as a chord diagram. In Figure 6 below we have plotted the population of 2017 asylum seekers, grouping each country into larger geographic regions. This plot shows the numbers of estimated asylum-seekers moving between the different regions, which is quantified by the width of the flow at both the region of origin and asylum (given in 1000s). The colour of the flow can be used to identify where asylum seekers are moving from and to, with there being a larger gap between the flow and the region that asylum seekers are leaving. We can use this plot to better understand the population size of asylum-seekers in each region and the composition of the asylum-seeker population in terms of where they have come from and where they have travelled to.

From this figure, we can conclude the following:

- Western Europe has the largest number of asylum-seekers in mid-2017 (approximately 1.2 million).
- The majority of asylum-seekers in Western Europe originated from the Middle East & North Africa, South Asia, Sub-Saharan Africa, the Former Soviet Union (FSU) and Central and Eastern Europe.
- North America has approximately 600,000 asylum-seekers, who mainly originated from Latin America & the Caribbean and Central Asia & China.
- There is evidence of internal movement within certain regions. For example, more than 250,000 asylum-seekers moved between countries in the Sub-Saharan Africa, more than 150,000 in the Middle East & North Africa and around 50,000 to 100,000 in Other Pacific Asia.

In this blog we have looked at different ways of visualising the populations of both refugees and asylum-seekers using the UNHCR annual statistics. We’ve used a combination of maps, time series plots and a chord diagram to demonstrate how many more insights you can gain from a data set when you use innovative visualisations. Not only have we discovered how recent conflicts have impacted the population of refugees within the UK, but we’ve been able to better understand which countries are more likely to have a successful asylum-seeker application as well as provide a whole World snapshot of asylum seekers in one simple plot.

The post Visualising Refugee and Asylum-Seeker Data appeared first on Select Statistical Consultants.

]]>The post The Select Team is Growing! appeared first on Select Statistical Consultants.

]]>Both have spent the summer working on their dissertations. Sally looked at data from an NHS clinical trial to explore a new approach for estimating quality-adjusted life years (a metric bringing together information on both a person’s life expectancy and their quality of life during that time), which would improve on the accuracy of the current industry standard techniques. Louise focused on machine learning techniques, investigating whether they could more accurately predict the risk of a patient developing Type 2 Diabetes compared to the standard statistical approach as well as examining the viability of using this approach in medical practice.

“We’re delighted to welcome both Sally and Louise to Select.” says Managing Director, Lynsey McColl, “Having both just finished their Master’s, they are clearly enthusiastic and excited about using their statistical know-how and coding expertise on Real-World client problems. I’m sure they’ll be a great asset to our consulting team.”

The post The Select Team is Growing! appeared first on Select Statistical Consultants.

]]>The post Cumulative Gains and Lift Curves: Measuring the Performance of a Marketing Campaign appeared first on Select Statistical Consultants.

]]>What returns will I get from running my marketing campaign?

In this context, we want to understand what benefit the predictive model can offer in predicting which customers will be responders versus non-responders in a new campaign (compared to targeting them at random). This can be achieved by examining the ** cumulative gains** and

In this blog, we describe the steps required to calculate the cumulative gains and lift associated with a predictive classification model.

Continuing with the direct marketing example, using the fitted model we can compare the observed outcomes from the historical marketing campaign, i.e., who responded and who did not, with the predicted probabilities of responding for each customer contacted in that campaign. (Note that, in practice, we would fit the model to a subset of our data and use this model to predict the probability of responding for each customer in a “hold-out” sample to get a more accurate assessment of how the model would perform for new customers.)

We first sort the customers by their predicted probabilities, in decreasing order from highest (closest to one) to lowest (closest to zero). Splitting the customers into equally sized segments, we create groups containing the same numbers of customers, for example, 10 decile groups each containing 10% of the customer base. So, those customers who we predict are most likely to respond are in decile group 1, the next most likely in decile group 2, and so on. Examining each of the decile groups, we can produce a decile summary, as shown in Table 1, summarising the numbers and proportions of customers and responders in each decile.

The historical data may show that overall, and therefore when mailing the customer base at random, approximately 5% of customers respond (506 out of 10,000 customers). So, if you mail 1,000 customers you expect to see around 50 responders. But, if we look at the response rates achieved in each of the decile groups in Table 1, we see that the top groups have a higher response rate than this, they are our best prospects.

Decile Group |
Predicted Probability Range |
Number of Custom-ers |
Cumulative No. of Customers |
Cumulative % of Customers |
Respond-ers |
Response Rate |
Cumulative No. of Respond-ers |
Cumulative % of Respond-ers |
Lift |

1 | 0.129-1.000 | 1,000 | 1,000 | 10.0% | 143 | 14.3% | 143 | 28.3% | 2.83 |

2 | 0.105-0.129 | 1,000 | 2,000 | 20.0% | 118 | 11.8% | 261 | 51.6% | 2.58 |

3 | 0.073-0.105 | 1,000 | 3,000 | 30.0% | 96 | 9.6% | 357 | 70.6% | 2.35 |

4 | 0.040-0.073 | 1,000 | 4,000 | 40.0% | 51 | 5.1% | 408 | 80.6% | 2.02 |

5 | 0.025-0.040 | 1,000 | 5,000 | 50.0% | 32 | 3.2% | 440 | 87.0% | 1.74 |

6 | 0.018-0.025 | 1,000 | 6,000 | 60.0% | 19 | 1.9% | 459 | 90.7% | 1.51 |

7 | 0.015-0.018 | 1,000 | 7,000 | 70.0% | 17 | 1.7% | 476 | 94.1% | 1.34 |

8 | 0.012-0.015 | 1,000 | 8,000 | 80.0% | 14 | 1.4% | 490 | 96.8% | 1.21 |

9 | 0.006-0.012 | 1,000 | 9,000 | 90.0% | 11 | 1.1% | 501 | 99.0% | 1.10 |

10 | 0.000-0.006 | 1,000 | 10,000 | 100.0% | 5 | 0.5% | 506 | 100.0% | 1.00 |

For example, we find that in decile group 1 the response rate was 14.3% (there were 143 responders out of the 1,000 customers), compared with the overall response rate of 5.1%. We can also visualise the results from the decile summary in a waterfall plot, as shown in Figure 1. This illustrates that all of the customers in decile groups 1, 2 and 3 have a higher response rate using the predictive model.

From the decile summary, we can also calculate the ** cumulative gains** provided by the model. We compare the cumulative percentage of customers who are responders with the cumulative percentage of customers contacted in the marketing campaign across the groups. This describes the ‘gain’ in targeting a given percentage of the total number of customers using the highest modelled probabilities of responding, rather than targeting them at random.

For example, the top 10% of customers with the highest predicted probabilities (decile 1), contain approximately 28.3% of the responders (143/506). So, rather than capturing 10% of the responders, we have found 28.3% of the responders having mailed only 10% of the customer base. Including a further 10% of customers (deciles 1 and 2), we find that the top 20% of customers contain approximately 51.6% of the responders. These figures can be displayed in a cumulative gains chart, as shown in Figure 2.

The dashed line in Figure 2 corresponds with “no gain”, i.e., what we would expect to achieve by contacting customers at random. The closer the cumulative gains line is to the top-left corner of the chart, the greater the gain; the higher the proportion of the responders that are reached for the lower proportion of customers contacted.

Depending on the costs associated with sending each piece of direct mail and the expected revenue from each responder, the cumulative gains chart can be used to decide upon the optimum number of customers to contact. There will likely be a *tipping point* at which we have reached a sufficiently high proportion of responders, and where the costs of contacting a greater proportion of customers are too great given the diminishing returns. This will generally correspond with a flattening-off of the cumulative gains curve, where further contacts (corresponding with additional deciles) are not expected to provide many additional responders. In practice, rather than grouping customers into deciles, a larger number of groups could be examined, allowing greater flexibility in the proportion of customers we might consider contacting.

We can also look at the **lift** achieved by targeting increasing percentages of the customer base, ordered by decreasing probability. The lift is simply the ratio of the percentage of responders reached to the percentage of customers contacted.

So, a lift of 1 is equivalent to no gain compared with contacting customers at random. Whereas a lift of 2, for example, corresponds with there being twice the number of responders reached compared with the number you’d expect by contacting the same number of customers at random. So, we may have only contacted 40% of the customers, but we may have reached 80% of the responders in the customer base. Therefore, we have doubled the number of responders reached by targeting this group compared with mailing a random sample of customers.

These figures can be displayed in a lift curve, as shown in Figure 3. Ideally, we want the lift curve to extend as high as possible into the top-left corner of the figure, indicating that we have a large lift associated with contacting a small proportion of customers.

In a previous blog post we discussed how ROC curves can be used in assessing how good a model is at classifying (i.e., predicting an outcome). As well as understanding the predictive accuracy of a model used for classification, it can also be helpful to understand what benefit is offered by the model compared with trying to identify an outcome without it.

Cumulative gains and lift curves are a simple and useful approach to understand what returns you are likely to get from running a marketing campaign and how many customers you should contact, based on targeting the most promising customers using a predictive model. These approaches could similarly be applied in the context of predicting which individuals will default on a personal loan in order to decide who could be offered a credit card, for example. In this case, the aim is to minimise the number of people likely to default on the loan, whilst maximising the number of credit cards offered to those who will not default. The predictive model in each case could be any appropriate statistical approach for generating a probability for a binary outcome, be that a logistic regression model, a random forest, or a neural network, for example.

The post Cumulative Gains and Lift Curves: Measuring the Performance of a Marketing Campaign appeared first on Select Statistical Consultants.

]]>The post Debunking the Myth of a North/South Divide in GCSE Performance appeared first on Select Statistical Consultants.

]]>Interestingly, his analysis, conducted on three annual cohorts of pupils, finds the same results as our single cohort school-level analysis reported in our recent blog; that differences between GCSE performance are not driven by a North/South divide and that factors affecting performance are, in fact, multifaceted and complex.

The article advises against just using high-level statistics, and highlights the importance of undertaking in-depth analyses. In our analyses of the available data, we fit a statistical model to the data and found deprivation to be a driver of performance; areas of high-levels of deprivation tended to have lower GCSE performance. Interestingly, Stephen Gorard’s research has delved into this a little deeper and shows that it is not just whether or not pupils are eligible for free school meals that affects attainment, but that a more important factor is the length of time that pupils have faced disadvantage. The article says that the current measure of deprivation, whether a child is eligible for free school meals or not, does not capture enough of the aspects of socio-economic deprivation or disadvantage.

Of course, in the education sector and other fields that use observational studies, whilst we include as many of the influential and relevant factors in any analysis, we must always be aware that analyses are often limited by the factors you can include, or more importantly, what you can’t include. Many analyses of student outcomes can’t, for example, take account of factors such as motivation, the effect of inspirational teachers, or home resources since these are not simple to measure.

Given past headlines stating the existence of a North/South educational divide, how do we know that an analysis has been conducted appropriately and whether or not to believe a headline? In our experience, clear and honest reporting is vital; detailing not only the results, but also the methods, any assumptions and limitations. By being clear about what is and isn’t included in your analyses and what it does and doesn’t tell you enables others to appropriately evaluate the evidence themselves.

The post Debunking the Myth of a North/South Divide in GCSE Performance appeared first on Select Statistical Consultants.

]]>