Case Studies

The Challenge

Customer Satisfaction Surveys (CSS’s) are a useful tool for identifying strengths and weaknesses in a commercial organisation providing goods or services. However, these types of surveys suffer from a self-selection bias that can negatively influence the quality and reliability of the results and can lead to an inaccurate evaluation of customer satisfaction. These errors occur because whether or not a survey is completed is not an entirely random event and the people who choose to respond to a survey may be systematically different from those who do not respond.

For example, if a survey is conducted by calling a random sample of publicly available telephone numbers during a weekday, it will not include the responses of people with unlisted telephone numbers or those people who are unable to answer the phone because they are at work. Instead, it is likely to include a disproportionate number of respondents who have traditional land-line telephone services, and who are at home during normal working hours (for example the elderly).

Research has shown that CSS surveys have a particularly low response rate of around 20%. This may be due to the reluctance of some to fill in questionnaires simply because of the number of CSS’s they are asked to fill in (by hotels, tour operators, course administrators, service providers etc.) or because they have a neutral attitude to the service (neither good nor bad).

The Approach

Since organisations often hold personal information about their customers this information, in conjunction with the knowledge of who undertook a survey and who did not, can be used to model the probability of response and then to correct the bias caused by self-selection. Examples of the type of personal information that is available include nationality, age and gender in addition to specific details about the level of service received (for example for an airline, this would be the cabin you most often flew in and whether you are a member of their executive club).

The statistical method most used to correct the self-selection bias is known as Propensity Score Matching. In this method, we use a statistical model to estimate the probability that an individual will respond to the survey (also known as the propensity score) based on their known personal information. It is then possible to match those individuals who have not responded to the questionnaire to those who have (using their propensity scores) and attribute the same level of satisfaction to the non-respondent.

The Value

Since there are often a large number of non-responses in CSS’s, the resulting estimate of the level of customer satisfaction can be very inaccurate. By employing statistical techniques that aim to eliminate self-selection errors, the reliability and accuracy of the results are improved. This means that businesses will better understand the needs of their customers and provide services of higher value.