Survey non-response biases
Collecting survey data is not easy. One issue is that the people completing the survey (survey respondents) may not be representative of the population it is meant to study in terms of their characteristics. If such characteristics are correlated with survey answers, non-response biases can occur, where survey estimates differ from those that would be obtained if all population members responded. These can lead to false conclusions being drawn from the survey data.
The possibility of false conclusions being drawn from survey data is a concern, so survey designers try to minimise biases. Efforts are made to ensure that the people asked to complete the survey (the survey sample) are representative of the study population, and that responses are obtained from all population subgroups in terms of characteristics. In addition, after data collection methods are used to correct for remaining biases. The most widely used of these methods, and the topic of this explainer, is the production of survey weights.
What do survey weights do, and how are they calculated?
What survey weights do is balance the influence of survey answers from members of different subgroups in analyses of the data so that they reflect the population distribution of subgroups. These weights have several components. The first of these is the design weight, which aims to correct biases arising during sampling. If all population members have the same chance of being sampled, all sample members have a design weight of 1. However, this may not be the case. Sometimes for instance, certain groups are sampled more often than others, to ensure data from enough members is collected to enable analyses. If, to give a very simple example (surveys often make more complicated adjustments using more characteristics, though the principle remains the same), people in one region are twice as likely to be sampled as people in other regions, the weights for them would be 1 / 2 = 0.5, whereas the weights for people in the other regions would be 1 / 0.5 = 2. Note that throughout this explainer the numbers used are also chosen specifically to simplify our explanations.
The second component is the non-response adjustment, which aims to correct biases arising from differential non-response in population subgroups. This is calculated in a similar way to the design weight. To stay with our region example, if people in the less sampled regions are twice as likely to respond as people in the more sampled region, the adjustment for them would be 1 / 2 = 0.5, and the adjustment for people in the more sampled region would be 1 / 0.5 = 2 (again, surveys often make more complex adjustments). These adjustments are then multiplied by the design weights to produce the non-response weights i.e. for those in the more sampled – low response region the weight would be 0.5 * 2 = 1, and for those in less sampled – high response regions it would be 2 * 0.5 = 1.
The third component is the poststratification adjustment. This adjusts the non-response weighted survey subgroup totals to estimates of population subgroup totals, for example from a census. To explain this adjustment, we’ll use an example with only two subgroups: males and females (once more, surveys often make more complex adjustments). Assume that there are 1000 people of each sex in the population, and that we have responses from 250 males and 500 females. The poststratification weight for each person in a subgroup is calculated as the population subgroup total divided by the non-response weighted survey subgroup total i.e. given the non-response weights from the previous paragraph, for males it is:
1000 / (1 * 250) = 4000,
and for females, it is:
1000 / (1 * 500) = 2000.
To produce the final weights, these weights are then rescaled to have a mean of 1 i.e. each is divided by the overall weight mean. In this case, the overall weight mean is:
((4000 * (1 * 250)) + (2000 * (1 * 500))) / ((1 * 250) + (1 * 500)) = 2666.667.
Hence, for each male, the final poststratification weight is:
4000 / 2666.667 = 1.5,
and for each female it is:
2000 / 2666.667 = 0.75.
At the Institute for Social and Economic Research, the long running Understanding Society – the UK Household Longitudinal Study (UKHLS) is produced. The information it collects on the UK population is used by many researchers and policy makers. The team responsible for the survey, of which the author is a member, release a range of weights with the survey datasets. Each has a different purpose: for example, separate weights for respondents to the adult and youth questionnaires are released, along with weights for all respondents to each wave of data collection (participants are surveyed annually) and also weights for only respondents to the wave and all waves previous. All though, have the same aim: to ensure that with their use respondents reflect (the relevant part of) the population. All also include the three weight components described above in their calculation, though exact procedures used are weight specific and more complex. By using these weights in analyses, researchers and policy makers can ensure that conclusions drawn from the UKHLS data are free from biases.
Jamie Moore is a Research Fellow at the Institute for Social and Economic Research