An experimental project designed to test whether changes to questionnaire design will remove bias and improve answers to sensitive questions.
Background
Social surveys often ask questions on sensitive topics. Getting answers which are thoughtful and accurate can be difficult. Often, where sensitive topics are concerned respondents may give answers which are more positive than their true state (for example when asked how much alcohol they consume). The result is that the survey tends to underestimate socially unpleasant behaviour.
At the same time, recent research suggests that respondents do not always fully read through a list of response options, especially where respondents rush their answers. In addition respondents tend to align themselves with the top options in a vertical list, known as the primacy effect.
By inverting a vertical scale so that the most negative response is at the top of the scale respondents will be aware of the negative options and will consider these before moving down the scale. At the same time it is when respondents are highly motivated they take more care in completing survey questions and this leads to higher quality responses.
Project aims
The aims of this experimental project are:
- To test whether a top negative scale is less likely to obtain socially desirable
answers than questions with a top positive scale.
- To see whether motivating respondents to respond carefully increases the reporting of socially undesirable answers and to see whether this effect varies with the type of scale used
- To assess the effect of the above on respondents who spend less time on survey questions
Data was drawn from the result of a randomised experiment using 90 volunteers who were asked to complete a questionnaire using questions selected from the Understanding Society adult self-completion questionnaire. The experiment consisted of two related tests.
In the first test, half the respondents were asked to complete the questionnaire using a response scale which starts with positive category at the top (top positive condition). The other half will complete the same questionnaire using a reversed version of the scale (top-negative condition).
The second test is related to motivation. During the questionnaire, half the respondents were asked to make a special effort to take more care when responding to the survey questions.
The data was analysed at question level with account for clustering within respondents in Stata.
Team member
Survey Statistician - ISER - University of Essex
Olena's is a Survey Statistician at ISER. Her research interests include: survey methods, weighting, mixed mode surveys, non-response and measurement errors and the interaction among them in longitudinal surveys.
