http://www.iser.essex.ac.uk/publications/working-papers/iser/2008-39

The way you ask a question often affects the answer you get. This is just as true in survey research as it is in ordinary life. In recent years there has been a shift of interest in the policy debate from financial measures of well-being (income, wealth, etc.) to broader concepts of welfare (happiness, satisfaction, etc.) This is undoubtedly a good thing but it raises the question of how best to measure illdefined concepts like satisfaction. The usual method in social science research is to use largescale surveys, with questionnaires containing direct questions on satisfaction with various aspects of life, and work. Survey participants are then asked to locate their degree of satisfaction on a numerical scale from (say) 1 to 7. The British Household Panel Survey (BHPS) has been widely used for research on life and job satisfaction and, since its inception in 1991, there have been some significant changes in the way the satisfaction questions have been asked. In this paper, we ask whether the answers that people give to these questions have been influenced significantly by the way the questions are asked. We focus on two features of the BHPS. First, in 1992, there was an apparently minor change to the questions, which involved explanatory textual labels being added to more of the response categories numbered 1-7. Consequently, from 1992 onwards, interviewees were given a clearer explanation of what the response scale means. Second, from 1996, a self-completion paper questionnaire was added, so that we know both the answer that each individual gave in open interview and the much more private answer given in the selfcompletion questionnaire. There are six main conclusions: (1) The apparently minor re-design of the satisfaction questions in 1992 caused a very large change in the pattern of answers, particularly for women, who seem to respond better when the levels of satisfaction are given verbal as well as numerical meaning. (2) Oral interviews conducted by an interviewer tend to produce more positive reports of satisfaction than private self-completion questionnaires – the “let’s put on a good show for the interviewer” effect. (3) When children are present during the interview, adult interviewees tend to give still more positive responses – the “not in front of the children” effect. (4) The presence of the interviewee’s partner during the interview tends to depress the level of reported satisfaction – the “don’t show your partner how satisfied you are” effect, which we speculate may have something to do with the desire to maintain a strong bargaining position within the relationship. (5) These distortions of survey responses are important for research findings. For example, it is often reported by researchers that women’s job satisfaction is little affected by their hours and rate of pay. We cast doubt on this finding. When information from the more private selfcompletion questionnaire is used for the analysis, there is strong evidence that, like men, women’s degree of job satisfaction is influenced by both. (6) In future surveys asking about subjective well-being, happiness or satisfaction, it is important where possible to ask these questions by a suitably ‘private’ mode rather than by open oral interview.

Surveys differ in the way they measure satisfaction and happiness, so comparative research findings are vulnerable to distortion by survey design differences. We examine this using the British Household Panel Survey, exploiting its changes in question design and parallel use of different interview modes. We find significant biases in econometric results, particularly for gender differences in attitudes to the wage and hours of work. Results suggest that the common empirical finding that women care less than men about their wage and more about their hours may be an artifact of survey design rather than a real behavioural difference. Working Paper 2008-39 2008-12-01 Gabriella Conti Department of Economics, University of Chicago http://home.uchicago.edu/~gconti/ Stephen Pudney Institute for Social and Economic Research spudney@essex.ac.uk http://www.iser.essex.ac.uk/home/spudney