Skip to content

Journal Article

Assessing the effect of data collection mode on measurement

Authors

Publication date

2010

Summary

Rising administration costs and falling response rates mean that many surveys that would previously have been carried out in one preferred mode of data collection are having to consider the use of mixed modes. For example, increasing numbers of surveys use a mix of modes, starting with a cheaper mode (such as telephone interviewing) which typically produces lower response rates, and following up non-respondents with face-to-face interviews. In order to decide about suitable data collection designs, survey practitioners must assess the trade-off between the potential advantages (for example in terms of financial costs and response rates) and disadvantages (for example in terms of data comparability) of mixing modes.
We discuss some of the challenges in evaluating the effects of using mixed modes on measurement and hence data comparability. The main argument is that it is very difficult to provide the information survey practitioners would need, about whether and to what extent using mixed modes would affect substantive conclusions. We briefly review theories about why different modes can lead to differences in survey responses. We then discuss the methods typically used to assess mode effects on measurement and then focus on some of the challenges. These include 1) the need to avoid confounding effects and what kinds of mode effects are actually identified, 2) the sensitivity of conclusions about the existence of mode effects to statistical methods used for the analysis of experimental mode comparison data, 3) the difficulty of assessing whether measurement differences matter in practice, and 4) the assessment of which mode provides better measurement. The main focus of the paper is on analysis methods. The points raised for discussion here arose in the context of the European Social Survey (ESS), which is conducting a programme of experimental research to inform the decision about whether to allow telephone interviewing in addition to face-to-face in its future rounds. We use some examples from the ESS experiments to illustrate how we tried to deal with these issues and to stimulate discussion. The paper concludes with an outlook of how the findings from the experimental studies are informing the decision process about whether or not to mix modes of data collection on the ESS and with general implications for mixed modes research.

Published in

International Statistical Review

Volume and page numbers

78 , 3 -20

DOI

http://dx.doi.org/10.1111/j.1751-5823.2010.00102.x

Links

http://serlib0.essex.ac.uk/record=b1641083~S5

Notes

56th Session of the International Statistical Institute, held 22-29 August 2007, Lisbon, Portugal; Albert Sloman Library Periodicals *restricted to Univ. Essex registered users*


Related publications

  1. Assessing the effect of data collection mode on measurement

    Annette Jäckle, Caroline Roberts, and Peter Lynn

  2. Assessing the Effect of Data Collection Mode on Measurement

    Annette Jäckle, Caroline Roberts, and Peter Lynn

#513520


Research home

Research home

News

Latest findings, new research

Publications search

Search all research by subject and author

Podcasts

Researchers discuss their findings and what they mean for society

Projects

Background and context, methods and data, aims and outputs

Events

Conferences, seminars and workshops

Survey methodology

Specialist research, practice and study

Taking the long view

ISER's annual report

Themes

Key research themes and areas of interest