Survey data is collected using a variety of methods (modes). Face to face (F2F) or telephone interviews may be conducted, questionnaires may be mailed for completion, or they may be completable online using a computer, tablet or smartphone (web mode). Each mode has its advantages and disadvantages (mode effects), as we shall describe in this explainer before outlining how they are used to collect data in current surveys.
Mode effects
Interview mode can impact on various aspects of a survey. The first is survey coverage. Participants must first be contacted. F2F and mail modes can make use of address lists, which are mostly complete, to select sample members. Telephone mode though, requires use of a directory to select samples. Some people do not have a telephone, or, especially if they only have a mobile phone, their number may not be listed, so may be excluded from sample selection. In these circumstances, if certain population subgroups are more likely to be excluded, coverage errors can occur where the survey sample does not reflect the population being studied. These errors can also occur with web mode, due to the non-existence of nationwide lists of email addresses and some population subgroups such as older adults being less likely to use the internet.
The second aspect of a survey that can be impacted on by interview mode is sample member non-response. This is an issue because it reduces dataset size. In addition, if respondent characteristics differ from those of non-respondents and those characteristics are correlated with survey answers, it can cause non-response bias, where survey estimates differ from those that would be obtained if information from all population members was collected. This can lead to inaccurate conclusions being drawn from survey data. Response rates tend to be highest for F2F and telephone modes, then web and finally mail, though patterns can change: for example, telephone response rates have declined in recent years because people tend not to answer mobile phone calls from numbers they do not recognise. Patterns of response across different population subgroups can also vary. For instance, younger adults tend to have lower response rates compared to other subgroups to F2F, telephone and mail modes, but higher rates to web mode, whereas older adults tend to respond less often by web.
The third aspect of a survey that can be impacted on by interview mode is measurement quality. Participant answers to questions can be mode dependent. This can include whether questions are answered at all. Often, fewer are answered with mail and web modes (known as item non-response) than with F2F or telephone modes, because with the latter two modes interviewers can probe for answers and / or provide extra information to participants who are unsure of how to reply. However, it can also include the answers given by participants (measurement effects). This can again be due to mail or web participants being unable to obtain more information about questions, but also some questions can be affected by social desirability bias, where participants over-report desirable behaviours and under-report undesirable behaviours to F2F or telephone interviewers, but give more accurate answers by mail or web. Examples of the latter questions include those on voting, smoking and sexual behaviour
The fourth aspect of a survey that can be impacted on by interview mode is cost. F2F and telephone modes are most costly, due to the need to train and pay interviewers and in the case of F2F for them to travel to the participant’s home. Mail mode is next, due to the cost of printing and posting questionnaires to participants and also paying for them to be posted back. Web mode is least costly, although costs are still non-negligible since questionnaires must be specifically designed to be answered online.
Mode use in current surveys
Traditionally, F2F, telephone or mail modes have been used to collect survey data. Web mode has only been used for the last 15 years or so. Currently, many surveys use more than one mode (mixed mode surveys). Research has shown that these survey designs reduce coverage errors and non-response biases compared to single mode designs. Participants may be allowed to choose between modes, or may be offered one mode first with non-respondents then followed up in another mode (sequential mixed mode surveys). Of the latter, one increasingly popular form are web-first sequential mixed mode surveys, in which participants are first offered web mode then non-respondents followed up by more costly F2F, telephone or mail modes. Research has shown that these designs can reduce survey costs while retaining the benefits of mixed modes in terms of reducing coverage errors and non-response biases.
An issue with mixed mode surveys though, is measurement quality. As mentioned earlier, participant answers can be subject to measurement effects. If the nature of such dependencies is known, for example under- or over-reporting, in single mode surveys it can be accounted for when interpreting the results of analyses. However, in mixed mode surveys, if participant choice of mode depends on their characteristics and such characteristics are correlated with their answers to questions, impacts on analysis results can be more complex and difficult to predict, potentially leading to false conclusions being drawn. Ways of identifying when this occurs and correcting for it so that false conclusions can be prevented is an active field of research among survey methodologists.
At the Institute for Social and Economic Research, the long running Understanding Society – the UK Household Longitudinal Study (UKHLS) is produced. The information it collects on the UK population is used by many researchers and policy makers. Currently, the survey offers F2F, telephone and (increasingly) web modes to participants, mostly sequentially. To address the issue of possible measurement effects, the survey team (of which the author of this explainer is a member) recommend that researchers compare the results of their analyses to similar analyses carried out on a ringfenced 20% subset of the survey sample that are only interviewed F2F. That way, any findings that are due to measurement effects can be identified, preventing false conclusions from being drawn from the data.