Moving online with Understanding Society
We expect to accomplish everyday tasks online, from buying a pint of milk to binge-watching TV seasons, booking a gym class or a long-haul flight and hotel accommodation on the other side of the world. Taking part in surveys is also increasingly done online, with numerous opt-in panels offering points and prizes for completing questionnaires. However, online surveys which use high-quality probability sampling are rare. The recent British Polling Council report on the 2015 election indicated that the sampling and fieldwork procedures on the political polls were one of the reasons for the ‘failure of the polls’. Well-drawn probability samples, and fieldwork strategies which motivate interviewers to make multiple calls to make contact with sample members, is the key to high quality survey research.
This does not mean that established social surveys should ignore the potential of the web. Online surveys of an established sample have the potential to make the work of data collection more cost-efficient. By making it easier for some people to participate, we can then focus resources on those who may need more efforts. Moving a face-to-face survey onto the web is not without risks, however. Understanding Society is in a fortunate position, in that the study includes an “Innovation Panel” (IP) of around 1,500 households. This IP is used to test different methods of data collection, through experimentation with question wording, responses or how the questions are asked. We have been using the IP for the past few years to test the effect of using different methods of collecting data on response and data quality.
We are now taking what we have learned using the IP and starting to implement it on the main Understanding Society sample. To begin with, at Wave 7, we invited adults who were in households that had not participated at the previous wave to complete their survey online. After a couple of weeks (and a reminder or two), interviewers started calling on those who had not yet completed online, to give them an opportunity to take part in a face-to-face interview. These people had not participated at the previous wave when an interviewer had called, our experimental testing showed that they were more likely to ‘return’ to the study if they could complete the survey online.
From Wave 8 (which started in January 2016), we are extending the invitation to complete online to adults in some of the households that took part at Wave 7 with an interviewer.
Using the IP, our researchers have analysed what characteristics of a household are associated with completing the survey online. Using this information, we are inviting the households in the main sample who are more likely to participate online to do the survey on the web. Once again, those that do not complete their interview in the first few weeks are issued to interviewers so they can be contacted face-to-face. At Wave 8, 40% of households are invited to take part on-line, with this proportion scheduled to increase over the next few waves.
We are also using our experience with the IP to introduce an “adaptive design” to the main sample of Understanding Society. That is, we are introducing changes in the standard fieldwork design which we think will increase the proportion of households that complete online, where all adults in the household use the web to answer their survey. We are introducing these changes experimentally, with the design which proves most effective then being adopted for future months. So far, we’ve introduced an additional £10 token of appreciation if the sample member completes online in the first couple of weeks – before the household is issued to an interviewer. This adaptation doubled the proportion of household that completed in the first two weeks of fieldwork, from 18% to 36%. We’ve also tested the trigger for this additional bonus; completion by the adult sample member, or the whole household. Early results indicate that the individual-completion trigger is most effective at increasing household completion and so this has been adopted from July onwards. From July to September we will be experimenting with the length of the online period and use of reminders. At each stage, experienced researchers examine the results and propose changes in the data collection strategy which seek to maximise survey quality within a tight budget.
Understanding Society, leads the world when it comes to innovating with this type of longitudinal household survey. Using the Innovation Panel to ensure that the design decisions we make are backed by evidence, we’re taking a leading role in survey design. There is much interest from the US and across Europe and even as far as Australia, in the experience of Understanding Society in moving to a mixed mode design. We are all seeking the goal of collecting high quality, robust, social science data in a way which is cost efficient and sustainable over the long-term.