The accuracy of using paradata to detect and understand interviewer question-reading deviations

Publication type

Conference Paper

Series

Understanding Society Scientific Conference 2017, 11-13 July 2017, University of Essex, Colchester, UK

Author

Publication date

July 13, 2017

Summary:

Deviations from reading survey questions exactly as worded may change the validity of the questions, thus increasing measurement error. Hence, organizations train their interviewers to read questions verbatim. To ensure interviewers are reading questions verbatim, organizations rely on interview recordings. However, this takes a significant amount of resources. Therefore, some organizations are using paradata generated by the survey software, specifically timestamps, to try to detect when interviewers deviate from reading the question verbatim. However, there is no established method on how to use timestamps to detect question-reading deviations and little is known about the level of accuracy for the different methods currently used. This study evaluates the current methods used for detecting question-reading deviations using interview recordings and paradata from Wave 3 of the Understanding Society Innovation Panel. Using interview recordings allows a direct comparison of the different detection methods to how the interviewers actually administered the question and thus measure the accuracy of each detection method. Deviations are also coded for the extent (i.e., minor or major) and type of deviation. This analysis will give better insight on the scope and types of deviations interviewers are engaging in and practical guidance on how to best detect question-reading deviations.

Subjects

Link

https://www.understandingsociety.ac.uk/scientific-conference-2017/papers/136

#524368

News

Latest findings, new research

Publications search

Search all research by subject and author

Podcasts

Researchers discuss their findings and what they mean for society

Projects

Background and context, methods and data, aims and outputs

Events

Conferences, seminars and workshops

Survey methodology

Specialist research, practice and study

Taking the long view

ISER's annual report

Themes

Key research themes and areas of interest