Publication type
Survey Futures Working Paper Series
Series Number
25
Series
Survey Futures Working Paper Series
Authors
Publication date
July 30, 2026
Summary:
Occupation is a key indicator in social surveys, but office coding of open-text job descriptions is costly and time-consuming. Respondent-led look-up tools could offer an efficient alternative by assigning codes during data collection, yet evidence on their performance remains limited. We evaluated the feasibility, validity, and reliability of a look-up tool using a two-wave randomised mode experiment with 1,195 adults in England who remained in the same job across waves. Participants completed surveys online, by video interview, or in person, selected an occupation code using the tool, and provided open-text descriptions for independent office coding. The tool achieved an 86.8% coding success rate, and 97.2% of respondents reported that their selected code described their job fairly or very well. However, four-digit agreement with office coding was 64.7%, compared with 95.1% agreement between office coders, and cross wave consistency was lower for the look-up tool than for office coding (60% vs. 73.7%). Online respondents reported lower subjective accuracy than those in interviewer assisted modes. Longer inputs were associated with lower subjective accuracy and agreement, while lexical, but not semantic, similarity predicted consistency. Overall, look-up coding is feasible but does not yet match expert office coding in validity or reliability.
Subjects
Paper download#589119