Gauging Perceived Political Sensitivity in Political Opinion Polls: A Multi-Measure Design

Political scientists often seek answers to research questions using public opinion polls. In Hong Kong, the introduction of the national security law may have fundamentally undermined opinion surveys as a tool to gauge public opinion if ordinary citizens become unwilling to provide truthful answers to questions deemed politically sensitive for fear of unwanted legal repercussions. The falsification of political preferences will adversely affect the validity of survey results if not defeat the purpose of the survey. Although social scientists have developed tools to detect preference falsification, researchers often apply them to survey questions known (or assumed) to be sensitive. In other words, these tools are often used to confirm researchers’ prior beliefs. Problems arise if their prior beliefs are flawed, such that they overlook questions vulnerable to false responses, or they lack any strong prior belief. These are practical conundrums that haunt political scientists who run surveys in Hong Kong and beyond. The purpose of this study is threefold. First and foremost, unlike extant studies, which focus on testing the sensitivity of a few questions that are already known to be sensitive, we develop a measure of political sensitivity over a large set of questions commonly asked by political science researchers. Second, we test if experts and non-experts have different perceptions of political sensitivity. Third, we aim to identify behavioral correlates of politically sensitive questions. For instance, would it take longer to answer politically sensitive questions? Would respondents be more likely to select “don’t know?” More importantly, are sensitive questions prone to preference falsification? With these correlates, we can model political sensitivity based on a robust multi-measure design. The proposed study involves three phases. In Phase 1, we will compile a comprehensive questionnaire by drawing questions from well-established political surveys. In Phase 2, we will recruit experts and non-experts to rate the political sensitivity of each question. In Phase 3, we will recruit another group of respondents to answer all these questions. Experimental methods (e.g., list experiment) will be employed, and paradata (e.g., response time) will be collected. We will then use the expert and crowd-sourced sensitivity measures obtained from Phase 2 to predict outcomes (e.g., response time) collected in Phase 3 so that we can find out the impacts of perceived political sensitivity on survey responses, behavioral indicators of preference falsification, and the gap between experts’ and non-experts’ perceptions of political sensitivity.