When Voice Meets Touch: Conflict Analysis in Mobile Applications

Suwan Li; Lei Bu; Shangqing Liu; Guangdong Bai; Fuman Xie; Kai Chen; Chang Yue

doi:10.1109/TSE.2026.3656691

When Voice Meets Touch: Conflict Analysis in Mobile Applications

Suwan Li
, Lei Bu^*
, Shangqing Liu
, Guangdong Bai
, Fuman Xie
, Kai Chen
, Chang Yue

^*Corresponding author for this work

Department of Computer Science

Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review

8 Downloads (CityUHK Scholars)

Abstract

The recent advancement of the automatic speech recognition (ASR) contributes to the voice user interface (VUI), which is broadly embedded into mobile apps. The VUI implemented on modern mobile operating systems like Android naturally involves multiple threads, and brings new race issues and challenges in defining and identifying them. Specifically, when the GUI and VUI (GV) actions both access to the same resource simultanously, the data race named GV-race may occur. GV-race can lead to wrong behavior and even crashes. However, to the best of our knowledge, this problem has not been adequately studied.
In this paper, we present the first study of GV-race on Android apps. However, the involvement of the VUI complicates the concurrency model, affects the temporal relationship and brings state space explosion in global analysis. To tackle these challenges, we firstly define primitives and their happen-before rules to abstract GV interaction patterns. Using these primitives, we are able to characterize and formally define GV-race. We then develop Roma (GV-race detector on mobile apps) to detect both app-level and system-level GV-race automatically. Through static program analysis, Roma extracts GV related call graphs for each pair of conflicting GV actions to reduce the state space, and generates a universal GV interaction graph using our pre-defined primitives. It encodes happen-before constraints to formally specify the freeness of GV-race, so that the detection of GV-race can be reduced to constraint solving with SMT solvers. We apply Roma to analyze 266 apps. Roma finds 52 apps with app-level GV-race and 56 apps with system-level GV-race. We confirm that 101 apps are true positives.
© 2026 IEEE.

Original language	English
Journal	IEEE Transactions on Software Engineering
DOIs	https://doi.org/10.1109/TSE.2026.3656691
Publication status	Online published - 26 Jan 2026

Funding

We are grateful for the constructive feedback of all the anonymous reviewers to improve this manuscript. This work was supported by the National Natural Science Foundation of China (No. 62232008, 62172200, 62572256, U24A20236). In addition, the authors from Nanjing University are supported in part by the National Key Research and Development Program of China (2024YFB2505604). The author from City University of Hong Kong is partly supported by CityUHK’s Start-up Grant. The IIE authors are supported in part by the CAS Project for Young Scientists in Basic Research (No. YSBR-118).

Research Keywords

Data Race
Graphical User Interface
Mobile Applications
Voice User Interface

Publisher's Copyright Statement

COPYRIGHT TERMS OF DEPOSITED POSTPRINT FILE: © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Li, S., Bu, L., Liu, S., Bai, G., Xie, F., Chen, K., & Yue, C. (2026). When Voice Meets Touch: Conflict Analysis in Mobile Applications. IEEE Transactions on Software Engineering. Advance online publication. https://doi.org/10.1109/TSE.2026.3656691

Access Output

10.1109/TSE.2026.3656691

428661928.pdfPost-print, 1.41 MB

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@article{4703538b9fca4aa0bf6f2d646a6699b5,

title = "When Voice Meets Touch: Conflict Analysis in Mobile Applications",

abstract = "The recent advancement of the automatic speech recognition (ASR) contributes to the voice user interface (VUI), which is broadly embedded into mobile apps. The VUI implemented on modern mobile operating systems like Android naturally involves multiple threads, and brings new race issues and challenges in defining and identifying them. Specifically, when the GUI and VUI (GV) actions both access to the same resource simultanously, the data race named GV-race may occur. GV-race can lead to wrong behavior and even crashes. However, to the best of our knowledge, this problem has not been adequately studied. In this paper, we present the first study of GV-race on Android apps. However, the involvement of the VUI complicates the concurrency model, affects the temporal relationship and brings state space explosion in global analysis. To tackle these challenges, we firstly define primitives and their happen-before rules to abstract GV interaction patterns. Using these primitives, we are able to characterize and formally define GV-race. We then develop Roma (GV-race detector on mobile apps) to detect both app-level and system-level GV-race automatically. Through static program analysis, Roma extracts GV related call graphs for each pair of conflicting GV actions to reduce the state space, and generates a universal GV interaction graph using our pre-defined primitives. It encodes happen-before constraints to formally specify the freeness of GV-race, so that the detection of GV-race can be reduced to constraint solving with SMT solvers. We apply Roma to analyze 266 apps. Roma finds 52 apps with app-level GV-race and 56 apps with system-level GV-race. We confirm that 101 apps are true positives. {\textcopyright} 2026 IEEE.",

keywords = "Data Race, Graphical User Interface, Mobile Applications, Voice User Interface",

author = "Suwan Li and Lei Bu and Shangqing Liu and Guangdong Bai and Fuman Xie and Kai Chen and Chang Yue",

year = "2026",

month = jan,

day = "26",

doi = "10.1109/TSE.2026.3656691",

language = "English",

journal = "IEEE Transactions on Software Engineering",

issn = "0098-5589",

publisher = "IEEE",

}

TY - JOUR

T1 - When Voice Meets Touch

T2 - Conflict Analysis in Mobile Applications

AU - Li, Suwan

AU - Bu, Lei

AU - Liu, Shangqing

AU - Bai, Guangdong

AU - Xie, Fuman

AU - Chen, Kai

AU - Yue, Chang

PY - 2026/1/26

Y1 - 2026/1/26

N2 - The recent advancement of the automatic speech recognition (ASR) contributes to the voice user interface (VUI), which is broadly embedded into mobile apps. The VUI implemented on modern mobile operating systems like Android naturally involves multiple threads, and brings new race issues and challenges in defining and identifying them. Specifically, when the GUI and VUI (GV) actions both access to the same resource simultanously, the data race named GV-race may occur. GV-race can lead to wrong behavior and even crashes. However, to the best of our knowledge, this problem has not been adequately studied. In this paper, we present the first study of GV-race on Android apps. However, the involvement of the VUI complicates the concurrency model, affects the temporal relationship and brings state space explosion in global analysis. To tackle these challenges, we firstly define primitives and their happen-before rules to abstract GV interaction patterns. Using these primitives, we are able to characterize and formally define GV-race. We then develop Roma (GV-race detector on mobile apps) to detect both app-level and system-level GV-race automatically. Through static program analysis, Roma extracts GV related call graphs for each pair of conflicting GV actions to reduce the state space, and generates a universal GV interaction graph using our pre-defined primitives. It encodes happen-before constraints to formally specify the freeness of GV-race, so that the detection of GV-race can be reduced to constraint solving with SMT solvers. We apply Roma to analyze 266 apps. Roma finds 52 apps with app-level GV-race and 56 apps with system-level GV-race. We confirm that 101 apps are true positives. © 2026 IEEE.

AB - The recent advancement of the automatic speech recognition (ASR) contributes to the voice user interface (VUI), which is broadly embedded into mobile apps. The VUI implemented on modern mobile operating systems like Android naturally involves multiple threads, and brings new race issues and challenges in defining and identifying them. Specifically, when the GUI and VUI (GV) actions both access to the same resource simultanously, the data race named GV-race may occur. GV-race can lead to wrong behavior and even crashes. However, to the best of our knowledge, this problem has not been adequately studied. In this paper, we present the first study of GV-race on Android apps. However, the involvement of the VUI complicates the concurrency model, affects the temporal relationship and brings state space explosion in global analysis. To tackle these challenges, we firstly define primitives and their happen-before rules to abstract GV interaction patterns. Using these primitives, we are able to characterize and formally define GV-race. We then develop Roma (GV-race detector on mobile apps) to detect both app-level and system-level GV-race automatically. Through static program analysis, Roma extracts GV related call graphs for each pair of conflicting GV actions to reduce the state space, and generates a universal GV interaction graph using our pre-defined primitives. It encodes happen-before constraints to formally specify the freeness of GV-race, so that the detection of GV-race can be reduced to constraint solving with SMT solvers. We apply Roma to analyze 266 apps. Roma finds 52 apps with app-level GV-race and 56 apps with system-level GV-race. We confirm that 101 apps are true positives. © 2026 IEEE.

KW - Data Race

KW - Graphical User Interface

KW - Mobile Applications

KW - Voice User Interface

UR - https://www.scopus.com/pages/publications/105029061941

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-105029061941&origin=recordpage

U2 - 10.1109/TSE.2026.3656691

DO - 10.1109/TSE.2026.3656691

M3 - RGC 21 - Publication in refereed journal

SN - 0098-5589

JO - IEEE Transactions on Software Engineering

JF - IEEE Transactions on Software Engineering

ER -

When Voice Meets Touch: Conflict Analysis in Mobile Applications

Abstract

Funding

Research Keywords

Publisher's Copyright Statement

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this