VITAS: Guided Model-based VUI Testing of VPA Apps

Suwan Li; Lei Bu; Guangdong Bai; Zhixiu Guo; Kai Chen; Hanlin Wei

doi:10.1145/3551349.3556957

VITAS: Guided Model-based VUI Testing of VPA Apps

Suwan Li, Lei Bu^*, Guangdong Bai, Zhixiu Guo, Kai Chen, Hanlin Wei

^*Corresponding author for this work

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

8 Citations (Scopus)

Abstract

Virtual personal assistant (VPA) services, e.g. Amazon Alexa and Google Assistant, are becoming increasingly popular recently. Users interact with them through voice-based apps, e.g. Amazon Alexa skills and Google Assistant actions. Unlike the desktop and mobile apps which have visible and intuitive graphical user interface (GUI) to facilitate interaction, VPA apps convey information purely verbally through the voice user interface (VUI), which is known to be limited in its invisibility, single mode and high demand of user attention. This may lead to various problems on the usability and correctness of VPA apps.
In this work, we propose a model-based framework named Vitas to handle VUI testing of VPA apps. Vitas interacts with the app VUI, and during the testing process, it retrieves semantic information from voice feedbacks by natural language processing. It incrementally constructs the finite state machine (FSM) model of the app with a weighted exploration strategy guided by key factors such as the coverage of app functionality. We conduct a large-scale testing on 41,581 VPA apps (i.e., skills) of Amazon Alexa, the most popular VPA service, and find that 51.29% of them have weaknesses. They largely suffer from problems such as unexpected exit/start, privacy violation and so on. Our work reveals the immaturity of the VUI designs and implementations in VPA apps, and sheds light on the improvement of several crucial aspects of VPA apps. © 2022 Association for Computing Machinery.

Original language	English
Title of host publication	ASE '22
Subtitle of host publication	Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering
Publisher	Association for Computing Machinery
Number of pages	12
ISBN (Print)	978-1-4503-9475-8
DOIs	https://doi.org/10.1145/3551349.3556957
Publication status	Published - Oct 2022
Externally published	Yes
Event	37th IEEE/ACM International Conference on Automated Software Engineering (ASE 2022) - Oakland Center, Rochester, United States Duration: 10 Oct 2022 → 14 Oct 2022 https://conf.researchr.org/home/ase-2022

Conference

Conference	37th IEEE/ACM International Conference on Automated Software Engineering (ASE 2022)
Abbreviated title	ASE '22
Place	United States
City	Rochester
Period	10/10/22 → 14/10/22
Internet address	https://conf.researchr.org/home/ase-2022

Funding

We are grateful for the constructive feedback of all the anonymous reviewers to improve this manuscript. The authors from Nanjing University are supported in part by the Leading-edge Technology Program of Jiangsu Natural Science Foundation (No. BK20202001), the National Natural Science Foundation of China (No. 62172200 and No.62032010) and the Fundamental Research Funds for the Central Universities (No. 020214380094). The authors from IIE, Chinese Academy of Science, are supported in part by NSFC U1836211, Beijing Natural Science Foundation (No.M22004), the Anhui Department of Science and Technology under Grant 202103a05020009, Youth Innovation Promotion Association CAS, Beijing Academy of Artificial Intelligence (BAAI). The authors from University of Queensland are supported in part by UQ’s NSRSG grant and Oracle Labs Australia under the CR grant.

Access Output

10.1145/3551349.3556957

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@inproceedings{687c1c322d564a6dbc0bba86ced254de,

title = "VITAS: Guided Model-based VUI Testing of VPA Apps",

abstract = "Virtual personal assistant (VPA) services, e.g. Amazon Alexa and Google Assistant, are becoming increasingly popular recently. Users interact with them through voice-based apps, e.g. Amazon Alexa skills and Google Assistant actions. Unlike the desktop and mobile apps which have visible and intuitive graphical user interface (GUI) to facilitate interaction, VPA apps convey information purely verbally through the voice user interface (VUI), which is known to be limited in its invisibility, single mode and high demand of user attention. This may lead to various problems on the usability and correctness of VPA apps. In this work, we propose a model-based framework named Vitas to handle VUI testing of VPA apps. Vitas interacts with the app VUI, and during the testing process, it retrieves semantic information from voice feedbacks by natural language processing. It incrementally constructs the finite state machine (FSM) model of the app with a weighted exploration strategy guided by key factors such as the coverage of app functionality. We conduct a large-scale testing on 41,581 VPA apps (i.e., skills) of Amazon Alexa, the most popular VPA service, and find that 51.29% of them have weaknesses. They largely suffer from problems such as unexpected exit/start, privacy violation and so on. Our work reveals the immaturity of the VUI designs and implementations in VPA apps, and sheds light on the improvement of several crucial aspects of VPA apps. {\textcopyright} 2022 Association for Computing Machinery.",

author = "Suwan Li and Lei Bu and Guangdong Bai and Zhixiu Guo and Kai Chen and Hanlin Wei",

year = "2022",

month = oct,

doi = "10.1145/3551349.3556957",

language = "English",

isbn = "978-1-4503-9475-8",

booktitle = "ASE '22",

publisher = "Association for Computing Machinery",

address = "United States",

note = "37th IEEE/ACM International Conference on Automated Software Engineering (ASE 2022), ASE '22 ; Conference date: 10-10-2022 Through 14-10-2022",

url = "https://conf.researchr.org/home/ase-2022",

}

Li, S, Bu, L, Bai, G, Guo, Z, Chen, K & Wei, H 2022, VITAS: Guided Model-based VUI Testing of VPA Apps. in ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering., 115, Association for Computing Machinery, 37th IEEE/ACM International Conference on Automated Software Engineering (ASE 2022), Rochester, Michigan, United States, 10/10/22. https://doi.org/10.1145/3551349.3556957

VITAS: Guided Model-based VUI Testing of VPA Apps. / Li, Suwan; Bu, Lei; Bai, Guangdong et al.
ASE '22: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. Association for Computing Machinery, 2022. 115.

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - VITAS

T2 - 37th IEEE/ACM International Conference on Automated Software Engineering (ASE 2022)

AU - Li, Suwan

AU - Bu, Lei

AU - Bai, Guangdong

AU - Guo, Zhixiu

AU - Chen, Kai

AU - Wei, Hanlin

PY - 2022/10

Y1 - 2022/10

N2 - Virtual personal assistant (VPA) services, e.g. Amazon Alexa and Google Assistant, are becoming increasingly popular recently. Users interact with them through voice-based apps, e.g. Amazon Alexa skills and Google Assistant actions. Unlike the desktop and mobile apps which have visible and intuitive graphical user interface (GUI) to facilitate interaction, VPA apps convey information purely verbally through the voice user interface (VUI), which is known to be limited in its invisibility, single mode and high demand of user attention. This may lead to various problems on the usability and correctness of VPA apps. In this work, we propose a model-based framework named Vitas to handle VUI testing of VPA apps. Vitas interacts with the app VUI, and during the testing process, it retrieves semantic information from voice feedbacks by natural language processing. It incrementally constructs the finite state machine (FSM) model of the app with a weighted exploration strategy guided by key factors such as the coverage of app functionality. We conduct a large-scale testing on 41,581 VPA apps (i.e., skills) of Amazon Alexa, the most popular VPA service, and find that 51.29% of them have weaknesses. They largely suffer from problems such as unexpected exit/start, privacy violation and so on. Our work reveals the immaturity of the VUI designs and implementations in VPA apps, and sheds light on the improvement of several crucial aspects of VPA apps. © 2022 Association for Computing Machinery.

AB - Virtual personal assistant (VPA) services, e.g. Amazon Alexa and Google Assistant, are becoming increasingly popular recently. Users interact with them through voice-based apps, e.g. Amazon Alexa skills and Google Assistant actions. Unlike the desktop and mobile apps which have visible and intuitive graphical user interface (GUI) to facilitate interaction, VPA apps convey information purely verbally through the voice user interface (VUI), which is known to be limited in its invisibility, single mode and high demand of user attention. This may lead to various problems on the usability and correctness of VPA apps. In this work, we propose a model-based framework named Vitas to handle VUI testing of VPA apps. Vitas interacts with the app VUI, and during the testing process, it retrieves semantic information from voice feedbacks by natural language processing. It incrementally constructs the finite state machine (FSM) model of the app with a weighted exploration strategy guided by key factors such as the coverage of app functionality. We conduct a large-scale testing on 41,581 VPA apps (i.e., skills) of Amazon Alexa, the most popular VPA service, and find that 51.29% of them have weaknesses. They largely suffer from problems such as unexpected exit/start, privacy violation and so on. Our work reveals the immaturity of the VUI designs and implementations in VPA apps, and sheds light on the improvement of several crucial aspects of VPA apps. © 2022 Association for Computing Machinery.

UR - http://www.scopus.com/inward/record.url?scp=85146945067&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85146945067&origin=recordpage

U2 - 10.1145/3551349.3556957

DO - 10.1145/3551349.3556957

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 978-1-4503-9475-8

BT - ASE '22

PB - Association for Computing Machinery

Y2 - 10 October 2022 through 14 October 2022

ER -

VITAS: Guided Model-based VUI Testing of VPA Apps

Abstract

Conference

Funding

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this