LipText: Lip Tracking Based Text Entry in VR

Jiaye Leng; Zijun Wang; Jian Wu; Lili Wang

doi:10.1007/978-981-96-3679-2_11

LipText: Lip Tracking Based Text Entry in VR

Jiaye Leng
, Zijun Wang
, Jian Wu^*
, Lili Wang

^*Corresponding author for this work

School of Creative Media

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

Abstract

Text entry is an important task in virtual reality (VR), and most existing methods require hand involvement, while hands-free typing has great potential for applications in mobile scenarios. Existing hands-free text entry methods are usually implemented by combining the head and eyes with techniques such as Dwell, Blink and Gesture, which can easily fatigue the user. In this paper, we propose LipText, a lip-tracking-based text entry method in VR. We use a neural network to perform letter-level prediction on the lip data captured by the facial tracker and use head-based selection as an auxiliary to improve the accuracy. We conduct a user study to evaluate our method, the results show a typing speed of 8.63 WPM for the novice group, 9.81 WPM for the potential expert group, and the highest recorded typing speed is 11.13 WPM achieved by a potential expert. Our method is also novice-friendly, and their typing speed increased by 64.38% over a six-day practice. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.

Original language	English
Title of host publication	Extended Reality
Subtitle of host publication	First International Conference, ICXR 2024, Xiamen, China, November 14–17, 2024, Proceedings
Editors	Weitao Song, Frank Guan, Shuai Li, Guofeng Zhang
Place of Publication	Singapore
Publisher	Springer
Pages	162-178
ISBN (Electronic)	978-981-96-3679-2
ISBN (Print)	9789819636785
DOIs	https://doi.org/10.1007/978-981-96-3679-2_11
Publication status	Published - 2025
Event	1st International Conference on Extended Reality (ICXR 2024) - Xiamen, China Duration: 14 Nov 2024 → 17 Nov 2024

Publication series

Name	Lecture Notes in Computer Science
Volume	15461
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	1st International Conference on Extended Reality (ICXR 2024)
Place	China
City	Xiamen
Period	14/11/24 → 17/11/24

Funding

This work was supported by the National Natural Science Foundation of China through Projects 61932003 and 62372026, by Beijing Science and Technology Plan Project Z221100007722004, and by NationalKey R&D plan 2019YFC1521102.

Research Keywords

Hands-free
Text entry
Virtual reality

Access Output

10.1007/978-981-96-3679-2_11

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Leng, J., Wang, Z., Wu, J., & Wang, L. (2025). LipText: Lip Tracking Based Text Entry in VR. In W. Song, F. Guan, S. Li, & G. Zhang (Eds.), Extended Reality: First International Conference, ICXR 2024, Xiamen, China, November 14–17, 2024, Proceedings (pp. 162-178). (Lecture Notes in Computer Science; Vol. 15461). Springer . https://doi.org/10.1007/978-981-96-3679-2_11

@inproceedings{889ee022e055468482b7e407427230d6,

title = "LipText: Lip Tracking Based Text Entry in VR",

abstract = "Text entry is an important task in virtual reality (VR), and most existing methods require hand involvement, while hands-free typing has great potential for applications in mobile scenarios. Existing hands-free text entry methods are usually implemented by combining the head and eyes with techniques such as Dwell, Blink and Gesture, which can easily fatigue the user. In this paper, we propose LipText, a lip-tracking-based text entry method in VR. We use a neural network to perform letter-level prediction on the lip data captured by the facial tracker and use head-based selection as an auxiliary to improve the accuracy. We conduct a user study to evaluate our method, the results show a typing speed of 8.63 WPM for the novice group, 9.81 WPM for the potential expert group, and the highest recorded typing speed is 11.13 WPM achieved by a potential expert. Our method is also novice-friendly, and their typing speed increased by 64.38\% over a six-day practice. {\textcopyright} The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.",

keywords = "Hands-free, Text entry, Virtual reality",

author = "Jiaye Leng and Zijun Wang and Jian Wu and Lili Wang",

year = "2025",

doi = "10.1007/978-981-96-3679-2\_11",

language = "English",

isbn = "9789819636785",

series = "Lecture Notes in Computer Science",

publisher = "Springer ",

pages = "162--178",

editor = "Weitao Song and Frank Guan and Shuai Li and Guofeng Zhang",

booktitle = "Extended Reality",

note = "1st International Conference on Extended Reality (ICXR 2024) ; Conference date: 14-11-2024 Through 17-11-2024",

}

Leng, J, Wang, Z, Wu, J & Wang, L 2025, LipText: Lip Tracking Based Text Entry in VR. in W Song, F Guan, S Li & G Zhang (eds), Extended Reality: First International Conference, ICXR 2024, Xiamen, China, November 14–17, 2024, Proceedings. Lecture Notes in Computer Science, vol. 15461, Springer , Singapore, pp. 162-178, 1st International Conference on Extended Reality (ICXR 2024), Xiamen, China, 14/11/24. https://doi.org/10.1007/978-981-96-3679-2_11

LipText: Lip Tracking Based Text Entry in VR. / Leng, Jiaye; Wang, Zijun; Wu, Jian et al.
Extended Reality: First International Conference, ICXR 2024, Xiamen, China, November 14–17, 2024, Proceedings. ed. / Weitao Song; Frank Guan; Shuai Li; Guofeng Zhang. Singapore: Springer , 2025. p. 162-178 (Lecture Notes in Computer Science; Vol. 15461).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - LipText

T2 - 1st International Conference on Extended Reality (ICXR 2024)

AU - Leng, Jiaye

AU - Wang, Zijun

AU - Wu, Jian

AU - Wang, Lili

PY - 2025

Y1 - 2025

N2 - Text entry is an important task in virtual reality (VR), and most existing methods require hand involvement, while hands-free typing has great potential for applications in mobile scenarios. Existing hands-free text entry methods are usually implemented by combining the head and eyes with techniques such as Dwell, Blink and Gesture, which can easily fatigue the user. In this paper, we propose LipText, a lip-tracking-based text entry method in VR. We use a neural network to perform letter-level prediction on the lip data captured by the facial tracker and use head-based selection as an auxiliary to improve the accuracy. We conduct a user study to evaluate our method, the results show a typing speed of 8.63 WPM for the novice group, 9.81 WPM for the potential expert group, and the highest recorded typing speed is 11.13 WPM achieved by a potential expert. Our method is also novice-friendly, and their typing speed increased by 64.38% over a six-day practice. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.

AB - Text entry is an important task in virtual reality (VR), and most existing methods require hand involvement, while hands-free typing has great potential for applications in mobile scenarios. Existing hands-free text entry methods are usually implemented by combining the head and eyes with techniques such as Dwell, Blink and Gesture, which can easily fatigue the user. In this paper, we propose LipText, a lip-tracking-based text entry method in VR. We use a neural network to perform letter-level prediction on the lip data captured by the facial tracker and use head-based selection as an auxiliary to improve the accuracy. We conduct a user study to evaluate our method, the results show a typing speed of 8.63 WPM for the novice group, 9.81 WPM for the potential expert group, and the highest recorded typing speed is 11.13 WPM achieved by a potential expert. Our method is also novice-friendly, and their typing speed increased by 64.38% over a six-day practice. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2025.

KW - Hands-free

KW - Text entry

KW - Virtual reality

UR - https://www.scopus.com/pages/publications/105002587909

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-105002587909&origin=recordpage

U2 - 10.1007/978-981-96-3679-2_11

DO - 10.1007/978-981-96-3679-2_11

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9789819636785

T3 - Lecture Notes in Computer Science

SP - 162

EP - 178

BT - Extended Reality

A2 - Song, Weitao

A2 - Guan, Frank

A2 - Li, Shuai

A2 - Zhang, Guofeng

PB - Springer

CY - Singapore

Y2 - 14 November 2024 through 17 November 2024

ER -

LipText: Lip Tracking Based Text Entry in VR

Abstract

Publication series

Conference

Funding

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this