Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays

Yunpeng Bai; Aleksi Ikkala; Antti Oulasvirta; Shengdong Zhao; Lucia J Wang; Pengzhi Yang; Peisen Xu

doi:10.1145/3613904.3642540

Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays

Yunpeng Bai
, Aleksi Ikkala
, Antti Oulasvirta^*
, Shengdong Zhao^*
, Lucia J Wang
, Pengzhi Yang
, Peisen Xu

^*Corresponding author for this work

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

10 Citations (Scopus)

46 Downloads (CityUHK Scholars)

Abstract

Optical Head-Mounted Displays (OHMDs) allow users to read digital content while walking. A better understanding of how users allocate attention between these two tasks is crucial for improving OHMD interfaces. This paper introduces a computational model for simulating users' attention switches between reading and walking. We model users' decision to deploy visual attention as a hierarchical reinforcement learning problem, wherein a supervisory controller optimizes attention allocation while considering both reading activity and walking safety. Our model simulates the control of eye movements and locomotion as an adaptation to the given task priority, design of digital content, and walking speed. The model replicates key multitasking behaviors during OHMD reading while walking, including attention switches, changes in reading and walking speeds, and reading resumptions. © 2024 Copyright held by the owner/author(s).

Original language	English
Title of host publication	CHI’24
Subtitle of host publication	Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems
Publisher	Association for Computing Machinery
ISBN (Print)	9798400703300
DOIs	https://doi.org/10.1145/3613904.3642540
Publication status	Published - 2024
Event	2024 CHI Conference on Human Factors in Computing Sytems (CHI 2024): "Surfing the World" - Hawaiʻi Convention Center, Honolulu, United States Duration: 11 May 2024 → 16 May 2024 https://chi2024.acm.org/for-attendees/

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings

Conference

Conference	2024 CHI Conference on Human Factors in Computing Sytems (CHI 2024)
Place	United States
City	Honolulu
Period	11/05/24 → 16/05/24
Internet address	https://chi2024.acm.org/for-attendees/

Research Keywords

bounded optimal control
computational rationality
deep reinforcement learning
heads-up computing
multitasking

Publisher's Copyright Statement

This full text is made available under CC-BY 4.0. https://creativecommons.org/licenses/by/4.0/

Access Output

10.1145/3613904.3642540

229051657.pdfFinal Published version, 3.12 MBLicence: CC BY

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Bai, Y., Ikkala, A., Oulasvirta, A., Zhao, S., Wang, L. J., Yang, P., & Xu, P. (2024). Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays. In CHI’24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems Article 79 (Conference on Human Factors in Computing Systems - Proceedings). Association for Computing Machinery. https://doi.org/10.1145/3613904.3642540

@inproceedings{83a6c35f6993449e9b4dc4f010df9e75,

title = "Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays",

abstract = "Optical Head-Mounted Displays (OHMDs) allow users to read digital content while walking. A better understanding of how users allocate attention between these two tasks is crucial for improving OHMD interfaces. This paper introduces a computational model for simulating users' attention switches between reading and walking. We model users' decision to deploy visual attention as a hierarchical reinforcement learning problem, wherein a supervisory controller optimizes attention allocation while considering both reading activity and walking safety. Our model simulates the control of eye movements and locomotion as an adaptation to the given task priority, design of digital content, and walking speed. The model replicates key multitasking behaviors during OHMD reading while walking, including attention switches, changes in reading and walking speeds, and reading resumptions. {\textcopyright} 2024 Copyright held by the owner/author(s).",

keywords = "bounded optimal control, computational rationality, deep reinforcement learning, heads-up computing, multitasking",

author = "Yunpeng Bai and Aleksi Ikkala and Antti Oulasvirta and Shengdong Zhao and Wang, \{Lucia J\} and Pengzhi Yang and Peisen Xu",

year = "2024",

doi = "10.1145/3613904.3642540",

language = "English",

isbn = "9798400703300",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "Association for Computing Machinery",

booktitle = "CHI{\textquoteright}24",

address = "United States",

note = "2024 CHI Conference on Human Factors in Computing Sytems (CHI 2024) : {"}Surfing the World{"} ; Conference date: 11-05-2024 Through 16-05-2024",

url = "https://chi2024.acm.org/for-attendees/",

}

Bai, Y, Ikkala, A, Oulasvirta, A, Zhao, S, Wang, LJ, Yang, P & Xu, P 2024, Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays. in CHI’24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems., 79, Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, 2024 CHI Conference on Human Factors in Computing Sytems (CHI 2024), Honolulu, United States, 11/05/24. https://doi.org/10.1145/3613904.3642540

Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays. / Bai, Yunpeng; Ikkala, Aleksi; Oulasvirta, Antti et al.
CHI’24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2024. 79 (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Heads-Up Multitasker

T2 - 2024 CHI Conference on Human Factors in Computing Sytems (CHI 2024)

AU - Bai, Yunpeng

AU - Ikkala, Aleksi

AU - Oulasvirta, Antti

AU - Zhao, Shengdong

AU - Wang, Lucia J

AU - Yang, Pengzhi

AU - Xu, Peisen

PY - 2024

Y1 - 2024

N2 - Optical Head-Mounted Displays (OHMDs) allow users to read digital content while walking. A better understanding of how users allocate attention between these two tasks is crucial for improving OHMD interfaces. This paper introduces a computational model for simulating users' attention switches between reading and walking. We model users' decision to deploy visual attention as a hierarchical reinforcement learning problem, wherein a supervisory controller optimizes attention allocation while considering both reading activity and walking safety. Our model simulates the control of eye movements and locomotion as an adaptation to the given task priority, design of digital content, and walking speed. The model replicates key multitasking behaviors during OHMD reading while walking, including attention switches, changes in reading and walking speeds, and reading resumptions. © 2024 Copyright held by the owner/author(s).

AB - Optical Head-Mounted Displays (OHMDs) allow users to read digital content while walking. A better understanding of how users allocate attention between these two tasks is crucial for improving OHMD interfaces. This paper introduces a computational model for simulating users' attention switches between reading and walking. We model users' decision to deploy visual attention as a hierarchical reinforcement learning problem, wherein a supervisory controller optimizes attention allocation while considering both reading activity and walking safety. Our model simulates the control of eye movements and locomotion as an adaptation to the given task priority, design of digital content, and walking speed. The model replicates key multitasking behaviors during OHMD reading while walking, including attention switches, changes in reading and walking speeds, and reading resumptions. © 2024 Copyright held by the owner/author(s).

KW - bounded optimal control

KW - computational rationality

KW - deep reinforcement learning

KW - heads-up computing

KW - multitasking

UR - https://www.scopus.com/pages/publications/85194844992

UR - https://www.scopus.com/pages/publications/85194844992#tab=impact

U2 - 10.1145/3613904.3642540

DO - 10.1145/3613904.3642540

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9798400703300

T3 - Conference on Human Factors in Computing Systems - Proceedings

BT - CHI’24

PB - Association for Computing Machinery

Y2 - 11 May 2024 through 16 May 2024

ER -

Bai Y, Ikkala A, Oulasvirta A, Zhao S, Wang LJ, Yang P et al. Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays. In CHI’24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery. 2024. 79. (Conference on Human Factors in Computing Systems - Proceedings). Epub 2024 May 11. doi: 10.1145/3613904.3642540

Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted Displays

Abstract

Publication series

Conference

Research Keywords

Publisher's Copyright Statement

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this