"What's this?": Understanding User Interaction Behaviour with Multimodal Input Information Retrieval System

Silang Wang; Hyeongcheol Kim; Nuwan Janaka; Kun Yue; Hoang-Long Nguyen; Shengdong Zhao; Haiming Liu; Khanh-Duy Le

doi:10.1145/3640471.3680230

"What's this?": Understanding User Interaction Behaviour with Multimodal Input Information Retrieval System

Silang Wang, Hyeongcheol Kim, Nuwan Janaka, Kun Yue, Hoang-Long Nguyen, Shengdong Zhao^*, Haiming Liu^*, Khanh-Duy Le^*

^*Corresponding author for this work

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

2 Citations (Scopus)

Abstract

Human communication relies on integrated multimodal channels to facilitate rich information exchange. Building on this foundation, researchers have long speculated about the potential benefits of incorporating multimodal input channels into conventional information retrieval (IR) systems to support users' complex daily IR tasks more effectively. However, the true benefits of such integration remain uncertain. This paper presents a series of exploratory pilot tests comparing Multimodal Input IR (MIIR) with Unimodal Input IR (UIIR) across various IR scenarios, concluding that MIIR offers distinct advantages over UIIR in terms of user experiences. Our preliminary results suggest that MIIR could reduce the cognitive load associated with IR query formulation by allowing users to formulate different query-component in a unified manner across different input modalities, particularly when conducting complex exploratory search tasks in unfamiliar, in-situ contexts. The discussions stemming from this finding draw scholarly attention and suggest new angles for designing and developing MIIR systems. © 2024 Copyright held by the owner/author(s).

Original language	English
Title of host publication	MobileHCI '24 Adjunct
Subtitle of host publication	Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction
Editors	Tilman Dingler, Flora Salim, Marion Koelle, Niels van Berkel, Jenny Waycott
Publisher	Association for Computing Machinery
ISBN (Print)	979-8-4007-0506-9
DOIs	https://doi.org/10.1145/3640471.3680230
Publication status	Published - Sept 2024
Event	26th International Conference on Mobile Human-Computer Interaction (MobileHCI 2024): Connecting Cultures - Melbourne, Australia Duration: 30 Sept 2024 → 3 Oct 2024 https://mobilehci.acm.org/2024/

Publication series

Name	MobileHCI Adjunct Proceedings - Publication of International Conference on Mobile Human-Computer Interaction

Conference

Conference	26th International Conference on Mobile Human-Computer Interaction (MobileHCI 2024)
Abbreviated title	MobileHCI'24
Country/Territory	Australia
City	Melbourne
Period	30/09/24 → 3/10/24
Internet address	https://mobilehci.acm.org/2024/

Research Keywords

Heads-up Computing
Information Retrieval
Multimodal Interaction
User Search Behavior

Access Output

10.1145/3640471.3680230

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Wang, S., Kim, H., Janaka, N., Yue, K., Nguyen, H.-L., Zhao, S., Liu, H., & Le, K.-D. (2024). "What's this?": Understanding User Interaction Behaviour with Multimodal Input Information Retrieval System. In T. Dingler, F. Salim, M. Koelle, N. V. Berkel, & J. Waycott (Eds.), MobileHCI '24 Adjunct: Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction Article 3 (MobileHCI Adjunct Proceedings - Publication of International Conference on Mobile Human-Computer Interaction). Association for Computing Machinery. https://doi.org/10.1145/3640471.3680230

Wang, Silang ; Kim, Hyeongcheol ; Janaka, Nuwan et al. / "What's this?" : Understanding User Interaction Behaviour with Multimodal Input Information Retrieval System. MobileHCI '24 Adjunct: Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction. editor / Tilman Dingler ; Flora Salim ; Marion Koelle ; Niels van Berkel ; Jenny Waycott. Association for Computing Machinery, 2024. (MobileHCI Adjunct Proceedings - Publication of International Conference on Mobile Human-Computer Interaction).

@inproceedings{dc653c648b13472e84bd6bdc613e3c7f,

title = "{"}What's this?{"}: Understanding User Interaction Behaviour with Multimodal Input Information Retrieval System",

abstract = "Human communication relies on integrated multimodal channels to facilitate rich information exchange. Building on this foundation, researchers have long speculated about the potential benefits of incorporating multimodal input channels into conventional information retrieval (IR) systems to support users' complex daily IR tasks more effectively. However, the true benefits of such integration remain uncertain. This paper presents a series of exploratory pilot tests comparing Multimodal Input IR (MIIR) with Unimodal Input IR (UIIR) across various IR scenarios, concluding that MIIR offers distinct advantages over UIIR in terms of user experiences. Our preliminary results suggest that MIIR could reduce the cognitive load associated with IR query formulation by allowing users to formulate different query-component in a unified manner across different input modalities, particularly when conducting complex exploratory search tasks in unfamiliar, in-situ contexts. The discussions stemming from this finding draw scholarly attention and suggest new angles for designing and developing MIIR systems. {\textcopyright} 2024 Copyright held by the owner/author(s).",

keywords = "Heads-up Computing, Information Retrieval, Multimodal Interaction, User Search Behavior",

author = "Silang Wang and Hyeongcheol Kim and Nuwan Janaka and Kun Yue and Hoang-Long Nguyen and Shengdong Zhao and Haiming Liu and Khanh-Duy Le",

year = "2024",

month = sep,

doi = "10.1145/3640471.3680230",

language = "English",

isbn = "979-8-4007-0506-9",

series = "MobileHCI Adjunct Proceedings - Publication of International Conference on Mobile Human-Computer Interaction",

publisher = "Association for Computing Machinery",

editor = "Tilman Dingler and Flora Salim and Marion Koelle and Berkel, {Niels van} and Jenny Waycott",

booktitle = "MobileHCI '24 Adjunct",

address = "United States",

note = "26th International Conference on Mobile Human-Computer Interaction (MobileHCI 2024) : Connecting Cultures, MobileHCI'24 ; Conference date: 30-09-2024 Through 03-10-2024",

url = "https://mobilehci.acm.org/2024/",

}

Wang, S, Kim, H, Janaka, N, Yue, K, Nguyen, H-L, Zhao, S, Liu, H & Le, K-D 2024, "What's this?": Understanding User Interaction Behaviour with Multimodal Input Information Retrieval System. in T Dingler, F Salim, M Koelle, NV Berkel & J Waycott (eds), MobileHCI '24 Adjunct: Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction., 3, MobileHCI Adjunct Proceedings - Publication of International Conference on Mobile Human-Computer Interaction, Association for Computing Machinery, 26th International Conference on Mobile Human-Computer Interaction (MobileHCI 2024), Melbourne, Victoria, Australia, 30/09/24. https://doi.org/10.1145/3640471.3680230

"What's this?": Understanding User Interaction Behaviour with Multimodal Input Information Retrieval System. / Wang, Silang; Kim, Hyeongcheol; Janaka, Nuwan et al.
MobileHCI '24 Adjunct: Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction. ed. / Tilman Dingler; Flora Salim; Marion Koelle; Niels van Berkel; Jenny Waycott. Association for Computing Machinery, 2024. 3 (MobileHCI Adjunct Proceedings - Publication of International Conference on Mobile Human-Computer Interaction).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - "What's this?"

T2 - 26th International Conference on Mobile Human-Computer Interaction (MobileHCI 2024)

AU - Wang, Silang

AU - Kim, Hyeongcheol

AU - Janaka, Nuwan

AU - Yue, Kun

AU - Nguyen, Hoang-Long

AU - Zhao, Shengdong

AU - Liu, Haiming

AU - Le, Khanh-Duy

PY - 2024/9

Y1 - 2024/9

N2 - Human communication relies on integrated multimodal channels to facilitate rich information exchange. Building on this foundation, researchers have long speculated about the potential benefits of incorporating multimodal input channels into conventional information retrieval (IR) systems to support users' complex daily IR tasks more effectively. However, the true benefits of such integration remain uncertain. This paper presents a series of exploratory pilot tests comparing Multimodal Input IR (MIIR) with Unimodal Input IR (UIIR) across various IR scenarios, concluding that MIIR offers distinct advantages over UIIR in terms of user experiences. Our preliminary results suggest that MIIR could reduce the cognitive load associated with IR query formulation by allowing users to formulate different query-component in a unified manner across different input modalities, particularly when conducting complex exploratory search tasks in unfamiliar, in-situ contexts. The discussions stemming from this finding draw scholarly attention and suggest new angles for designing and developing MIIR systems. © 2024 Copyright held by the owner/author(s).

AB - Human communication relies on integrated multimodal channels to facilitate rich information exchange. Building on this foundation, researchers have long speculated about the potential benefits of incorporating multimodal input channels into conventional information retrieval (IR) systems to support users' complex daily IR tasks more effectively. However, the true benefits of such integration remain uncertain. This paper presents a series of exploratory pilot tests comparing Multimodal Input IR (MIIR) with Unimodal Input IR (UIIR) across various IR scenarios, concluding that MIIR offers distinct advantages over UIIR in terms of user experiences. Our preliminary results suggest that MIIR could reduce the cognitive load associated with IR query formulation by allowing users to formulate different query-component in a unified manner across different input modalities, particularly when conducting complex exploratory search tasks in unfamiliar, in-situ contexts. The discussions stemming from this finding draw scholarly attention and suggest new angles for designing and developing MIIR systems. © 2024 Copyright held by the owner/author(s).

KW - Heads-up Computing

KW - Information Retrieval

KW - Multimodal Interaction

KW - User Search Behavior

UR - http://www.scopus.com/inward/record.url?scp=85206188191&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85206188191&origin=recordpage

U2 - 10.1145/3640471.3680230

DO - 10.1145/3640471.3680230

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 979-8-4007-0506-9

T3 - MobileHCI Adjunct Proceedings - Publication of International Conference on Mobile Human-Computer Interaction

BT - MobileHCI '24 Adjunct

A2 - Dingler, Tilman

A2 - Salim, Flora

A2 - Koelle, Marion

A2 - Berkel, Niels van

A2 - Waycott, Jenny

PB - Association for Computing Machinery

Y2 - 30 September 2024 through 3 October 2024

ER -

Wang S, Kim H, Janaka N, Yue K, Nguyen HL, Zhao S et al. "What's this?": Understanding User Interaction Behaviour with Multimodal Input Information Retrieval System. In Dingler T, Salim F, Koelle M, Berkel NV, Waycott J, editors, MobileHCI '24 Adjunct: Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction. Association for Computing Machinery. 2024. 3. (MobileHCI Adjunct Proceedings - Publication of International Conference on Mobile Human-Computer Interaction). Epub 2024 Sept 21. doi: 10.1145/3640471.3680230

"What's this?": Understanding User Interaction Behaviour with Multimodal Input Information Retrieval System

Abstract

Publication series

Conference

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this