Demonstrating PANDALens: Enhancing Daily Activity Documentation with AI-assisted In-Context Writing on OHMD

Nuwan Janaka; Runze Cai; Shengdong Zhao; David Hsu

doi:10.1145/3613905.3648644

Demonstrating PANDALens: Enhancing Daily Activity Documentation with AI-assisted In-Context Writing on OHMD

Nuwan Janaka, Runze Cai^*, Shengdong Zhao^*, David Hsu

^*Corresponding author for this work

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

4 Citations (Scopus)

Abstract

We introduce PANDALens, a Proactive AI Narrative Documentation Assistant built on an Optical See-Through Head-Mounted Display that transforms the in-context writing tool into an intelligent companion during daily activities. PANDALens observes multimodal contextual information from user behaviors and the environment to detect interesting moments and elicit contemplation. It also employs Large Language Models to transform such multimodal information into coherent narratives with significantly reduced user effort. PANDALens was iteratively designed through a formative study identifying the user requirements. We verify its utility in a real-world travel scenario in improving writing quality and travel enjoyment while minimizing user effort. © 2024 Copyright held by the owner/author(s).

Original language	English
Title of host publication	CHI’24
Subtitle of host publication	Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems
Publisher	Association for Computing Machinery
ISBN (Print)	9798400703317
DOIs	https://doi.org/10.1145/3613905.3648644
Publication status	Published - 2024
Event	2024 CHI Conference on Human Factors in Computing Sytems (CHI 2024): "Surfing the World" - Hawaiʻi Convention Center, Honolulu, United States Duration: 11 May 2024 → 16 May 2024 https://chi2024.acm.org/for-attendees/

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings

Conference

Conference	2024 CHI Conference on Human Factors in Computing Sytems (CHI 2024)
Place	United States
City	Honolulu
Period	11/05/24 → 16/05/24
Internet address	https://chi2024.acm.org/for-attendees/

Funding

This research is supported by the National Research Foundation, Singapore, under its AI Singapore Programme (AISG Award No: AISG2-RP-2020-016). It is also supported in part by the Ministry of Education, Singapore, under its MOE Academic Research Fund Tier 2 programme (MOE-T2EP20221-0010), and by a research grant #22-5913-A0001 from the Ministry of Education of Singapore. Additionally, the CityU Start-up Grant also provides partial support.

Research Keywords

AI
HMD
Human-AI collaborative writing
in-context writing
large language model
multimodal information
smart glasses
travel blog

Access Output

10.1145/3613905.3648644

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Janaka, N., Cai, R., Zhao, S., & Hsu, D. (2024). Demonstrating PANDALens: Enhancing Daily Activity Documentation with AI-assisted In-Context Writing on OHMD. In CHI’24: Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems Article 397 (Conference on Human Factors in Computing Systems - Proceedings). Association for Computing Machinery. https://doi.org/10.1145/3613905.3648644

@inproceedings{1292b2643c854e9abfcbd90297285a5f,

title = "Demonstrating PANDALens: Enhancing Daily Activity Documentation with AI-assisted In-Context Writing on OHMD",

abstract = "We introduce PANDALens, a Proactive AI Narrative Documentation Assistant built on an Optical See-Through Head-Mounted Display that transforms the in-context writing tool into an intelligent companion during daily activities. PANDALens observes multimodal contextual information from user behaviors and the environment to detect interesting moments and elicit contemplation. It also employs Large Language Models to transform such multimodal information into coherent narratives with significantly reduced user effort. PANDALens was iteratively designed through a formative study identifying the user requirements. We verify its utility in a real-world travel scenario in improving writing quality and travel enjoyment while minimizing user effort. {\textcopyright} 2024 Copyright held by the owner/author(s).",

keywords = "AI, HMD, Human-AI collaborative writing, in-context writing, large language model, multimodal information, smart glasses, travel blog",

author = "Nuwan Janaka and Runze Cai and Shengdong Zhao and David Hsu",

year = "2024",

doi = "10.1145/3613905.3648644",

language = "English",

isbn = "9798400703317",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "Association for Computing Machinery",

booktitle = "CHI{\textquoteright}24",

address = "United States",

note = "2024 CHI Conference on Human Factors in Computing Sytems (CHI 2024) : {"}Surfing the World{"} ; Conference date: 11-05-2024 Through 16-05-2024",

url = "https://chi2024.acm.org/for-attendees/",

}

Janaka, N, Cai, R, Zhao, S & Hsu, D 2024, Demonstrating PANDALens: Enhancing Daily Activity Documentation with AI-assisted In-Context Writing on OHMD. in CHI’24: Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems., 397, Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery, 2024 CHI Conference on Human Factors in Computing Sytems (CHI 2024), Honolulu, United States, 11/05/24. https://doi.org/10.1145/3613905.3648644

Demonstrating PANDALens: Enhancing Daily Activity Documentation with AI-assisted In-Context Writing on OHMD. / Janaka, Nuwan; Cai, Runze; Zhao, Shengdong et al.
CHI’24: Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, 2024. 397 (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Demonstrating PANDALens

T2 - 2024 CHI Conference on Human Factors in Computing Sytems (CHI 2024)

AU - Janaka, Nuwan

AU - Cai, Runze

AU - Zhao, Shengdong

AU - Hsu, David

PY - 2024

Y1 - 2024

N2 - We introduce PANDALens, a Proactive AI Narrative Documentation Assistant built on an Optical See-Through Head-Mounted Display that transforms the in-context writing tool into an intelligent companion during daily activities. PANDALens observes multimodal contextual information from user behaviors and the environment to detect interesting moments and elicit contemplation. It also employs Large Language Models to transform such multimodal information into coherent narratives with significantly reduced user effort. PANDALens was iteratively designed through a formative study identifying the user requirements. We verify its utility in a real-world travel scenario in improving writing quality and travel enjoyment while minimizing user effort. © 2024 Copyright held by the owner/author(s).

AB - We introduce PANDALens, a Proactive AI Narrative Documentation Assistant built on an Optical See-Through Head-Mounted Display that transforms the in-context writing tool into an intelligent companion during daily activities. PANDALens observes multimodal contextual information from user behaviors and the environment to detect interesting moments and elicit contemplation. It also employs Large Language Models to transform such multimodal information into coherent narratives with significantly reduced user effort. PANDALens was iteratively designed through a formative study identifying the user requirements. We verify its utility in a real-world travel scenario in improving writing quality and travel enjoyment while minimizing user effort. © 2024 Copyright held by the owner/author(s).

KW - AI

KW - HMD

KW - Human-AI collaborative writing

KW - in-context writing

KW - large language model

KW - multimodal information

KW - smart glasses

KW - travel blog

UR - http://www.scopus.com/inward/record.url?scp=85194192929&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85194192929&origin=recordpage

U2 - 10.1145/3613905.3648644

DO - 10.1145/3613905.3648644

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9798400703317

T3 - Conference on Human Factors in Computing Systems - Proceedings

BT - CHI’24

PB - Association for Computing Machinery

Y2 - 11 May 2024 through 16 May 2024

ER -

Janaka N, Cai R, Zhao S, Hsu D. Demonstrating PANDALens: Enhancing Daily Activity Documentation with AI-assisted In-Context Writing on OHMD. In CHI’24: Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery. 2024. 397. (Conference on Human Factors in Computing Systems - Proceedings). Epub 2024 May 11. doi: 10.1145/3613905.3648644

Demonstrating PANDALens: Enhancing Daily Activity Documentation with AI-assisted In-Context Writing on OHMD

Abstract

Publication series

Conference

Funding

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this