Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation

Susan Lin; Jeremy Warner; J.D. Zamfirescu-Pereira; Matthew G. Lee; Sauhard Jain; Shanqing Cai; Piyawat Lertvittayakumjorn; Michael Xuelin Huang; Shumin Zhai; Björn Hartmann; Can Liu

doi:10.1145/3613904.3642217

Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation

Susan Lin, Jeremy Warner, J.D. Zamfirescu-Pereira, Matthew G. Lee, Sauhard Jain, Shanqing Cai, Piyawat Lertvittayakumjorn, Michael Xuelin Huang, Shumin Zhai, Björn Hartmann, Can Liu^*

^*Corresponding author for this work

School of Creative Media

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

29 Citations (Scopus)

Abstract

Dictation enables efficient text input on mobile devices. However, writing with speech can produce disfluent, wordy, and incoherent text and thus requires heavy post-processing. This paper presents Rambler, an LLM-powered graphical user interface that supports gist-level manipulation of dictated text with two main sets of functions: gist extraction and macro revision. Gist extraction generates keywords and summaries as anchors to support the review and interaction with spoken text. LLM-assisted macro revisions allow users to respeak, split, merge, and transform dictated text without specifying precise editing locations. Together they pave the way for interactive dictation and revision that help close gaps between spontaneously spoken words and well-structured writing. In a comparative study with 12 participants performing verbal composition tasks, Rambler outperformed the baseline of a speech-to-text editor + ChatGPT, as it better facilitates iterative revisions with enhanced user control over the content while supporting surprisingly diverse user strategies. © 2024 Copyright held by the owner/author(s).

Original language	English
Title of host publication	CHI’24
Subtitle of host publication	Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems
Place of Publication	New York, NY
Publisher	Association for Computing Machinery
ISBN (Print)	9798400703300
DOIs	https://doi.org/10.1145/3613904.3642217
Publication status	Published - 2024
Event	2024 ACM CHI Conference on Human Factors in Computing Systems (CHI 2024): Surfing the World - Hybrid, Honolulu, United States Duration: 11 May 2024 → 16 May 2024 https://chi2024.acm.org/

Conference

Conference	2024 ACM CHI Conference on Human Factors in Computing Systems (CHI 2024)
Abbreviated title	CHI '24
Place	United States
City	Honolulu
Period	11/05/24 → 16/05/24
Internet address	https://chi2024.acm.org/

Funding

This project was funded in part by the Berkeley Artificial Intelligence Research Lab - Open Research Commons, and by the National Natural Science Foundation of China - Young Scientists Fund (CityU 62202397). We thank the Android Input Research team at Google for discussions and feedback.

Research Keywords

AI
LLM
STT
dictation
speech
speech-to-text
text composition
writing

Access Output

10.1145/3613904.3642217

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Lin, S., Warner, J., Zamfirescu-Pereira, J. D., Lee, M. G., Jain, S., Cai, S., Lertvittayakumjorn, P., Huang, M. X., Zhai, S., Hartmann, B., & Liu, C. (2024). Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation. In CHI’24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems Article 1043 Association for Computing Machinery. https://doi.org/10.1145/3613904.3642217

@inproceedings{b7281f5626d04f0fb65208ade3968fc9,

title = "Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation",

abstract = "Dictation enables efficient text input on mobile devices. However, writing with speech can produce disfluent, wordy, and incoherent text and thus requires heavy post-processing. This paper presents Rambler, an LLM-powered graphical user interface that supports gist-level manipulation of dictated text with two main sets of functions: gist extraction and macro revision. Gist extraction generates keywords and summaries as anchors to support the review and interaction with spoken text. LLM-assisted macro revisions allow users to respeak, split, merge, and transform dictated text without specifying precise editing locations. Together they pave the way for interactive dictation and revision that help close gaps between spontaneously spoken words and well-structured writing. In a comparative study with 12 participants performing verbal composition tasks, Rambler outperformed the baseline of a speech-to-text editor + ChatGPT, as it better facilitates iterative revisions with enhanced user control over the content while supporting surprisingly diverse user strategies. {\textcopyright} 2024 Copyright held by the owner/author(s).",

keywords = "AI, LLM, STT, dictation, speech, speech-to-text, text composition, writing",

author = "Susan Lin and Jeremy Warner and J.D. Zamfirescu-Pereira and Lee, {Matthew G.} and Sauhard Jain and Shanqing Cai and Piyawat Lertvittayakumjorn and Huang, {Michael Xuelin} and Shumin Zhai and Bj{\"o}rn Hartmann and Can Liu",

year = "2024",

doi = "10.1145/3613904.3642217",

language = "English",

isbn = "9798400703300",

booktitle = "CHI{\textquoteright}24",

publisher = "Association for Computing Machinery",

address = "United States",

note = "2024 ACM CHI Conference on Human Factors in Computing Systems (CHI 2024) : Surfing the World, CHI '24 ; Conference date: 11-05-2024 Through 16-05-2024",

url = "https://chi2024.acm.org/",

}

Lin, S, Warner, J, Zamfirescu-Pereira, JD, Lee, MG, Jain, S, Cai, S, Lertvittayakumjorn, P, Huang, MX, Zhai, S, Hartmann, B & Liu, C 2024, Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation. in CHI’24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems., 1043, Association for Computing Machinery, New York, NY, 2024 ACM CHI Conference on Human Factors in Computing Systems (CHI 2024), Honolulu, Hawaii, United States, 11/05/24. https://doi.org/10.1145/3613904.3642217

Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation. / Lin, Susan; Warner, Jeremy; Zamfirescu-Pereira, J.D. et al.
CHI’24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. New York, NY: Association for Computing Machinery, 2024. 1043.

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Rambler

T2 - 2024 ACM CHI Conference on Human Factors in Computing Systems (CHI 2024)

AU - Lin, Susan

AU - Warner, Jeremy

AU - Zamfirescu-Pereira, J.D.

AU - Lee, Matthew G.

AU - Jain, Sauhard

AU - Cai, Shanqing

AU - Lertvittayakumjorn, Piyawat

AU - Huang, Michael Xuelin

AU - Zhai, Shumin

AU - Hartmann, Björn

AU - Liu, Can

PY - 2024

Y1 - 2024

N2 - Dictation enables efficient text input on mobile devices. However, writing with speech can produce disfluent, wordy, and incoherent text and thus requires heavy post-processing. This paper presents Rambler, an LLM-powered graphical user interface that supports gist-level manipulation of dictated text with two main sets of functions: gist extraction and macro revision. Gist extraction generates keywords and summaries as anchors to support the review and interaction with spoken text. LLM-assisted macro revisions allow users to respeak, split, merge, and transform dictated text without specifying precise editing locations. Together they pave the way for interactive dictation and revision that help close gaps between spontaneously spoken words and well-structured writing. In a comparative study with 12 participants performing verbal composition tasks, Rambler outperformed the baseline of a speech-to-text editor + ChatGPT, as it better facilitates iterative revisions with enhanced user control over the content while supporting surprisingly diverse user strategies. © 2024 Copyright held by the owner/author(s).

AB - Dictation enables efficient text input on mobile devices. However, writing with speech can produce disfluent, wordy, and incoherent text and thus requires heavy post-processing. This paper presents Rambler, an LLM-powered graphical user interface that supports gist-level manipulation of dictated text with two main sets of functions: gist extraction and macro revision. Gist extraction generates keywords and summaries as anchors to support the review and interaction with spoken text. LLM-assisted macro revisions allow users to respeak, split, merge, and transform dictated text without specifying precise editing locations. Together they pave the way for interactive dictation and revision that help close gaps between spontaneously spoken words and well-structured writing. In a comparative study with 12 participants performing verbal composition tasks, Rambler outperformed the baseline of a speech-to-text editor + ChatGPT, as it better facilitates iterative revisions with enhanced user control over the content while supporting surprisingly diverse user strategies. © 2024 Copyright held by the owner/author(s).

KW - AI

KW - LLM

KW - STT

KW - dictation

KW - speech

KW - speech-to-text

KW - text composition

KW - writing

U2 - 10.1145/3613904.3642217

DO - 10.1145/3613904.3642217

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9798400703300

BT - CHI’24

PB - Association for Computing Machinery

CY - New York, NY

Y2 - 11 May 2024 through 16 May 2024

ER -

Rambler: Supporting Writing With Speech via LLM-Assisted Gist Manipulation

Abstract

Conference

Funding

Research Keywords

Access Output

Other Access Options

Permanent Link

Fingerprint

Cite this