Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text Composition

Brinda Mehra; Kejia Shen; Hen Chen Yen; Can Liu

doi:10.1145/3571884.3597134

Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text Composition

Brinda Mehra
, Kejia Shen
, Hen Chen Yen
, Can Liu^*

^*Corresponding author for this work

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

8 Citations (Scopus)

Abstract

Recent interest in speech-to-text applications has found speech to be an efficient modality for text input. However, the spontaneity of speech makes direct transcriptions of spoken compositions effortful to edit. While previous works in Human-Computer Interaction (HCI) domain focus on improving error correction, there is a lack of theoretical ground around the understanding of speech as an input modality. This work explores literature from Cognitive Science to synthesize relevant theories and findings for the HCI audience to reference. Motivated by the literature indicating a fast memory decay of speech production and a preference towards gist abstraction in memory traces, an experiment was conducted to observe users' immediate recall of their verbal composition. Based on the theories and findings, we introduce new interaction concepts and workflows that adapt to the characteristics of speech input. © 2023 ACM.

Original language	English
Title of host publication	CUI '23: Proceedings of the 5th International Conference on Conversational User Interfaces
Editors	Minha Lee, Cosmin Munteanu, Martin Porcheron, Johanne Trippas, Sarah Theres Völkel
Publisher	Association for Computing Machinery
ISBN (Print)	9798400700149
DOIs	https://doi.org/10.1145/3571884.3597134
Publication status	Published - Jul 2023
Event	5th International Conference on Conversational User Interfaces (CUI 2023) - Eindhoven University of Technology, Eindhoven, Netherlands Duration: 19 Jul 2023 → 21 Jul 2023 https://cui.acm.org/2023/

Publication series

Name	Proceedings of the International Conference on Conversational User Interfaces, CUI

Conference

Conference	5th International Conference on Conversational User Interfaces (CUI 2023)
Place	Netherlands
City	Eindhoven
Period	19/07/23 → 21/07/23
Internet address	https://cui.acm.org/2023/

Bibliographical note

Full text of this publication does not contain sufficient affiliation information. Research Unit(s) information for this record is based on his previous affiliation.

Research Keywords

dictation
speech
speech-to-text
STT
text composition
text editing
text entry

RGC Funding Information

RGC-funded

Access Output

10.1145/3571884.3597134

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Mehra, B., Shen, K., Yen, H. C., & Liu, C. (2023). Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text Composition. In M. Lee, C. Munteanu, M. Porcheron, J. Trippas, & S. T. Völkel (Eds.), CUI '23: Proceedings of the 5th International Conference on Conversational User Interfaces Article 15 (Proceedings of the International Conference on Conversational User Interfaces, CUI). Association for Computing Machinery. https://doi.org/10.1145/3571884.3597134

Mehra, Brinda ; Shen, Kejia ; Yen, Hen Chen et al. / Gist and Verbatim : Understanding Speech to Inform New Interfaces for Verbal Text Composition. CUI '23: Proceedings of the 5th International Conference on Conversational User Interfaces. editor / Minha Lee ; Cosmin Munteanu ; Martin Porcheron ; Johanne Trippas ; Sarah Theres Völkel. Association for Computing Machinery, 2023. (Proceedings of the International Conference on Conversational User Interfaces, CUI).

@inproceedings{8e4f7a99e3944f12b36aa6a60e904d85,

title = "Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text Composition",

abstract = "Recent interest in speech-to-text applications has found speech to be an efficient modality for text input. However, the spontaneity of speech makes direct transcriptions of spoken compositions effortful to edit. While previous works in Human-Computer Interaction (HCI) domain focus on improving error correction, there is a lack of theoretical ground around the understanding of speech as an input modality. This work explores literature from Cognitive Science to synthesize relevant theories and findings for the HCI audience to reference. Motivated by the literature indicating a fast memory decay of speech production and a preference towards gist abstraction in memory traces, an experiment was conducted to observe users' immediate recall of their verbal composition. Based on the theories and findings, we introduce new interaction concepts and workflows that adapt to the characteristics of speech input. {\textcopyright} 2023 ACM.",

keywords = "dictation, speech, speech-to-text, STT, text composition, text editing, text entry",

author = "Brinda Mehra and Kejia Shen and Yen, \{Hen Chen\} and Can Liu",

note = "Full text of this publication does not contain sufficient affiliation information. Research Unit(s) information for this record is based on his previous affiliation.; 5th International Conference on Conversational User Interfaces (CUI 2023) ; Conference date: 19-07-2023 Through 21-07-2023",

year = "2023",

month = jul,

doi = "10.1145/3571884.3597134",

language = "English",

isbn = "9798400700149",

series = "Proceedings of the International Conference on Conversational User Interfaces, CUI",

publisher = "Association for Computing Machinery",

editor = "Minha Lee and Cosmin Munteanu and Martin Porcheron and Johanne Trippas and V{\"o}lkel, \{Sarah Theres\}",

booktitle = "CUI '23: Proceedings of the 5th International Conference on Conversational User Interfaces",

address = "United States",

url = "https://cui.acm.org/2023/",

}

Mehra, B, Shen, K, Yen, HC & Liu, C 2023, Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text Composition. in M Lee, C Munteanu, M Porcheron, J Trippas & ST Völkel (eds), CUI '23: Proceedings of the 5th International Conference on Conversational User Interfaces., 15, Proceedings of the International Conference on Conversational User Interfaces, CUI, Association for Computing Machinery, 5th International Conference on Conversational User Interfaces (CUI 2023), Eindhoven, Netherlands, 19/07/23. https://doi.org/10.1145/3571884.3597134

Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text Composition. / Mehra, Brinda; Shen, Kejia; Yen, Hen Chen et al.
CUI '23: Proceedings of the 5th International Conference on Conversational User Interfaces. ed. / Minha Lee; Cosmin Munteanu; Martin Porcheron; Johanne Trippas; Sarah Theres Völkel. Association for Computing Machinery, 2023. 15 (Proceedings of the International Conference on Conversational User Interfaces, CUI).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Gist and Verbatim

T2 - 5th International Conference on Conversational User Interfaces (CUI 2023)

AU - Mehra, Brinda

AU - Shen, Kejia

AU - Yen, Hen Chen

AU - Liu, Can

N1 - Full text of this publication does not contain sufficient affiliation information. Research Unit(s) information for this record is based on his previous affiliation.

PY - 2023/7

Y1 - 2023/7

N2 - Recent interest in speech-to-text applications has found speech to be an efficient modality for text input. However, the spontaneity of speech makes direct transcriptions of spoken compositions effortful to edit. While previous works in Human-Computer Interaction (HCI) domain focus on improving error correction, there is a lack of theoretical ground around the understanding of speech as an input modality. This work explores literature from Cognitive Science to synthesize relevant theories and findings for the HCI audience to reference. Motivated by the literature indicating a fast memory decay of speech production and a preference towards gist abstraction in memory traces, an experiment was conducted to observe users' immediate recall of their verbal composition. Based on the theories and findings, we introduce new interaction concepts and workflows that adapt to the characteristics of speech input. © 2023 ACM.

AB - Recent interest in speech-to-text applications has found speech to be an efficient modality for text input. However, the spontaneity of speech makes direct transcriptions of spoken compositions effortful to edit. While previous works in Human-Computer Interaction (HCI) domain focus on improving error correction, there is a lack of theoretical ground around the understanding of speech as an input modality. This work explores literature from Cognitive Science to synthesize relevant theories and findings for the HCI audience to reference. Motivated by the literature indicating a fast memory decay of speech production and a preference towards gist abstraction in memory traces, an experiment was conducted to observe users' immediate recall of their verbal composition. Based on the theories and findings, we introduce new interaction concepts and workflows that adapt to the characteristics of speech input. © 2023 ACM.

KW - dictation

KW - speech

KW - speech-to-text

KW - STT

KW - text composition

KW - text editing

KW - text entry

UR - https://www.scopus.com/pages/publications/85167819039

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85167819039&origin=recordpage

U2 - 10.1145/3571884.3597134

DO - 10.1145/3571884.3597134

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9798400700149

T3 - Proceedings of the International Conference on Conversational User Interfaces, CUI

BT - CUI '23: Proceedings of the 5th International Conference on Conversational User Interfaces

A2 - Lee, Minha

A2 - Munteanu, Cosmin

A2 - Porcheron, Martin

A2 - Trippas, Johanne

A2 - Völkel, Sarah Theres

PB - Association for Computing Machinery

Y2 - 19 July 2023 through 21 July 2023

ER -

Mehra B, Shen K, Yen HC, Liu C. Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text Composition. In Lee M, Munteanu C, Porcheron M, Trippas J, Völkel ST, editors, CUI '23: Proceedings of the 5th International Conference on Conversational User Interfaces. Association for Computing Machinery. 2023. 15. (Proceedings of the International Conference on Conversational User Interfaces, CUI). Epub 2023 Jul 19. doi: 10.1145/3571884.3597134

Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text Composition

Abstract

Publication series

Conference

Bibliographical note

Research Keywords

RGC Funding Information

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this