Bridging ASR Limitations with Human Judgment: A Hybrid Approach to Oral Presentation Feedback

Pauli Lai; Christy Chan; Julia Chen; Irene Yiqiao Song; Chiho Chan

doi:10.1007/978-3-032-29788-4_28

Bridging ASR Limitations with Human Judgment: A Hybrid Approach to Oral Presentation Feedback

Pauli Lai^*
, Christy Chan
, Julia Chen
, Irene Yiqiao Song
, Chiho Chan

^*Corresponding author for this work

Chan Feng Men-ling Chan Shuk-lin Language Centre

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

Abstract

The growing popularity of AI speaking assessment platforms in educational technology is often attributed to their capacity to offer immediate, consistent, and scalable feedback, thereby supporting student self-learning. However, such platforms have been found to have suffered from notable limitations, such as the emphasis on surface-level speech elements and the potential algorithmic bias across the different Automatic Speech Recognition (ASR) tools adopted by the platforms. To mitigate these limitations of fully automated AI-based speaking assessment platforms, we developed NinjOrAItor, a hybrid intelligence platform designed to integrate artificial intelligence with human oversight and expertise. Within this framework, AI acts as an assistive agent that delivers objective feedback on linguistic features, whereas human intelligence in the form of self-assessment, peer, and teacher feedback supplies contextual depth, nuanced interpretation, and personalized guidance to interpret AI outputs, correct inaccuracies, mitigate biases, and convert feedback into actionable self-regulated learning (SRL) strategies. In a mixed-methods study designed to evaluate the effectiveness of NinjOrAltor, a hybrid intelligence platform, 161 undergraduate students in Hong Kong’s English-medium instruction (EMI) context showed a statistically significant 4.80% overall improvement in oral presentation performance, as indicated by paired pre- and post-intervention results (p < 0.001). Students reported valuing the transparency of the AI-generated vocal-filler detection tool as a catalyst for greater self-awareness, while crediting human feedback for improving the trustworthiness and interpretability of the AI output. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2026

Original language	English
Title of host publication	Artificial Intelligence in Education. Late Breaking Results, WideAIED, Practitioners, Industry and Policies, Blue Sky, Doctoral Consortium, FoL Workshops and Tutorials, FoL Invited Papers
Subtitle of host publication	27th International Conference, AIED 2026, Seoul, South Korea, June 27 – July 3, 2026, Proceedings, Part I
Editors	Emmanuel G. Blanchard, Guanliang Chen, Min Chi, Seiji Isotani
Place of Publication	Cham
Publisher	Springer
Pages	195-201
ISBN (Electronic)	978-3-032-29788-4
ISBN (Print)	978-3-032-29787-7
DOIs	https://doi.org/10.1007/978-3-032-29788-4_28
Publication status	Published - 2026
Event	27th International Conference on Artificial Intelligence in Education (AIED 2026): From Tools To Teammates: Human-AI Synergy For Augmented Learning - Seoul, Korea, Republic of Duration: 27 Jun 2026 → 3 Jul 2026 https://www.aied-conference.org/2026

Publication series

Name	Communications in Computer and Information Science
Volume	3031
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	27th International Conference on Artificial Intelligence in Education (AIED 2026)
Abbreviated title	AIED2026
Place	Korea, Republic of
City	Seoul
Period	27/06/26 → 3/07/26
Internet address	https://www.aied-conference.org/2026

Bibliographical note

Research Unit(s) information for this publication is provided by the author(s) concerned.

Funding

This research was supported by the University Grants Committee (UGC)’s Teaching Development Grant (TDG) Inter-institutional Collaborative Activities (IICA) 2022–25.

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 4 Quality Education

Research Keywords

Automatic speech recognition (ASR)
Hybrid intelligence
Oral presentations
Personalized feedback
Self-regulated learning (SRL)

Access Output

10.1007/978-3-032-29788-4_28

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Lai, P., Chan, C., Chen, J., Song, I. Y., & Chan, C. (2026). Bridging ASR Limitations with Human Judgment: A Hybrid Approach to Oral Presentation Feedback. In E. G. Blanchard, G. Chen, M. Chi, & S. Isotani (Eds.), Artificial Intelligence in Education. Late Breaking Results, WideAIED, Practitioners, Industry and Policies, Blue Sky, Doctoral Consortium, FoL Workshops and Tutorials, FoL Invited Papers: 27th International Conference, AIED 2026, Seoul, South Korea, June 27 – July 3, 2026, Proceedings, Part I (pp. 195-201). (Communications in Computer and Information Science; Vol. 3031). Springer . https://doi.org/10.1007/978-3-032-29788-4_28

Lai, Pauli ; Chan, Christy ; Chen, Julia et al. / Bridging ASR Limitations with Human Judgment : A Hybrid Approach to Oral Presentation Feedback. Artificial Intelligence in Education. Late Breaking Results, WideAIED, Practitioners, Industry and Policies, Blue Sky, Doctoral Consortium, FoL Workshops and Tutorials, FoL Invited Papers: 27th International Conference, AIED 2026, Seoul, South Korea, June 27 – July 3, 2026, Proceedings, Part I. editor / Emmanuel G. Blanchard ; Guanliang Chen ; Min Chi ; Seiji Isotani. Cham : Springer , 2026. pp. 195-201 (Communications in Computer and Information Science).

@inproceedings{63646b003ea8459c9aab7e950b76580e,

title = "Bridging ASR Limitations with Human Judgment: A Hybrid Approach to Oral Presentation Feedback",

abstract = "The growing popularity of AI speaking assessment platforms in educational technology is often attributed to their capacity to offer immediate, consistent, and scalable feedback, thereby supporting student self-learning. However, such platforms have been found to have suffered from notable limitations, such as the emphasis on surface-level speech elements and the potential algorithmic bias across the different Automatic Speech Recognition (ASR) tools adopted by the platforms. To mitigate these limitations of fully automated AI-based speaking assessment platforms, we developed NinjOrAItor, a hybrid intelligence platform designed to integrate artificial intelligence with human oversight and expertise. Within this framework, AI acts as an assistive agent that delivers objective feedback on linguistic features, whereas human intelligence in the form of self-assessment, peer, and teacher feedback supplies contextual depth, nuanced interpretation, and personalized guidance to interpret AI outputs, correct inaccuracies, mitigate biases, and convert feedback into actionable self-regulated learning (SRL) strategies. In a mixed-methods study designed to evaluate the effectiveness of NinjOrAltor, a hybrid intelligence platform, 161 undergraduate students in Hong Kong{\textquoteright}s English-medium instruction (EMI) context showed a statistically significant 4.80\% overall improvement in oral presentation performance, as indicated by paired pre- and post-intervention results (p < 0.001). Students reported valuing the transparency of the AI-generated vocal-filler detection tool as a catalyst for greater self-awareness, while crediting human feedback for improving the trustworthiness and interpretability of the AI output. {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2026",

keywords = "Automatic speech recognition (ASR), Hybrid intelligence, Oral presentations, Personalized feedback, Self-regulated learning (SRL)",

author = "Pauli Lai and Christy Chan and Julia Chen and Song, \{Irene Yiqiao\} and Chiho Chan",

note = "Research Unit(s) information for this publication is provided by the author(s) concerned. ; 27th International Conference on Artificial Intelligence in Education (AIED 2026) : From Tools To Teammates: Human-AI Synergy For Augmented Learning, AIED2026 ; Conference date: 27-06-2026 Through 03-07-2026",

year = "2026",

doi = "10.1007/978-3-032-29788-4\_28",

language = "English",

isbn = "978-3-032-29787-7",

series = "Communications in Computer and Information Science",

publisher = "Springer ",

pages = "195--201",

editor = "Blanchard, \{Emmanuel G. \} and Chen, \{Guanliang \} and Min Chi and Seiji Isotani",

booktitle = "Artificial Intelligence in Education. Late Breaking Results, WideAIED, Practitioners, Industry and Policies, Blue Sky, Doctoral Consortium, FoL Workshops and Tutorials, FoL Invited Papers",

url = "https://www.aied-conference.org/2026",

}

Lai, P, Chan, C, Chen, J, Song, IY & Chan, C 2026, Bridging ASR Limitations with Human Judgment: A Hybrid Approach to Oral Presentation Feedback. in EG Blanchard, G Chen, M Chi & S Isotani (eds), Artificial Intelligence in Education. Late Breaking Results, WideAIED, Practitioners, Industry and Policies, Blue Sky, Doctoral Consortium, FoL Workshops and Tutorials, FoL Invited Papers: 27th International Conference, AIED 2026, Seoul, South Korea, June 27 – July 3, 2026, Proceedings, Part I. Communications in Computer and Information Science, vol. 3031, Springer , Cham, pp. 195-201, 27th International Conference on Artificial Intelligence in Education (AIED 2026), Seoul, Korea, Republic of, 27/06/26. https://doi.org/10.1007/978-3-032-29788-4_28

Bridging ASR Limitations with Human Judgment: A Hybrid Approach to Oral Presentation Feedback. / Lai, Pauli; Chan, Christy; Chen, Julia et al.
Artificial Intelligence in Education. Late Breaking Results, WideAIED, Practitioners, Industry and Policies, Blue Sky, Doctoral Consortium, FoL Workshops and Tutorials, FoL Invited Papers: 27th International Conference, AIED 2026, Seoul, South Korea, June 27 – July 3, 2026, Proceedings, Part I. ed. / Emmanuel G. Blanchard; Guanliang Chen; Min Chi; Seiji Isotani. Cham: Springer , 2026. p. 195-201 (Communications in Computer and Information Science; Vol. 3031).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Bridging ASR Limitations with Human Judgment

T2 - 27th International Conference on Artificial Intelligence in Education (AIED 2026)

AU - Lai, Pauli

AU - Chan, Christy

AU - Chen, Julia

AU - Song, Irene Yiqiao

AU - Chan, Chiho

N1 - Research Unit(s) information for this publication is provided by the author(s) concerned.

PY - 2026

Y1 - 2026

N2 - The growing popularity of AI speaking assessment platforms in educational technology is often attributed to their capacity to offer immediate, consistent, and scalable feedback, thereby supporting student self-learning. However, such platforms have been found to have suffered from notable limitations, such as the emphasis on surface-level speech elements and the potential algorithmic bias across the different Automatic Speech Recognition (ASR) tools adopted by the platforms. To mitigate these limitations of fully automated AI-based speaking assessment platforms, we developed NinjOrAItor, a hybrid intelligence platform designed to integrate artificial intelligence with human oversight and expertise. Within this framework, AI acts as an assistive agent that delivers objective feedback on linguistic features, whereas human intelligence in the form of self-assessment, peer, and teacher feedback supplies contextual depth, nuanced interpretation, and personalized guidance to interpret AI outputs, correct inaccuracies, mitigate biases, and convert feedback into actionable self-regulated learning (SRL) strategies. In a mixed-methods study designed to evaluate the effectiveness of NinjOrAltor, a hybrid intelligence platform, 161 undergraduate students in Hong Kong’s English-medium instruction (EMI) context showed a statistically significant 4.80% overall improvement in oral presentation performance, as indicated by paired pre- and post-intervention results (p < 0.001). Students reported valuing the transparency of the AI-generated vocal-filler detection tool as a catalyst for greater self-awareness, while crediting human feedback for improving the trustworthiness and interpretability of the AI output. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2026

AB - The growing popularity of AI speaking assessment platforms in educational technology is often attributed to their capacity to offer immediate, consistent, and scalable feedback, thereby supporting student self-learning. However, such platforms have been found to have suffered from notable limitations, such as the emphasis on surface-level speech elements and the potential algorithmic bias across the different Automatic Speech Recognition (ASR) tools adopted by the platforms. To mitigate these limitations of fully automated AI-based speaking assessment platforms, we developed NinjOrAItor, a hybrid intelligence platform designed to integrate artificial intelligence with human oversight and expertise. Within this framework, AI acts as an assistive agent that delivers objective feedback on linguistic features, whereas human intelligence in the form of self-assessment, peer, and teacher feedback supplies contextual depth, nuanced interpretation, and personalized guidance to interpret AI outputs, correct inaccuracies, mitigate biases, and convert feedback into actionable self-regulated learning (SRL) strategies. In a mixed-methods study designed to evaluate the effectiveness of NinjOrAltor, a hybrid intelligence platform, 161 undergraduate students in Hong Kong’s English-medium instruction (EMI) context showed a statistically significant 4.80% overall improvement in oral presentation performance, as indicated by paired pre- and post-intervention results (p < 0.001). Students reported valuing the transparency of the AI-generated vocal-filler detection tool as a catalyst for greater self-awareness, while crediting human feedback for improving the trustworthiness and interpretability of the AI output. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2026

KW - Automatic speech recognition (ASR)

KW - Hybrid intelligence

KW - Oral presentations

KW - Personalized feedback

KW - Self-regulated learning (SRL)

UR - http://www.scopus.com/inward/record.url?scp=105043902212&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-105043902212&origin=recordpage

U2 - 10.1007/978-3-032-29788-4_28

DO - 10.1007/978-3-032-29788-4_28

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 978-3-032-29787-7

T3 - Communications in Computer and Information Science

SP - 195

EP - 201

BT - Artificial Intelligence in Education. Late Breaking Results, WideAIED, Practitioners, Industry and Policies, Blue Sky, Doctoral Consortium, FoL Workshops and Tutorials, FoL Invited Papers

A2 - Blanchard, Emmanuel G.

A2 - Chen, Guanliang

A2 - Chi, Min

A2 - Isotani, Seiji

PB - Springer

CY - Cham

Y2 - 27 June 2026 through 3 July 2026

ER -

Lai P, Chan C, Chen J, Song IY, Chan C. Bridging ASR Limitations with Human Judgment: A Hybrid Approach to Oral Presentation Feedback. In Blanchard EG, Chen G, Chi M, Isotani S, editors, Artificial Intelligence in Education. Late Breaking Results, WideAIED, Practitioners, Industry and Policies, Blue Sky, Doctoral Consortium, FoL Workshops and Tutorials, FoL Invited Papers: 27th International Conference, AIED 2026, Seoul, South Korea, June 27 – July 3, 2026, Proceedings, Part I. Cham: Springer . 2026. p. 195-201. (Communications in Computer and Information Science). Epub 2026 Jun 27. doi: 10.1007/978-3-032-29788-4_28

Bridging ASR Limitations with Human Judgment: A Hybrid Approach to Oral Presentation Feedback

Abstract

Publication series

Conference

Bibliographical note

Funding

UN SDGs

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this