Reference decisions enhance LLM performance, amplified by source disclosure

Yongxiang Zhang; Zhaobin Liu; Shaosen Bai; Ting Xu; Raymond YK Lau

doi:10.1177/20552076251342078

Reference decisions enhance LLM performance, amplified by source disclosure

Yongxiang Zhang
, Zhaobin Liu
, Shaosen Bai
, Ting Xu^*
, Raymond YK Lau^*

^*Corresponding author for this work

Department of Information Systems

Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review

2 Citations (Scopus)

22 Downloads (CityUHK Scholars)

Abstract

Objective: The rapid integration of large language models (LLMs) has propelled advancements in automated dialog technologies, improving the public's access to healthcare services. Drawing inspiration from the collaborative decision-making practices of medical professionals in complex cases, we investigated whether LLMs could enhance their diagnostic accuracy through interaction.

Methods: An experimental study was conducted in China (September–December 2024) to investigate the impact of LLM-generated reference decisions and source disclosure on LLMs’ diagnostic performance. We used a Chinese clinical diagnostic task in a controlled comparative design, where three Chinese LLMs interpreted symptoms and conditions based on patient queries. LLMs’ outcomes were evaluated through accuracy and weighted F1 score metrics, with statistical analysis to determine significance.

Results: Analysis of variance on LLMs’ diagnostic accuracy scores demonstrated that incorporating LLM-generated decisions as a reference significantly improved diagnostic outcomes, with source disclosure amplifying this improvement.

Conclusion: Our findings underscore the potential of LLM collaboration in healthcare, offering strategies to refine response generation and decision-making across various applications.

© The Author(s) 2025.

Original language	English
Article number	20552076251342078
Journal	Digital Health
Volume	11
Online published	23 May 2025
DOIs	https://doi.org/10.1177/20552076251342078
Publication status	Published - 2025

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Research Grants Council of the Hong Kong Special Administrative Region, China (Grant No. CityU 11507323).

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being
SDG 10 Reduced Inequalities
SDG 16 Peace, Justice and Strong Institutions

Research Keywords

clinical diagnostics
healthcare application
Large language model
reference decision
source disclosure

Publisher's Copyright Statement

This full text is made available under CC-BY-NC-ND 4.0. https://creativecommons.org/licenses/by-nc-nd/4.0/

RGC Funding Information

RGC-funded

Access Output

10.1177/20552076251342078

288142813.pdfFinal Published version, 769 KBLicence: CC BY-NC-ND

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@article{d1a4915b98d24d85975684d2caeb425e,

title = "Reference decisions enhance LLM performance, amplified by source disclosure",

abstract = "Objective: The rapid integration of large language models (LLMs) has propelled advancements in automated dialog technologies, improving the public's access to healthcare services. Drawing inspiration from the collaborative decision-making practices of medical professionals in complex cases, we investigated whether LLMs could enhance their diagnostic accuracy through interaction. Methods: An experimental study was conducted in China (September–December 2024) to investigate the impact of LLM-generated reference decisions and source disclosure on LLMs{\textquoteright} diagnostic performance. We used a Chinese clinical diagnostic task in a controlled comparative design, where three Chinese LLMs interpreted symptoms and conditions based on patient queries. LLMs{\textquoteright} outcomes were evaluated through accuracy and weighted F1 score metrics, with statistical analysis to determine significance. Results: Analysis of variance on LLMs{\textquoteright} diagnostic accuracy scores demonstrated that incorporating LLM-generated decisions as a reference significantly improved diagnostic outcomes, with source disclosure amplifying this improvement. Conclusion: Our findings underscore the potential of LLM collaboration in healthcare, offering strategies to refine response generation and decision-making across various applications. {\textcopyright} The Author(s) 2025.",

keywords = "clinical diagnostics, healthcare application, Large language model, reference decision, source disclosure",

author = "Yongxiang Zhang and Zhaobin Liu and Shaosen Bai and Ting Xu and Lau, \{Raymond YK\}",

year = "2025",

doi = "10.1177/20552076251342078",

language = "English",

volume = "11",

journal = "Digital Health",

issn = "2055-2076",

publisher = "Sage Publications",

}

TY - JOUR

T1 - Reference decisions enhance LLM performance, amplified by source disclosure

AU - Zhang, Yongxiang

AU - Liu, Zhaobin

AU - Bai, Shaosen

AU - Xu, Ting

AU - Lau, Raymond YK

PY - 2025

Y1 - 2025

N2 - Objective: The rapid integration of large language models (LLMs) has propelled advancements in automated dialog technologies, improving the public's access to healthcare services. Drawing inspiration from the collaborative decision-making practices of medical professionals in complex cases, we investigated whether LLMs could enhance their diagnostic accuracy through interaction. Methods: An experimental study was conducted in China (September–December 2024) to investigate the impact of LLM-generated reference decisions and source disclosure on LLMs’ diagnostic performance. We used a Chinese clinical diagnostic task in a controlled comparative design, where three Chinese LLMs interpreted symptoms and conditions based on patient queries. LLMs’ outcomes were evaluated through accuracy and weighted F1 score metrics, with statistical analysis to determine significance. Results: Analysis of variance on LLMs’ diagnostic accuracy scores demonstrated that incorporating LLM-generated decisions as a reference significantly improved diagnostic outcomes, with source disclosure amplifying this improvement. Conclusion: Our findings underscore the potential of LLM collaboration in healthcare, offering strategies to refine response generation and decision-making across various applications. © The Author(s) 2025.

AB - Objective: The rapid integration of large language models (LLMs) has propelled advancements in automated dialog technologies, improving the public's access to healthcare services. Drawing inspiration from the collaborative decision-making practices of medical professionals in complex cases, we investigated whether LLMs could enhance their diagnostic accuracy through interaction. Methods: An experimental study was conducted in China (September–December 2024) to investigate the impact of LLM-generated reference decisions and source disclosure on LLMs’ diagnostic performance. We used a Chinese clinical diagnostic task in a controlled comparative design, where three Chinese LLMs interpreted symptoms and conditions based on patient queries. LLMs’ outcomes were evaluated through accuracy and weighted F1 score metrics, with statistical analysis to determine significance. Results: Analysis of variance on LLMs’ diagnostic accuracy scores demonstrated that incorporating LLM-generated decisions as a reference significantly improved diagnostic outcomes, with source disclosure amplifying this improvement. Conclusion: Our findings underscore the potential of LLM collaboration in healthcare, offering strategies to refine response generation and decision-making across various applications. © The Author(s) 2025.

KW - clinical diagnostics

KW - healthcare application

KW - Large language model

KW - reference decision

KW - source disclosure

UR - https://www.scopus.com/pages/publications/105005967917

UR - https://www.scopus.com/pages/publications/105005967917#tab=impact

U2 - 10.1177/20552076251342078

DO - 10.1177/20552076251342078

M3 - RGC 21 - Publication in refereed journal

SN - 2055-2076

VL - 11

JO - Digital Health

JF - Digital Health

M1 - 20552076251342078

ER -

Reference decisions enhance LLM performance, amplified by source disclosure

Abstract

Funding

UN SDGs

Research Keywords

Publisher's Copyright Statement

RGC Funding Information

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

GRF: Multimodal Data Augmentation-enhanced Deep Learning Framework for Corporate Credit Risk Assessment

Cite this

Reference decisions enhance LLM performance, amplified by source disclosure

Abstract

Funding

UN SDGs

Research Keywords

Publisher's Copyright Statement

RGC Funding Information

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Projects

GRF: Multimodal Data Augmentation-enhanced Deep Learning Framework for Corporate Credit Risk Assessment

Cite this