ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph

Langming Liu; Haibin Chen; Yuhao Wang; Yujin Yuan; Shilei Liu; Wenbo Su; Xiangyu Zhao; Bo Zheng

doi:10.1145/3746252.3761613

ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph

Langming Liu
, Haibin Chen
, Yuhao Wang
, Yujin Yuan
, Shilei Liu
, Wenbo Su
, Xiangyu Zhao
, Bo Zheng^*

^*Corresponding author for this work

Department of Data Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

Abstract

Large language models (LLMs) have demonstrated their capabilities across various natural language processing (NLP) tasks. Their potential in e-commerce is also substantial, evidenced by existing implementations in scenarios such as platform search and recommender systems. One obstinate concern associated with LLMs is the factuality issue (e.g., hallucination), which is urgent in e-commerce due to its significant impact on user experience and revenue. While some methods aim to evaluate the factuality of LLMs, issues such as lack of objectivity, high consumption, and lack of domain expertise arise. To this end, leveraging a collected knowledge graph (KG) as a reliable source, we propose ECKGBench, a question-answering dataset to assess LLMs' capacity in e-commerce. Specifically, each question is automatically generated based on one KG triple through a standardized pipeline, guaranteeing evaluation quality and reliability. We evaluate advanced LLMs using ECKGBench and provide insights into experimental results. The dataset is available online at∼ https://github.com/OpenStellarTeam/ECKGBench. © 2025 Copyright held by the owner/author(s).

Original language	English
Title of host publication	CIKM '25 - Proceedings of the 34th ACM International Conference on Information and Knowledge Management
Publisher	Association for Computing Machinery
Pages	6461-6465
ISBN (Print)	9798400720406
DOIs	https://doi.org/10.1145/3746252.3761613
Publication status	Published - Nov 2025
Event	34th ACM International Conference on Information and Knowledge Management (CIKM 2025) - COEX, Seoul, Korea, Republic of Duration: 10 Nov 2025 → 14 Nov 2025 https://cikm2025.org/

Publication series

Name	CIKM - Proceedings of the ACM International Conference on Information and Knowledge Management

Conference

Conference	34th ACM International Conference on Information and Knowledge Management (CIKM 2025)
Abbreviated title	CIKM '25
Place	Korea, Republic of
City	Seoul
Period	10/11/25 → 14/11/25
Internet address	https://cikm2025.org/

Bibliographical note

Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s).

Research Keywords

e-commerce
factuality evaluation
large language models

Access Output

10.1145/3746252.3761613

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Liu, L., Chen, H., Wang, Y., Yuan, Y., Liu, S., Su, W., Zhao, X., & Zheng, B. (2025). ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph. In CIKM '25 - Proceedings of the 34th ACM International Conference on Information and Knowledge Management (pp. 6461-6465). (CIKM - Proceedings of the ACM International Conference on Information and Knowledge Management). Association for Computing Machinery. https://doi.org/10.1145/3746252.3761613

Liu, Langming ; Chen, Haibin ; Wang, Yuhao et al. / ECKGBench : Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph. CIKM '25 - Proceedings of the 34th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, 2025. pp. 6461-6465 (CIKM - Proceedings of the ACM International Conference on Information and Knowledge Management).

@inproceedings{30e73c02d3734df9b4a53f2124ff7341,

title = "ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph",

abstract = "Large language models (LLMs) have demonstrated their capabilities across various natural language processing (NLP) tasks. Their potential in e-commerce is also substantial, evidenced by existing implementations in scenarios such as platform search and recommender systems. One obstinate concern associated with LLMs is the factuality issue (e.g., hallucination), which is urgent in e-commerce due to its significant impact on user experience and revenue. While some methods aim to evaluate the factuality of LLMs, issues such as lack of objectivity, high consumption, and lack of domain expertise arise. To this end, leveraging a collected knowledge graph (KG) as a reliable source, we propose ECKGBench, a question-answering dataset to assess LLMs' capacity in e-commerce. Specifically, each question is automatically generated based on one KG triple through a standardized pipeline, guaranteeing evaluation quality and reliability. We evaluate advanced LLMs using ECKGBench and provide insights into experimental results. The dataset is available online at∼ https://github.com/OpenStellarTeam/ECKGBench. {\textcopyright} 2025 Copyright held by the owner/author(s).",

keywords = "e-commerce, factuality evaluation, large language models",

author = "Langming Liu and Haibin Chen and Yuhao Wang and Yujin Yuan and Shilei Liu and Wenbo Su and Xiangyu Zhao and Bo Zheng",

note = "Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s).; 34th ACM International Conference on Information and Knowledge Management (CIKM 2025), CIKM '25 ; Conference date: 10-11-2025 Through 14-11-2025",

year = "2025",

month = nov,

doi = "10.1145/3746252.3761613",

language = "English",

isbn = "9798400720406",

series = "CIKM - Proceedings of the ACM International Conference on Information and Knowledge Management",

publisher = "Association for Computing Machinery",

pages = "6461--6465",

booktitle = "CIKM '25 - Proceedings of the 34th ACM International Conference on Information and Knowledge Management",

address = "United States",

url = "https://cikm2025.org/",

}

Liu, L, Chen, H, Wang, Y, Yuan, Y, Liu, S, Su, W, Zhao, X & Zheng, B 2025, ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph. in CIKM '25 - Proceedings of the 34th ACM International Conference on Information and Knowledge Management. CIKM - Proceedings of the ACM International Conference on Information and Knowledge Management, Association for Computing Machinery, pp. 6461-6465, 34th ACM International Conference on Information and Knowledge Management (CIKM 2025), Seoul, Korea, Republic of, 10/11/25. https://doi.org/10.1145/3746252.3761613

ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph. / Liu, Langming; Chen, Haibin; Wang, Yuhao et al.
CIKM '25 - Proceedings of the 34th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery, 2025. p. 6461-6465 (CIKM - Proceedings of the ACM International Conference on Information and Knowledge Management).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - ECKGBench

T2 - 34th ACM International Conference on Information and Knowledge Management (CIKM 2025)

AU - Liu, Langming

AU - Chen, Haibin

AU - Wang, Yuhao

AU - Yuan, Yujin

AU - Liu, Shilei

AU - Su, Wenbo

AU - Zhao, Xiangyu

AU - Zheng, Bo

N1 - Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s).

PY - 2025/11

Y1 - 2025/11

N2 - Large language models (LLMs) have demonstrated their capabilities across various natural language processing (NLP) tasks. Their potential in e-commerce is also substantial, evidenced by existing implementations in scenarios such as platform search and recommender systems. One obstinate concern associated with LLMs is the factuality issue (e.g., hallucination), which is urgent in e-commerce due to its significant impact on user experience and revenue. While some methods aim to evaluate the factuality of LLMs, issues such as lack of objectivity, high consumption, and lack of domain expertise arise. To this end, leveraging a collected knowledge graph (KG) as a reliable source, we propose ECKGBench, a question-answering dataset to assess LLMs' capacity in e-commerce. Specifically, each question is automatically generated based on one KG triple through a standardized pipeline, guaranteeing evaluation quality and reliability. We evaluate advanced LLMs using ECKGBench and provide insights into experimental results. The dataset is available online at∼ https://github.com/OpenStellarTeam/ECKGBench. © 2025 Copyright held by the owner/author(s).

AB - Large language models (LLMs) have demonstrated their capabilities across various natural language processing (NLP) tasks. Their potential in e-commerce is also substantial, evidenced by existing implementations in scenarios such as platform search and recommender systems. One obstinate concern associated with LLMs is the factuality issue (e.g., hallucination), which is urgent in e-commerce due to its significant impact on user experience and revenue. While some methods aim to evaluate the factuality of LLMs, issues such as lack of objectivity, high consumption, and lack of domain expertise arise. To this end, leveraging a collected knowledge graph (KG) as a reliable source, we propose ECKGBench, a question-answering dataset to assess LLMs' capacity in e-commerce. Specifically, each question is automatically generated based on one KG triple through a standardized pipeline, guaranteeing evaluation quality and reliability. We evaluate advanced LLMs using ECKGBench and provide insights into experimental results. The dataset is available online at∼ https://github.com/OpenStellarTeam/ECKGBench. © 2025 Copyright held by the owner/author(s).

KW - e-commerce

KW - factuality evaluation

KW - large language models

UR - https://www.scopus.com/pages/publications/105023165018

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-105023165018&origin=recordpage

U2 - 10.1145/3746252.3761613

DO - 10.1145/3746252.3761613

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 9798400720406

T3 - CIKM - Proceedings of the ACM International Conference on Information and Knowledge Management

SP - 6461

EP - 6465

BT - CIKM '25 - Proceedings of the 34th ACM International Conference on Information and Knowledge Management

PB - Association for Computing Machinery

Y2 - 10 November 2025 through 14 November 2025

ER -

Liu L, Chen H, Wang Y, Yuan Y, Liu S, Su W et al. ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph. In CIKM '25 - Proceedings of the 34th ACM International Conference on Information and Knowledge Management. Association for Computing Machinery. 2025. p. 6461-6465. (CIKM - Proceedings of the ACM International Conference on Information and Knowledge Management). Epub 2025 Nov 10. doi: 10.1145/3746252.3761613

ECKGBench: Benchmarking Large Language Models in E-commerce Leveraging Knowledge Graph

Abstract

Publication series

Conference

Bibliographical note

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this