Uncertainty-Based Metamorphic Testing for Validating Plagiarism Detection Systems

Pak Yuen Patrick Chan; Jacky Keung; Zhen Yang

doi:10.1007/978-981-99-8255-4_26

Uncertainty-Based Metamorphic Testing for Validating Plagiarism Detection Systems

Pak Yuen Patrick Chan^*, Jacky Keung, Zhen Yang

^*Corresponding author for this work

Department of Computer Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

Abstract

Plagiarism is a severe issue in academia, and uncertainty in plagiarism detection systems might lead to inconsistent detections. Thus, evaluating the system is essential; however, it is also a test oracle problem as it is challenging to distinguish correct behaviour from potentially incorrect behaviour of the system. To alleviate this challenge, we develop a feasible approach by applying an uncertainty matrix to identify the uncertainty of the plagiarism detection systems and derive metamorphic relations of metamorphic testing from the identified uncertainty for validation. We experimented with three plagiarism detection systems in a classroom scenario where students were hypothesized to use tools to generate answers for assignments. These answers were fed into the systems for validation by comparing the systems’ similarity scores of the tool-generated answers. Results showed that the proposed approach can effectively validate plagiarism detection systems. Future studies can apply this approach to locate uncertainties to enhance systems’ robustness. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024

Original language	English
Title of host publication	Technology in Education. Innovative Practices for the New Normal
Subtitle of host publication	6th International Conference on Technology in Education, ICTE 2023, Hong Kong, China, December 19–21, 2023, Proceedings
Editors	Simon K. S. Cheung, Fu Lee Wang, Naraphorn Paoprasert, Peerayuth Charnsethikul, Kam Cheong Li, Kongkiti Phusavat
Publisher	Springer
Pages	299-314
ISBN (Electronic)	978-981-99-8255-4
ISBN (Print)	978-981-99-8254-7
DOIs	https://doi.org/10.1007/978-981-99-8255-4_26
Publication status	Published - 2024
Event	6th International Conference on Technology in Education (ICTE 2023) - Hong Kong, China Duration: 19 Dec 2023 → 21 Dec 2023

Publication series

Name	Communications in Computer and Information Science
Volume	1974
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	6th International Conference on Technology in Education (ICTE 2023)
Place	China
City	Hong Kong
Period	19/12/23 → 21/12/23

Funding

This work is supported in part by the General Research Fund of the Research Grants Council of Hong Kong and the research funds of the City University of Hong Kong (6000796).

Research Keywords

Validation
Metamorphic testing
Uncertainty
Natural language processing
Plagiarism detection

Access Output

10.1007/978-981-99-8255-4_26

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Chan, P. Y. P., Keung, J., & Yang, Z. (2024). Uncertainty-Based Metamorphic Testing for Validating Plagiarism Detection Systems. In S. K. S. Cheung, F. L. Wang, N. Paoprasert, P. Charnsethikul, K. C. Li, & K. Phusavat (Eds.), Technology in Education. Innovative Practices for the New Normal: 6th International Conference on Technology in Education, ICTE 2023, Hong Kong, China, December 19–21, 2023, Proceedings (pp. 299-314). (Communications in Computer and Information Science; Vol. 1974). Springer . https://doi.org/10.1007/978-981-99-8255-4_26

Chan, Pak Yuen Patrick ; Keung, Jacky ; Yang, Zhen. / Uncertainty-Based Metamorphic Testing for Validating Plagiarism Detection Systems. Technology in Education. Innovative Practices for the New Normal: 6th International Conference on Technology in Education, ICTE 2023, Hong Kong, China, December 19–21, 2023, Proceedings. editor / Simon K. S. Cheung ; Fu Lee Wang ; Naraphorn Paoprasert ; Peerayuth Charnsethikul ; Kam Cheong Li ; Kongkiti Phusavat. Springer , 2024. pp. 299-314 (Communications in Computer and Information Science).

@inproceedings{cf1e3d09d1254a72be7041f559ed1210,

title = "Uncertainty-Based Metamorphic Testing for Validating Plagiarism Detection Systems",

abstract = "Plagiarism is a severe issue in academia, and uncertainty in plagiarism detection systems might lead to inconsistent detections. Thus, evaluating the system is essential; however, it is also a test oracle problem as it is challenging to distinguish correct behaviour from potentially incorrect behaviour of the system. To alleviate this challenge, we develop a feasible approach by applying an uncertainty matrix to identify the uncertainty of the plagiarism detection systems and derive metamorphic relations of metamorphic testing from the identified uncertainty for validation. We experimented with three plagiarism detection systems in a classroom scenario where students were hypothesized to use tools to generate answers for assignments. These answers were fed into the systems for validation by comparing the systems{\textquoteright} similarity scores of the tool-generated answers. Results showed that the proposed approach can effectively validate plagiarism detection systems. Future studies can apply this approach to locate uncertainties to enhance systems{\textquoteright} robustness. {\textcopyright} The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024",

keywords = "Validation, Metamorphic testing, Uncertainty, Natural language processing, Plagiarism detection",

author = "Chan, {Pak Yuen Patrick} and Jacky Keung and Zhen Yang",

year = "2024",

doi = "10.1007/978-981-99-8255-4_26",

language = "English",

isbn = "978-981-99-8254-7",

series = "Communications in Computer and Information Science",

publisher = "Springer ",

pages = "299--314",

editor = "Cheung, {Simon K. S.} and Wang, {Fu Lee} and Naraphorn Paoprasert and Peerayuth Charnsethikul and Li, {Kam Cheong} and Kongkiti Phusavat",

booktitle = "Technology in Education. Innovative Practices for the New Normal",

note = "6th International Conference on Technology in Education (ICTE 2023) ; Conference date: 19-12-2023 Through 21-12-2023",

}

Chan, PYP , Keung, J & Yang, Z 2024, Uncertainty-Based Metamorphic Testing for Validating Plagiarism Detection Systems. in SKS Cheung, FL Wang, N Paoprasert, P Charnsethikul, KC Li & K Phusavat (eds), Technology in Education. Innovative Practices for the New Normal: 6th International Conference on Technology in Education, ICTE 2023, Hong Kong, China, December 19–21, 2023, Proceedings. Communications in Computer and Information Science, vol. 1974, Springer , pp. 299-314, 6th International Conference on Technology in Education (ICTE 2023), Hong Kong, China, 19/12/23. https://doi.org/10.1007/978-981-99-8255-4_26

Uncertainty-Based Metamorphic Testing for Validating Plagiarism Detection Systems. / Chan, Pak Yuen Patrick ; Keung, Jacky ; Yang, Zhen.
Technology in Education. Innovative Practices for the New Normal: 6th International Conference on Technology in Education, ICTE 2023, Hong Kong, China, December 19–21, 2023, Proceedings. ed. / Simon K. S. Cheung; Fu Lee Wang; Naraphorn Paoprasert; Peerayuth Charnsethikul; Kam Cheong Li; Kongkiti Phusavat. Springer , 2024. p. 299-314 (Communications in Computer and Information Science; Vol. 1974).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Uncertainty-Based Metamorphic Testing for Validating Plagiarism Detection Systems

AU - Chan, Pak Yuen Patrick

AU - Keung, Jacky

AU - Yang, Zhen

PY - 2024

Y1 - 2024

N2 - Plagiarism is a severe issue in academia, and uncertainty in plagiarism detection systems might lead to inconsistent detections. Thus, evaluating the system is essential; however, it is also a test oracle problem as it is challenging to distinguish correct behaviour from potentially incorrect behaviour of the system. To alleviate this challenge, we develop a feasible approach by applying an uncertainty matrix to identify the uncertainty of the plagiarism detection systems and derive metamorphic relations of metamorphic testing from the identified uncertainty for validation. We experimented with three plagiarism detection systems in a classroom scenario where students were hypothesized to use tools to generate answers for assignments. These answers were fed into the systems for validation by comparing the systems’ similarity scores of the tool-generated answers. Results showed that the proposed approach can effectively validate plagiarism detection systems. Future studies can apply this approach to locate uncertainties to enhance systems’ robustness. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024

AB - Plagiarism is a severe issue in academia, and uncertainty in plagiarism detection systems might lead to inconsistent detections. Thus, evaluating the system is essential; however, it is also a test oracle problem as it is challenging to distinguish correct behaviour from potentially incorrect behaviour of the system. To alleviate this challenge, we develop a feasible approach by applying an uncertainty matrix to identify the uncertainty of the plagiarism detection systems and derive metamorphic relations of metamorphic testing from the identified uncertainty for validation. We experimented with three plagiarism detection systems in a classroom scenario where students were hypothesized to use tools to generate answers for assignments. These answers were fed into the systems for validation by comparing the systems’ similarity scores of the tool-generated answers. Results showed that the proposed approach can effectively validate plagiarism detection systems. Future studies can apply this approach to locate uncertainties to enhance systems’ robustness. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2024

KW - Validation

KW - Metamorphic testing

KW - Uncertainty

KW - Natural language processing

KW - Plagiarism detection

UR - http://www.scopus.com/inward/record.url?scp=85177190598&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85177190598&origin=recordpage

U2 - 10.1007/978-981-99-8255-4_26

DO - 10.1007/978-981-99-8255-4_26

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 978-981-99-8254-7

T3 - Communications in Computer and Information Science

SP - 299

EP - 314

BT - Technology in Education. Innovative Practices for the New Normal

A2 - Cheung, Simon K. S.

A2 - Wang, Fu Lee

A2 - Paoprasert, Naraphorn

A2 - Charnsethikul, Peerayuth

A2 - Li, Kam Cheong

A2 - Phusavat, Kongkiti

PB - Springer

T2 - 6th International Conference on Technology in Education (ICTE 2023)

Y2 - 19 December 2023 through 21 December 2023

ER -

Chan PYP , Keung J , Yang Z. Uncertainty-Based Metamorphic Testing for Validating Plagiarism Detection Systems. In Cheung SKS, Wang FL, Paoprasert N, Charnsethikul P, Li KC, Phusavat K, editors, Technology in Education. Innovative Practices for the New Normal: 6th International Conference on Technology in Education, ICTE 2023, Hong Kong, China, December 19–21, 2023, Proceedings. Springer . 2024. p. 299-314. (Communications in Computer and Information Science). Epub 2023 Nov 9. doi: 10.1007/978-981-99-8255-4_26

Uncertainty-Based Metamorphic Testing for Validating Plagiarism Detection Systems

Abstract

Publication series

Conference

Funding

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

TDG(CityU): Innovative Learning using Gamification to enhance Teaching and Learning Software Engineering

Cite this

Uncertainty-Based Metamorphic Testing for Validating Plagiarism Detection Systems

Abstract

Publication series

Conference

Funding

Research Keywords

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Projects

TDG(CityU): Innovative Learning using Gamification to enhance Teaching and Learning Software Engineering

Cite this