An Ensemble Text Classification-based Hierarchical Fine-tuning Approach for Scientific Large Language Models

Abstract

This dissertation explores how large language models (LLMs) can learn knowledge from scientific documents and how they can assist with scientific management tasks. More specifically, this dissertation designed an ensemble text classification-based large language model fine-tuning framework for better LLMs learning. This framework contains two components, which are represented as two studies. One investigates the knowledge fusion of human experts and deep learning algorithms for better scientific text classification. Based on the first study, the other one studies the learning path of LLMs within a hierarchical knowledge context for specialized disciplinary large language models.

The first essay focus on scientific knowledge comprehension of human experts and deep learning algorithms. In proposal peer review matching context, existing studies mainly apply the rule-based or deep learning method to capture the different sources of scientific knowledge to predict the target proposal into the proper discipline. However, existing studies fail to utilize academic experts’ knowledge, which could be a strong support for scientific knowledge comprehension. To fill this gap, an expert-driven rule-based method is first introduced to inject the experts’ knowledge into the rule-based models. Then, an ensemble model for proposal classification is proposed by fully taking advantage of expert-driven rule-based and deep learning methods in terms of expert knowledge and contextual semantic knowledge enhancement. The experiment results show that the proposed ensemble model produces higher classification accuracy than any base methods. The performance of the ensemble model indicates its advantage in scientific proposal classification, which can effectively assist following disciplinary hierarchical LLMs fine-tuning.

The second essay focuses on how LLMs learn within a hierarchical knowledge context. The incorporation of LLMs into academia has garnered considerable attention for their potential in academic writing assisting capability. However, existing studies mainly use domain-specific data for one-step fine-tuning, often omitting the hierarchical structure of scientific knowledge. To address this gap, this essay proposes a hierarchical two-stage fine-tuning method specially for academic domains, and I hypothesize that combining relevant neighbor disciplines can also enhance model performance for scientific downstream tasks within a target discipline. A case study on the targeted computer science discipline was conducted to test our hypothesis. Experimental results show that the hierarchical two-stage fine-tuned model outperforms traditional one-stage fine-tuned models. This work theoretically demonstrates that LLMs should be learned in hierarchical stages and explains how disciplinary relevance positively influences the performance of LLMs in downstream tasks.

Date of Award	9 Sept 2024
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Jian MA (Supervisor)