Intention-based Modular Retrieval-Augmented Generation for Research Policy Question Answering System

Abstract

In the e-governance era, efficient access and accurate understanding of government policies are crucial for social development and public welfare. Government policy question answering (QA) systems have become vital tools to boost administrative efficiency, bridging the gap between policymakers and the public by offering quick, precise responses to policy queries. The key to solving the core task of question-answering system is to recognize the intention behind user's inquiry. Intentions in QA systems are usually designed and mapped to pre-defined conversational categories in an intention schema. However, existing government QA systems often fall short in recognize the underlying policy inquiry intentions due to the following critical challenges in government context. Firstly, manually outlining a comprehensive fine-grained policy inquiry intention framework is difficult. Secondly, policy inquiry intentions for specific government affairs are interrelated and dispersed across various policy documents. Lastly, the linguistic complexity of government policy jargons poses significant barriers for alignment between policy inquiry and relevant policy documents.

The previous studies on traditional QA system has evolved from statistical-based to neural-based approaches, yet these methods remain inadequate for recognizing underlying policy inquiry intents in government contexts. The state-of-the-art Large Language Models (LLMs) have advanced QA systems paradigm through their advanced semantic understanding capabilities under Retrieval-Augmented Generation (RAG) framework. However, most existing RAG-based QA solutions adopt a single-channel architecture, where LLMs serially process inputs through one unified pipeline without validating their outputs. Such single-channel design brings various issues such as limited retrieval information sources and limited in-domain understanding that fails to address data biases when processing complex queries. These issues lead to hallucinatory response, producing disastrously incorrect responses which is totally intolerable for high factuality-demanding requirement in government QA context.

To address these challenges, this thesis proposes an Intention-based Modular RAG system within a multi-channel framework for government research policy question answering. Specifically, the system consists of multiple modules specifically designed for policy QA tasks in 3 stages of retrieval: (1) In pre-retrieval stage, a few-shot prompting-based Query Re-writer is utilized, which transforms original query to mimic the writing style of semantically similar queries in the local knowledge base. (2) The retrieval stage uses a Policy Intention Recognition Module and a Policy Intention Knowledge Graph-based Pruning Module to identify core policy inquiry intentions for each rewritten query. A threshold-based retrieval weighting formula is utilized to merge query retrieval and policy intention keyword retrieval to produce a ranking list for potential policy answers for each independent retrieval channel. (3) The post-retrieval stage employs a weighted Reciprocal Rank Fusion (RRF) and LLM joint ensemble strategy to generate one final ranking list from multiple retrievers. The multi-channel ensemble method mitigate variance in single retrieval channels and enhancing output consistency by reranking policy answers across all independent retrieval channels.

The main experiment results with ablation studies and analysis across query and policy categories validate that our proposed method outperforms baseline models and reduced models, demonstrating the effectiveness of our designed framework for government QA tasks. We draw the following conclusions from the study: (1) Our intention-based policy QA model identifies core policy inquiry intentions via two steps through intention prediction and intention entity pruning. Respectively, performance improvement are primarily from policy categories showing higher intention keyword distribution balance and shorter average intention entity distances in the Policy Intention KG. (2) Our Policy Intention KG-enhanced pruning strategy collaborates with the query rewriter module to reduce variance from excessive query rewriting and enhance overall retrieval performance. (3) The Multi-channel setting performs best with our proposed ensemble strategy and outperforms the results from Single-channel setting. (4) LLMs combined with traditional retrieval algorithms can effectively boost retrieval performance across varying LLM configurations and data sample characteristics, such as short queries get more performance gains under few-shot prompt guidance in our experiment.

The contributions of this thesis are as follows: Firstly, we present a Policy Intention Knowledge Graph-enhanced Intention Recognition solution for policy QA, using LLMs to automatically extract fine-grained policy keywords from documents and eliminating the need for time-consuming manual intention mapping to conversational categories in previous studies. Experiments validate the design's effectiveness in aligning user queries with policy answers via standardized policy intention keywords as an intermediary. Secondly, we introduce a multi-channel RAG framework combined with an ensemble ranking strategy, which effectively alleviates the hallucination and variance problems common in existing single-channel RAG systems. Thirdly, our study presents pioneering work in integrating a modular RAG architecture into policy QA systems design, discovering strong interconnections and mutual reinforcement among proposed modules in different retrieval-stage. These synergistic working modules leverage their combined capabilities to enhance the overall accuracy of policy question answering.

Date of Award	22 Oct 2025
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Jian MA (Supervisor) & Junming LIU (Supervisor)