External Knowledge-Guided Intelligent Medical Image Analysis

Abstract

Medical imaging plays a crucial role in clinical practice, offering non-invasive visualization and assessment of diseases. Recent advances in deep learning have significantly improved automatic medical image analysis, leading to remarkable gains in diagnostic accuracy and disease characterization. The integration of external knowledge, such as expert annotations and structured medical ontologies, has emerged as a promising approach to enhance image interpretation and strengthen clinical decision-making. Despite its potential, acquiring external knowledge for medical data remains a challenging endeavor. Expert annotations typically demand significant time and specialized expertise, while privacy regulations often restrict access to medical resources, limiting the availability of valuable external knowledge for clinical applications. This thesis addresses several key challenges in medical image analysis by generating external knowledge, with a focus on data scarcity, emerging disease adaptation, and resolution degradation. Specifically, (1) The scarcity of rare disease cases limits the availability of sufficient labeled data, leading to biased and incomplete disease knowledge that undermines the reliability and generalizability of diagnostic models. (2) Data privacy regulations often limit access to historical medical data, hindering learned knowledge retention during model adaptation to emerging diseases and leading to catastrophic forgetting, which compromises overall diagnostic performance. (3) Limited access to advanced imaging equipment restricts the acquisition of high-quality medical images. Resolution degradation of medical images compromises critical pathological features, diminishing accurate disease assessment and diagnosis.

In this thesis, we present a series of techniques that address the aforementioned challenges by generating and leveraging external knowledge. First, to overcome the limitations of diagnostic models in dealing with unknown rare diseases, we propose to generate external reciprocal samples and design a Semantic-guided unknown-aware Rare Disease Diagnosis (SRDD) model. We first introduce a Semantic-guided Saliency Discovery (SSD) module to explore semantic saliency information within images using category labels and decompose images into semantically related information (SRI) and image instance template (IIT). A Reciprocal Samples Synthesis (RSS) strategy is then developed to generate reciprocal samples using SRI and IIT, which are integrated into boundary learning to facilitate compact feature space for known classes while preserving data space for unknown classes.

Second, to mitigate the limitations of incomplete disease knowledge caused by data scarcity, we propose the Dynamic Attribute-guided Few-shot Open-set Network (DAFON) to generate pseudo-samples for rare or unseen diseases. DAFON first introduces a Global Attribute Generator (GAG) to create attributes from closed-set medical images. An attribute space is then constructed and employed to generate pseudo samples with a designed Open-set Data Sampler (ODS). To further enhance category boundary learning and feature discrimination, we propose a Dynamic Attribute Guided Alignment (DAGA) module that aligns the feature and attribute spaces, thereby improving the reliability and accuracy of diagnostic models.

Third, we consider a practical clinical scenario where the diagnostic model is required to recognize emerging diseases without access to historical disease data. To mitigate performance degradation caused by catastrophic forgetting, we propose an LLM-guided Decoupled Probabilistic Prompt (LDPP) framework to generate expert knowledge for continual learning. Specifically, we develop an Expert Knowledge Generation (EKG) module that leverages pre-trained large language models (LLM) to acquire decoupled expert knowledge for various diseases.
This knowledge is used to construct a Decoupled Probabilistic Prompt-pool (DePP), which provides globally shared, diverse, and flexible prompts for medical image interpretation. Additionally, we design a Steering Prompt Pool (SPP) to learn task-specific non-shared prompts, enhancing intra-class compactness and further improving model performance.

Lastly, to address resolution degradation in magnetic resonance imaging (MRI), we propose a Unified Codebook-prior guided Anatomical alignment for MRI Super-Resolution (UCASR) method that integrates anatomical knowledge into the reconstruction process. UCASR first introduces an anatomical knowledge generation (AKG) module to extract regions of interest from MRI slices and generate anatomical knowledge. A prior-guided unified codebook generation module (PUC) is then developed to enable the codebook to effectively capture anatomical information. Additionally, a Prior Matching Alignment (PMA) module is devised to align codebook index probabilities between adjacent slices, as well as across low-resolution and high-resolution domains, effectively preserving image details and enhancing super-resolution quality.

In conclusion, this thesis aims to efficiently generate and integrate external knowledge for medical image analysis, addressing issues from the perspectives of instance-level knowledge, semantic knowledge, and visual priors. DAFON addresses data scarcity by generating attribute-guided pseudo-samples, LDPP leverages LLM to generate medical-related semantic prompts for continual learning, and PGASR incorporates generated anatomical priors to improve MRI super-resolution. In general, the proposed methods advance the accuracy and generalizability of various medical image analysis tasks by generating external knowledge, enabling more reliable and adaptable clinical decision support systems.

Date of Award	2 Jan 2026
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Ho Man CHAN (Supervisor), Kwok Leung CHAN (Supervisor) & Yixuan Yuan (External Co-Supervisor)