The rapid development of medical image analysis is hampered by the laborious and resource-intensive burden of annotating large-scale medical image datasets. Manual annotation of medical images requires domain expertise, extensive time, and considerable financial investment, posing a bottleneck to the timely advancement of machine learning algorithms in this domain. The need for a substantial amount of labeled data for training accurate models further exacerbates this challenge.

Adaptation learning emerges as a promising approach to mitigate the burden of data annotation. By leveraging information from related domains, adaptation learning seeks to expedite model training and enhance the model's ability to generalize to the target domain. However, the successful application of adaptation learning in medical imaging encounters unique hurdles. Transferring knowledge from one domain to another in medical imaging is limited due to stringent privacy protection regulations, concerns regarding secure data storage and transmission, high computation burden, and substantial costs associated with data sharing and transfer.

In this thesis, we address these challenges by focusing on robust adaptation under various data access privileges for medical image analysis. We categorize the access privileges into four distinct scenarios: (1) Unsupervised Domain Adaptation (UDA), where full access to the source data is available for adaptation; (2) Source-Free Domain Adaptation (SFDA), where permission is granted only for the source model instead of access to the source data; (3) Multi-Source-Free Domain Adaptation (MSFDA), where permission is granted only for multiple source models instead of access to multiple source data; (4) Federated Learning (FL), where the server side can hold and distribute client models while the client side maintains local data. Each scenario presents unique adaptation challenges and requires tailored strategies to effectively navigate the constraints posed by limited data access, ensuring robust and efficient adaptation in medical image analysis.

In the first part, We introduce a mutual-prototype adaptation network for transferring source domain knowledge to enable adaptation in unlabeled target domains. The mutual-prototype adaptation framework is designed to facilitate the indirect alignment of source and target prototypes by leveraging self-domain and cross-domain information refinement. Additionally, we incorporate two auxiliary modules, progressive self-training and disentangled reconstruction, to enhance the model's robustness.

In the second part, considering privacy concerns of the source data, we are the first to transfer the knowledge of source model rather than source data to achieve adaptation on unlabeled target domain and propose a fourier style mining method with two stages. The generation stage aims to generate high quality source-like target data by model inversion and mutual fourier transform. The adaptation stage applies a Contrastive Domain Distillation and a Compact-Aware Domain Consistency to achieve feature-level and output-level adaptation.

In the third part, We introduce a transferability-guided multi-source model adaptation approach for transferring knowledge from multiple source models to adapt to a single unlabeled target domain. This method leverages a label-free transferability metric (LFTM) to assess the transferability of source models without requiring target domain annotations, thereby eliminating negative transfer. Using this metric, we compute an instance-level transferability matrix for pseudo label correction and a domain-level transferability matrix to improve the initialization of the target model.

In the fourth part, we design a domain-aware masked autoencoders to help personalized federated learning. Based on the ability of detecting ood data, distribution-aware local training and discrepancy-aware global aggregation are proposed to assign different importance on local samples and re-weight local models by domain relevance of DMAEs to tackle model drift.

In the last part, We introduce a parameter disentanglement method for federated open-set recognition. This approach is designed to separate an open-set network into distinct components, isolating specific and shared parts. We then align these corresponding components using optimal transport and combine them to create a global model, mitigating issues related to parameter misalignment.