Editing Pre-trained Vision Models for Multi-granularity Adaptation

Abstract

The rapid evolution of pre-trained vision models has revolutionized artificial intelligence; however, these models inevitably inherit biases and limitations from their pre-training tasks, which often fail to capture the full complexity and diversity of real-world scenarios. Consequently, targeted model adaptation enables data-efficient updates that refine model behavior for specific tasks without complete retraining. Existing editing methods face critical challenges in effectively modifying behavior within adaptation scopes while preserving original functionality outside the adaptation scope, at various levels of granularity. This thesis advances multi-granularity adaptation through a "locate-then-edit" framework that decouples editing into key subproblems: where-to-edit and how-to-edit.

At the concept-level model adaptation, we address undesirable concept erasure in text-to-image diffusion models, where unsafe (e.g., nudity) or copyrighted (e.g., Snoopy) concepts should be removed without affecting unrelated content. Existing methods compromise semantically proximate concepts; our solution identifies critical cross-attention modules related to concept generation via systematic greedy search (where-to-edit), then implements memory-based editing to spatially constrain erasure to target regions using concept-level feature similarity (how-to-edit). This approach efficiently eliminates target concepts while preserving the locality of non-target concepts, particularly those semantically proximate to targets.

At the sample-level model adaptation, we correct predictive errors arising from subpopulation shifts while ensuring generalization to neighboring failures and locality to minimize unintended effects on unrelated samples. While significant progress has been made in editing Transformer-based large language models, effective strategies for editing vision Transformers (ViTs) in computer vision remain largely unexplored. In this thesis, we take initial steps towards correcting predictive errors of ViTs, particularly those arising from subpopulation shifts. Taking a locate-then-edit approach, we first address the where-to-edit challenge by meta-learning a hypernetwork on CutMix-augmented data generated for editing reliability. This trained hypernetwork produces generalizable binary masks that identify a sparse subset of structured model parameters, responsive to real-world failure samples. Afterward, we solve the how-to-edit problem by simply fine-tuning the identified parameters using a variant of gradient descent to achieve successful edits. To validate our method, we construct an editing benchmark that introduces subpopulation shifts towards natural underrepresented images and AI-generated images, thereby revealing the limitations of pre-trained ViTs for object recognition. Our approach not only achieves superior performance on the proposed benchmark but also allows for adjustable trade-offs between generalization and locality.

At the task-level model adaptation, we adapt pre-trained models to downstream tasks. Our empirical results reveal that off-the-shelf fine-tuning techniques inadequately mitigate negative transfer caused by underperforming features in a pre-trained model, including rare features and spuriously correlated features. Without requiring preservation of pre-training task behavior, we designate the entire model as the edit location (where-to-edit) to optimize downstream performance. To alleviate negative transfer, we focus on how-to-edit and propose a Concept-wise fine-tuning (Concept-tuning) approach, which refines feature representations at the level of patches, with each patch encoding a concept. Concept-tuning minimizes the negative impacts of rare features and spuriously correlated features by 1) maximizing the mutual information between examples in the same category with regard to a slice of rare features (a patch) and 2) applying front-door adjustment via attention neural networks in channels and feature slices (patches). Concept-tuning consistently outperforms state-of-the-art methods across datasets, diverse pre-training strategies (supervised/self-supervised), various architectures, and downstream dataset sizes.

Collectively, these contributions establish foundational insights and methodologies for multi-granularity model adaptation, paving the way for safer, more robust vision systems.

Date of Award	5 Sept 2025
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Kede MA (Supervisor) & Ying Wei (External Co-Supervisor)