Skip to main navigation Skip to search Skip to main content

MTSAM: MULTI-TASK FINE-TUNING FOR SEGMENT ANYTHING MODEL

  • Xuehao Wang
  • , Zhan Zhuang
  • , Feiyang Ye
  • , Yu Zhang*
  • *Corresponding author for this work

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

Abstract

The Segment Anything Model (SAM), with its remarkable zero-shot capability, has the potential to be a foundation model for multi-task learning. However, adopting SAM to multi-task learning faces two challenges: (a) SAM has difficulty generating task-specific outputs with different channel numbers, and (b) how to fine-tune SAM to adapt multiple downstream tasks simultaneously remains unexplored. To address these two challenges, in this paper, we propose the Multi-Task SAM (MTSAM) framework, which enables SAM to work as a foundation model for multi-task learning. MTSAM modifies SAM's architecture by removing the prompt encoder and implementing task-specific no-mask embeddings and mask decoders, enabling the generation of task-specific outputs. Furthermore, we introduce Tensorized low-Rank Adaptation (ToRA) to perform multi-task fine-tuning on SAM. Specifically, ToRA injects an update parameter tensor into each layer of the encoder in SAM and leverages a low-rank tensor decomposition method to incorporate both task-shared and task-specific information. Extensive experiments conducted on benchmark datasets substantiate the efficacy of MTSAM in enhancing the performance of multi-task learning. Our code is available at https://github.com/XuehaoWangFi/MTSAM.
Original languageEnglish
Title of host publicationInternational Conference on Representation Learning 2025 (ICLR 2025)
EditorsY. Yue, A. Garg, N. Peng, F. Sha, R. Yu
PublisherInternational Conference on Learning Representations, ICLR
Number of pages22
ISBN (Print)9798331320850
Publication statusPublished - Apr 2025
Event13th International Conference on Learning Representations (ICLR 2025) - Singapore EXPO, Singapore, Singapore
Duration: 24 Apr 2025 → 28 Apr 2025
https://iclr.cc/Conferences/2025

Publication series

NameInternational Conference on Learning Representations

Conference

Conference13th International Conference on Learning Representations (ICLR 2025)
Abbreviated titleICLR 2025
PlaceSingapore
CitySingapore
Period24/04/25 → 28/04/25
Internet address

Bibliographical note

Research Unit(s) information for this publication is provided by the author(s) concerned.

Funding

This work is supported by National Key R&D Program of China 2022ZD0160300 and NSFC key grant under grant no. 62136005.

Fingerprint

Dive into the research topics of 'MTSAM: MULTI-TASK FINE-TUNING FOR SEGMENT ANYTHING MODEL'. Together they form a unique fingerprint.

Cite this