Abstract
The Segment Anything Model (SAM), with its remarkable zero-shot capability, has the potential to be a foundation model for multi-task learning. However, adopting SAM to multi-task learning faces two challenges: (a) SAM has difficulty generating task-specific outputs with different channel numbers, and (b) how to fine-tune SAM to adapt multiple downstream tasks simultaneously remains unexplored. To address these two challenges, in this paper, we propose the Multi-Task SAM (MTSAM) framework, which enables SAM to work as a foundation model for multi-task learning. MTSAM modifies SAM's architecture by removing the prompt encoder and implementing task-specific no-mask embeddings and mask decoders, enabling the generation of task-specific outputs. Furthermore, we introduce Tensorized low-Rank Adaptation (ToRA) to perform multi-task fine-tuning on SAM. Specifically, ToRA injects an update parameter tensor into each layer of the encoder in SAM and leverages a low-rank tensor decomposition method to incorporate both task-shared and task-specific information. Extensive experiments conducted on benchmark datasets substantiate the efficacy of MTSAM in enhancing the performance of multi-task learning. Our code is available at https://github.com/XuehaoWangFi/MTSAM.
| Original language | English |
|---|---|
| Title of host publication | International Conference on Representation Learning 2025 (ICLR 2025) |
| Editors | Y. Yue, A. Garg, N. Peng, F. Sha, R. Yu |
| Publisher | International Conference on Learning Representations, ICLR |
| Number of pages | 22 |
| ISBN (Print) | 9798331320850 |
| Publication status | Published - Apr 2025 |
| Event | 13th International Conference on Learning Representations (ICLR 2025) - Singapore EXPO, Singapore, Singapore Duration: 24 Apr 2025 → 28 Apr 2025 https://iclr.cc/Conferences/2025 |
Publication series
| Name | International Conference on Learning Representations |
|---|
Conference
| Conference | 13th International Conference on Learning Representations (ICLR 2025) |
|---|---|
| Abbreviated title | ICLR 2025 |
| Place | Singapore |
| City | Singapore |
| Period | 24/04/25 → 28/04/25 |
| Internet address |
Bibliographical note
Research Unit(s) information for this publication is provided by the author(s) concerned.Funding
This work is supported by National Key R&D Program of China 2022ZD0160300 and NSFC key grant under grant no. 62136005.
Fingerprint
Dive into the research topics of 'MTSAM: MULTI-TASK FINE-TUNING FOR SEGMENT ANYTHING MODEL'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver