Curvature-Guided Mixing for MLLM Adaptation

Jinglong Yang; Jiaxuan He; Wenjian Huang; Zhan Zhuang; Jianguo Zhang

Curvature-Guided Mixing for MLLM Adaptation

Jinglong Yang (Co-first Author)
, Jiaxuan He (Co-first Author)
, Wenjian Huang
, Zhan Zhuang
, Jianguo Zhang^*

^*Corresponding author for this work

Department of Computer Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

1 Downloads (CityUHK Scholars)

Abstract

Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities. Existing model merging methods to combat this are often heuristic or use sub-optimal objectives. We propose CurvatureGuided Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models. CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form “soft mixing” ratio. This ratio intelligently blends parameters based on their relative task-specific curvatures. We also introduce CGM † , a robust “hard mixing” variant that performs sparse parameter selection guided by a novel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen2.5VL across multiple downstream tasks show that CGM and CGM† consistently improve the trade-off between task specialization and general knowledge retention over existing methods. Code is available at github.com/zzsyjl/CGM-ECCV-2026.

Original language	English
Title of host publication	Computer Vision – ECCV 2026
Subtitle of host publication	19th European Conference, Malmö, Sweden, September 8–12, 2026, Proceedings
Editors	Paolo Favaro, Zuzana Kukelova, Atsuto Maki, Anna Rohrbach, Konrad Schindler, Federico Tombari
Publisher	Springer, Cham
Publication status	Accepted/In press/Filed - Sept 2026
Event	The 19th European Conference on Computer Vision - Malmö, Sweden Duration: 8 Sept 2026 → 12 Sept 2026 https://eccv.ecva.net/Conferences/2026

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer Cham
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	The 19th European Conference on Computer Vision
Abbreviated title	ECCV 2026
Place	Sweden
City	Malmö
Period	8/09/26 → 12/09/26
Internet address	https://eccv.ecva.net/Conferences/2026

Bibliographical note

Since this conference is yet to commence, the information for this record is subject to revision.

Funding

This work is supported by National Natural Science Foundation of China (Grant No. 62276121) and Innovation Team and Talents Cultivation Program of National Administration of Traditional Chinese Medicine, No. ZYYCXTD-D-202403.

Research Keywords

Continual learning
MLLM adaptation
model merging

Permanent Link

Right click to copy link

Cite this

Yang, Jinglong ; He, Jiaxuan ; Huang, Wenjian et al. / Curvature-Guided Mixing for MLLM Adaptation. Computer Vision – ECCV 2026: 19th European Conference, Malmö, Sweden, September 8–12, 2026, Proceedings. editor / Paolo Favaro ; Zuzana Kukelova ; Atsuto Maki ; Anna Rohrbach ; Konrad Schindler ; Federico Tombari. Springer, Cham, 2026. (Lecture Notes in Computer Science).

@inproceedings{9952c762acd84489b230145cf5ff87aa,

title = "Curvature-Guided Mixing for MLLM Adaptation",

abstract = "Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities. Existing model merging methods to combat this are often heuristic or use sub-optimal objectives. We propose CurvatureGuided Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models. CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form “soft mixing” ratio. This ratio intelligently blends parameters based on their relative task-specific curvatures. We also introduce CGM † , a robust “hard mixing” variant that performs sparse parameter selection guided by a novel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen2.5VL across multiple downstream tasks show that CGM and CGM† consistently improve the trade-off between task specialization and general knowledge retention over existing methods. Code is available at github.com/zzsyjl/CGM-ECCV-2026.",

keywords = "Continual learning, MLLM adaptation, model merging",

author = "Jinglong Yang and Jiaxuan He and Wenjian Huang and Zhan Zhuang and Jianguo Zhang",

note = "Since this conference is yet to commence, the information for this record is subject to revision.; The 19th European Conference on Computer Vision, ECCV 2026 ; Conference date: 08-09-2026 Through 12-09-2026",

year = "2026",

month = sep,

language = "English",

series = "Lecture Notes in Computer Science",

publisher = "Springer, Cham",

editor = "Paolo Favaro and Zuzana Kukelova and Atsuto Maki and Anna Rohrbach and Konrad Schindler and Federico Tombari",

booktitle = "Computer Vision – ECCV 2026",

url = "https://eccv.ecva.net/Conferences/2026",

}

Yang, J, He, J, Huang, W, Zhuang, Z & Zhang, J 2026, Curvature-Guided Mixing for MLLM Adaptation. in P Favaro, Z Kukelova, A Maki, A Rohrbach, K Schindler & F Tombari (eds), Computer Vision – ECCV 2026: 19th European Conference, Malmö, Sweden, September 8–12, 2026, Proceedings. Lecture Notes in Computer Science, Springer, Cham, The 19th European Conference on Computer Vision, Malmö, Sweden, 8/09/26.

Curvature-Guided Mixing for MLLM Adaptation. / Yang, Jinglong (Co-first Author); He, Jiaxuan (Co-first Author); Huang, Wenjian et al.
Computer Vision – ECCV 2026: 19th European Conference, Malmö, Sweden, September 8–12, 2026, Proceedings. ed. / Paolo Favaro; Zuzana Kukelova; Atsuto Maki; Anna Rohrbach; Konrad Schindler; Federico Tombari. Springer, Cham, 2026. (Lecture Notes in Computer Science).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Curvature-Guided Mixing for MLLM Adaptation

AU - Yang, Jinglong

AU - He, Jiaxuan

AU - Huang, Wenjian

AU - Zhuang, Zhan

AU - Zhang, Jianguo

N1 - Since this conference is yet to commence, the information for this record is subject to revision.

PY - 2026/9

Y1 - 2026/9

N2 - Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities. Existing model merging methods to combat this are often heuristic or use sub-optimal objectives. We propose CurvatureGuided Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models. CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form “soft mixing” ratio. This ratio intelligently blends parameters based on their relative task-specific curvatures. We also introduce CGM † , a robust “hard mixing” variant that performs sparse parameter selection guided by a novel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen2.5VL across multiple downstream tasks show that CGM and CGM† consistently improve the trade-off between task specialization and general knowledge retention over existing methods. Code is available at github.com/zzsyjl/CGM-ECCV-2026.

AB - Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities. Existing model merging methods to combat this are often heuristic or use sub-optimal objectives. We propose CurvatureGuided Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models. CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form “soft mixing” ratio. This ratio intelligently blends parameters based on their relative task-specific curvatures. We also introduce CGM † , a robust “hard mixing” variant that performs sparse parameter selection guided by a novel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen2.5VL across multiple downstream tasks show that CGM and CGM† consistently improve the trade-off between task specialization and general knowledge retention over existing methods. Code is available at github.com/zzsyjl/CGM-ECCV-2026.

KW - Continual learning

KW - MLLM adaptation

KW - model merging

M3 - RGC 32 - Refereed conference paper (with host publication)

T3 - Lecture Notes in Computer Science

BT - Computer Vision – ECCV 2026

A2 - Favaro, Paolo

A2 - Kukelova, Zuzana

A2 - Maki, Atsuto

A2 - Rohrbach, Anna

A2 - Schindler, Konrad

A2 - Tombari, Federico

PB - Springer, Cham

T2 - The 19th European Conference on Computer Vision

Y2 - 8 September 2026 through 12 September 2026

ER -

Curvature-Guided Mixing for MLLM Adaptation

Abstract

Publication series

Conference

Bibliographical note

Funding

Research Keywords

Permanent Link

Fingerprint

Cite this