AROMA: Autonomous Rank-one Matrix Adaptation

Hao Nan Sheng; Zhi-Yong Wang; Hing Cheung So; Mingrui Yang

doi:10.18653/v1/2025.emnlp-main.170

AROMA: Autonomous Rank-one Matrix Adaptation

Hao Nan Sheng, Zhi-Yong Wang, Hing Cheung So^*, Mingrui Yang

^*Corresponding author for this work

Department of Electrical Engineering

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

3 Downloads (CityUHK Scholars)

Abstract

As large language models continue to grow in size, parameter-efficient fine-tuning (PEFT) has become increasingly crucial. While low-rank adaptation (LoRA) offers a solution through low-rank updates, its static rank allocation may yield suboptimal results. Adaptive low-rank adaptation (AdaLoRA) improves this with dynamic allocation but remains sensitive to initial and target rank configurations. We introduce AROMA, a framework that automatically constructs layer-specific updates by iteratively building up rank-one components with very few trainable parameters that gradually diminish to zero. Unlike existing methods that employ rank reduction mechanisms, AROMA introduces a dual-loop architecture for rank growth. The inner loop extracts information from each rank-one subspace, while the outer loop determines the number of rank-one subspaces, i.e., the optimal rank. We reset optimizer states to maintain subspace independence. AROMA significantly reduces parameters compared to LoRA and AdaLoRA while achieving superior performance on natural language understanding and generation, commonsense reasoning, offering new insights into adaptive PEFT. The code is available at https://github.com/ShuDun23/AROMA.
©2025 Association for Computational Linguistics

Original language	English
Title of host publication	Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Editors	Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, Violet Peng
Publisher	Association for Computational Linguistics
Pages	3443-3459
Number of pages	17
ISBN (Print)	979-8-89176-332-6
DOIs	https://doi.org/10.18653/v1/2025.emnlp-main.170
Publication status	Published - Nov 2025
Event	30th Conference on Empirical Methods in Natural Language Processing (EMNLP 2025) - Suzhou, China Duration: 4 Nov 2025 → 9 Nov 2025 https://aclanthology.org/volumes/2025.emnlp-main/

Conference

Conference	30th Conference on Empirical Methods in Natural Language Processing (EMNLP 2025)
Abbreviated title	30th EMNLP
Place	China
City	Suzhou
Period	4/11/25 → 9/11/25
Internet address	https://aclanthology.org/volumes/2025.emnlp-main/

Bibliographical note

Research Unit(s) information for this publication is provided by the author(s) concerned.

Funding

This work was supported by the Research Grant of Shenzhen Research Institute, City University of Hong Kong, Shenzhen, China under Project RIND25501.

Publisher's Copyright Statement

This full text is made available under CC-BY 4.0. https://creativecommons.org/licenses/by/4.0/

Access Output

10.18653/v1/2025.emnlp-main.170

425920651.pdfFinal Published version, 666 KBLicence: CC BY

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@inproceedings{16d93a90f0bc4108b4ebc534bb7698a7,

title = "AROMA: Autonomous Rank-one Matrix Adaptation",

abstract = "As large language models continue to grow in size, parameter-efficient fine-tuning (PEFT) has become increasingly crucial. While low-rank adaptation (LoRA) offers a solution through low-rank updates, its static rank allocation may yield suboptimal results. Adaptive low-rank adaptation (AdaLoRA) improves this with dynamic allocation but remains sensitive to initial and target rank configurations. We introduce AROMA, a framework that automatically constructs layer-specific updates by iteratively building up rank-one components with very few trainable parameters that gradually diminish to zero. Unlike existing methods that employ rank reduction mechanisms, AROMA introduces a dual-loop architecture for rank growth. The inner loop extracts information from each rank-one subspace, while the outer loop determines the number of rank-one subspaces, i.e., the optimal rank. We reset optimizer states to maintain subspace independence. AROMA significantly reduces parameters compared to LoRA and AdaLoRA while achieving superior performance on natural language understanding and generation, commonsense reasoning, offering new insights into adaptive PEFT. The code is available at https://github.com/ShuDun23/AROMA.{\textcopyright}2025 Association for Computational Linguistics",

author = "Sheng, {Hao Nan} and Zhi-Yong Wang and So, {Hing Cheung} and Mingrui Yang",

note = "Research Unit(s) information for this publication is provided by the author(s) concerned.; 30th Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), 30th EMNLP ; Conference date: 04-11-2025 Through 09-11-2025",

year = "2025",

month = nov,

doi = "10.18653/v1/2025.emnlp-main.170",

language = "English",

isbn = "979-8-89176-332-6",

pages = "3443--3459",

editor = "Christos Christodoulopoulos and Tanmoy Chakraborty and Carolyn Rose and Violet Peng",

booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",

publisher = "Association for Computational Linguistics",

address = "United States",

url = "https://aclanthology.org/volumes/2025.emnlp-main/",

}

Sheng, HN , Wang, Z-Y , So, HC & Yang, M 2025, AROMA: Autonomous Rank-one Matrix Adaptation. in C Christodoulopoulos, T Chakraborty, C Rose & V Peng (eds), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 3443-3459, 30th Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), Suzhou, China, 4/11/25. https://doi.org/10.18653/v1/2025.emnlp-main.170

AROMA: Autonomous Rank-one Matrix Adaptation. / Sheng, Hao Nan ; Wang, Zhi-Yong ; So, Hing Cheung et al.
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. ed. / Christos Christodoulopoulos; Tanmoy Chakraborty; Carolyn Rose; Violet Peng. Association for Computational Linguistics, 2025. p. 3443-3459.

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - AROMA

T2 - 30th Conference on Empirical Methods in Natural Language Processing (EMNLP 2025)

AU - Sheng, Hao Nan

AU - Wang, Zhi-Yong

AU - So, Hing Cheung

AU - Yang, Mingrui

N1 - Research Unit(s) information for this publication is provided by the author(s) concerned.

PY - 2025/11

Y1 - 2025/11

N2 - As large language models continue to grow in size, parameter-efficient fine-tuning (PEFT) has become increasingly crucial. While low-rank adaptation (LoRA) offers a solution through low-rank updates, its static rank allocation may yield suboptimal results. Adaptive low-rank adaptation (AdaLoRA) improves this with dynamic allocation but remains sensitive to initial and target rank configurations. We introduce AROMA, a framework that automatically constructs layer-specific updates by iteratively building up rank-one components with very few trainable parameters that gradually diminish to zero. Unlike existing methods that employ rank reduction mechanisms, AROMA introduces a dual-loop architecture for rank growth. The inner loop extracts information from each rank-one subspace, while the outer loop determines the number of rank-one subspaces, i.e., the optimal rank. We reset optimizer states to maintain subspace independence. AROMA significantly reduces parameters compared to LoRA and AdaLoRA while achieving superior performance on natural language understanding and generation, commonsense reasoning, offering new insights into adaptive PEFT. The code is available at https://github.com/ShuDun23/AROMA.©2025 Association for Computational Linguistics

AB - As large language models continue to grow in size, parameter-efficient fine-tuning (PEFT) has become increasingly crucial. While low-rank adaptation (LoRA) offers a solution through low-rank updates, its static rank allocation may yield suboptimal results. Adaptive low-rank adaptation (AdaLoRA) improves this with dynamic allocation but remains sensitive to initial and target rank configurations. We introduce AROMA, a framework that automatically constructs layer-specific updates by iteratively building up rank-one components with very few trainable parameters that gradually diminish to zero. Unlike existing methods that employ rank reduction mechanisms, AROMA introduces a dual-loop architecture for rank growth. The inner loop extracts information from each rank-one subspace, while the outer loop determines the number of rank-one subspaces, i.e., the optimal rank. We reset optimizer states to maintain subspace independence. AROMA significantly reduces parameters compared to LoRA and AdaLoRA while achieving superior performance on natural language understanding and generation, commonsense reasoning, offering new insights into adaptive PEFT. The code is available at https://github.com/ShuDun23/AROMA.©2025 Association for Computational Linguistics

U2 - 10.18653/v1/2025.emnlp-main.170

DO - 10.18653/v1/2025.emnlp-main.170

M3 - RGC 32 - Refereed conference paper (with host publication)

SN - 979-8-89176-332-6

SP - 3443

EP - 3459

BT - Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing

A2 - Christodoulopoulos, Christos

A2 - Chakraborty, Tanmoy

A2 - Rose, Carolyn

A2 - Peng, Violet

PB - Association for Computational Linguistics

Y2 - 4 November 2025 through 9 November 2025

ER -

AROMA: Autonomous Rank-one Matrix Adaptation

Abstract

Conference

Bibliographical note

Funding

Publisher's Copyright Statement

Access Output

Other Access Options

Permanent Link

Fingerprint

Cite this