Towards Efficient Collaboration for Large Language Models

Abstract

The rapid advancement of Large Language Models (LLMs) has fundamentally transformed the landscape of artificial intelligence, enabling sophisticated capabilities across diverse domains ranging from creative generation to complex reasoning.

However, the widespread deployment and continuous adaptation of large language models remain constrained by the substantial computational and memory costs of traditional full fine-tuning. As the variety of pre-trained models grows, a key challenge emerges: how to efficiently adapt these models without retraining, while enabling them to accommodate dynamic data distributions. This challenge manifests in two distinct scenarios. First, for heterogeneous models, architectural and vocabulary discrepancies often lead to feature space mismatch in traditional ensembling approaches. Such mismatch not only degrades predictive performance but also incurs considerable inference overhead, limiting practical applicability. Second, for homogeneous models, while parameter merging offers a viable path, existing methods often suffer from catastrophic forgetting and lack the capability for continuous evolution. This limitation restricts effective model cooperation and prevents full utilization of their complementary strengths.

For heterogeneous collaboration, we initially analyze the critical barriers and propose an efficient solution. Practically, models often possess distinct vocabularies and architectural topologies, resulting in traditional ensembling being ineffective due to feature space mismatch and computational redundancy. We empirically investigate the factors influencing ensemble performance, identifying model performance, vocabulary size, and response style as key determinants, revealing that compatibility among models is essential for effective ensembling. Building on this analysis, we introduce UniTE (Top-k Union Ensembling), a framework specifically designed to mitigate these structural overheads while preserving the predictive diversity of multiple experts. UniTE implements a topology-compatible selection strategy at the probability-level that identifies the most complementary models via a top-k union mechanism, simultaneously employing a novel vocabulary alignment protocol to reconcile heterogeneous tokenizers. By harmonizing the input spaces of disparate models, UniTE preserves knowledge diversity while significantly reducing inference latency and memory footprint compared to naive ensembling. This approach establishes a robust foundation for flexible multi-model synergy, demonstrating that effective collaboration can be achieved without parameter unification when model heterogeneity is high.

For homogeneous challenges, we focus on unification to achieve extreme deployment efficiency, progressing from static consolidation to dynamic evolution. We initiate this journey with Activation-Guided Consensus Merging (ACM), which solves the capability forgetting problem inherent in naive weight averaging. To resolve the interference between task-specific parameters, ACM leverages layer-adaptive activation mutual information to derive precise, non-uniform merging coefficients. This mechanism ensures that critical task-specific capabilities are retained within the merged weights by aligning semantic subspaces without requiring any additional training or access to original data. However, recognizing that static unification assumes a fixed optimization landscape, we further evolve this paradigm to handle continuous data streams through Activation-Guided Rotation-Aware Merging (ARM). ARM transcends the limitations of offline merging by approximating gradient dynamics through activation subspace rotation, enabling a flow-like merging process. This approach allows the model to iteratively refine its parameters over time, surpassing the convergence limits of supervised fine-tuning while maintaining stability against catastrophic forgetting. This progression from static consensus (ACM) to dynamic evolution (ARM) represents a significant theoretical advancement, bridging the gap between offline model merging and online continuous learning.

Empirical results from multiple benchmarks demonstrate that our approaches provide flexible and superior solutions for different deployment constraints. Specifically, UniTE performs exceptionally well in scenarios requiring maximum performance utilization of heterogeneous resources and achieves good results in multi-model collaboration tasks. Meanwhile, the evolution from ACM to ARM provides a scalable path for efficient continuous model enhancement in homogeneous scenarios, consistently outperforming traditional fine-tuning and baseline merging methods in both accuracy and efficiency. Experimental outputs verify that different integration strategies are required for different structural constraints, but both can be optimized through carefully designed compatibility and alignment mechanisms.

In summary, this thesis proposes systematic approaches for collaborative large language model enhancement. It establishes new frameworks for scalable, training-free, and continuously adaptable model ensembling and merging, providing a robust pathway for efficient model integration. By clarifying the different challenges between heterogeneous ensembles and homogeneous merging, this paper not only contributes specific algorithmic innovations but also provides novel theoretical perspectives on the sustainable management and enhancement of large-scale models accordingly. This research opens up new avenues for future work in artificial intelligence under resource-constrained environments, pointing out that the future of large language model adaptation lies not in monolithic training but in the intelligent combination and continuous evolution of existing model assets.

Date of Award	29 Jul 2026
Original language	English
Awarding Institution	City University of Hong Kong
Supervisor	Linqi SONG (Supervisor)