FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference

Hongchao Du; Shangyu Wu; Arina Kharlamova; Nan Guan; Chun Jason Xue

doi:10.1145/3721146.3721961

FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference

Hongchao Du (Co-first Author), Shangyu Wu (Co-first Author), Arina Kharlamova, Nan Guan, Chun Jason Xue

Department of Computer Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

11 Downloads (CityUHK Scholars)

Abstract

Large Language Models (LLMs) face challenges for on-device inference due to high memory demands. Traditional methods to reduce memory usage often compromise performance and lack adaptability. We propose FlexInfer, an optimized offloading framework for on-device inference, addressing these issues with techniques like asynchronous prefetching, balanced memory locking, and flexible tensor preservation. These strategies enhance memory efficiency and mitigate I/O bottlenecks, ensuring high performance within userspecified resource constraints. Experiments demonstrate that FlexInfer significantly improves throughput under limited resources, achieving up to 12.5 times better performance than existing methods and facilitating the deployment of large models on resource-constrained devices. © 2025 Copyright held by the owner/author(s).

Original language	English
Title of host publication	EuroMLSys '25
Subtitle of host publication	Proceedings of the 5th Workshop on Machine Learning and Systems
Publisher	Association for Computing Machinery
Pages	56-65
ISBN (Electronic)	979-8-4007-1538-9
DOIs	https://doi.org/10.1145/3721146.3721961
Publication status	Published - 1 Apr 2025
Event	EuroMLSys '25: 5th Workshop on Machine Learning and Systems - Rotterdam, Netherlands Duration: 30 Mar 2025 → 3 Apr 2025

Conference

Conference	EuroMLSys '25: 5th Workshop on Machine Learning and Systems
Country/Territory	Netherlands
City	Rotterdam
Period	30/03/25 → 3/04/25

Bibliographical note

Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s)

Research Keywords

LLM
On-Device Inference
Offloading
ResourceConstrained Devices

Publisher's Copyright Statement

This full text is made available under CC-BY 4.0. https://creativecommons.org/licenses/by/4.0/

Access Output

10.1145/3721146.3721961

282652967.pdfFinal Published version, 1.39 MBLicence: CC BY

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@inproceedings{2b89eca844e44ea9adaee1c910b1ce02,

title = "FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference",

abstract = "Large Language Models (LLMs) face challenges for on-device inference due to high memory demands. Traditional methods to reduce memory usage often compromise performance and lack adaptability. We propose FlexInfer, an optimized offloading framework for on-device inference, addressing these issues with techniques like asynchronous prefetching, balanced memory locking, and flexible tensor preservation. These strategies enhance memory efficiency and mitigate I/O bottlenecks, ensuring high performance within userspecified resource constraints. Experiments demonstrate that FlexInfer significantly improves throughput under limited resources, achieving up to 12.5 times better performance than existing methods and facilitating the deployment of large models on resource-constrained devices. {\textcopyright} 2025 Copyright held by the owner/author(s).",

keywords = "LLM, On-Device Inference, Offloading, ResourceConstrained Devices",

author = "Hongchao Du and Shangyu Wu and Arina Kharlamova and Nan Guan and Xue, {Chun Jason}",

note = "Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s); EuroMLSys '25: 5th Workshop on Machine Learning and Systems ; Conference date: 30-03-2025 Through 03-04-2025",

year = "2025",

month = apr,

day = "1",

doi = "10.1145/3721146.3721961",

language = "English",

pages = "56--65",

booktitle = "EuroMLSys '25",

publisher = "Association for Computing Machinery",

address = "United States",

}

Du, H , Wu, S, Kharlamova, A, Guan, N & Xue, CJ 2025, FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference. in EuroMLSys '25: Proceedings of the 5th Workshop on Machine Learning and Systems. Association for Computing Machinery, pp. 56-65, EuroMLSys '25: 5th Workshop on Machine Learning and Systems, Rotterdam, Netherlands, 30/03/25. https://doi.org/10.1145/3721146.3721961

FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference. / Du, Hongchao (Co-first Author); Wu, Shangyu (Co-first Author); Kharlamova, Arina et al.
EuroMLSys '25: Proceedings of the 5th Workshop on Machine Learning and Systems. Association for Computing Machinery, 2025. p. 56-65.

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - FlexInfer

T2 - EuroMLSys '25: 5th Workshop on Machine Learning and Systems

AU - Du, Hongchao

AU - Wu, Shangyu

AU - Kharlamova, Arina

AU - Guan, Nan

AU - Xue, Chun Jason

N1 - Full text of this publication does not contain sufficient affiliation information. With consent from the author(s) concerned, the Research Unit(s) information for this record is based on the existing academic department affiliation of the author(s)

PY - 2025/4/1

Y1 - 2025/4/1

N2 - Large Language Models (LLMs) face challenges for on-device inference due to high memory demands. Traditional methods to reduce memory usage often compromise performance and lack adaptability. We propose FlexInfer, an optimized offloading framework for on-device inference, addressing these issues with techniques like asynchronous prefetching, balanced memory locking, and flexible tensor preservation. These strategies enhance memory efficiency and mitigate I/O bottlenecks, ensuring high performance within userspecified resource constraints. Experiments demonstrate that FlexInfer significantly improves throughput under limited resources, achieving up to 12.5 times better performance than existing methods and facilitating the deployment of large models on resource-constrained devices. © 2025 Copyright held by the owner/author(s).

AB - Large Language Models (LLMs) face challenges for on-device inference due to high memory demands. Traditional methods to reduce memory usage often compromise performance and lack adaptability. We propose FlexInfer, an optimized offloading framework for on-device inference, addressing these issues with techniques like asynchronous prefetching, balanced memory locking, and flexible tensor preservation. These strategies enhance memory efficiency and mitigate I/O bottlenecks, ensuring high performance within userspecified resource constraints. Experiments demonstrate that FlexInfer significantly improves throughput under limited resources, achieving up to 12.5 times better performance than existing methods and facilitating the deployment of large models on resource-constrained devices. © 2025 Copyright held by the owner/author(s).

KW - LLM

KW - On-Device Inference

KW - Offloading

KW - ResourceConstrained Devices

UR - http://www.scopus.com/inward/record.url?scp=105003622298&partnerID=8YFLogxK

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-105003622298&origin=recordpage

U2 - 10.1145/3721146.3721961

DO - 10.1145/3721146.3721961

M3 - RGC 32 - Refereed conference paper (with host publication)

SP - 56

EP - 65

BT - EuroMLSys '25

PB - Association for Computing Machinery

Y2 - 30 March 2025 through 3 April 2025

ER -

FlexInfer: Breaking Memory Constraint via Flexible and Efficient Offloading for On-Device LLM Inference

Abstract

Conference

Bibliographical note

Research Keywords

Publisher's Copyright Statement

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this