Data-efficient Large Vision Models through Sequential Autoregression

Zhiwei Hao; Jianyuan Guo; Chengcheng Wang; Yehui Tang; Han Wu; Han Hu; Kai Han; Chang Xu

Data-efficient Large Vision Models through Sequential Autoregression

Zhiwei Hao
, Jianyuan Guo (Co-first Author)
, Chengcheng Wang
, Yehui Tang
, Han Wu
, Han Hu^*
, Kai Han
, Chang Xu^*

^*Corresponding author for this work

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

3 Citations (Scopus)

Abstract

Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended to not only comprehend but also seamlessly transit to out-of-domain tasks. However, current endeavors are hamstrung by an over-reliance on colossal models, exemplified by models with upwards of 3B parameters, and the necessity for an extensive corpus of visual data, often comprising a staggering 400B tokens (Bai et al., 2023). In this paper, we delve into the development of an efficient, autoregression-based vision model, innovatively architected to operate on a limited dataset. We meticulously demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding during the testing phase. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint, and a marked decrease in training data requirements, thereby paving the way for more sustainable and accessible advancements in the field of generalist vision models. The code is available at https://github.com/ggjy/DeLVM.
Copyright 2024 by the author(s).

Original language	English
Title of host publication	Proceedings of the 41st International Conference on Machine Learning
Editors	Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, Felix Berkenkamp
Place of Publication	United States
Publisher	ML Research Press
Pages	17572-17596
Number of pages	25
Publication status	Published - 21 Jul 2024
Externally published	Yes
Event	41st International Conference on Machine Learning (ICML 2024) - Messe Wien Exhibition Congress Center, Vienna, Austria Duration: 21 Jul 2024 → 27 Jul 2024 https://proceedings.mlr.press/v235/ https://icml.cc/

Publication series

Name	Proceedings of Machine Learning Research
Publisher	ML Research Press
Volume	235
ISSN (Print)	2640-3498

Conference

Conference	41st International Conference on Machine Learning (ICML 2024)
Place	Austria
City	Vienna
Period	21/07/24 → 27/07/24
Internet address	https://proceedings.mlr.press/v235/ https://icml.cc/

Bibliographical note

Publisher Copyright:
Copyright 2024 by the author(s)

Funding

This paper is supported by National Key Research and Development Program of China under No. 2021YFC3300128, and Joint Funds of the National Natural Science Foundation of China No. U2336211. Chang Xu was supported in part by the Australian Research Council under Projects DP240101848 and FT230100549.

Access Output

https://proceedings.mlr.press/v235/hao24b.html

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Hao, Z., Guo, J., Wang, C., Tang, Y., Wu, H., Hu, H., Han, K., & Xu, C. (2024). Data-efficient Large Vision Models through Sequential Autoregression. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, & F. Berkenkamp (Eds.), Proceedings of the 41st International Conference on Machine Learning (pp. 17572-17596). (Proceedings of Machine Learning Research; Vol. 235). ML Research Press. https://proceedings.mlr.press/v235/hao24b.html

Hao, Zhiwei ; Guo, Jianyuan ; Wang, Chengcheng et al. / Data-efficient Large Vision Models through Sequential Autoregression. Proceedings of the 41st International Conference on Machine Learning. editor / Ruslan Salakhutdinov ; Zico Kolter ; Katherine Heller ; Adrian Weller ; Nuria Oliver ; Jonathan Scarlett ; Felix Berkenkamp. United States : ML Research Press, 2024. pp. 17572-17596 (Proceedings of Machine Learning Research).

@inproceedings{12f70c9fce544c2094e2fd2e6b336977,

title = "Data-efficient Large Vision Models through Sequential Autoregression",

abstract = "Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended to not only comprehend but also seamlessly transit to out-of-domain tasks. However, current endeavors are hamstrung by an over-reliance on colossal models, exemplified by models with upwards of 3B parameters, and the necessity for an extensive corpus of visual data, often comprising a staggering 400B tokens (Bai et al., 2023). In this paper, we delve into the development of an efficient, autoregression-based vision model, innovatively architected to operate on a limited dataset. We meticulously demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding during the testing phase. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint, and a marked decrease in training data requirements, thereby paving the way for more sustainable and accessible advancements in the field of generalist vision models. The code is available at https://github.com/ggjy/DeLVM. Copyright 2024 by the author(s).",

author = "Zhiwei Hao and Jianyuan Guo and Chengcheng Wang and Yehui Tang and Han Wu and Han Hu and Kai Han and Chang Xu",

year = "2024",

month = jul,

day = "21",

language = "English",

series = "Proceedings of Machine Learning Research",

publisher = "ML Research Press",

pages = "17572--17596",

editor = "Ruslan Salakhutdinov and Zico Kolter and Katherine Heller and Adrian Weller and Nuria Oliver and Jonathan Scarlett and Felix Berkenkamp",

booktitle = "Proceedings of the 41st International Conference on Machine Learning",

url = "https://proceedings.mlr.press/v235/, https://icml.cc/",

}

Hao, Z, Guo, J, Wang, C, Tang, Y, Wu, H, Hu, H, Han, K & Xu, C 2024, Data-efficient Large Vision Models through Sequential Autoregression. in R Salakhutdinov, Z Kolter, K Heller, A Weller, N Oliver, J Scarlett & F Berkenkamp (eds), Proceedings of the 41st International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 235, ML Research Press, United States, pp. 17572-17596, 41st International Conference on Machine Learning (ICML 2024), Vienna, Austria, 21/07/24. <https://proceedings.mlr.press/v235/hao24b.html>

Data-efficient Large Vision Models through Sequential Autoregression. / Hao, Zhiwei; Guo, Jianyuan (Co-first Author); Wang, Chengcheng et al.
Proceedings of the 41st International Conference on Machine Learning. ed. / Ruslan Salakhutdinov; Zico Kolter; Katherine Heller; Adrian Weller; Nuria Oliver; Jonathan Scarlett; Felix Berkenkamp. United States: ML Research Press, 2024. p. 17572-17596 (Proceedings of Machine Learning Research; Vol. 235).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Data-efficient Large Vision Models through Sequential Autoregression

AU - Hao, Zhiwei

AU - Guo, Jianyuan

AU - Wang, Chengcheng

AU - Tang, Yehui

AU - Wu, Han

AU - Hu, Han

AU - Han, Kai

AU - Xu, Chang

PY - 2024/7/21

Y1 - 2024/7/21

N2 - Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended to not only comprehend but also seamlessly transit to out-of-domain tasks. However, current endeavors are hamstrung by an over-reliance on colossal models, exemplified by models with upwards of 3B parameters, and the necessity for an extensive corpus of visual data, often comprising a staggering 400B tokens (Bai et al., 2023). In this paper, we delve into the development of an efficient, autoregression-based vision model, innovatively architected to operate on a limited dataset. We meticulously demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding during the testing phase. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint, and a marked decrease in training data requirements, thereby paving the way for more sustainable and accessible advancements in the field of generalist vision models. The code is available at https://github.com/ggjy/DeLVM. Copyright 2024 by the author(s).

AB - Training general-purpose vision models on purely sequential visual data, eschewing linguistic inputs, has heralded a new frontier in visual understanding. These models are intended to not only comprehend but also seamlessly transit to out-of-domain tasks. However, current endeavors are hamstrung by an over-reliance on colossal models, exemplified by models with upwards of 3B parameters, and the necessity for an extensive corpus of visual data, often comprising a staggering 400B tokens (Bai et al., 2023). In this paper, we delve into the development of an efficient, autoregression-based vision model, innovatively architected to operate on a limited dataset. We meticulously demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding during the testing phase. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint, and a marked decrease in training data requirements, thereby paving the way for more sustainable and accessible advancements in the field of generalist vision models. The code is available at https://github.com/ggjy/DeLVM. Copyright 2024 by the author(s).

UR - https://www.scopus.com/pages/publications/85203786785

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-84989848639&origin=recordpage

M3 - RGC 32 - Refereed conference paper (with host publication)

AN - SCOPUS:85203786785

T3 - Proceedings of Machine Learning Research

SP - 17572

EP - 17596

BT - Proceedings of the 41st International Conference on Machine Learning

A2 - Salakhutdinov, Ruslan

A2 - Kolter, Zico

A2 - Heller, Katherine

A2 - Weller, Adrian

A2 - Oliver, Nuria

A2 - Scarlett, Jonathan

A2 - Berkenkamp, Felix

PB - ML Research Press

CY - United States

T2 - 41st International Conference on Machine Learning (ICML 2024)

Y2 - 21 July 2024 through 27 July 2024

ER -

Hao Z, Guo J, Wang C, Tang Y, Wu H, Hu H et al. Data-efficient Large Vision Models through Sequential Autoregression. In Salakhutdinov R, Kolter Z, Heller K, Weller A, Oliver N, Scarlett J, Berkenkamp F, editors, Proceedings of the 41st International Conference on Machine Learning. United States: ML Research Press. 2024. p. 17572-17596. (Proceedings of Machine Learning Research).

Data-efficient Large Vision Models through Sequential Autoregression

Abstract

Publication series

Conference

Bibliographical note

Funding

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this