High Efficiency Image Compression for Large Visual-Language Models
Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review
Author(s)
Related Research Unit(s)
Detail(s)
Original language | English |
---|---|
Journal / Publication | IEEE Transactions on Circuits and Systems for Video Technology |
Publication status | Online published - 30 Oct 2024 |
Link(s)
Abstract
In recent years, large visual language models (LVLMs) have shown impressive performance and promising generalization capability in multi-modal tasks, thus replacing humans as receivers of visual information in various application scenarios. In this paper, we pioneer to propose a variable bitrate image compression scheme consisting of a pre-editing module and an end-to-end codec to achieve promising rate-accuracy performance for different LVLMs. In particular, instead of optimizing an adaptive pre-editing network towards a particular task or several representative tasks, we propose a new optimization strategy tailored for LVLMs, which is designed based on the representation and discrimination capability with token-level distortion and rank. The pre-editing module and the variable bitrate end-to-end image codec are jointly trained by the losses based on semantic tokens of the large model, which introduce enhanced generalization capability for various data and tasks. Experimental results demonstrate that the proposed framework could efficiently achieve much better rate-accuracy performance compared to the state-of-the-art coding standard, Versatile Video Coding. Meanwhile, experiments with multi-modal tasks have revealed the robustness and generalization capability of the proposed framework. © 2024 IEEE.
Research Area(s)
- Image compression for machine, large visual-language model, pre-editing process
Citation Format(s)
High Efficiency Image Compression for Large Visual-Language Models. / Li, Binzhe; Wang, Shurun; Wang, Shiqi et al.
In: IEEE Transactions on Circuits and Systems for Video Technology, 30.10.2024.
In: IEEE Transactions on Circuits and Systems for Video Technology, 30.10.2024.
Research output: Journal Publications and Reviews › RGC 21 - Publication in refereed journal › peer-review