Projects per year
Abstract
This paper addresses the problem of cross-modal object tracking from RGB videos and event data. Rather than constructing a complex cross-modal fusion network, we explore the great potential of a pre-trained vision Transformer (ViT). Particularly, we delicately investigate plug-and-play training augmentations that encourage the ViT to bridge the vast distribution gap between the two modalities, enabling comprehensive cross-modal information interaction and thus enhancing its ability. Specifically, we propose a mask modeling strategy that randomly masks a specific modality of some tokens to enforce the interaction between tokens from different modalities interacting proactively. To mitigate network oscillations resulting from the masking strategy and further amplify its positive effect, we then theoretically propose an orthogonal high-rank loss to regularize the attention matrix. Extensive experiments demonstrate that our plug-and-play training augmentation techniques can significantly boost state-of-the-art one-stream and two-stream trackers to a large extent in terms of both tracking precision and success rate. Our new perspective and findings will potentially bring insights to the field of leveraging powerful pre-trained ViTs to model cross-modal data. The code is publicly available at https://github.com/ZHU-Zhiyu/High-Rank_RGB-Event_Tracker.
© 2023 IEEE
© 2023 IEEE
Original language | English |
---|---|
Title of host publication | Proceedings - 2023 IEEE/CVF International Conference on Computer Vision (ICCV 2023) |
Publisher | IEEE |
Pages | 21988-21998 |
ISBN (Electronic) | 979-8-3503-0718-4 |
DOIs | |
Publication status | Published - Oct 2023 |
Event | IEEE International Conference on Computer Vision 2023 (ICCV 2023) - Paris Convention Center , Paris, France Duration: 2 Oct 2023 → 6 Oct 2023 https://iccv2023.thecvf.com/ |
Conference
Conference | IEEE International Conference on Computer Vision 2023 (ICCV 2023) |
---|---|
Abbreviated title | ICCV23 |
Country/Territory | France |
City | Paris |
Period | 2/10/23 → 6/10/23 |
Internet address |
Bibliographical note
Research Unit(s) information for this publication is provided by the author(s) concerned.Funding
This work was supported in part by the Hong Kong Research Grants Council under Grant 11218121 and Grant 11202320, and in part by the Hong Kong Innovation and Technology Fund under Grant MHP/117/21.
Fingerprint
Dive into the research topics of 'Cross-modal Orthogonal High-rank Augmentation for RGB-Event Transformer-trackers'. Together they form a unique fingerprint.-
GRF: Learning from 4D Light Fields for Clear Vision in Poor Visibility Environments
HOU, J. (Principal Investigator / Project Coordinator)
1/01/22 → …
Project: Research
-
ITF: Wide FoV and High Resolution Video Perception and Efficient Coding
HOU, J. (Principal Investigator / Project Coordinator)
1/01/23 → 31/12/24
Project: Research
-
GRF: Learning-based Three-dimensional Point Cloud Data Reconstruction and Processing
HOU, J. (Principal Investigator / Project Coordinator)
1/01/21 → 23/12/24
Project: Research