Circle-rope: Cone-like decoupled rotary positional embedding for vision-language models

Chengcheng Wang; Jianyuan Guo; Hongguang Li; Yuchuan Tian; Ying Nie; Chang Xu; Kai Han

Circle-rope: Cone-like decoupled rotary positional embedding for vision-language models

Chengcheng Wang (Co-first Author)
, Jianyuan Guo (Co-first Author)
, Hongguang Li (Co-first Author)
, Yuchuan Tian
, Ying Nie
, Chang Xu^* (Co-last Author)
, Kai Han^* (Co-last Author)

^*Corresponding author for this work

Department of Computer Science

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

Abstract

Rotary Position Embedding (RoPE) is widely adopted in large language models, but when applied to vision-language models (VLMs) it couples text and image position indices and can introduce spurious cross-modal relative-position bias. We propose Per-Token Distance (PTD) to quantify cross-modal positional disentanglement, and we prove that is a sufficient condition to eliminate the geometric attention bias induced by RoPE. Guided by this criterion, we introduce Circle-RoPE, which remaps 2D image-token coordinates onto an annulus orthogonal to the text position axis, yielding a cone-like geometry where each text token is equidistant to all image tokens while preserving intra-image spatial structure. We further propose Alternating Geometry Encoding (AGE) to improve robustness by alternating RoPE variants across layers, and experiments on diverse VLM backbones and multimodal benchmarks show consistent gains in spatial grounding and visual reasoning.
Copyright 2026 by the author(s).

Original language	English
Title of host publication	Proceedings of the 43rd International Conference on Machine Learning
Publication status	Published - Jul 2026
Event	43rd International Conference on Machine Learning (ICML 2026) - Seoul, Korea, Republic of Duration: 6 Jul 2026 → 11 Jul 2026 https://icml.cc/Conferences/2026

Conference

Conference	43rd International Conference on Machine Learning (ICML 2026)
Abbreviated title	ICML 2026
Place	Korea, Republic of
City	Seoul
Period	6/07/26 → 11/07/26
Internet address	https://icml.cc/Conferences/2026

Funding

This work is funded by Peking University–BHP Carbon and Climate Wei-Ming PhD Scholars Program (Program Name: Research on Low-Carbon and Energy-Efficient Large Model Architectures; Program Number: WM202505).

Access Output

https://icml.cc/virtual/2026/poster/61523

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

@inproceedings{16f0adb36dd94829a45535c02d214787,

title = "Circle-rope: Cone-like decoupled rotary positional embedding for vision-language models",

abstract = "Rotary Position Embedding (RoPE) is widely adopted in large language models, but when applied to vision-language models (VLMs) it couples text and image position indices and can introduce spurious cross-modal relative-position bias. We propose Per-Token Distance (PTD) to quantify cross-modal positional disentanglement, and we prove that is a sufficient condition to eliminate the geometric attention bias induced by RoPE. Guided by this criterion, we introduce Circle-RoPE, which remaps 2D image-token coordinates onto an annulus orthogonal to the text position axis, yielding a cone-like geometry where each text token is equidistant to all image tokens while preserving intra-image spatial structure. We further propose Alternating Geometry Encoding (AGE) to improve robustness by alternating RoPE variants across layers, and experiments on diverse VLM backbones and multimodal benchmarks show consistent gains in spatial grounding and visual reasoning. Copyright 2026 by the author(s).",

author = "Chengcheng Wang and Jianyuan Guo and Hongguang Li and Yuchuan Tian and Ying Nie and Chang Xu and Kai Han",

year = "2026",

month = jul,

language = "English",

booktitle = "Proceedings of the 43rd International Conference on Machine Learning",

note = "43rd International Conference on Machine Learning (ICML 2026), ICML 2026 ; Conference date: 06-07-2026 Through 11-07-2026",

url = "https://icml.cc/Conferences/2026",

}

Circle-rope: Cone-like decoupled rotary positional embedding for vision-language models. / Wang, Chengcheng (Co-first Author); Guo, Jianyuan (Co-first Author); Li, Hongguang (Co-first Author) et al.
Proceedings of the 43rd International Conference on Machine Learning. 2026.

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Circle-rope

T2 - 43rd International Conference on Machine Learning (ICML 2026)

AU - Wang, Chengcheng

AU - Guo, Jianyuan

AU - Li, Hongguang

AU - Tian, Yuchuan

AU - Nie, Ying

AU - Xu, Chang

AU - Han, Kai

PY - 2026/7

Y1 - 2026/7

N2 - Rotary Position Embedding (RoPE) is widely adopted in large language models, but when applied to vision-language models (VLMs) it couples text and image position indices and can introduce spurious cross-modal relative-position bias. We propose Per-Token Distance (PTD) to quantify cross-modal positional disentanglement, and we prove that is a sufficient condition to eliminate the geometric attention bias induced by RoPE. Guided by this criterion, we introduce Circle-RoPE, which remaps 2D image-token coordinates onto an annulus orthogonal to the text position axis, yielding a cone-like geometry where each text token is equidistant to all image tokens while preserving intra-image spatial structure. We further propose Alternating Geometry Encoding (AGE) to improve robustness by alternating RoPE variants across layers, and experiments on diverse VLM backbones and multimodal benchmarks show consistent gains in spatial grounding and visual reasoning. Copyright 2026 by the author(s).

AB - Rotary Position Embedding (RoPE) is widely adopted in large language models, but when applied to vision-language models (VLMs) it couples text and image position indices and can introduce spurious cross-modal relative-position bias. We propose Per-Token Distance (PTD) to quantify cross-modal positional disentanglement, and we prove that is a sufficient condition to eliminate the geometric attention bias induced by RoPE. Guided by this criterion, we introduce Circle-RoPE, which remaps 2D image-token coordinates onto an annulus orthogonal to the text position axis, yielding a cone-like geometry where each text token is equidistant to all image tokens while preserving intra-image spatial structure. We further propose Alternating Geometry Encoding (AGE) to improve robustness by alternating RoPE variants across layers, and experiments on diverse VLM backbones and multimodal benchmarks show consistent gains in spatial grounding and visual reasoning. Copyright 2026 by the author(s).

M3 - RGC 32 - Refereed conference paper (with host publication)

BT - Proceedings of the 43rd International Conference on Machine Learning

Y2 - 6 July 2026 through 11 July 2026

ER -

Circle-rope: Cone-like decoupled rotary positional embedding for vision-language models

Abstract

Conference

Funding

Access Output

Other Access Options

Permanent Link

Fingerprint

Cite this