Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning

Shuang Qiu; Lingxiao Wang; Chenjia Bai; Zhuoran Yang; Zhaoran Wang

Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning

Shuang Qiu^*
, Lingxiao Wang^*
, Chenjia Bai^*
, Zhuoran Yang
, Zhaoran Wang

^*Corresponding author for this work

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

27 Citations (Scopus)

Abstract

In view of its power in extracting feature representation, contrastive self-supervised learning has been successfully integrated into the practice of (deep) reinforcement learning (RL), leading to efficient policy learning on various applications. Despite its tremendous empirical successes, the understanding of contrastive learning for RL remains elusive. To narrow such a gap, we study contrastive-learning empowered RL for a class of Markov decision processes (MDPs) and Markov games (MGs) with low-rank transitions. For both models, we propose to extract the correct feature representations of the low-rank model by minimizing a contrastive loss. Moreover, under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs. We further theoretically prove that our algorithm recovers the true representations and simultaneously achieves sample efficiency in learning the optimal policy and Nash equilibrium in MDPs and MGs. We also provide empirical studies to demonstrate the efficacy of the UCB-based contrastive learning method for RL. To the best of our knowledge, we provide the first provably efficient online RL algorithm that incorporates contrastive learning for representation learning. Copyright © 2022 by the author(s)

Original language	English
Title of host publication	Proceedings of the 39^th International Conference on Machine Learning
Publisher	ML Research Press
Pages	18168-18210
Volume	162
Publication status	Published - Jul 2022
Externally published	Yes
Event	39th International Conference on Machine Learning (ICML 2022) - Hybrid, Baltimore, United States Duration: 17 Jul 2022 → 23 Jul 2022 https://icml.cc/virtual/2022/index.html https://icml.cc/Conferences/2022 https://proceedings.mlr.press/v162/

Publication series

Name	Proceedings of Machine Learning Research
ISSN (Print)	2640-3498

Conference

Conference	39th International Conference on Machine Learning (ICML 2022)
Place	United States
City	Baltimore
Period	17/07/22 → 23/07/22
Internet address	https://icml.cc/virtual/2022/index.html https://icml.cc/Conferences/2022 https://proceedings.mlr.press/v162/

Funding

The authors would like to thank all reviewers for valuable comments. The authors would like to thank Sirui Zheng for helpful discussions. Zhaoran Wang acknowledges National Science Foundation (Awards 2048075, 2008827, 2015568, 1934931), Simons Institute (Theory of Reinforcement Learning), Amazon, J.P. Morgan, and Two Sigma for their supports.

Access Output

https://proceedings.mlr.press/v162/qiu22c.html

Other Access Options

Check CityUHK Library

Permanent Link

Right click to copy link

Cite this

Qiu, S., Wang, L., Bai, C., Yang, Z., & Wang, Z. (2022). Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning. In Proceedings of the 39^th International Conference on Machine Learning (Vol. 162, pp. 18168-18210). (Proceedings of Machine Learning Research). ML Research Press. https://proceedings.mlr.press/v162/qiu22c.html

@inproceedings{f0a3bb725d6c4a259822e57684a76861,

title = "Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning",

abstract = "In view of its power in extracting feature representation, contrastive self-supervised learning has been successfully integrated into the practice of (deep) reinforcement learning (RL), leading to efficient policy learning on various applications. Despite its tremendous empirical successes, the understanding of contrastive learning for RL remains elusive. To narrow such a gap, we study contrastive-learning empowered RL for a class of Markov decision processes (MDPs) and Markov games (MGs) with low-rank transitions. For both models, we propose to extract the correct feature representations of the low-rank model by minimizing a contrastive loss. Moreover, under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs. We further theoretically prove that our algorithm recovers the true representations and simultaneously achieves sample efficiency in learning the optimal policy and Nash equilibrium in MDPs and MGs. We also provide empirical studies to demonstrate the efficacy of the UCB-based contrastive learning method for RL. To the best of our knowledge, we provide the first provably efficient online RL algorithm that incorporates contrastive learning for representation learning. Copyright {\textcopyright} 2022 by the author(s)",

author = "Shuang Qiu and Lingxiao Wang and Chenjia Bai and Zhuoran Yang and Zhaoran Wang",

year = "2022",

month = jul,

language = "English",

volume = "162",

series = "Proceedings of Machine Learning Research",

publisher = "ML Research Press",

pages = "18168--18210",

booktitle = "Proceedings of the 39th International Conference on Machine Learning",

note = "39th International Conference on Machine Learning (ICML 2022) ; Conference date: 17-07-2022 Through 23-07-2022",

url = "https://icml.cc/virtual/2022/index.html, https://icml.cc/Conferences/2022, https://proceedings.mlr.press/v162/",

}

Qiu, S, Wang, L, Bai, C, Yang, Z & Wang, Z 2022, Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning. in Proceedings of the 39^th International Conference on Machine Learning. vol. 162, Proceedings of Machine Learning Research, ML Research Press, pp. 18168-18210, 39th International Conference on Machine Learning (ICML 2022), Baltimore, Maryland, United States, 17/07/22. <https://proceedings.mlr.press/v162/qiu22c.html>

Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning. / Qiu, Shuang; Wang, Lingxiao; Bai, Chenjia et al.
Proceedings of the 39^th International Conference on Machine Learning. Vol. 162 ML Research Press, 2022. p. 18168-18210 (Proceedings of Machine Learning Research).

Research output: Chapters, Conference Papers, Creative and Literary Works › RGC 32 - Refereed conference paper (with host publication) › peer-review

TY - GEN

T1 - Contrastive UCB

T2 - 39th International Conference on Machine Learning (ICML 2022)

AU - Qiu, Shuang

AU - Wang, Lingxiao

AU - Bai, Chenjia

AU - Yang, Zhuoran

AU - Wang, Zhaoran

PY - 2022/7

Y1 - 2022/7

N2 - In view of its power in extracting feature representation, contrastive self-supervised learning has been successfully integrated into the practice of (deep) reinforcement learning (RL), leading to efficient policy learning on various applications. Despite its tremendous empirical successes, the understanding of contrastive learning for RL remains elusive. To narrow such a gap, we study contrastive-learning empowered RL for a class of Markov decision processes (MDPs) and Markov games (MGs) with low-rank transitions. For both models, we propose to extract the correct feature representations of the low-rank model by minimizing a contrastive loss. Moreover, under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs. We further theoretically prove that our algorithm recovers the true representations and simultaneously achieves sample efficiency in learning the optimal policy and Nash equilibrium in MDPs and MGs. We also provide empirical studies to demonstrate the efficacy of the UCB-based contrastive learning method for RL. To the best of our knowledge, we provide the first provably efficient online RL algorithm that incorporates contrastive learning for representation learning. Copyright © 2022 by the author(s)

AB - In view of its power in extracting feature representation, contrastive self-supervised learning has been successfully integrated into the practice of (deep) reinforcement learning (RL), leading to efficient policy learning on various applications. Despite its tremendous empirical successes, the understanding of contrastive learning for RL remains elusive. To narrow such a gap, we study contrastive-learning empowered RL for a class of Markov decision processes (MDPs) and Markov games (MGs) with low-rank transitions. For both models, we propose to extract the correct feature representations of the low-rank model by minimizing a contrastive loss. Moreover, under the online setting, we propose novel upper confidence bound (UCB)-type algorithms that incorporate such a contrastive loss with online RL algorithms for MDPs or MGs. We further theoretically prove that our algorithm recovers the true representations and simultaneously achieves sample efficiency in learning the optimal policy and Nash equilibrium in MDPs and MGs. We also provide empirical studies to demonstrate the efficacy of the UCB-based contrastive learning method for RL. To the best of our knowledge, we provide the first provably efficient online RL algorithm that incorporates contrastive learning for representation learning. Copyright © 2022 by the author(s)

UR - https://www.scopus.com/pages/publications/85163119748

UR - https://www.scopus.com/record/pubmetrics.uri?eid=2-s2.0-85163119748&origin=recordpage

M3 - RGC 32 - Refereed conference paper (with host publication)

VL - 162

T3 - Proceedings of Machine Learning Research

SP - 18168

EP - 18210

BT - Proceedings of the 39th International Conference on Machine Learning

PB - ML Research Press

Y2 - 17 July 2022 through 23 July 2022

ER -

Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learning

Abstract

Publication series

Conference

Funding

Access Output

Other Access Options

Permanent Link

Other Files and Links

Fingerprint

Cite this