Policy Gradient in Robust MDPs with Global Convergence Guarantee

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review

2 Scopus Citations
View graph of relations

Author(s)

Related Research Unit(s)

Detail(s)

Original languageEnglish
Title of host publicationProceedings of the 40th International Conference on Machine Learning
PublisherPMLR
Pages35763-35797
Publication statusPublished - Jul 2023

Publication series

NameProceedings of Machine Learning Research
Volume202
ISSN (Print)2640-3498

Conference

Title40th International Conference on Machine Learning (ICML 2023)
LocationHawaii Convention Center
PlaceUnited States
CityHonolulu
Period23 - 29 July 2023

Abstract

Robust Markov decision processes (RMDPs) provide a promising framework for computing reliable policies in the face of model errors. Many successful reinforcement learning algorithms build on variations of policy-gradient methods, but adapting these methods to RMDPs has been challenging. As a result, the applicability of RMDPs to large, practical domains remains limited. This paper proposes a new Double-Loop Robust Policy Gradient (DRPG), the first generic policy gradient method for RMDPs. In contrast with prior robust policy gradient algorithms, DRPG monotonically reduces approximation errors to guarantee convergence to a globally optimal policy in tabular RMDPs. We introduce a novel parametric transition kernel and solve the inner loop robust policy via a gradient-based method. Finally, our numerical results demonstrate the utility of our new algorithm and confirm its global convergence properties.

© The authors and PMLR 2023. MLResearchPress

Citation Format(s)

Policy Gradient in Robust MDPs with Global Convergence Guarantee. / Wang, Qiuhao; Ho, Chin Pang; Petrik, Marek.
Proceedings of the 40th International Conference on Machine Learning. PMLR, 2023. p. 35763-35797 (Proceedings of Machine Learning Research; Vol. 202).

Research output: Chapters, Conference Papers, Creative and Literary WorksRGC 32 - Refereed conference paper (with host publication)peer-review