On Robust Regret for Sequential Decision-Making Problem

Markov decision processes (MDPs) are widely used models for sequential decision-making. To formulate an MDP, one needs to specify the transition kernel, which is often estimated from available data using statistical analysis. However, MDPs are highly sensitive to errors in the estimation of the transition kernel, which can result in policies that are overly optimistic and perform poorly in practice.To address this issue, robust Markov decision processes (MDPs) have emerged as a promising framework. Unlike traditional MDPs, robust MDPs consider a set of possible transition kernels, referred to as the ambiguity set, rather than relying on a single estimate. The output policies of robust MDPs are reliable because they hedge against all potential kernels simultaneously. However, these policies are often overly conservative as they optimize the worst-case performance.In this project, we propose robust regret MDPs, which aim to minimize regret under model ambiguity. By optimizing this criterion, one can obtain policies that minimize regret relative to the best possible outcome at each time step, rather than focusing solely on worst-case performance. As a result, these policies provide reliable performance without being overly conservative.We aim to focus on three main tasks. First, we aim to propose a time-consistent robust regret MDPs and derive their dynamic programming formulation. We also aim to design tractable algorithms that are efficient and scalable.Second, we aim to extend robust regret MDPs by incorporating distributionally robust safety constraints. This will ensure that policies remain safe even when the cost function is uncertain and ambiguous. We aim to explore various optimization algorithms and policy gradient methods to solve this problem efficiently.Third, we aim to apply our framework to a case study in supply chain inventory management. We aim to model this problem as a robust regret MDP and investigate how traditional policies, such as base-stock policies, can be improved under ambiguity.Assuming positive outcomes, the developed models and algorithms will provide a reliable framework for sequential decision problems under uncertainty. Our results will contribute to the literature on robust MDPs and optimization. In addition, we aim to provide open-source implementations of our algorithms, thereby facilitating their adoption by researchers and practitioners.