Robust Phi-divergence MDPs
File(s)
Accepted version
Author(s)
Ho, Chin Pang
Petrik, Marek
Wiesemann, Wolfram
Type
Conference Paper
Abstract
In recent years, robust Markov decision processes (MDPs) have emerged as a prominent modeling framework for dynamic decision problems affected by uncertainty. In contrast to classical MDPs, which only account for stochasticity by modeling the dynamics through a stochastic process with a known transition kernel, robust MDPs additionally account for ambiguity by optimizing in view of the most adverse transition kernel from a prescribed ambiguity set. In this paper, we develop a novel solution framework for robust MDPs with s-rectangular ambiguity sets that decomposes the problem into a sequence of robust Bellman updates and simplex projections. Exploiting the rich structure present in the simplex projections corresponding to φ-divergence ambiguity sets, we show that the associated s-rectangular
robust MDPs can be solved substantially faster than with state-of-the-art commercial solvers as well as a recent first-order solution scheme, thus rendering them attractive alternatives to classical MDPs in practical applications.
robust MDPs can be solved substantially faster than with state-of-the-art commercial solvers as well as a recent first-order solution scheme, thus rendering them attractive alternatives to classical MDPs in practical applications.
Date Issued
2022-11-28
Date Acceptance
2022-09-15
Citation
Advances in Neural Information Processing Systems, 2022, 35
ISBN
9781713871088
ISSN
1049-5258
Publisher
NeurIPS
Journal / Book Title
Advances in Neural Information Processing Systems
Volume
35
Copyright Statement
© 2022 The Author(s).
Source
NeurIPS 2022
Publication Status
Published
Start Date
2022-11-28
Finish Date
2022-12-09
Coverage Spatial
New Orleans, LA, USA