A unifying view of optimism in episodic reinforcement learning
File(s)
Accepted version
Author(s)
Neu, Gergely
Pike-Burke, Ciara
Type
Conference Paper
Abstract
The principle of “optimism in the face of uncertainty” underpins many theoretically successful reinforcement learning algorithms. In this paper we provide a general framework for designing, analyzing and implementing such algorithms in the episodic reinforcement learning problem. This framework is built upon Lagrangian duality, and demonstrates that every model-optimistic algorithm that constructs anoptimistic MDP has an equivalent representation as a value-optimistic dynamic programming algorithm. Typically, it was thought that these two classes of algorithms were distinct, with model-optimistic algorithms benefiting from a cleaner probabilistic analysis while value-optimistic algorithms are easier to implement and thus more practical. With the framework developed in this paper, we show that it is possible to get the best of both worlds by providing a class of algorithms which have a computationally efficient dynamic-programming implementation and also a simple probabilistic analysis. Besides being able to capture many existing algorithms in the tabular setting, our framework can also address large-scale problems under realizable function approximation, where it enables a simple model-based analysis of some recently proposed methods.
Date Issued
2020-12-06
Date Acceptance
2020-09-25
Citation
Advances in neural information processing systems, 2020, 33
ISSN
1049-5258
Journal / Book Title
Advances in neural information processing systems
Volume
33
Copyright Statement
© 2020 Neural Information Processing System.
Identifier
https://papers.nips.cc/paper/2020/hash/0f0e13216262f4a201bec128044dd30f-Abstract.html
Source
Neural Information Processing Systems (NeurIPS 2020)
Subjects
1701 Psychology
1702 Cognitive Sciences
Publication Status
Published
Start Date
2020-12-06
Finish Date
2020-12-12
Coverage Spatial
Virtual