Potential based reward shaping for hierarchical reinforcement learning
File(s)
Accepted version
Author(s)
Toni, F
Gao, Y
Type
Conference Paper
Abstract
Hierarchical Reinforcement Learning (HRL) outperforms
many ‘flat’ Reinforcement Learning (RL)
algorithms in some application domains. However,
HRL may need longer time to obtain the optimal
policy because of its large action space. Potential
Based Reward Shaping (PBRS) has been widely
used to incorporate heuristics into flat RL algorithms
so as to reduce their exploration. In this
paper, we investigate the integration of PBRS and
HRL, and propose a new algorithm: PBRS-MAXQ-
0. We prove that under certain conditions, PBRSMAXQ-0
is guaranteed to converge. Empirical results
show that PBRS-MAXQ-0 significantly outperforms
MAXQ-0 given good heuristics, and can
converge even when given misleading heuristics.
many ‘flat’ Reinforcement Learning (RL)
algorithms in some application domains. However,
HRL may need longer time to obtain the optimal
policy because of its large action space. Potential
Based Reward Shaping (PBRS) has been widely
used to incorporate heuristics into flat RL algorithms
so as to reduce their exploration. In this
paper, we investigate the integration of PBRS and
HRL, and propose a new algorithm: PBRS-MAXQ-
0. We prove that under certain conditions, PBRSMAXQ-0
is guaranteed to converge. Empirical results
show that PBRS-MAXQ-0 significantly outperforms
MAXQ-0 given good heuristics, and can
converge even when given misleading heuristics.
Date Issued
2015-07-31
Date Acceptance
2015-04-16
Citation
Proceedings of the twenty-fourth International Joint Conference on Artificial Intelligence, 2015
Publisher
AAAI Press/ International Joint Conference on Artificial Intelligence
Journal / Book Title
Proceedings of the twenty-fourth International Joint Conference on Artificial Intelligence
Copyright Statement
© 2015 International Joint Conferences on Artificial Intelligence
Identifier
https://www.ijcai.org/Abstract/15/493
Source
24th International Joint Conference on Artificial Intelligence
Subjects
Science & Technology
Technology
Computer Science, Artificial Intelligence
Computer Science, Interdisciplinary Applications
Computer Science
Publication Status
Published
Start Date
2015-07-25
Finish Date
2015-07-31
Coverage Spatial
Buenos Aires
Date Publish Online
2015-07-31