A pragmatic look at deep imitation learning
File(s)
Working paper
Author(s)
Arulkumaran, Kai
Lillrank, Dan Ogawa
Type
Working Paper
Abstract
The introduction of the generative adversarial imitation learning (GAIL)
algorithm has spurred the development of scalable imitation learning approaches
using deep neural networks. The GAIL objective can be thought of as 1) matching
the expert policy's state distribution; 2) penalising the learned policy's
state distribution; and 3) maximising entropy. While theoretically motivated,
in practice GAIL can be difficult to apply, not least due to the instabilities
of adversarial training. In this paper, we take a pragmatic look at GAIL and
related imitation learning algorithms. We implement and automatically tune a
range of algorithms in a unified experimental setup, presenting a fair
evaluation between the competing methods. From our results, our primary
recommendation is to consider non-adversarial methods. Furthermore, we discuss
the common components of imitation learning objectives, and present promising
avenues for future research.
algorithm has spurred the development of scalable imitation learning approaches
using deep neural networks. The GAIL objective can be thought of as 1) matching
the expert policy's state distribution; 2) penalising the learned policy's
state distribution; and 3) maximising entropy. While theoretically motivated,
in practice GAIL can be difficult to apply, not least due to the instabilities
of adversarial training. In this paper, we take a pragmatic look at GAIL and
related imitation learning algorithms. We implement and automatically tune a
range of algorithms in a unified experimental setup, presenting a fair
evaluation between the competing methods. From our results, our primary
recommendation is to consider non-adversarial methods. Furthermore, we discuss
the common components of imitation learning objectives, and present promising
avenues for future research.
Date Issued
2021-08-04
Citation
2021
Publisher
arXiv
Copyright Statement
© 2021 The Author(s)
Identifier
http://arxiv.org/abs/2108.01867v1
Subjects
cs.LG
cs.LG
cs.NE
stat.ML
Publication Status
Published