Policy manifold search: exploring the manifold hypothesis for diversity-based neuroevolution
File(s)
Published version
Author(s)
Rakicevic, Nemanja
Cully, Antoine
Kormushev, Petar
Type
Conference Paper
Abstract
Neuroevolution is an alternative to gradient-based optimisation that has the potential to avoid local minima and allows parallelisation. The main limiting factor is that usually it does not scale well with parameter space dimensionality. Inspired by recent work examining neural network intrinsic dimension and loss landscapes, we hypothesise that there exists a low-dimensional manifold, embedded in the policy network parameter space, around which a high-density of diverse and useful policies are located. This paper proposes a novel method for diversity-based policy search via Neuroevolution, that leverages learned representations of the policy network parameters, by performing policy search in this learned representation space. Our method relies on the Quality-Diversity (QD) framework which provides a principled approach to policy search, and maintains a collection of diverse policies, used as a dataset for learning policy representations. Further, we use the Jacobian of the inverse-mapping function to guide the search in the representation space. This ensures that the generated samples remain in the high-density regions, after mapping back to the original space. Finally, we evaluate our contributions on four continuous-control tasks in simulated environments, and compare to diversity-based baselines.
Date Issued
2021-06-26
Date Acceptance
2021-03-26
Citation
Proceedings of the Genetic and Evolutionary Computation Conference, 2021, pp.901-909
ISBN
9781450383509
Start Page
901
End Page
909
Journal / Book Title
Proceedings of the Genetic and Evolutionary Computation Conference
Copyright Statement
© 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in GECCO '21: Proceedings of the Genetic and Evolutionary Computation Conference June 2021 Pages 901–909 https://doi.org/10.1145/3449639.3459320
Identifier
http://arxiv.org/abs/2104.13424v1
Source
Genetic and Evolutionary Computation Conference (GECCO '21)
Subjects
cs.LG
cs.LG
cs.NE
Publication Status
Published
Start Date
2021-07-10
Finish Date
2021-07-14
Coverage Spatial
Lille, France
Date Publish Online
2021-06-26