Empirical analysis of PGA-MAP-Elites for neuroevolution in uncertain domains
File(s)
Author(s)
Flageat, Manon
Chalumeau, Felix
Cully, Antoine
Type
Journal Article
Abstract
Quality-Diversity algorithms, among which MAP-Elites, have emerged as powerful alternatives to performance-only optimisation approaches as they enable generating collections of diverse and high-performing solutions to an optimisation problem. However, they are often limited to low-dimensional search spaces and deterministic environments. The recently introduced Policy Gradient Assisted MAP-Elites (PGA-MAP-Elites) algorithm overcomes this limitation by pairing the traditional Genetic operator of MAP-Elites with a gradient-based operator inspired by Deep Reinforcement Learning. This new operator guides mutations toward high-performing solutions using policy-gradients. In this work, we propose an in-depth study of PGA-MAP-Elites. We demonstrate the benefits of policy-gradients on the performance of the algorithm and the reproducibility of the generated solutions when considering uncertain domains. We first prove that PGA-MAP-Elites is highly performant in both deterministic and uncertain high-dimensional environments, decorrelating the two challenges it tackles. Secondly, we show that in addition to outperforming all the considered baselines, the collections of solutions generated by PGA-MAP-Elites are highly reproducible in uncertain environments, approaching the reproducibility of solutions found by Quality-Diversity approaches built specifically for uncertain applications. Finally, we propose an ablation and in-depth analysis of the dynamic of the policy-gradients-based variation. We demonstrate that the policy-gradient variation operator is determinant to guarantee the performance of PGA-MAP-Elites but is only essential during the early stage of the process, where it finds high-performing regions of the search space.
Date Issued
2023-03-29
Date Acceptance
2023-01-01
Citation
ACM Transactions on Evolutionary Learning and Optimization, 2023, 3 (1), pp.1-32
ISSN
2688-299X
Publisher
Association for Computing Machinery (ACM)
Start Page
1
End Page
32
Journal / Book Title
ACM Transactions on Evolutionary Learning and Optimization
Volume
3
Issue
1
Copyright Statement
© 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Evolutionary Learning and Optimization, (09 Jan 2023) https://dl.acm.org/doi/10.1145/3577203
Identifier
https://dl.acm.org/doi/10.1145/3577203
Publication Status
Published
Article Number
1
Date Publish Online
2023-01-09