Scaling policy gradient quality-diversity with massive parallelization via behavioral variations
File(s)
Published version
Author(s)
Mitsides, Konstantinos
Faldor, Maxence
Cully, Antoine
Type
Conference Paper
Abstract
Quality-Diversity optimization comprises a family of evolutionary algorithms aimed at generating a collection of diverse and high-performing solutions. MAP-Elites (ME), a notable example, is used effectively in "elds like evolutionary robotics. However, the reliance of ME on random mutations from Genetic Algorithms limits its ability to evolve high-dimensional solutions. Methods proposed to overcome this include using gradient-based operators like policy
gradients or natural evolution strategies. While successful at scaling ME for neuroevolution, these methods often suffer from slow training speeds, or difficulties in scaling with massive parallelization due to high computational demands or reliance on centralized actor-critic training. In this work, we introduce a fast, sample-efficient ME based algorithm capable of scaling with massive parallelization, significantly reducing runtimes without compromising performance.
Our method, ASCII-ME, unlike existing policy gradient quality-diversity methods, does not rely on centralized actor-critic training. It performs behavioral variations based on time step performance metrics and maps these variations to solutions using policy gradients. Our experiments show that ASCII-ME can generate a diverse collection of high-performing deep neural network policies in less than 250 seconds on a single GPU. Additionally, it operates on
average, five times faster than state-of-the-art algorithms while maintaining competitive sample efficiency.
gradients or natural evolution strategies. While successful at scaling ME for neuroevolution, these methods often suffer from slow training speeds, or difficulties in scaling with massive parallelization due to high computational demands or reliance on centralized actor-critic training. In this work, we introduce a fast, sample-efficient ME based algorithm capable of scaling with massive parallelization, significantly reducing runtimes without compromising performance.
Our method, ASCII-ME, unlike existing policy gradient quality-diversity methods, does not rely on centralized actor-critic training. It performs behavioral variations based on time step performance metrics and maps these variations to solutions using policy gradients. Our experiments show that ASCII-ME can generate a diverse collection of high-performing deep neural network policies in less than 250 seconds on a single GPU. Additionally, it operates on
average, five times faster than state-of-the-art algorithms while maintaining competitive sample efficiency.
Date Acceptance
2025-03-19
Citation
GECCO '25: Proceedings of the Genetic and Evolutionary Computation Conference
ISBN
9798400714658
Publisher
Association for Computing Machinery (ACM)
Journal / Book Title
GECCO '25: Proceedings of the Genetic and Evolutionary Computation Conference
Copyright Statement
Subject to copyright. This paper is embargoed until publication. Once published the author’s accepted manuscript will be made available under a CC-BY License in accordance with Imperial’s Research Publications Open Access policy (www.imperial.ac.uk/oa-policy).
Source
GECCO'25 - The Genetic and Evolutionary Computation Conference
Publication Status
Accepted
Start Date
2025-07-14
Finish Date
2025-07-18
Coverage Spatial
Malaga, Spain