Whether, How and When Modern Evolutionary Strategies Can Be Improved

Recent advances in the development of effective Evolutionary Strategies (ESs) have paved the way to their usage in a broad variety of problems. Among the others, the OpenAI Evolutionary Strategy (OpenAI-ES) has emerged as a notable method to discover solutions for robotic problems like locomotion and aggregation, or for classic tasks like pole balancing. The main advantage of OpenAI-ES over its counterparts is the usage of momentum vectors storing information about the relationship between parameter variations and performance, with no need for computational demanding covariance matrices. However, previous studies show that OpenAI-ES may achieve low performance in rather complex scenarios or when the function used to evaluate performance is not designed for ESs. This makes us hypothesize that there is room for enhancements of OpenAI-ES. This work delves into the design and analysis of three variants of OpenAI-ES and performs a thorough comparison of OpenAI- ES, its variants and the Stochastic Steady State with Hill Climbing (SSSHC) on a broad set of problems, including several benchmarks from the literature. Our results prove that the variants introduced in this work are competitive with OpenAI-ES in many cases. Moreover, SSSHC outperforms the other methods in most of the considered problems, while OpenAI- ES excels in robot locomotion. This work contributes to shed the light on both the pitfalls OpenAI-ES can encounter and new perspectives to further improve it.