A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents

A lot of recent machine learning research papers have ``open-ended learning''in their title. But very few of them attempt to define what they mean whenusing the term. Even worse, when looking more closely there seems to be noconsensus on what distinguishes open-ended learning from related concepts suchas continual learning, lifelong learning or autotelic learning. In this paper,we contribute to fixing this situation. After illustrating the genealogy of theconcept and more recent perspectives about what it truly means, we outline thatopen-ended learning is generally conceived as a composite notion encompassing aset of diverse properties. In contrast with previous approaches, we propose toisolate a key elementary property of open-ended processes, which is to produceelements from time to time (e.g., observations, options, reward functions, andgoals), over an infinite horizon, that are considered novel from an observer'sperspective. From there, we build the notion of open-ended learning problemsand focus in particular on the subset of open-ended goal-conditionedreinforcement learning problems in which agents can learn a growing repertoireof goal-driven skills. Finally, we highlight the work that remains to beperformed to fill the gap between our elementary definition and the moreinvolved notions of open-ended learning that developmental AI researchers mayhave in mind.