FORM: Learning expressive and transferable first-order logic reward machines
File(s)
Author(s)
Russo, Alessandra
Ardon, Leo
Parac, Roko
Furelos Blanco, Daniel
Type
Conference Paper
Abstract
Reward machines (RMs) are an effective approach for addressing non-Markovian rewards in reinforcement learning (RL) through finite-state machines. Traditional RMs, which label edges with propositional logic formulae, inherit the limited expressivity of propositional logic. This limitation hinders the learnability and transferability of RMs since complex tasks will require numerous states and edges. To overcome these challenges, we propose First-Order Reward Machines (FORMs), which use first-order logic to label edges, resulting in more compact and transferable RMs. We introduce a novel method for learning FORMs and a multi-agent formulation for exploiting them and facilitate their transferability,
where multiple agents collaboratively learn policies for a shared FORM. Our experimental results demonstrate the scalability of FORMs with respect to traditional RMs. Specifically, we show that FORMs can be effectively learnt for tasks where traditional RM learning approaches fail. We also show significant improvements in learning
speed and task transferability thanks to the multi-agent learning framework and the abstraction provided by the first-order language.
where multiple agents collaboratively learn policies for a shared FORM. Our experimental results demonstrate the scalability of FORMs with respect to traditional RMs. Specifically, we show that FORMs can be effectively learnt for tasks where traditional RM learning approaches fail. We also show significant improvements in learning
speed and task transferability thanks to the multi-agent learning framework and the abstraction provided by the first-order language.
Date Acceptance
2024-12-19
Citation
24th International Conference on Autonomous Agents and Multiagent Systems
Publisher
ACM
Journal / Book Title
24th International Conference on Autonomous Agents and Multiagent Systems
Copyright Statement
Subject to copyright. This paper is embargoed until publication. Once published will be available open access under a CC-BY Licence.
License URL
Sponsor
US Army (US)
Engineering & Physical Science Research Council (E
Grant Number
0160 G LB679
EP/X040518/1
Source
International Conference on Autonomous Agents and Multiagent Systems
Publication Status
Accepted
Start Date
2025-05-21
Finish Date
2025-05-23
Coverage Spatial
Detroit, Michigan, USA