Probabilistic counterexample guidance for safer reinforcement learning
File(s)
Accepted version
Author(s)
Ji, Xiaotong
Filieri, Antonio
Type
Conference Paper
Abstract
Safe exploration aims at addressing the limitations of Reinforcement Learning (RL) in safety-critical scenarios, where failures during trial-and-error learning may incur high costs. Several methods exist to incorporate external knowledge or to use proximal sensor data to limit the exploration of unsafe states. However, reducing exploration risks in unknown environments, where an agent must discover safety threats during exploration, remains challenging.
In this paper, we target the problem of safe exploration by guiding the training with counterexamples of the safety requirement. Our method abstracts both continuous and discrete state-space systems into compact abstract models representing the safety-relevant knowledge acquired by the agent during exploration. We then exploit probabilistic counterexample generation to construct minimal simulation submodels eliciting safety requirement violations, where the agent can efficiently train offline to refine its policy towards minimising the risk of safety violations during the subsequent online exploration.
We demonstrate our method’s effectiveness in reducing safety violations during online exploration in preliminary experiments by an average of 40.3% compared with QL and DQN standard algorithms and 29.1% compared with previous related work, while achieving comparable cumulative rewards with respect to unrestricted exploration and alternative approaches.
In this paper, we target the problem of safe exploration by guiding the training with counterexamples of the safety requirement. Our method abstracts both continuous and discrete state-space systems into compact abstract models representing the safety-relevant knowledge acquired by the agent during exploration. We then exploit probabilistic counterexample generation to construct minimal simulation submodels eliciting safety requirement violations, where the agent can efficiently train offline to refine its policy towards minimising the risk of safety violations during the subsequent online exploration.
We demonstrate our method’s effectiveness in reducing safety violations during online exploration in preliminary experiments by an average of 40.3% compared with QL and DQN standard algorithms and 29.1% compared with previous related work, while achieving comparable cumulative rewards with respect to unrestricted exploration and alternative approaches.
Date Issued
2023-09-15
Date Acceptance
2023-09-01
Citation
Lecture Notes in Computer Science, 2023, 14287, pp.311-328
ISBN
9783031438349
ISSN
0302-9743
Publisher
Springer Nature Switzerland
Start Page
311
End Page
328
Journal / Book Title
Lecture Notes in Computer Science
Volume
14287
Copyright Statement
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG. The Version of Record is available online at: https://link.springer.com/chapter/10.1007/978-3-031-43835-6_22
Identifier
http://dx.doi.org/10.1007/978-3-031-43835-6_22
Source
QEST 2023
Publication Status
Published
Start Date
2023-09-20
Finish Date
2023-09-22
Coverage Spatial
Antwerp, Belgium
Date Publish Online
2023-09-15