Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control
File(s)
Published version
Author(s)
Kamthe, Sanket
Deisenroth, Marc Peter
Type
Working Paper
Abstract
Trial-and-error based reinforcement learning (RL) has seen rapid advancements
in recent times, especially with the advent of deep neural networks. However,
the majority of autonomous RL algorithms either rely on engineered features or
a large number of interactions with the environment. Such a large number of
interactions may be impractical in many real-world applications. For example,
robots are subject to wear and tear and, hence, millions of interactions may
change or damage the system. Moreover, practical systems have limitations in
the form of the maximum torque that can be safely applied. To reduce the number
of system interactions while naturally handling constraints, we propose a
model-based RL framework based on Model Predictive Control (MPC). In
particular, we propose to learn a probabilistic transition model using Gaussian
Processes (GPs) to incorporate model uncertainties into long-term predictions,
thereby, reducing the impact of model errors. We then use MPC to find a control
sequence that minimises the expected long-term cost. We provide theoretical
guarantees for the first-order optimality in the GP-based transition models
with deterministic approximate inference for long-term planning. The proposed
framework demonstrates superior data efficiency and learning rates compared to
the current state of the art.
in recent times, especially with the advent of deep neural networks. However,
the majority of autonomous RL algorithms either rely on engineered features or
a large number of interactions with the environment. Such a large number of
interactions may be impractical in many real-world applications. For example,
robots are subject to wear and tear and, hence, millions of interactions may
change or damage the system. Moreover, practical systems have limitations in
the form of the maximum torque that can be safely applied. To reduce the number
of system interactions while naturally handling constraints, we propose a
model-based RL framework based on Model Predictive Control (MPC). In
particular, we propose to learn a probabilistic transition model using Gaussian
Processes (GPs) to incorporate model uncertainties into long-term predictions,
thereby, reducing the impact of model errors. We then use MPC to find a control
sequence that minimises the expected long-term cost. We provide theoretical
guarantees for the first-order optimality in the GP-based transition models
with deterministic approximate inference for long-term planning. The proposed
framework demonstrates superior data efficiency and learning rates compared to
the current state of the art.
Date Issued
2017-12-31
Date Acceptance
2017-11-15
Copyright Statement
© The Author
Identifier
http://arxiv.org/abs/1706.06491v1
Source
International Conference on Artificial Intelligence and Statistics
Subjects
cs.SY
cs.SY
stat.ML