Data-efficient reinforcement learning (RL) in continuous state-action spaces
using very high-dimensional observations remains a key challenge in developing
fully autonomous systems. We consider a particularly important instance of this
challenge, the pixels-to-torques problem, where an RL agent learns a
closed-loop control policy ("torques") from pixel information only. We
introduce a data-efficient, model-based reinforcement learning algorithm that
learns such a closed-loop policy directly from pixel information. The key
ingredient is a deep dynamical model for learning a low-dimensional feature
embedding of images jointly with a predictive model in this low-dimensional
feature space. Joint learning is crucial for long-term predictions, which lie
at the core of the adaptive nonlinear model predictive control strategy that we
use for closed-loop control. Compared to state-of-the-art RL methods for
continuous states and actions, our approach learns quickly, scales to
high-dimensional state spaces, is lightweight and an important step toward
fully autonomous end-to-end learning from pixels to torques.