The ability of deep learning-based methods to perceive, reason about, and react to complex sensory signals has the potential to give robots the capability to manipulate the world around them, and interact with unstructured and unpredictable environments such as human homes.
In order to practically use these methods for robotics however, the question of data availability needs to be addressed. A way of doing so that naturally comes to mind is to use simulation and synthetic data in order to generate any datasets required for learning. Yet, this is a challenging task. Models deployed in the real- world that are naively trained using simulated data will fail due to the reality gap, and it is also not clear which parts of a robotics pipeline can or should be addressed using learning and sim-to-real transfer.
As such, in this thesis we explore how to best use simulation and sim-to-real transfer to enable real-world, learning-based robot-manipulation without using real- world data. We start by conducting an in-depth study on sim-to-real transfer for dynamics with end-to-end control, benchmarking several alternative approaches. We then investigate methods and frameworks that incorporate simulation-trained, learning-based components into otherwise well structured robotics pipelines. Specifically, (1) we develop a framework that can achieve sub-millimetre precision in the control while generalising to wide task spaces, (2) we show how sim-to-real transfer can be used for eye-in-hand camera calibration, an often necessary step in robotics pipelines, and (3) we present a framework for one-shot imitation learning that can perform a task immediately after one demonstration, without the need for further real-world data collection or training.