Embedded deep learning systems for robot visual intelligence

Abstract

Over the past decade, the breakthrough of deep learning (DL) has revolutionised myriads of machine vision applications in the wild. As such, the Artificial Intelligence (AI) disruption has boosted the deployment of mobile robot systems, typically equipped with visual sensors, in real- world environments. However, the tremendous advancement of Deep Neural Networks (DNNs), which lead to their state-of-the-art accuracy, comes with constantly increasing computational and memory requirements. This poses significant challenges to their deployment in real-time applications, especially those related to mobile robots and Unmanned Aerial Vehicles (UAVs), that constitute inherently resource-constrained platforms due to their limited payload and power envelopes. Often confined by a strict latency budget and limited network availability upon deployment, o -loading computation to remote servers on the cloud is usually not a feasible option. This translates to sole reliance on the limited compute-capability devices available on-board, such as embedded Graphic Processing Units (GPUs) and Field-Programmable Gate Arrays (FPGAs). Therefore, alleviating the workload burden of modern DNNs is of primary importance towards their e cient deployment in the embedded space. Additionally, taking into account the important interplay between the model (software) and architecture (hardware) design, e ciency can arise from incorporating knowledge about the one to the design process of the other (i.e. hardware- aware model design, or model-aware hardware design) or even co-optimisation (model-hardware co-design), instead of studying the two in isolation. This thesis addresses the challenge of e cient DNN deployment, focusing on mobile robot and UAV applications. The main vehicle employed for e ciency is application-specific customisation (either at the hardware or model level), materialised through approximation methodologies applied all-across the deployment stack of DNNs. This spans from the (lowest) computation- level approaches (e.g. approximating DNN calculations by adopting low-precision and custom hardware accelerators), through the model and data layers to the (highest) application-level (e.g. approximating unseen camera views of an object through DNNs). The contributions of this work include: a multi-precision model cascade for e cient input-dependent classification; a progressive-refinement based approximate inference methodology for autonomous driving; a region-selection methodology for e cient UAV-based detection exploiting prior domain knowledge and multi-modal sensory inputs to dynamically reduce the workload; a self-supervised DL-based approach for indoor autonomous navigation through spatio-temporal representation learning; and an object-level depth inpainting pipeline for e cient occlusion-free 3D reconstruction. Each of the above works exploits approximation and/or application-specific customisation methodologies to introduce and exploit a performance-accuracy trade-o at di erent abstraction levels, pushing the limits of e cient deployment of DNNs on-board mobile robot platforms, towards robot visual intelligence.