Data learning for human pose tracking

Abstract

One of the most pressing problems in data-driven models is how to include latent data information in the model-building process. This could help to reduce the amount of data required for training in machine learning applications. Pose tracking is a field currently dominated by data-driven models and where the collection of large, labeled datasets is difficult and time intensive. We believe this is an application that could benefit significantly from the inclusion of structure in the interpretation of available data. Improving human pose tracking methods is significant, as existing methods are not robust enough for application in real-world scenarios such as remote physiotherapy or mobility monitoring. Data Assimilation and Machine Learning are both fields that allow a prediction to be made using a forecasting model. We propose to integrate methods from these two fields to improve the application of pose tracking methods in real-world scenarios. To the best of our knowledge, this is the first application of methods of this kind to the field of human pose tracking. In particular, this Thesis presents two areas of work applying Data Assimilation methods to human pose tracking. First, we show how to apply adjoint methods (from Data Assimilation (DA)) to 3D human pose tracking. This method successfully recovers 3D pose from only Inertial Measurement Units orientation data without the need for a learnt prior, joint limits or additional constraints. The method also does not require a full motion sequence for optimisation, allowing the algorithm to run online. The second collection of work is concerned with 2D human pose tracking in RGB images. Here we show three methods for Data Assimilation inspired modifications to traditional pose estimation methods. First, we apply a Kalman filter layer to traditional Convolutional Neural Network-based pose estimation methods to improve speed, consistency and accuracy of joint (or keypoint) location. Second, to additionally improve performance of these filters, we present a novel application of covariance extraction from feature heatmaps outputted by pose estimation Convolutional Neural Networks (CNNs). Third, we change the convolution function of the keypoint location network of the open-source pose estimation framework OpenPose to resemble a covariance calculation. There is significant potential for this work to continue to be applied in human pose tracking and other fields. To this end, the equations presented in this work are general to allow them to be applied to other areas with only minor modification.