Kernel-based adaptive estimation: multidimensional and state-space approaches

Abstract

Several disciplines, from engineering to social sciences, critically depend on adaptive signal estimation to either remove observation noise (filtering), or to approximate quantities before they become available (prediction). When an optimal estimator cannot be expressed in closed form, e.g. due to model uncertainty or complexity, machine learning algorithms have proven to successfully learn a model which captures rich relationships from large datasets. This thesis proposes two novel approaches to signal estimation based on support vector regression (SVR): high-dimensional kernel learning (HDKL) and kernel-based state-spaces modelling (KSSM).

In real-world applications, signal dynamics usually depend on both time and the value of the signal itself. The HDKL concept extends the standard, single-kernel, SVR estimation approach by considering a feature space constructed as an ensemble of real-valued feature spaces; the resulting feature space provides highly-localised estimation by averaging the subkernels estimates and is well-suited for multichannel signals, as it captures interchannel data-dependency. This thesis then provides a rigorous account for the existence of such higher-dimensional RKHS and their corresponding kernels by considering the complex-, quaternion- and vector-valued cases.

Current kernel adaptive filters employ nonlinear autoregressive models and express the current value of the signal as a function of past values with added noise. The motivation for the second main contribution of this thesis is to depart from this class of models and propose a state-space model designed using kernels (KSSM), whereby the signal of interest is a latent state and the observations are noisy measurements of the hidden process. This formulation allows for jointly estimating the signal (state) and the parameters, and is robust to observation noise. The posterior density of the kernel mixing parameters is then found in an unsupervised fashion using Markov chain Monte Carlo and particle filters, and both the offline and online cases are addressed.

The capabilities of the proposed algorithms are initially illustrated by simulation examples using synthetic data in a controlled environment. Finally, both the HDKL and the KSSM approaches are validated in the estimation of real-world signals including body-motion trajectories, bivariate wind speed, point-of-gaze location, and national grid frequency.