Bayesian learning for nonlinear system identification

Abstract

Prediction and control of behaviour and abnormalities in any complex dynamical systems, and in particular those encountered in biology, physics, engineering require the development of multivariate mechanistic and predictive models that integrate large datasets from different sources. Although, a large amount of data are being collected on a daily basis, very few methods allow the automatic creation from these data of nonlinear dynamical models for understanding and (re-)design/control, and an inordinate amount of time is still being spent on the manual aggregation of information and development of models that explains these data.

In particular, this thesis considers sparse modelling and estimation for a selection of nonlinear dynamical systems classes. There are two key features of modern time series data, i.e., high dimensionality and large scale. The dimensionality, or the complexity, grew with the sample size, and ``ultra-high'' refers to the case where the dimensionality increased at a non-polynomial rate. Scale, or the size, refers to the dimension of the system, i.e., the number of state variables. This work aims to design a framework and associated algorithms for the identification of a variety of nonlinear dynamical systems encountered in practice from high-dimensional and large-scale time series data.

In the first part of the thesis, we introduce the type of time series data and the class of nonlinear dynamical system considered in this thesis. Both a selection of time-invariant and time-varying nonlinear dynamical systems are covered. For time-invariant system, the classic nonlinear system identification problem from single dataset is addressed in the beginning. Then we move to a more practical and significant yet complicated scenario where heterogeneous datasets are used simultaneously. Such datasets typically contain (a) data from several replicates of an experiment performed on a biological system of interest and/or (b) data measured from a biochemical system subjected to different experimental conditions, for example, changes/perturbations in biological inductions, temperature, gene knock-out, gene over-expression, etc. For time-varying systems, the regime-switch system identification problem is considered, i.e., the problem of identifying both the switching points and the nonlinear model structure within each regime. Then the abrupt change point detection problem is considered. Using these, the classic trending filtering and fault diagnosis problems are revisited. All the identification problems are formulated as various $\ell_0$ type optimisation problems. In the end, we discuss some technical issues on data processing arising from practical applications.

In the second part of the thesis, a repository of algorithms are derived respectively for each identification problem formulated in the first part. These algorithms are not distinct and can be formulated in a unified way using Bayesian Learning with structural sparse prior. Furthermore, we suggest a series of iterative reweighted convex relaxation schemes for connecting these algorithms to popular algorithms including Lasso, Group-Lasso, Generalised-Lasso, Fused-Lasso and Graphical-Lasso. In this part, we go beyond from simple nonlinear model class to more general class; from data likelihood in Gaussian distribution to the more general exponential family. The estimation of the stochastic term also discussed including ARMA and ARCH. Many optimisation framework, such as (stochastic) gradient descent, Newton method, Quasi-Newton method, alternating direction method of multiplier can be seamlessly integrated into our formulation as either centralised or distributed optimisation strategy to address high dimensionality and large scale problems. These algorithms largely enrich not only the family of time series modelling algorithms but also sparse signal recovery/modelling/estimation algorithms in various communities.

In the third part of the thesis, several time series modelling applications from systems biology, complex networks and power systems are given to illustrate the effectiveness of our modelling framework.

Last but not least, two future research directions based on the output of this thesis are pointed out, both related to ``brains''. The first is focusing on theory and algorithm about modelling/identification/learning on deep neural networks. The second is focusing applications in neuroscience: understanding the neural basis of decision making using mathematical modelling from big data.