Convergence of sparse variational inference in Gaussian processes regression
File(s)
Published version
Author(s)
Burt, David R
Rasmussen, Carl Edward
van der Wilk, Mark
Type
Journal Article
Abstract
Gaussian processes are distributions over functions that are versatile and mathematically convenient priors in Bayesian modelling. However, their use is often impeded for data with large numbers of observations, N, due to the cubic (in N) cost of matrix operations used in exact inference. Many solutions have been proposed that rely on M ≪ N inducing variables to form an approximation at a cost of O(NM2). While the computational cost appears linear in N, the true complexity depends on how M must scale with N to ensure a certain quality of the approximation. In this work, we investigate upper and lower bounds on how M needs to grow with N to ensure high quality approximations. We show that we can make the KL-divergence between the approximate model and the exact posterior arbitrarily small for a Gaussian-noise regression model with M ≪ N. Specifically, for the popular squared exponential kernel and D-dimensional Gaussian distributed covariates, M = O((logN)D) suffice and a method with an overall computational cost of ON(log N)2D(log log N)2D can be used to perform inference.
Date Issued
2020-08-01
Date Acceptance
2020-03-26
Citation
Journal of Machine Learning Research, 2020, 21, pp.1-63
ISSN
1532-4435
Publisher
Microtome Publishing
Start Page
1
End Page
63
Journal / Book Title
Journal of Machine Learning Research
Volume
21
Copyright Statement
© 2020 David R. Burt, Carl Edward Rasmussen, Mark van der Wilk.
License: CC-BY 4.0, see https://creativecommons.org/licenses/by/4.0/. Attribution requirements are provided
at http://jmlr.org/papers/v21/19-1015.html
License: CC-BY 4.0, see https://creativecommons.org/licenses/by/4.0/. Attribution requirements are provided
at http://jmlr.org/papers/v21/19-1015.html
License URL
Identifier
https://w.jmlr.org/papers/volume21/19-1015/19-1015.pdf
Subjects
Science & Technology
Technology
Automation & Control Systems
Computer Science, Artificial Intelligence
Computer Science
Gaussian processes
approximate inference
variational methods
Bayesian non-parameterics
kernel methods
EIGENVALUES
MATRIX
ERROR
Publication Status
Published
Article Number
131
Date Publish Online
2020-07-01