Conservation laws as inductive biases
File(s)
Author(s)
Saemundsson, Steindor
Type
Thesis
Abstract
A basic pattern in nature is invariance: the notion that properties (e.g. energy) of a system remain unchanged through a transformation (e.g. time). However, learning such patterns from data can be challenging since they are often non-trivially disguised as variation in observed phenomena. The motivation for the work in the thesis is improved data efficiency when learning predictive models of physical dynamical systems. Building on ideas from machine learning and physics, it explores learning algorithms using conserved quantities and conservation laws as general purpose model components, with the aim of more efficient learning. Chapter 2 develops learning algorithms for task structured problems where the notion of a task is identified with an unobserved conserved quantity to be learned from data. The main contribution is a model that accounts for globally invariant sources of variation (e.g. the laws of physics) and task-specific sources of variation (e.g. system parameters). The idea is to encourage modularity: a separation of reusable components of the model from task-specific ones. The chapter empirically studies the model in the context of learning predictive models of dynamical systems. It is found to be useful as an inductive bias for modularity, as measured by data efficiency in multi-task, transfer- and meta-learning settings. Chapter 3 develops expressive function classes with inbuilt physical geometry such as conservation laws. The main contribution is a tying together of the theory of variational integrators and neural networks. This produces a scheme for deriving symplectic and momentum-preserving architectures (variational integrator networks). The architectures are studied empirically in the context of noisy, and image observations, of physical systems. In the former, they are found to be efficient and flexible learners. In the latter, they are found to learn physically meaningful geometric representations, enabling accurate long-term forecasts in image space. Learning modular task representations is potentially important for developing practically useful meta-learning algorithms. In chapter 2 the representations are non-hierarchical and require labelling at the task-level. Extending the idea to hierarchical and unsupervised settings is an interesting future direction. Physical geometry is an elegant example of compact general-purpose representations. Extending chapter 3 to different and more general physics, building on the literature on variational integrators, is also an interesting direction.
Version
Open Access
Date Issued
2021-08
Date Awarded
2022-03
Copyright Statement
Creative Commons Attribution NonCommercial Licence
License URL
Advisor
Deisenroth, Marc
Publisher Department
Computing
Publisher Institution
Imperial College London
Qualification Level
Doctoral
Qualification Name
Doctor of Philosophy (PhD)