We formulate a general framework for building structural causal models (SCMs)
with deep learning components. The proposed approach employs normalising flows
and variational inference to enable tractable inference of exogenous noise
variables - a crucial step for counterfactual inference that is missing from
existing deep causal learning methods. Our framework is validated on a
synthetic dataset built on MNIST as well as on a real-world medical dataset of
brain MRI scans. Our experimental results indicate that we can successfully
train deep SCMs that are capable of all three levels of Pearl's ladder of
causation: association, intervention, and counterfactuals, giving rise to a
powerful new approach for answering causal questions in imaging applications
and beyond. The code for all our experiments is available at
https://github.com/biomedia-mira/deepscm.