mage registration with deep neural networks has become anactive field of research and exciting avenue for a long standing problem inmedical imaging. The goal is to learn a complex function that maps theappearance of input image pairs to parameters of a spatial transforma-tion in order to align corresponding anatomical structures. We argue andshow that the current direct, non-iterative approaches are sub-optimal,in particular if we seek accurate alignment of Structures-of-Interest (SoI).Information about SoI is often available at training time, for example,in form of segmentations or landmarks. We introduce a novel, genericframework, Image-and-Spatial Transformer Networks (ISTNs), to lever-age SoI information allowing us to learn new image representations thatare optimised for the downstream registration task. Thanks to these rep-resentations we can employ a test-specific, iterative refinement over thetransformation parameters which yields highly accurate registration evenwith very limited training data. Performance is demonstrated on pairwise3D brain registration and illustrative synthetic data.