Machine learning has evolved from more traditional statistical approaches and is able to tackle problems that were previously impossible to solve. However, we lack a proper understanding of why and how these new highly expressive models work, which leads to difficulties in applying them as effectively as could be done. In this thesis, I show that properly modelling the constraints inherent in a particular application domain can lead to better performance with a terser, simpler model (chapter 2), and that even a complicated model cannot learn when the data input into the model is uninformative (chapter 3). Based on this work, I build a case to generalise methods for designing models and drawing inferences by applying a constructivist approach grounded in category theory and functional programming.