Partial differential equations (PDEs) are central to describing and modelling complex physical systems that arise in many disciplines across science and engineering. However, in many realistic applications PDE modelling provides an incomplete description of the physics of interest. Machine learning techniques have become increasingly popular in filling the knowledge gap between the fundamental physical laws, expressed as differential equations, and the real-world phenomena studied by scientists and engineers. The emergence of this approach urges the need for scientific simulation frameworks that allow for the efficient development and deployment of models coupling PDEs and ML. In this thesis, we employ a differentiable programming approach to build a highly efficient and composable interface that provides researchers, engineers, and domain experts with diverse backgrounds with a highly productive way to run high-performance simulations coupling PDEs, implemented using the finite element method (FEM) in Firedrake, and machine learning models, specified in PyTorch. The resulting framework maintains separation of concerns while only requiring trivial changes to existing code. This work paves the way for building models that harness the capabilities of machine learning in the context of PDE-based systems that arise in various scientific fields, spanning from geoscience to structural mechanics to quantum mechanics.