Minimizing the energy consumed on heating, ventilation and air conditioning (HVAC) systems of residential buildings, without impacting occupants? comfort has been highlighted as an important artificial intelligence (AI) challenge. Typically, approaches that seek to address this challenge use a model that captures the thermal dynamics within a building, also referred to as a thermal model. In this paper, we introduce a novel thermal model, which we refer to as a latent force thermal model of the thermal dynamics of a building or LFM-TM. Our model is derived from an existing grey-box thermal model, which is augmented with an extra term referred to as the learned residual. This term is capable of modelling the effect of any a priori unknown additional dynamic, which if not captured, appears as structure in a thermal models residual (the error induced by the model). More importantly, the learned residual can also capture the effects of physical elements such as a building?s envelope or the lags in a heating system, leading to a significant reduction in complexity compared to existing models. We evaluate the performance of LFM-TM on two independent data sources: the FlexHouse data, which was previously used for evaluating the efficacy of existing grey-box models [Bacher and Madsen 2011], and heating data logged within homes located on University of Southampton campus. On both datasets, we show that our approach outperforms existing models in its ability to accurately fit the observed data, generate accurate day-ahead internal temperature predictions and explain a large amount of the variability in the future observations.