While the keypoint-based maps created by sparsemonocular Simultaneous Localisation and Mapping (SLAM)systems are useful for camera tracking, dense 3D recon-structions may be desired for many robotic tasks. Solutionsinvolving depth cameras are limited in range and to indoorspaces, and dense reconstruction systems based on minimisingthe photometric error between frames are typically poorlyconstrained and suffer from scale ambiguity. To address theseissues, we propose a 3D reconstruction system that leverages theoutput of a Convolutional Neural Network (CNN) to producefully dense depth maps for keyframes that include metric scale.Our system, DeepFusion, is capable of producing real-timedense reconstructions on a GPU. It fuses the output of a semi-dense multiview stereo algorithm with the depth and gradientpredictions of a CNN in a probabilistic fashion, using learneduncertainties produced by the network. While the network onlyneeds to be run once per keyframe, we are able to optimise forthe depth map with each new frame so as to constantly makeuse of new geometric constraints. Based on its performanceon synthetic and real world datasets, we demonstrate thatDeepFusion is capable of performing at least as well as othercomparable systems.