Cardiac magnetic resonance (CMR) is regarded as the reference examination for cardiac morphology in tetralogy of Fallot (ToF) patients allowing images of high spatial resolution and high contrast. The detailed knowledge of the right ventricular anatomy is critical in ToF management. The segmentation of the right ventricle (RV) in CMR images from ToF patients is a challenging task due to the high shape and image quality variability. In this paper we propose a fully automatic deep learning-based framework to segment the RV from CMR anatomical images of the whole heart. We adopt a 3D multi-scale deep convolutional neural network to identify pixels that belong to the RV. Our robust segmentation framework was tested on 26 ToF patients achieving a Dice similarity coefficient of 0.8281±0.1010 with reference to manual annotations performed by expert cardiologists. The proposed technique is also computationally efficient, which may further facilitate its adoption in the clinical routine.