Accurate delineation of heart substructures is a prerequisite for abnormality detection, for making quantitative and functional measurements, and for computer-aided diagnosis and treatment planning. Late Gadolinium-Enhanced Cardiac MRI (LGE-CMRI) is an emerging imaging technology for myocardial infarction or scar detection based on the differences in the volume of residual gadolinium distribution between scar and healthy tissues. While LGE-CMRI is a well-established non-invasive tool for detecting myocardial scar tissues in the ventricles, its application to left atrium (LA) imaging is more challenging due to its very thin wall of the LA and poor quality images, which may be produced because of motion artefacts and low signal-to-noise ratio. As the LGE-CMRI scan is designed to highlight scar tissues by altering the gadolinium kinetics, the anatomy among different heart substructures has less distinguishable boundaries. An accurate, robust and reproducible method for LA segmentation is highly in demand because it can not only provide valuable information of the heart function but also be helpful for the further delineation of scar tissue and measuring the scar percentage. In this study, we proposed a novel deep learning framework working on LGE-CMRI images directly by combining sequential learning and dilated residual learning to delineate LA and pulmonary veins fully automatically. The achieved results showed accurate segmentation results compared to the state-of-the-art methods. The proposed framework leads to an automatic generation of a patient-specific model that can potentially enable an objective atrial scarring assessment for the atrial fibrillation patients.