Magnetic resonance (MR) imaging studies have demonstrated reduced global and regional brain volumes in infants with congenital heart disease (CHD). This study aimed to provide a more detailed evaluation of altered structural brain development in newborn infants with CHD compared to healthy controls using tensor-based morphometry (TBM). We compared brain development in 64 infants with CHD to 192 age- and sex-matched healthy controls. T2-weighted MR images obtained prior to surgery were analysed to compare voxel-wise differences in structure across the whole brain between groups. Cerebral oxygen delivery (CDO2) was measured in infants with CHD (n = 49) using phase contrast MR imaging and the relationship between CDO2 and voxel-wise brain structure was assessed using TBM. After correcting for global scaling differences, clusters of significant volume reduction in infants with CHD were demonstrated bilaterally within the basal ganglia, thalami, corpus callosum, occipital, temporal, parietal and frontal lobes, and right hippocampus (p < 0.025 after family-wise error correction). Clusters of significant volume expansion in infants with CHD were identified in cerebrospinal fluid spaces (p < 0.025). After correcting for global brain size, there was no significant association between voxel-wise brain structure and CDO2. This study localizes abnormal brain development in infants with CHD, identifying areas of particular vulnerability.