Fe contamination is a serious composition barrier for Al recycling. In Fe-containing Al-Si-Cu alloy, a brittle and plate-shaped β phase forms, degrading the mechanical properties. Here, 4D (3D plus time) synchrotron X-ray tomography was used to observe the directional solidification of Fe-containing Al-Si-Cu alloy. The quantification of the coupled growth of the primary and β phase via machine learning and particle tracking, demonstrates that the final size of the β intermetallics were strongly influenced by the solute segregation and space available for growth whereas the β orientation was controlled by the temperature gradient direction. The work can be used to validate predictive models.