This paper uses a multi-scale crystal plasticity modelling approach to investigate the role of microstructure in the damage of Sn-3Ag-0.5Cu (wt%, SAC305) solder joints subject to thermal cycling. Faithful microstructure modelling has been developed to explicitly represent the complex microstructure characterized in the experiments. The mechanisms for the experimental finding that single crystal joints systematically outperform interlaced and beachball joints, in terms of damage development during thermal cycling, are found to be the localization of stored energy due to the grain boundaries. Interlaced regions, however, benefit the lifetime of solder joints versus a beachball microstructure by virtue of energy diffusion. The modelling results, together with experimental characterization, provide practical suggestions on how the microstructure could be designed to optimize the in-service lifetime of solder joints subject to thermal fatigue.