Rates of fatigue crack growth in Zircaloy-4 are highly microstructurally sensitive and dependent upon texture. Crystal plasticity finite element modelling with the eXtended Finite Element Method and a stored energy density fracture criterion are used to simulate crack propagation in single crystals and polycrystalline microstructures for comparison with experimental crack growth rate data. Results demonstrate that growth rate fluctuations at microstructural features are driven primarily by elastic anisotropy and yield stress mismatch. Additionally, reduced rate of growth in soft hexagonal close packed grains (relative to the remote loading direction) can be linked with crack path tortuosity, which is shown to be controlled by the cyclic development of an in-plane shear back stress. Most importantly, stored energy density is shown to accurately capture major microstructure-driven differences in crack growth rate. Comparison of simulated fatigue crack growth rates with experimental data enables estimation of the critical stored energy density for crack propagation in Zircaloy-4 to be 250 J/m2.