A stored energy based methodology for calculating the driving force for crack growth is introduced which can capture the highly local microstructural sensitivity. This has been implemented in the context of crystal plasticity finite element simulations with explicit representation of the crack with the eXtended Finite Element Method (XFEM), with non-local approaches for both stored energy and J-integral calculation. The model is shown to have good agreement with discrete dislocation plasticity (DDP) models in terms of the crack-tip dislocation configurational energy, and with experimental observations of long and very short (microstructurally-sensitive) cracks for both fracture toughness and crack growth rate data. The method is shown to capture the microstructural sensitivity, in contrast with the widely used J-Integral method. By modelling different crack lengths, the diminution of the microstructural sensitivity with increasing crack length is quantified and a critical length defined above which the microstructural sensitivity is insignificant.