In-depth understanding of the recrystallization process in alloys is critical for generating desirable small grains and weak textured microstructure, which provides high strength and toughness for metal formed parts. The manufacturing industry has a high demand for a valid computational model to accurately predict the level of recrystallization and recrystallized grain size under different strain paths and temperatures. However, current understanding and numerical calculation have not been linked properly for a reliable, physically based model to simulate the deformation and annealing process. Our phase-field model coupled with crystal plasticity simulations, which is based on the theory of stored energy minimization, enables a reliable prediction on the microstructure evolution under different processing routes. We hope that this modelling work provides a solution for the prediction of some long standing microstructure formation problems.