Microstructure and ductile damage have a significant influence on the deformation behavior of high-speed railway axles during hot cross wedge rolling (CWR) and its final performance. In this study, based on the continuum damage mechanics, a multiaxial constitutive model coupling microstructure and ductile damage was established to predict the evolution of microstructure and ductile damage of 25CrMo4 during hot CWR processes. Material constants within the multiaxial constitutive model were determined by Genetic Algorithm (GA) optimization techniques from thermo-mechanical test data. The derived multiaxial constitutive model was embedded into the DEFORM-3D software through a user subroutine. FE simulation of CWR was performed to predict the microstructure evolution and ductile damage. CWR experiments were also carried out to validate the proposed model. The predicted grain size and ductile damage agree well with the experimental results. Good agreements indicate that the derived multiaxial constitutive model is reliable and can be used to predict the evolution of microstructure and ductile damage during CWR process.