In design for forming, it is becoming increasingly significant to develop surrogate models of high-fidelity finite element analysis (FEA) simulations of forming processes, to achieve effective component feasibility assessment as well as process and component optimizations. However, surrogate models using traditional scalar-based machine learning methods (SBMLMs) fall short on accuracy and generalizability. This is because SBMLMs fail to harness the location information available from the simulations. To overcome this shortcoming, the theoretical feasibility and practical advantages of innovatively applying image-based machine learning methods (IBMLMs) in developing surrogate models of sheet stamp forming simulations are explored in this study. To demonstrate the advantages of IBMLMs, the effect of the location information on both design variables and simulated physical fields is firstly proposed and analyzed. Based on a sheet steel stamping case study, a Res-SE-U-Net IBMLM surrogate model of stamping simulations is then developed and compared with a baseline multi-layer perceptron (MLP) SBMLM surrogate model. The results show that the IBMLM model is advantageous over the MLP SBMLM model in accuracy, generalizability, robustness, and informativeness. This paper presents a promising methodology in leveraging IBMLMs as surrogate models to make maximum use of information from stamp forming FEA results. Future prospective studies that are inspired by this paper are also discussed.