Aligning pedestal models and associated magnetohydrodynamic codes with experimental data is an important challenge in order to be able to generate predictions for future devices, e.g., ITER. Previous efforts to perform calibration of unknown model parameters have largely been a manual process. In this paper, we construct a framework for the automatic calibration of JOREK. More formally, we reformulate the calibration problem into a black-box optimization task, by defining a measure of the discrepancy between an experiment and a reference quantity. As this discrepancy relies on JOREK simulations, the objective becomes computationally intensive and, hence, we resort to batch Bayesian optimization methodology to allow for efficient, gradient-free optimization. We apply this methodology to two different test cases with different discrepancies and show that the calibration is achievable.