Zirconia (zirconium dioxide) and hafnia (hafnium dioxide) are binary oxides used in a range of applications. Because zirconium and hafnium are chemically equivalent, they have three similar polymorphs, and it is important to understand the properties and energetics of these polymorphs. However, while density functional theory calculations can get the correct energetic ordering, the energy differences between polymorphs depend very much on the specific density functional theory approach, as do other quantities such as lattice constants and bulk modulus. We have used highly accurate quantum Monte Carlo simulations to model the three zirconia and hafnia polymorphs. We compare our results for structural parameters, bulk modulus, and cohesive energy with results obtained from density functional theory calculations. We also discuss comparisons of our results with existing experimental data, in particular for structural parameters where extrapolation to zero temperature can be attempted. We hope our results of structural parameters as well as for cohesive energy and bulk modulus can serve as benchmarks for density-functional theory based calculations and as a guidance for future experiments.