A search for nonresonant production of Higgs boson pairs via gluon-gluon and vector boson fusion processes in final states with two bottom quarks and two photons is presented. The search uses data from proton-proton collisions at a center-of-mass energy of s√ = 13 TeV recorded with the CMS detector at the LHC, corresponding to an integrated luminosity of 137 fb−1. No significant deviation from the background-only hypothesis is observed. An upper limit at 95% confidence level is set on the product of the Higgs boson pair production cross section and branching fraction into γγbb¯¯¯. The observed (expected) upper limit is determined to be 0.67 (0.45) fb, which corresponds to 7.7 (5.2) times the standard model prediction. This search has the highest sensitivity to Higgs boson pair production to date. Assuming all other Higgs boson couplings are equal to their values in the standard model, the observed coupling modifiers of the trilinear Higgs boson self-coupling κλ and the coupling between a pair of Higgs bosons and a pair of vector bosons c2V are constrained within the ranges −3.3 < κλ < 8.5 and −1.3 < c2V < 3.5 at 95% confidence level. Constraints on κλ are also set by combining this analysis with a search for single Higgs bosons decaying to two photons, produced in association with top quark-antiquark pairs, and by performing a simultaneous fit of κλ and the top quark Yukawa coupling modifier κt.