The variation in the radiative output of the Sun, described in terms of solar irradiance, is important to climatology. A common assumption is that solar irradiance variability is driven by its surface magnetism. Verifying this assumption has, however, been hampered by the fact that models of solar irradiance variability based on solar surface magnetism have to be calibrated to observed variability. Making use of realistic three-dimensional magnetohydrodynamic simulations of the solar atmosphere and state-of-the-art solar magnetograms from the Solar Dynamics Observatory, we present a model of total solar irradiance (TSI) that does not require any such calibration. In doing so, the modeled irradiance variability is entirely independent of the observational record. (The absolute level is calibrated to the TSI record from the Total Irradiance Monitor.) The model replicates 95% of the observed variability between April 2010 and July 2016, leaving little scope for alternative drivers of solar irradiance variability at least over the time scales examined (days to years).