Our understanding of material properties in the broadest sense is based on our ability to observe and disentangle underlying mechanisms. This has been aided enormously by the discovery and exploitation of synchrotron radiation. The next generation of light sources will be based on free electron lasers with potentially much greater light intensity and time resolution. This requires the development of new photocathode materials with high quantum efficiency (QE) and low emittance that are chemically and mechanically robust. One prospect is the use of yttrium (Y) and/or magnesium (Mg) thin films, but here, a fundamental understanding of the photoemission process from realistic materials is lacking. Observations of photoemissive performance would appear to contradict simple models. It is well known that materials with a lower work function are expected to facilitate photoemission, but the measured QE of Mg is higher than that of Y despite its nominal work function (3.7 eV) being significantly higher than that of Y (3.1 eV). In this work, these apparently anomalous observations are explained and rationalized by combining a simple three-step model of photoemission with large scale density functional theory calculations to predict the QE for realistic models of both materials in a special chemical environment. This approach allows us to identify the material parameters that govern the efficiency of the photoemission process. A detailed comparison with the experimental data suggests that, in this case, hydride formation on the Y surface, invisible to most experimental probes, nevertheless has a surprisingly large influence and reduces the photoemission significantly.