Efficient sunlight-to-energy conversion requires materials that can generate long-lived charge carriers upon illumination. However, the targeted design of semiconductors possessing intrinsically long lifetimes remains a key challenge. Here using a series of transition metal oxides, we establish a link between carrier lifetime and electronic configuration in transition metal-based semiconductors. We identify a subpicosecond relaxation mechanism via metal-centred ligand field states that compromise quantum yields in open d-shell transition metal oxides (for example, Fe2O3, Co3O4, Cr2O3 and NiO), which is more reminiscent of molecular complexes than crystalline semiconductors. We found that materials with spin-forbidden ligand field transitions could partially mitigate this relaxation pathway, explaining why Fe2O3 achieves higher photoelectrochemical activity than other visible light-absorbing transition metal oxides. However, achieving high yields of long-lived charges requires transition metal oxides with d0 or d10 electronic configurations (for example, TiO2 and BiVO4), where ligand field states are absent. These trends translate to transition metal-containing semiconductors beyond oxides, enabling the design of photoabsorbers with better-controlled recombination channels in photovoltaics, photocatalysis and communication devices.