The energetic offset between the initial photoexcited state and charge-transfer (CT) state in organic heterojunction solar cells influences both charge generation and open-circuit voltage (Voc). Here, we use time-resolved spectroscopy and voltage loss measurements to analyze the effect of the exciton–CT state offset on charge transfer, separation, and recombination processes in blends of a low-band-gap polymer (INDT-S) with fullerene derivatives of different electron affinity (PCBM and KL). For the lower exciton–CT state offset blend (INDT-S:PCBM), both photocurrent generation and nonradiative voltage losses are lower. The INDT-S:PCBM blend shows different excited-state dynamics depending on whether the donor or acceptor is photoexcited. Surprisingly, the charge recombination dynamics in INDT-S:PCBM are distinctly faster than those in INDT-S:KL upon excitation of the donor. We reconcile these observations using a kinetic model and by considering hybridization between the lowest excitonic and CT states. The modeling results show that this hybridization can significantly reduce Voc losses while still allowing reasonable charge generation efficiency.