The performance of photovoltaic devices based on blends of conjugated polymers with nonfullerene acceptors depends on the phase behavior and microstructure of the binary, which in turn depends on the chemical structures of the molecular components and the blend composition. We investigate the correlation between the molecular structure, composition, phase behavior, and device performance of a model system consisting of semicrystalline poly-3-hexylthiophene (P3HT) as the donor polymer and three nonfullerene acceptors, two of which (O-IDTBR/EH-IDTBR) have a planar core with different side chains and one (O-IDFBR) of which has a twisted core. We combine differential scanning calorimetry with optical measurements including UV–Vis spectroscopy, photoluminescence, spectroscopic ellipsometry, and Raman spectroscopy and photovoltaic device performance measurements, all at varying blend composition. For P3HT:IDTBR blends, the crystallinity of polymer and acceptor is preserved over a wide composition range and the blend displays a eutectic phase behavior, with the optimum solar cell composition lying close to the eutectic composition. For P3HT:IDFBR blends, increasing acceptor content disrupts the polymer crystallinity, and the optimum device composition appears to be limited by polymer connectivity rather than being linked to the eutectic composition. The optical probes allow us to probe both the crystalline and amorphous phases, clearly revealing the compositions at which component mixing disrupts crystallinity.