To understand the limitations placed on the open-circuit voltage of bulk heterojunction (BHJ) organic solar cells, the energy levels of neat donor and acceptor samples are often characterized and applied to study BHJ blends. However, energy levels derived from neat samples may not necessarily reflect those at the donor:acceptor interface in blends. The properties of organic semiconductors are sensitive to microstructural changes, with non-fullerene acceptors (NFAs) in particular known to exhibit different thin-film polymorphs. To investigate the influence of differences in molecular packing in neat and blend films, temperature-dependent current–voltage characteristics are measured for bilayer (BL) and BHJ devices. Herein, the fullerene acceptor PC71BM is compared—whose energy levels are expected to be less sensitive to molecular packing—with the NFA ITIC, paired with the same donor polymer PTB7-Th. It is found that the interfacial energy levels differ for BL and BHJ devices for the PTB7-Th:ITIC system but remain the same for the PTB7-Th:PC71BM system. Furthermore, X-ray scattering measurements identify that ITIC exhibits a different packing mode in neat films and in BHJ blends. Such microstructure-dependent differences between neat and blend samples need to be considered when studying energy losses in NFA BHJ solar cells.