The introduction of nonfullerene acceptors (NFA) facilitated the realization of high-efficiency organic solar cells (OSCs); however, OSCs suffer from relatively large losses in open-circuit voltage (VOC) as compared to inorganic or perovskite solar cells. Further enhancement in power conversion efficiency requires an increase in VOC. In this work, we take advantage of the high dipole moment of twisted perylene-diimide (TPDI) as a nonfullerene acceptor (NFA) to enhance the VOC of OSCs. In multiple bulk heterojunction solar cells incorporating TPDI with three polymer donors (PTB7-Th, PM6 and PBDB-T), we observed a VOC enhancement by modifying the cathode with a polyethylenimine (PEIE) interlayer. We show that the dipolar interaction between the TPDI NFA and PEIE─enhanced by the general tendency of TPDI to form J-aggregates─plays a crucial role in reducing nonradiative voltage losses under a constant radiative limit of VOC. This is aided by comparative studies with PM6:Y6 bulk heterojunction solar cells. We hypothesize that incorporating NFAs with significant dipole moments is a feasible approach to improving the VOC of OSCs.