The formation of different types of structural polymorphs of poly(3-hexyl-thiophene) (P3HT) affects the performance of organic photovoltaic (OPV) devices that use thermally-annealed P3HT:PCBM[60] blend films as photoactive layer. Here it is demonstrated that, when densely-packed and non-densely packed P3HT polymorphs co-exist in the P3HT:PCBM[60] layer, non-geminate charge recombination is fast; however, in a device non-geminate recombination is effectively overruled by efficient and fast charge carrier extraction. In stark contrast, when only a less-densely packed P3HT polymorph is present in the blend, non-geminate charge recombination losses are less pronounced, and the charge carrier extraction efficiency is lower. The antagonistic non-geminate charge recombination and charge carrier extraction processes in these systems are monitored by time-delayed-collection field (TDCF) and ultrafast transient absorption (TA) experiments. Furthermore, resonance Raman spectroscopy reveals that in the absence of the densely-packed P3HT polymorph, the energetic disorder present in the P3HT:PCBM[60] blend is higher. High-resolution atomic force microscopy imaging further identifies pronounced differences in the layer morphology when the polymorph distribution varies between unimodal and bimodal. These results indicate that less-densely packed P3HT polymorphs increase disorder and impede charge collection, leading to a reduction of the device fill factor.