Semiconducting polymers are versatile materials for solar energy conversion and have gained popularity as photocatalysts for sunlight-driven hydrogen production. Organic polymers often contain residual metal impurities such as palladium (Pd) clusters that are formed during the polymerization reaction, and there is increasing evidence for a catalytic role of such metal clusters in polymer photocatalysts. Using transient and operando optical spectroscopies on nanoparticles of F8BT, P3HT, and the dibenzo[b,d]thiophene sulfone homopolymer, P10, we demonstrate how differences in the timescale of electron transfer to Pd clusters translate into hydrogen evolution activity optima at extremely different residual Pd concentrations. For F8BT nanoparticles with common Pd concentrations of >1000 ppm (>0.1 wt. %), we find that residual Pd clusters quench photogenerated excitons via energy and electron transfer on the fs – ns timescale, thus outcompeting reductive quenching. We spectroscopically identify reduced Pd clusters in our F8BT nanoparticles from the µs timescale onwards and show that the predominant location of long-lived electrons gradually shifts to the F8BT polymer when the Pd content is lowered. While a low yield of long-lived electrons limits the hydrogen evolution activity of F8BT, P10 exhibits a substantially higher hydrogen evolution activity which we demonstrate results from higher yields of long-lived electrons due to more efficient reductive quenching. Surprisingly, and despite the higher performance of P10, long-lived electrons reside on the P10 polymer rather than on the Pd clusters in P10 particles, even at very high Pd concentrations of 27,000 ppm (2.7 wt. %). In contrast, long-lived electrons in F8BT already reside on Pd clusters before the typical timescale of hydrogen evolution. This comparison shows that P10 exhibits efficient reductive quenching but slow electron transfer to residual Pd clusters whereas the opposite is the case for F8BT, suggesting that the development of even more efficient polymer photocatalysts must target materials that combine both rapid reductive quenching and rapid charge transfer to a metal-based co-catalyst.