A chief aim in singlet fission research is to develop new materials concepts for more efficient singlet fission. The typical approaches such as tuning π-overlap and charge-transfer interactions, enhancing delocalization, altering diradical character, or extending the conjugation length have profound effects simultaneously on the singlet and triplet energetics and the couplings between them. While these strategies have resulted in a handful of high-efficiency materials, the complex interplay of these factors makes systematic materials development challenging, and it would be useful to be able to selectively manipulate the properties and dynamics of just part of the singlet fission pathway. Here, we investigate the potential of heteroatom substitution as just such a selective tool. We explore the influence of heavy atoms within the main backbone of polythienylenevinylene and its selenophene and tellurophene derivatives. We find no significant effects on the prompt <300 fs intramolecular singlet fission dynamics but a clear heavy-atom effect on longer time scales.