Set-point viral load (SPVL), a common measure of human immunodeficiency virus (HIV)-1 virulence, is partially determined by viral genotype. Epidemiological evidence suggests that this viral property has been under stabilising selection, with a typical optimum for the virus between 104 and 105 copies of viral RNA per ml. Here we aimed to detect transmission fitness differences between viruses from individuals with different SPVLs directly from phylogenetic trees inferred from whole-genome sequences. We used the local branching index (LBI) as a proxy for transmission fitness. We found that LBI is more sensitive to differences in infectiousness than to differences in the duration of the infectious state. By analysing subtype-B samples from the Bridging the Evolution and Epidemiology of HIV in Europe project, we inferred a significant positive relationship between SPVL and LBI up to approximately 105 copies/ml, with some evidence for a peak around this value of SPVL. This is evidence of selection against low values of SPVL in HIV-1 subtype-B strains, likely related to lower infectiousness, and perhaps a peak in the transmission fitness in the expected range of SPVL. The less prominent signatures of selection against higher SPVL could be explained by an inherent limit of the method or the deployment of antiretroviral therapy.