Huntington’s disease can be predicted many years before symptom onset, and thus makes an ideal model for studying the earliest mechanisms of neurodegeneration. Diffuse patterns of structural connectivity loss occur in the basal ganglia and cortex early in the disease. However, the organizational principles that underlie these changes are unclear. By understanding such principles we can gain insight into the link between the cellular pathology caused by mutant huntingtin and its downstream effect at the macroscopic level. The ‘rich club’ is a pattern of organization established in healthy human brains, where specific hub ‘rich club’ brain regions are more highly connected to each other than other brain regions. We hypothesized that selective loss of rich club connectivity might represent an organizing principle underlying the distributed pattern of structural connectivity loss seen in Huntington’s disease. To test this hypothesis we performed diffusion tractography and graph theoretical analysis in a pseudo-longitudinal study of 50 premanifest and 38 manifest Huntington’s disease participants compared with 47 healthy controls. Consistent with our hypothesis we found that structural connectivity loss selectively affected rich club brain regions in premanifest and manifest Huntington’s disease participants compared with controls. We found progressive network changes across controls, premanifest Huntington’s disease and manifest Huntington’s disease characterized by increased network segregation in the premanifest stage and loss of network integration in manifest disease. These regional and whole brain network differences were highly correlated with cognitive and motor deficits suggesting they have pathophysiological relevance. We also observed greater reductions in the connectivity of brain regions that have higher network traffic and lower clustering of neighbouring regions. This provides a potential mechanism that results in a characteristic pattern of structural connectivity loss targeting highly connected brain regions with high network traffic and low clustering of neighbouring regions. Our findings highlight the role of the rich club as a substrate for the structural connectivity loss seen in Huntington’s disease and have broader implications for understanding the connection between molecular and systems level pathology in neurodegenerative disease.