Cerebral small vessel disease (cSVD) is a leading cause of stroke and dementia. Genetic risk loci for white matter hyperintensities (WMH), the most common MRI-marker of cSVD in older age, were recently shown to be significantly associated with white matter (WM) microstructure on diffusion tensor imaging (signal-based) in young adults. To provide new insights into these early changes in WM microstructure and their relation with cSVD, we sought to explore the genetic underpinnings of cutting-edge tissue-based diffusion imaging markers across the adult lifespan. We conducted a genome-wide association study of neurite orientation dispersion and density imaging (NODDI) markers in young adults (i-Share study: N = 1 758, (mean[range]) 22.1[18-35] years), with follow-up in young middle-aged (Rhineland Study: N = 714, 35.2[30-40] years) and late middle-aged to older individuals (UK Biobank: N = 33 224, 64.3[45-82] years). We identified 21 loci associated with NODDI markers across brain regions in young adults. The most robust association, replicated in both follow-up cohorts, was with Neurite Density Index (NDI) at chr5q14.3, a known WMH locus in VCAN. Two additional loci were replicated in UK Biobank, at chr17q21.2 with NDI, and chr19q13.12 with Orientation Dispersion Index (ODI). Transcriptome-wide association studies showed associations of STAT3 expression in arterial and adipose tissue (chr17q21.2) with NDI, and of several genes at chr19q13.12 with ODI. Genetic susceptibility to larger WMH volume, but not to vascular risk factors, was significantly associated with decreased NDI in young adults, especially in regions known to harbor WMH in older age. Individually, seven of 25 known WMH risk loci were associated with NDI in young adults. In conclusion, we identified multiple novel genetic risk loci associated with NODDI markers, particularly NDI, in early adulthood. These point to possible early-life mechanisms underlying cSVD and to processes involving remyelination, neurodevelopment and neurodegeneration, with a potential for novel approaches to prevention.