A leading explanation for translational failure in neurodegenerative disease is that new drugs are evaluated late in the disease course when clinical features have become irreversible. Here, to address this gap, we cognitively profiled 21,051 people aged 17-85 years as part of the Genes and Cognition cohort within the National Institute for Health and Care Research BioResource across England. We describe the cohort, present cognitive trajectories and show the potential utility. Surprisingly, when studied at scale, the APOE genotype had negligible impact on cognitive performance. Different cognitive domains had distinct genetic architectures, with one indicating brain region-specific activation of microglia and another with glycogen metabolism. Thus, the molecular and cellular mechanisms underpinning cognition are distinct from dementia risk loci, presenting different targets to slow down age-related cognitive decline. Participants can now be recalled stratified by genotype and cognitive phenotype for natural history and interventional studies of neurodegenerative and other disorders.