The impact of initial severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infecting strain on downstream immunity to heterologous variants of concern (VOC) is unknown. Studying a longitudinal healthcare worker cohort, we found that after three antigen exposures (infection+two vaccine doses), S1 antibody, memory B cells and heterologous neutralization of B.1.351, P.1 and B.1.617.2 plateaued, while B.1.1.7 neutralization and spike T cell responses increased. Serology using Wuhan Hu-1 spike receptor binding domain poorly predicted neutralizing immunity against VOCs. Neutralization potency against VOCs changed with heterologous virus encounter and number of antigen exposures. Neutralization potency fell differentially depending on targeted VOCs over 5-months from the second vaccine dose. Heterologous combinations of spike encountered during infection and vaccination shape subsequent cross-protection against VOC, with implications for future-proof next-generation vaccines.