Cross-reactive immune responses to SARS-CoV-2 have been observed in pre-pandemic cohorts and proposed to contribute to host protection. Here we assess 52 COVID-19 household contacts to capture immune responses at the earliest timepoints after SARS-CoV-2 exposure. Using a dual cytokine FLISpot assay on peripheral blood mononuclear cells, we enumerate the frequency of T cells specific for spike, nucleocapsid, membrane, envelope and ORF1 SARS-CoV-2 epitopes that cross-react with human endemic coronaviruses. We observe higher frequencies of cross-reactive (p=0.0139), and nucleocapsid-specific (p=0.0355) IL-2-secreting memory T cells in contacts who remained PCR-negative despite exposure (n=26), when compared with those who convert to PCR-positive (n=26); no significant difference in the frequency of responses to spike is observed, hinting at a limited protective function of spike-cross-reactive T cells. Our results are thus consistent with pre-existing non-spike cross-reactive memory T cells protecting SARS-CoV-2-naïve contacts from infection, thereby supporting the inclusion of non-spike antigens in second-generation vaccines.