Selectivity of cortical neurons for sensory stimuli can increase across days as animals learn their behavioral relevance and across seconds when animals switch attention. While both phenomena occur in the same circuit, it is unknown whether they rely on similar mechanisms. We imaged primary visual cortex as mice learned a visual discrimination task and subsequently performed an attention switching task. Selectivity changes due to learning and attention were uncorrelated in individual neurons. Selectivity increases after learning mainly arose from selective suppression of responses to one of the stimuli but from selective enhancement and suppression during attention. Learning and attention differentially affected interactions between excitatory and PV, SOM, and VIP inhibitory cells. Circuit modeling revealed that cell class-specific top-down inputs best explained attentional modulation, while reorganization of local functional connectivity accounted for learning-related changes. Thus, distinct mechanisms underlie increased discriminability of relevant sensory stimuli across longer and shorter timescales.