Relationships between viral load, severity of illness, and transmissibility of virus, are fundamental to understanding pathogenesis and devising better therapeutic and prevention strategies for COVID-19. Here we present within-host modelling of viral load dynamics observed in the upper respiratory tract (URT), drawing upon 2172 serial measurements from 605 subjects, collected from 17 different studies. We developed a mechanistic model to describe viral load dynamics and host response, and contrast this with simpler mixed-effects regression analysis of peak viral load and its subsequent decline. We observed wide variation in URT viral load between individuals, over 5 orders of magnitude, at any given point in time since symptom onset. This variation was not explained by age, sex, or severity of illness, and these variables were not associated with the modelled early or late phases of immune-mediated control of viral load. We explored the application of the mechanistic model to identify measured immune responses associated with control of viral load. Neutralizing antibody correlated strongly with modelled immune-mediated control of viral load amongst subjects who produced neutralizing antibody. Our models can be used to identify host and viral factors which control URT viral load dynamics, informing future treatment and transmission blocking interventions.