Individual cells exhibit specific proliferative responses to changes in microenvironmental conditions. Whether such potential is constrained by the cell density throughout the growth process is however unclear. Here, we identify a theoretical framework that captures how the information encoded in the initial density of cancer cell populations impacts their growth profile. By following the growth of hundreds of populations of cancer cells, we found that the time they need to adapt to the environment decreases as the initial cell density increases. Moreover, the population growth rate shows a maximum at intermediate initial densities. With the support of a mathematical model, we show that the observed interdependence of adaptation time and growth rate is significantly at odds both with standard logistic growth models and with the Monod-like function that governs the dependence of the growth rate on nutrient levels. Our results (i) uncover and quantify a previously unnoticed heterogeneity in the growth dynamics of cancer cell populations; (ii) unveil how population growth may be affected by single-cell adaptation times; (iii) contribute to our understanding of the clinically-observed dependence of the primary and metastatic tumor take rates on the initial density of implanted cancer cells.