The rapid relaxation of above-band-gap “hot” carriers (HCs) imposes the key efficiency limit in lead-halide perovskite (LHP) solar cells. Recent studies have indicated that HC cooling in these systems may be sensitive to materials composition, as well as the energy and density of excited states. However, the key parameters underpinning the cooling mechanism are currently under debate. Here we use a sequence of ultrafast optical pulses (visible pump–infrared push–infrared probe) to directly compare the intraband cooling dynamics in five common LHPs: FAPbI3, FAPbBr3, MAPbI3, MAPbBr3, and CsPbBr3. We observe ∼100–900 fs cooling times, with slower cooling at higher HC densities. This effect is strongest in the all-inorganic Cs-based system, compared to the hybrid analogues with organic cations. These observations, together with band structure calculations, allow us to quantify the origin of the “hot-phonon bottleneck” in LHPs and assert the thermodynamic contribution of a symmetry-breaking organic cation toward rapid HC cooling.