The layered semiconductor SnSe is one of the highest-performing thermoelectric materials known. We demonstrate, through a first-principles lattice-dynamics study, that the high-temperature Cmcm phase is a dynamic average over lower-symmetry minima separated by very small energetic barriers. Compared to the low-temperature Pnma phase, the Cmcm phase displays a phonon softening and enhanced three-phonon scattering, leading to an anharmonic damping of the low-frequency modes and hence the thermal transport. We develop a renormalization scheme to quantify the effect of the soft modes on the calculated properties, and confirm that the anharmonicity is an inherent feature of the Cmcm phase. These results suggest a design concept for thermal insulators and thermoelectric materials, based on displacive instabilities, and highlight the power of lattice-dynamics calculations for materials characterization.