Scattered neutron spectroscopy is a diagnostic technique commonly used to measure areal density in inertial confinement fusion experiments. Deleterious areal density asymmetries modify the shape of the scattered neutron spectrum. In this work, a novel analysis is developed, which can be used to fit the shape change. This will allow experimental scattered neutron spectroscopy to directly infer the amplitude and mode of the areal density asymmetries, with little sensitivity to confounding factors that affect other diagnostics for areal density. The model is tested on spectra produced by a neutron transport calculation with both isotropic and anisotropic primary fusion neutron sources. Multiple lines of sight are required to infer the areal density distribution over the whole sphere—we investigate the error propagation and optimal detector arrangement associated with the inference of mode 1 asymmetries.