Gravitational potentials that change in time induce fluctuations in the observed cosmic microwave background (CMB) temperature. Cosmological structure moving transverse to our line of sight provides a specific example known as the moving lens effect. Here, we explore how the observed CMB temperature fluctuations, combined with the observed matter overdensity, can be used to infer the transverse velocity of cosmological structures on large scales. We show that near-future CMB surveys and galaxy surveys will have the statistical power to make a first detection of the moving lens effect, and we discuss applications for the reconstructed transverse velocity.