By tuning the tolerance factor, $t$, of the Ruddlesden--Popper oxide Ca$_2$MnO$_4$ through isovalent substitutions we show that the uniaxial coefficient of linear thermal expansion (CLTE) of these systems can be systematically changed through large negative to positive values. High-resolution X-ray diffraction measurements show that the magnitude of uniaxial negative thermal expansion (NTE) increases as $t$ decreases across the stability window of the NTE phase. Transitions to phases with positive thermal expansion (PTE) are found to occur at both the high-$t$ and low-$t$ limits of stability. First-principles calculations demonstrate that reducing $t$ enhances the contribution to thermal expansion from the lowest frequency phonons, which have the character of octahedral tilts and have negative mode Gr\"uneisen parameter components along the NTE axis. By tuning $t$ to the lower edge of the NTE phase stability window, we are hence able to maximise the amplitudes of these vibrations and thereby maximise NTE with a CLTE of -8.1~ppm/K at 125~K. We also illustrate, at the other end of the phase diagram, that an enhancement in compliance of these materials associated with the rotational instability provides another mechanism by which NTE could be yet further enhanced in this and related systems.