In this work we study the scattering and transfer matrices for electric
fields defined with respect to an angular spectrum of plane waves. For these
matrices, we derive the constraints that are enforced by conservation of
energy, reciprocity and time reversal symmetry. Notably, we examine the general
case of vector fields in three dimensions and allow for evanescent field
components. Moreover, we consider fields described by both continuous and
discrete angular spectra, the latter being more relevant to practical
applications, such as optical scattering experiments. We compare our results to
better-known constraints, such as the unitarity of the scattering matrix for
far-field modes, and show that previous results follow from our framework as
special cases. Finally, we demonstrate our results numerically with a simple
example of wave propagation at a planar glass-air interface, including the
effects of total internal reflection. Our formalism makes minimal assumptions
about the nature of the scattering medium and is thus applicable to a wide
range of scattering problems.