We study a new object that can be attached to an abelian variety or a complex torus: the invariant Brauer group, as recently defined by Yang Cao. Over the field of complex numbers this is an elementary abelian 2-group with an explicit upper bound on the rank. We exhibit many cases in which the invariant Brauer group is zero, and construct complex abelian varieties in every dimension starting with 2, both simple and non-simple, with invariant Brauer group of order 2. We also address the situation in finite characteristic and over non-closed fields.