Most of the recent deep reinforcement learning
advances take an RL-centric perspective and focus on refinements of the training objective. We
diverge from this view and show we can recover
the performance of these developments not by
changing the objective, but by regularising the
value-function estimator. Constraining the Lipschitz constant of a single layer using spectral
normalisation is sufficient to elevate the performance of a Categorical-DQN agent to that of a
more elaborated RAINBOW agent on the challenging Atari domain. We conduct ablation studies
to disentangle the various effects normalisation
has on the learning dynamics and show that is
sufficient to modulate the parameter updates to
recover most of the performance of spectral normalisation. These findings hint towards the need
to also focus on the neural component and its
learning dynamics to tackle the peculiarities of
Deep Reinforcement Learning.