Deep networks have enabled reinforcement learning to scale to more complex
and challenging domains, but these methods typically require large quantities
of training data. An alternative is to use sample-efficient episodic control
methods: neuro-inspired algorithms which use non-/semi-parametric models that
predict values based on storing and retrieving previously experienced
transitions. One way to further improve the sample efficiency of these
approaches is to use more principled exploration strategies. In this work, we
therefore propose maximum entropy mellowmax episodic control (MEMEC), which
samples actions according to a Boltzmann policy with a state-dependent
temperature. We demonstrate that MEMEC outperforms other uncertainty- and
softmax-based exploration methods on classic reinforcement learning
environments and Atari games, achieving both more rapid learning and higher
final rewards.