Quantifying the uncertainty of neural networks (NNs) has been
required by many safety-critical applications such as autonomous
driving or medical diagnosis. Recently, Bayesian transformers have
demonstrated their capabilities in providing high-quality uncer-
tainty estimates paired with excellent accuracy. However, their
real-time deployment is limited by the compute-intensive attention
mechanism that is core to the transformer architecture, and the
repeated Monte Carlo sampling to quantify the predictive uncer-
tainty. To address these limitations, this paper accelerates Bayesian
transformers via both algorithmic and hardware optimizations. On
the algorithmic level, an evolutionary algorithm (EA)-based frame-
work is proposed to exploit the sparsity in Bayesian transformers
and ease their computational workload. On the hardware level,
we demonstrate that the sparsity brings hardware performance
improvement on our optimized CPU and GPU implementations.
An adaptable hardware architecture is also proposed to acceler-
ate Bayesian transformers on an FPGA. Extensive experiments
demonstrate that the EA-based framework, together with hardware
optimizations, reduce the latency of Bayesian transformers by up to
13, 12 and 20 times on CPU, GPU and FPGA platforms respectively,
while achieving higher algorithmic performance.