Gaussian processes are valuable tools for non-parametric modelling, where
typically an assumption of stationarity is employed. While removing this
assumption can improve prediction, fitting such models is challenging. In this
work, hierarchical models are constructed based on Gaussian Markov random
fields with stochastic spatially varying parameters. Importantly, this allows
for non-stationarity while also addressing the computational burden through a
sparse banded representation of the precision matrix. In this setting,
efficient Markov chain Monte Carlo (MCMC) sampling is challenging due to the
strong coupling a posteriori of the parameters and hyperparameters. We develop
and compare three adaptive MCMC schemes and make use of banded matrix
operations for faster inference. Furthermore, a novel extension to
multi-dimensional settings is proposed through an additive structure that
retains the flexibility and scalability of the model, while also inheriting
interpretability from the additive approach. A thorough assessment of the
efficiency and accuracy of the methods in nonstationary settings is presented
for both simulated experiments and a computer emulation problem.