The task of image segmentation is inherently noisy due to ambiguities
regarding the exact location of boundaries between anatomical structures. We
argue that this information can be extracted from the expert annotations at no
extra cost, and when integrated into state-of-the-art neural networks, it can
lead to improved calibration between soft probabilistic predictions and the
underlying uncertainty. We built upon label smoothing (LS) where a network is
trained on 'blurred' versions of the ground truth labels which has been shown
to be effective for calibrating output predictions. However, LS is not taking
the local structure into account and results in overly smoothed predictions
with low confidence even for non-ambiguous regions. Here, we propose Spatially
Varying Label Smoothing (SVLS), a soft labeling technique that captures the
structural uncertainty in semantic segmentation. SVLS also naturally lends
itself to incorporate inter-rater uncertainty when multiple labelmaps are
available. The proposed approach is extensively validated on four clinical
segmentation tasks with different imaging modalities, number of classes and
single and multi-rater expert annotations. The results demonstrate that SVLS,
despite its simplicity, obtains superior boundary prediction with improved
uncertainty and model calibration.