Background samples provide key contextual information for segmenting regions
of interest (ROIs). However, they always cover a diverse set of structures,
causing difficulties for the segmentation model to learn good decision
boundaries with high sensitivity and precision. The issue concerns the highly
heterogeneous nature of the background class, resulting in multi-modal
distributions. Empirically, we find that neural networks trained with
heterogeneous background struggle to map the corresponding contextual samples
to compact clusters in feature space. As a result, the distribution over
background logit activations may shift across the decision boundary, leading to
systematic over-segmentation across different datasets and tasks. In this
study, we propose context label learning (CoLab) to improve the context
representations by decomposing the background class into several subclasses.
Specifically, we train an auxiliary network as a task generator, along with the
primary segmentation model, to automatically generate context labels that
positively affect the ROI segmentation accuracy. Extensive experiments are
conducted on several challenging segmentation tasks and datasets. The results
demonstrate that CoLab can guide the segmentation model to map the logits of
background samples away from the decision boundary, resulting in significantly
improved segmentation accuracy. Code is available.