Uncertainty-aware vision transformers for medical image analysis
File(s)
Accepted version
OA Location
Author(s)
Erick, Franciskus Xaverius
Rezaei, Mina
Mueller, Johanna Paula
Kainz, Bernhard
Type
Conference Paper
Abstract
Contrastive Language Image Pre-training (CLIP) has recently demonstrated success across various tasks due to superior feature representation empowered by image-text contrastive learning. However, the instance discrimination method used by CLIP can hardly encode the semantic structure of training data. To handle this limitation, cluster discrimination has been proposed through iterative cluster assignment and classification. Nevertheless, most cluster discrimination approaches only define a single pseudo-label for each image, neglecting multi-label signals in the image. In this paper, we propose a novel Multi-Label Cluster Discrimination method named MLCD to enhance representation learning. In the clustering step, we first cluster the large-scale LAION-400M dataset into one million centers based on off-the-shelf embedding features. Considering that natural images frequently contain multiple visual objects or attributes, we select the multiple closest centers as auxiliary class labels. In the discrimination step, we design a novel multi-label classification loss, which elegantly separates losses from positive classes and negative classes, and alleviates ambiguity on decision boundary. We validate the proposed multi-label cluster discrimination method with experiments on different scales of models and pre-training datasets. Experimental results show that our method achieves state-of-the-art performance on multiple downstream tasks including linear probe, zero-shot classification, and image-text retrieval.
Editor(s)
Sudre, CH
Mehta, R
Ouyang, C
Qin, C
Rakic, M
Wells, WM
Date Issued
2025-01-01
Date Acceptance
2024-10-01
Citation
Lecture Notes in Computer Science, 2025, 15167, pp.171-180
ISBN
978-3-031-73157-0
ISSN
0302-9743
Publisher
Springer International Publishing AG
Start Page
171
End Page
180
Journal / Book Title
Lecture Notes in Computer Science
Volume
15167
Copyright Statement
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG. This is the author’s accepted manuscript made available under a CC-BY licence in accordance with Imperial’s Research Publications Open Access policy (www.imperial.ac.uk/oa-policy)
License URL
Source
6th International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging (UNSURE)
Subjects
Computer Science
Computer Science, Artificial Intelligence
Computer Science, Theory & Methods
DISTANCE
Life Sciences & Biomedicine
Out-of-Distribution Detection
Radiology, Nuclear Medicine & Medical Imaging
Science & Technology
Technology
Vision Transformers
Publication Status
Published
Start Date
2024-10-10
Finish Date
2024-10-10
Coverage Spatial
Marrakesh, Morocco
Date Publish Online
2024-11-03