Multi-label cluster discrimination for visual representation learning
File(s)
Accepted version
Author(s)
An, Xiang
Yang, Kaicheng
Dai, Xiangzi
Feng, Ziyong
Deng, Jiankang
Type
Conference Paper
Abstract
Contrastive Language Image Pre-training (CLIP) has recently demonstrated success across various tasks due to superior feature representation empowered by image-text contrastive learning. However, the instance discrimination method used by CLIP can hardly encode the semantic structure of training data. To handle this limitation, cluster discrimination has been proposed through iterative cluster assignment and classification. Nevertheless, most cluster discrimination approaches only define a single pseudo-label for each image, neglecting multi-label signals in the image. In this paper, we propose a novel Multi-Label Cluster Discrimination method named MLCD to enhance representation learning. In the clustering step, we first cluster the large-scale LAION-400M dataset into one million centers based on off-the-shelf embedding features. Considering that natural images frequently contain multiple visual objects or attributes, we select the multiple closest centers as auxiliary class labels. In the discrimination step, we design a novel multi-label classification loss, which elegantly separates losses from positive classes and negative classes, and alleviates ambiguity on decision boundary. We validate the proposed multi-label cluster discrimination method with experiments on different scales of models and pre-training datasets. Experimental results show that our method achieves state-of-the-art performance on multiple downstream tasks including linear probe, zero-shot classification, and image-text retrieval.
Editor(s)
Ricci, E
Leonardis, A
Roth, S
Russakovsky, O
Sattler, T
Varol, G
Date Issued
2025-01-01
Date Acceptance
2024-09-29
Citation
Lecture Notes in Computer Science, 2025, 15085, pp.428-444
ISBN
978-3-031-73382-6
ISSN
0302-9743
Publisher
Springer International Publishing AG
Start Page
428
End Page
444
Journal / Book Title
Lecture Notes in Computer Science
Volume
15085
Copyright Statement
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG.
License URL
Source
18th European Conference on Computer Vision (ECCV)
Subjects
Cluster Discrimination
Computer Science
Computer Science, Artificial Intelligence
Computer Science, Interdisciplinary Applications
Computer Science, Theory & Methods
Instance Discrimination
Multi-label Learning
Science & Technology
Technology
Visual Representation Learning
Publication Status
Published
Start Date
2024-09-29
Finish Date
2024-10-04
Coverage Spatial
Milan, Italy
Date Publish Online
2024-11-03