CLIP-Decoder : ZeroShot Multilabel Classification using Multimodal CLIP Aligned Representations

Document Type

Conference Proceeding

Publication Title

Proceedings - 2023 IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2023

Abstract

Multi-label classification is an essential task utilized in a wide variety of real-world applications. Multi-label zero-shot learning is a method for classifying images into multiple unseen categories for which no training data is available, while in general zero-shot situations, the test set may include observed classes. The CLIP-Decoder is a novel method based on the state-of-the-art ML-Decoder attention-based head. We introduce multi-modal representation learning in CLIP-Decoder, utilizing the text encoder to extract text features and the image encoder for image feature extraction. Furthermore, we minimize semantic mismatch by aligning image and word embeddings in the same dimension and comparing their respective representations using a combined loss, which comprises classification loss and CLIP loss. This strategy outperforms other methods and we achieve cutting-edge results on zero-shot multilabel classification tasks using CLIP-Decoder. Our method achieves an absolute increase of 3.9% in performance compared to existing methods for zero-shot learning multi-label classification tasks. Additionally, in the generalized zero-shot learning multi-label classification task, our method shows an impressive increase of almost 2.3%.

First Page

4677

Last Page

4681

DOI

10.1109/ICCVW60793.2023.00505

Publication Date

12-25-2023

Keywords

Representation learning, Computer vision, Head, Zero-shot learning, Conferences, Semantics, Training data

Comments

IR conditions: non-described

Link to Full Text

Share

COinS