Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection

Authors

Hanoona Rasheed, Mohamed bin Zayed University of Artificial IntelligenceFollow
Muhammad Maaz, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Muhammad Uzair Khattak, Mohamed bin Zayed University of Artificial IntelligenceFollow
Salman Khan, Mohamed bin Zayed University of Artificial Intelligence & Australian National University, AustraliaFollow
Fahad Shahbaz Khan, Mohamed bin Zayed University of Artificial Intelligence & Linköping University, SwedenFollow

Document Type

Article

Publication Title

arXiv

Abstract

Existing open-vocabulary object detectors typically enlarge their vocabulary sizes by leveraging different forms of weak supervision. This helps generalize to novel objects at inference. Two popular forms of weak-supervision used in open-vocabulary detection (OVD) include pretrained CLIP model and image-level supervision. We note that both these modes of supervision are not optimally aligned for the detection task: CLIP is trained with image-text pairs and lacks precise localization of objects while the image-level supervision has been used with heuristics that do not accurately specify local object regions. In this work, we propose to address this problem by performing object-centric alignment of the language embeddings from the CLIP model. Furthermore, we visually ground the objects with only image-level supervision using a pseudo-labeling process that provides high-quality object proposals and helps expand the vocabulary during training. We establish a bridge between the above two object-alignment strategies via a novel weight transfer function that aggregates their complimentary strengths. In essence, the proposed model seeks to minimize the gap between object and image-centric representations in the OVD setting. On the COCO benchmark, our proposed approach achieves 40.3 AP50on novel classes, an absolute 11.9 gain over the previous best performance. For LVIS, we surpass the state-of-the-art ViLD model by 5.0 mask AP for rare categories and 3.4 overall. © 2022, CC BY.

DOI

10.48550/arXiv.2207.03482

Publication Date

7-7-2022

Keywords

Computer vision, Object detection

Comments

Preprint: arXiv

Archived with thanks to arXiv

Preprint License: CC by 4.0

Uploaded 25 August 2022

Recommended Citation

H. Rasheed, M. Maaz, M.U. Khattak, S. Khan and F.S. Khan, "Bridging the Gap between Object and Image-level Representations for Open-Vocabulary Detection", 2022, arXiv:2207.03482