Enhancing Novel Object Detection via Cooperative Foundational Models

Author

Rohit Bharadwaj, Mohamed bin Zayed University of Artificial IntelligenceFollow

Date of Award

4-30-2024

Document Type

Thesis

Degree Name

Master of Science in Machine Learning

Department

Machine Learning

First Advisor

Dr. Salman Khan

Second Advisor

Dr. Hava Siegelmann

Abstract

"In this work, we address the challenging and emergent problem of novel object detection (NOD), focusing on the accurate detection of both known and novel object categories during inference. Traditional object detection algorithms are inherently closed set, limiting their capability to handle NOD. We present a novel approach to transform existing closed-set detectors into open-set detectors. This transformation is achieved by leveraging the complementary strengths of pre-trained foundational models, specifically CLIP and SAM, through our cooperative mechanism. Furthermore, by integrating this mechanism with state-of-the-art open-set detectors such as GDINO, we establish new benchmarks in object detection performance. Our method achieves 17.42 mAP in novel object detection and 42.08 mAP for known objects on the challenging LVIS dataset. Adapting our approach to the COCO OVD split, we surpass the current stateof- the-art by a margin of 7.2 AP50 for novel classes. Our code is available at https: //github.com/rohit901/cooperative-foundational-models."

Comments

Thesis submitted to the Deanship of Graduate and Postdoctoral Studies

In partial fulfilment of the requirements for the M.Sc degree in Machine Learning

Advisors: Salman Khan, Hava Siegelmann

Online access available for MBZUAI patrons

Recommended Citation

R. Bharadwaj, "Enhancing Novel Object Detection via Cooperative Foundational Models,", Apr 2024.