Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting

Authors

Syed Talal Wasim, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Muzammal Naseer, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Salman Khan, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Fahad Shahbaz Khan, Mohamed Bin Zayed University of Artificial IntelligenceFollow
Mubarak Shah, University of Central Florida

Document Type

Conference Proceeding

Publication Title

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Abstract

Adopting contrastive image-text pretrained models like CLIP towards video classification has gained attention due to its cost-effectiveness and competitive performance. However, recent works in this area face a trade-off. Finetuning the pretrained model to achieve strong supervised performance results in low zero-shot generalization. Similarly, freezing the backbone to retain zero-shot capability causes significant drop in supervised accuracy. Because of this, recent works in literature typically train separate models for supervised and zero-shot action recognition. In this work, we propose a multimodal prompt learning scheme that works to balance the supervised and zero-shot performance under a single unified training. Our prompting approach on the vision side caters for three aspects: 1) Global video-level prompts to model the data distribution; 2) Local frame-level prompts to provide per-frame discriminative conditioning; and 3) a summary prompt to extract a condensed video representation. Additionally, we define a prompting scheme on the text side to augment the textual context. Through this prompting scheme, we can achieve state-of-the-art zero-shot performance on Kinetics-600, HMDB51 and UCF101 while remaining competitive in the supervised setting. By keeping the pretrained backbone frozen, we optimize a much lower number of parameters and retain the existing general representation which helps achieve the strong zero-shot performance. Our codes/models will be released at https://github.com/TalalWasim/Vita-Clip..

First Page

23034

Last Page

23044

DOI

10.1109/CVPR52729.2023.02206

Publication Date

8-22-2023

Keywords

Training, Computer vision, Adaptation models, Image recognition, Text recognition, Face recognition, Data models

Comments

IR conditions: non-described

Recommended Citation

S. Wasim et al., "Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting," Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2023-June, pp. 23034 - 23044, Aug 2023.

The definitive version is available at https://doi.org/10.1109/CVPR52729.2023.02206

Additional Links

https://doi.org/10.1109/CVPR52729.2023.02206