Structured latent embeddings for recognizing unseen classes in unseen domains

Shivam Chandhok, Mohamed bin Zayed University of Artificial Intelligence
Sanath Narayan, Inception Institute of Artificial Intelligence
Hisham Cholakkal, Mohamed bin Zayed University of Artificial Intelligence
Rao Muhammad Anwer, Mohamed bin Zayed University of Artificial Intelligence
Vineeth N. Balasubramanian, Indian Institute of Technology
Fahad Shahbaz Khan, Mohamed bin Zayed University of Artificial Intelligence
Ling Shao, Inception Institute of Artificial Intelligence

Preprint: arXiv

Abstract

The need to address the scarcity of task-specific annotated data has resulted in concerted efforts in recent years for specific settings such as zero-shot learning (ZSL) and domain generalization (DG), to separately address the issues of semantic shift and domain shift, respectively. However, real-world applications often do not have constrained settings and necessitate handling unseen classes in unseen domains – a setting called Zero-shot Domain Generalization, which presents the issues of domain and semantic shifts simultaneously. In this work, we propose a novel approach that learns domain-agnostic structured latent embeddings by projecting images from different domains as well as class-specific semantic text-based representations to a common latent space. In particular, our method jointly strives for the following objectives: (i) aligning the multimodal cues from visual and text-based semantic concepts; (ii) partitioning the common latent space according to the domain-agnostic class-level semantic concepts; and (iii) learning a domain invariance w.r.t. the visual-semantic joint distribution for generalizing to unseen classes in unseen domains. Our experiments on the challenging DomainNet and DomainNet-LS benchmarks show the superiority of our approach over existing methods, with significant gains on difficult domains like quickdraw and sketch. © 2021, CC BY.