From known to the unknown: Transferring knowledge to answer questions about novel visual and semantic concepts

Authors

Moshiur R. Farazi, College of Engineering and Computer Science, Australian National University (ANU), Canberra ACT, 0200, Australia & Data61, Commonwealth Scientific and Industrial Research Organisation (CSIRO), Canberra ACT, 2601, Australia
Salman Khan, College of Engineering and Computer Science, Australian National University (ANU), Canberra ACT, 0200, Australia & Mohamed bin Zayed University of Artificial IntelligenceFollow
Nick Barnes, College of Engineering and Computer Science, Australian National University (ANU), Canberra ACT, 0200, AustraliaFollow

Document Type

Article

Publication Title

Image and Vision Computing

Abstract

Current Visual Question Answering (VQA) systems can answer intelligent questions about ‘known’ visual content. However, their performance drops significantly when questions about visually and linguistically ‘unknown’ concepts are presented during inference (‘Open-world’ scenario). A practical VQA system should be able to deal with novel concepts in real world settings. To address this problem, we propose an exemplar-based approach that transfers learning (i.e., knowledge) from previously ‘known’ concepts to answer questions about the ‘unknown’. We learn a highly discriminative joint embedding (JE) space, where visual and semantic features are fused to give a unified representation. Once novel concepts are presented to the model, it looks for the closest match from an exemplar set in the JE space. This auxiliary information is used alongside the given Image-Question pair to refine visual attention in a hierarchical fashion. Our novel attention model is based on a dual-attention mechanism that combines the complementary effect of spatial and channel attention. Since handling the high dimensional exemplars on large datasets can be a significant challenge, we introduce an efficient matching scheme that uses a compact feature description for search and retrieval. To evaluate our model, we propose a new dataset for VQA, separating unknown visual and semantic concepts from the training set. Our approach shows significant improvements over state-of-the-art VQA models on the proposed Open-World VQA dataset and other standard VQA datasets. © 2020 Elsevier B.V.

DOI

10.1016/j.imavis.2020.103985

Publication Date

8-4-2020

Keywords

Behavioral research, Semantics, Transfer learning, Attention mechanisms, Auxiliary information, Compact Features, Question Answering, Real world setting, Search and retrieval, Semantic features, Visual Attention, Large dataset

Comments

IR Deposit conditions:

OA version (pathway b): Accepted version

24 months embargo

Licence: CC BY-NC-ND

Recommended Citation

M.R. Farazi, S.H. Khan, and N. Barnes, "From known to the unknown: Transferring knowledge to answer questions about novel visual and semantic concepts", in Image and Vision Computing, 2020, doi:10.1016/j.imavis.2020.103985