Attention guided semantic relationship parsing for visual question answering

Authors

Moshiur R. Farazi, College of Engineering and Computer Science, Australian National University (ANU), Canberra ACT, 0200, Australia & Data61, Commonwealth Scientific and Industrial Research Organisation (CSIRO), Canberra ACT, 2601, Australia
Salman Khan, College of Engineering and Computer Science, Australian National University (ANU), Canberra ACT, 0200, Australia & Mohamed bin Zayed University of Artificial IntelligenceFollow
Nick Barnes, College of Engineering and Computer Science, Australian National University (ANU), Canberra ACT, 0200, AustraliaFollow

Document Type

Article

Publication Title

arXiv

Abstract

Humans explain inter-object relationships with semantic labels that demonstrate a high-level understanding required to perform complex Vision-Language tasks such as Visual Question Answering (VQA). However, existing VQA models represent relationships as a combination of object-level visual features which constrain a model to express interactions between objects in a single domain, while the model is trying to solve a multi-modal task. In this paper, we propose a general purpose semantic relationship parser which generates a semantic feature vector for each subject-predicate-object triplet in an image, and a Mutual and Self Attention (MSA) mechanism that learns to identify relationship triplets that are important to answer the given question. To motivate the significance of semantic relationships, we show an oracle setting with ground-truth relationship triplets, where our model achieves a ∼25% accuracy gain over the closest state-of-the-art model on the challenging GQA dataset. Further, with our semantic parser, we show that our model outperforms other comparable approaches on VQA and GQA datasets. Copyright © 2020, The Authors. All rights reserved.

DOI

arXiv:2010.01725

Publication Date

10-5-2020

Keywords

Artificial Intelligence (cs.AI), Computer Vision and Pattern Recognition (cs.CV)

Comments

IR Deposit conditions: non-described

Preprint available on arXiv

Recommended Citation

M.R. Farazi, S. Khan, and N. Barnes, "Attention guided semantic relationship parsing for visual question answering", 2020, arXiv:2010.01725