Visual Attention Methods in Deep Learning: An In-Depth Survey

Authors

Mohammed Hassanin, UniSA (University of South Australia), Faculty of Computers and Information, Fayoum University, Egypt
Anwar Saeed, The Australian National University, Canberra, Australia & Data61, CSIRO (The Commonwealth Scientific and Industrial Research Organisation), The University of Technology Sydney, University of Canberra, Australia
Ibrahim Radwan, University of Canberra, Australia
Fahad Shahbaz Khan, Mohamed bin Zayed University of Artificial Intelligence & Linköping University, SwedenFollow
Ajmal Mian, The University of Western Australia, Australia

Document Type

Article

Publication Title

arXiv

Abstract

Inspired by the human cognitive system, attention is a mechanism that imitates the human cognitive awareness about specific information, amplifying critical details to focus more on the essential aspects of data. Deep learning has employed attention to boost performance for many applications. Interestingly, the same attention design can suit processing different data modalities and can easily be incorporated into large networks. Furthermore, multiple complementary attention mechanisms can be incorporated in one network. Hence, attention techniques have become extremely attractive. However, the literature lacks a comprehensive survey specific to attention techniques to guide researchers in employing attention in their deep models. Note that, besides being demanding in terms of training data and computational resources, transformers only cover a single category in self-attention out of the many categories available. We fill this gap and provide an in-depth survey of 50 attention techniques categorizing them by their most prominent features. We initiate our discussion by introducing the fundamental concepts behind the success of attention mechanism. Next, we furnish some essentials such as the strengths and limitations of each attention category, describe their fundamental building blocks, basic formulations with primary usage, and applications specifically for computer vision. We also discuss the challenges and open questions related to attention mechanism in general. Finally, we recommend possible future research directions for deep attention. © 2022, CC BY.

DOI

10.48550/arXiv.2204.07756

Publication Date

4-16-2022

Keywords

Behavioral research, Cognitive systems, Data handling, Deep learning, Surveys, Attention in computer vision, Attention mechanisms, Attention module, Deep attention, Depth surveys, Human cognitive systems, Larger networks, Performance, Specific information, Visual attention methods, Computer vision, Artificial Intelligence (cs.AI), Computer Vision and Pattern Recognition (cs.CV), Image and Video Processing (eess.IV), Machine Learning (cs.LG)

Comments

Preprint: arXiv

Archived with thanks to arXiv

Preprint License: CC by 4.0

Uploaded 24 May 2022

Recommended Citation

M. Hassanin, A. Saeed, I. Radwan, F.S. Khan, and A. Mian, "Visual Attention Methods in Deep Learning: An In-Depth Survey", arXiv, Apr 2022, doi: 10.48550/arXiv.2204.07756