Document Type


Publication Title



Inspired by the human cognitive system, attention is a mechanism that imitates the human cognitive awareness about specific information, amplifying critical details to focus more on the essential aspects of data. Deep learning has employed attention to boost performance for many applications. Interestingly, the same attention design can suit processing different data modalities and can easily be incorporated into large networks. Furthermore, multiple complementary attention mechanisms can be incorporated in one network. Hence, attention techniques have become extremely attractive. However, the literature lacks a comprehensive survey specific to attention techniques to guide researchers in employing attention in their deep models. Note that, besides being demanding in terms of training data and computational resources, transformers only cover a single category in self-attention out of the many categories available. We fill this gap and provide an in-depth survey of 50 attention techniques categorizing them by their most prominent features. We initiate our discussion by introducing the fundamental concepts behind the success of attention mechanism. Next, we furnish some essentials such as the strengths and limitations of each attention category, describe their fundamental building blocks, basic formulations with primary usage, and applications specifically for computer vision. We also discuss the challenges and open questions related to attention mechanism in general. Finally, we recommend possible future research directions for deep attention. © 2022, CC BY.



Publication Date



Behavioral research, Cognitive systems, Data handling, Deep learning, Surveys, Attention in computer vision, Attention mechanisms, Attention module, Deep attention, Depth surveys, Human cognitive systems, Larger networks, Performance, Specific information, Visual attention methods, Computer vision, Artificial Intelligence (cs.AI), Computer Vision and Pattern Recognition (cs.CV), Image and Video Processing (eess.IV), Machine Learning (cs.LG)


Preprint: arXiv

Archived with thanks to arXiv

Preprint License: CC by 4.0

Uploaded 24 May 2022