Document Type

Article

Publication Title

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract

As information exists in various modalities in real world, effective interaction and fusion among multimodal information plays a key role for the creation and perception of multimodal data in computer vision and deep learning research. With superb power in modeling the interaction among multimodal information, multimodal image synthesis and editing has become a hot research topic in recent years. Instead of providing explicit guidance for network training, multimodal guidance offers intuitive and flexible means for image synthesis and editing. On the other hand, this field is also facing several challenges in alignment of multimodal features, synthesis of high-resolution images, faithful evaluation metrics, etc. In this survey, we comprehensively contextualize the advance of the recent multimodal image synthesis and editing and formulate taxonomies according to data modalities and model types. We start with an introduction to different guidance modalities in image synthesis and editing, and then describe multimodal image synthesis and editing approaches extensively according to their model types. After that, we describe benchmark datasets and evaluation metrics as well as corresponding experimental results. Finally, we provide insights about the current research challenges and possible directions for future research.

First Page

15098

Last Page

15119

DOI

10.1109/TPAMI.2023.3305243

Publication Date

12-1-2023

Keywords

Visualization, Image synthesis, Training, Surveys, Task analysis, Computational modeling, Transformers, Autoregressive models, Diffusion models, GANs, Image synthesis and editing, Multimodality, NeRF

Comments

Preprint version from arXiv

CC BY

Uploaded on June 21, 2024

Download

Share

COinS