A Unified Query-based Paradigm for Camouflaged Instance Segmentation

Authors

Bo Dong, College of Biomedical Engineering & Instrument Science, Zhejiang University
Jialun Pei, Chinese University of Hong Kong
Rongrong Gao, Hong Kong University of Science and Technology
Tian Zhu Xiang, G42 Healthcare
Shuo Wang, ETH Zürich
Huan Xiong, Mohamed Bin Zayed University of Artificial IntelligenceFollow

Document Type

Conference Proceeding

Publication Title

MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia

Abstract

Due to the high similarity between camouflaged instances and the background, the recently proposed camouflaged instance segmentation (CIS) faces challenges in accurate localization and instance segmentation. To this end, inspired by query-based transformers, we propose a unified query-based multi-task learning framework for camouflaged instance segmentation, termed UQFormer, which builds a set of mask queries and a set of boundary queries to learn a shared composed query representation and efficiently integrates global camouflaged object region and boundary cues, for simultaneous instance segmentation and instance boundary detection in camouflaged scenarios. Specifically, we design a composed query learning paradigm that learns a shared representation to capture object region and boundary features by the cross-attention interaction of mask queries and boundary queries in the designed multi-scale unified learning transformer decoder. Then, we present a transformer-based multi-task learning framework for simultaneous camouflaged instance segmentation and camouflaged instance boundary detection based on the learned composed query representation, which also forces the model to learn a strong instance-level query representation. Notably, our model views the instance segmentation as a query-based direct set prediction problem, without other post-processing such as non-maximal suppression. Compared with 14 state-of-the-art approaches, our UQFormer significantly improves the performance of camouflaged instance segmentation. Our code will be available at: https://github.com/dongbo811/UQFormer.

First Page

2131

Last Page

2138

DOI

10.1145/3581783.3612185

Publication Date

10-27-2023

Keywords

Camouflaged instance segmentation, Query learning, Transformer, Boundary detection, Object boundaries

Comments

Open Access version from Arxiv

Uploaded on June 11, 2024

Recommended Citation

B. Dong, J. Pei, R. Gao, T. Zhu Xiang, S. Wang and H. Xiong, "A Unified Query-based Paradigm for Camouflaged Instance Segmentation," MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia, pp. 2131 - 2138, Oct 2023. doi: 10.1145/3581783.3612185