M2Net : Multi-modal Multi-channel Network for Overall Survival Time Prediction of Brain Tumor Patients
Document Type
Conference Proceeding
Publication Title
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Abstract
Early and accurate prediction of overall survival (OS) time can help to obtain better treatment planning for brain tumor patients. Although many OS time prediction methods have been developed and obtain promising results, there are still several issues. First, conventional prediction methods rely on radiomic features at the local lesion area of a magnetic resonance (MR) volume, which may not represent the full image or model complex tumor patterns. Second, different types of scanners (i.e., multi-modal data) are sensitive to different brain regions, which makes it challenging to effectively exploit the complementary information across multiple modalities and also preserve the modality-specific properties. Third, existing methods focus on prediction models, ignoring complex data-to-label relationships. To address the above issues, we propose an end-to-end OS time prediction model; namely, Multi-modal Multi-channel Network. Specifically, we first project the 3D MR volume onto 2D images in different directions, which reduces computational costs, while preserving important information and enabling pre-trained models to be transferred from other tasks. Then, we use a modality-specific network to extract implicit and high-level features from different MR scans. A multi-modal shared network is built to fuse these features using a bilinear pooling model, exploiting their correlations to provide complementary information. Finally, we integrate the outputs from each modality-specific network and the multi-modal shared network to generate the final prediction result. Experimental results demonstrate the superiority of our model over other methods.
First Page
221
Last Page
231
DOI
10.1007/978-3-030-59713-9_22
Publication Date
9-29-2020
Keywords
Brain, Complex networks, Forecasting, Magnetic resonance, Medical computing, Medical imaging, Modal analysis, Patient treatment, Tumors
Recommended Citation
T. Zhou et al., "M2Net : Multi-modal Multi-channel Network for Overall Survival Time Prediction of Brain Tumor Patients," Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12262 LNCS, pp. 221 - 231, Sep 2020.
The definitive version is available at https://doi.org/10.1007/978-3-030-59713-9_22
Comments
IR conditions: non-described