Deep Learning for Cancer Diagnosis and Prognosis

Date of Award


Document Type


Degree Name

Doctor of Philosophy in Machine Learning


Machine Learning

First Advisor

Dr. Mohammad Yaqub

Second Advisor

Dr. Bin Gu


According to the World Health Organization (WHO), cancer claimed the lives of nearly 10 million people in 2020. Amidst this challenge, the medical community witnessed a notable advancement with the introduction of deep learning models. The development of these models has significantly advanced the methods of cancer diagnosis and prognosis, providing levels of accuracy and predictability previously unattainable. Given the inherent complexities and variations in different types of cancer, each presents its own set of challenges in terms of diagnosis and prognosis. Consequently, there's a pressing need for tailored deep-learning methods that can accurately address the specific nuances and intricacies of each cancer type. In spite of the significant advancements in cancer diagnosis and prognosis that deep learning has made, the field still faces significant challenges. Among them are the reliability and generalizability of diagnostic models, as well as the integration of multimodal data for the purpose of enhancing prognostic models. In this thesis, we focus on developing unique and effective solutions to address some of these challenges for two types of cancers, namely, head and neck and glioblastoma. Both of these cancer types are aggressive and challenging to treat. Towards this end, we present four primary research studies; the first three focus on distinct issues related to head and neck cancers, while the fourth is dedicated to glioblastoma. In the first study, Positron Emission Tomography (PET), Computed Tomography (CT) and Electronic Health Records (EHR) are used from the HECKTOR dataset to segment the tumors and predict the survival rate of patients. We propose a multi-faceted network that combines Convolutional Neural Networks (CNN), deep Multi-Task Logistic Regressing (MTLR), and Cox Proportional Hazard (Cox-PH) models. The extracted features of CT and PET scans are integrated into the patient's electronic health record in order to forecast their prognosis. We achieve state-of-the-art results in predicting the survival of these patients with a concordance index of 0.72. The integration of the image and clinical features in the proposed approach (and many other existing models) occurs at a later stage in the model architecture. Medical experts, however, synthesize information from a variety of sources, including medical imagery and histories of patients, in a unified manner in order to assess patient survival. Hence, in the second study, we introduced TMSS, a Transformer-based Multimodal network for Segmentation and Survival prediction to emulate the approach of oncologists. The proposed solution was validated on the HECKTOR dataset, where our proposed model achieves a concordance index of 0.763 and a dice score of 0.772, exceeding existing prognostic models and achieving a competitive dice score for the segmentation task. In the medical field, a challenge arises when trying to adapt a trained model for diagnosis or prognosis using new medical center data without "catastrophic forgetting." Often, merging old and new data for retraining isn't feasible due to privacy or storage constraints. Thus, in the third study, we introduced a fine-tuning technique to adapt pre-trained transformer models to new center data. This method consistently demonstrated robust performance on new data while maintaining accuracy for previous centers. Notably, it outperformed existing methods in computational efficiency and speed. In the fourth study, we investigated the gap between the acclaimed superiority of deep learning models in medical diagnosis and their limited real-world deployment. Focusing on glioblastoma, a lethal brain tumor in older adults, we examined the correlation between the MGMT promoter's methylation status, which influences chemotherapy efficacy, and the brain MRI scans. Using deep learning, we analyzed brain MRI scans from a substantial dataset of 585 participants to predict this methylation status. Despite various testing methods, our results found no correlation, highlighting the need for external data validation to ensure the accuracy of such models in cancer diagnosis.


Thesis submitted to the Deanship of Graduate and Postdoctoral Studies

In partial fulfilment of the requirements for the PhD degree in Machine Learning

Advisors: Mohammad Yaqub, Bin Gu

Online access available for MBZUAI patrons