Exploring the Transfer Learning Capabilities of CLIP in Domain Generalization for Diabetic Retinopathy
Document Type
Conference Proceeding
Publication Title
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Abstract
Diabetic Retinopathy (DR), a leading cause of vision impairment, requires early detection and treatment. Developing robust AI models for DR classification holds substantial potential, but a key challenge is ensuring their generalization in unfamiliar domains with varying data distributions. To address this, our paper investigates cross-domain generalization, also known as domain generalization (DG), within the context of DR classification. DG, a challenging problem in the medical domain, is complicated by the difficulty of gathering labeled data across different domains, such as patient demographics and disease stages. Some recent studies have shown the effectiveness of using CLIP to handle the DG problem in natural images. In this study, we investigate CLIP’s transfer learning capabilities and its potential for cross-domain generalization in diabetic retinopathy (DR) classification. We carry out comprehensive experiments to assess the efficacy and potential of CLIP in addressing DG for DR classification. Further, we introduce a multi-modal fine-tuning strategy named Context Optimization with Learnable Visual Tokens (CoOpLVT), which enhances context optimization by conditioning on visual features. Our findings demonstrate that the proposed method increases the F1-score by 1.8% over the baseline, thus underlining its promise for effective DG in DR classification. Our code is publicly available at https://github.com/Sanoojan/CLIP-DRDG.
First Page
444
Last Page
453
DOI
10.1007/978-3-031-45673-2_44
Publication Date
10-15-2023
Keywords
Cross-domain, Data distribution, Diabetic retinopathy, Generalisation, Labeled data, Learning capabilities, Medical domains, Optimisations, Transfer learning, Vision impairments
Recommended Citation
S. Baliah et al., "Exploring the Transfer Learning Capabilities of CLIP in Domain Generalization for Diabetic Retinopathy," Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14348 LNCS, pp. 444 - 453, Oct 2023.
The definitive version is available at https://doi.org/10.1007/978-3-031-45673-2_44
Comments
IR conditions: non-described