Multi Label Ocular Disease Detection in Ophthalmological Images and Question Answering System on Health Data

Abstract

In recent years, visual deep learning-based methods have achieved significant successes in medical analysis, particularly producing successful results in disease detection using visual data. Although current CNN-based approaches offer high performance in disease detection, they fall short in providing understandable and accessible explanations related to the diagnosed diseases. To address this deficiency, this study proposes a two-stage framework that integrates automatic eye disease detection with an information retrieval-based question-answering system.In the first stage, a multi-label classification task is performed on retinal images to detect eight eye diseases. For this purpose, various advanced Convolutional Neural Network (CNN) architectures have been evaluated. In the second stage, a Retrieval-Augmented Generation (RAG) method, powered by GPT-4o, is employed to provide users with information about the detected diseases and answers to their questions.This approach combines high-accuracy CNN-based classification with a semantically rich RAG question-answering system, enabling both reliable detection of eye diseases and the delivery of meaningful, contextually appropriate explanations to end users. Comparisons and objective evaluations of both stages of the proposed framework have been conducted, leading to the conclusion that the system is applicable in real-world settings. © 2025 Elsevier B.V., All rights reserved.