An online diabetic retinopathy screening tool for patients with type 2 diabetes

Wan-Yue Li, Ming Yang, Ya-Nan Song, Ling Luo, Chuang Nie, Mao-Nian Zhang

1Medical School of Chinese PLA, Beijing 100853, China

2Department of Ophthalmology, Chinese PLA General Hospital, Beijing 100853, China

3Medical Big Data Research Center, Medical Innovation Research Division of Chinese PLA General Hospital, Beijing 100853, China

4Department of Ophthalmology, Strategic Support Force Medical Centre, Beijing 100101, China

Abstract

INTRODUCTION

Diabetic retinopathy (DR) is the leading cause of permanent and irreversible blindness in working-age adults globally[1]. The preventive effects of therapy and the fact that patients with proliferative DR or macular edema may be asymptomatic provide strong support for screening to detect DR[2]. The American Diabetes Association (ADA) recommends dilated and comprehensive eye examinations every 1-2y for patients with diabetes without evidence of retinopathy, and more frequent examinations are necessary if any level of DR is present or if sight is threatened[2]. However, not every patient undergoes annual dilated and comprehensive eye examinations because of lack of knowledge of DR, economic burden, and unevenly distributed resources for ophthalmic care[3]. As an article has reported, only approximately 60% of patients with diabetes undergo an annual dilated eye examination in the United States[4].

A large number of studies on the automated identification of DR have been conducted using fundus images[5-7]. However,these screening systems cannot meet the needs of a significant proportion of patients with diabetes. In some medically underserved areas, patients with diabetes are examined for their health in primary medical institutions, such as community hospitals. Most primary medical institutions are not equipped with fundus cameras and do not have ophthalmologists.Doctors not belonging to the ophthalmology and endocrinology departments often do not have the awareness of preventing DR when receiving a patient with diabetes and may not provide recommendations for DR screening.

Hence, we aimed to establish a DR risk prediction model and develop a DR screening tool which is easy to promote and operate, and integrates disease education, risk prediction,medical advice function to help patients with type 2 diabetes mellitus (T2DM) to self-manage and assist primary medical institutions in making referral recommendations.

SUBJECTS AND METHODS

Ethical ApprovalThis retrospective study was approved by the Chinese PLA General Hospital Clinical Research Ethics Committee (No.S2019-326-02, February 25, 2020) and adhered to the tenets of the Declaration of Helsinki.

Development DatasetThe prediction model was developed from the clinical data of inpatients diagnosed with T2DM extracted from the Chinese PLA General Hospital electronic medical record system from January 1, 2019 to December 31, 2019. The development dataset was randomly divided into training and internal validation sets. The training set,containing 70% of the samples, was used to train the prediction model. The internal validation set, obtaining the rest of the samples, was used to assess the prediction performance of the model to optimize the parameters.

Data Extraction CriteriaThe first record of the measurements upon admission for each variable was extracted. Diagnostic information on the disease was extracted from the discharge diagnosis records. The retrieved variables included demographic characteristics (sex and age), duration of diabetes, living habits(smoking, drinking, and diet control), DR screening history[absence of DR screening, annual DR screening, and number of years until the first fundus examination after diabetes diagnosis (first DR screening)], diabetes complications(diabetic nephropathy, neuropathy, and retinopathy), other chronic diseases (hypertension, duration of hypertension,hyperlipidemia, atherosclerosis, and coronary heart disease),physical indicators (weight, height, and systolic and diastolic blood pressure), and laboratory parameters [levels of fasting blood glucose, postprandial blood glucose, glycosylated hemoglobin (HbA1c), triglycerides, total cholesterol, highdensity lipoproteins, low-density lipoproteins, urea, and serum creatinine]. Body mass index was calculated as weight (kg)divided by height in meters squared (m2).

External Validation DatasetThe clinical data of inpatients diagnosed with T2DM extracted from the Strategic Support Force Medical Centre electronic medical record system in the same period were used for external validation. To better verify the generalization ability of the model, 64 patients with T2DM with DR and 64 patients with T2DM without DR were randomly selected, with a ratio of 1:1. The variables retained in the prediction model were extracted using the same extraction criteria.

Inclusion and Exclusion CriteriaThe patients admitted in the Ophthalmology Department or had received an ophthalmic consultation were included in this study. The patients with missing data or those with cataracts, keratitis, corneal speckles,and other eye diseases that affect fundus examination were excluded.

Diagnostic CriteriaThe diagnostic criteria for T2DM followed those established by the 2013 ADA[8]. DR was diagnosed according to the 2017 ADA Position Statement of DR[9]using color fundus photography and indirect ophthalmoscopy when the pupils were dilated. Fundus examination, image reading, and diagnosis were performed by at least two experienced ophthalmologists for each patient,whether directly admitted to the ophthalmology department or for ophthalmic consultation in other departments. All patients with diabetic fundus lesions, including mild non-proliferative DR, were included in the DR group.

Statistical AnalysisIn this study, the R programming language (version 4.0.2) was used for statistical analyses.A baseline analysis was conducted on the development dataset. Continuous variables were expressed as mean±SD or median (interquartile range) after the normality distribution test. Categorical variables are expressed as numbers and percentages. The Student’st-test was performed on continuous variables that followed a normal distribution, while the Mann-WhitneyUtest was performed on those with skewed distributions. The Chi-squared test was used to compare categorical variables. Statistical significance was set atP＜0.05.

Prediction Model Training and Internal ValidationThe Lasso regression method was used to determine the optimal variables for feature selection. Feature selection was aimed at eliminating redundant factors without losing key information,and to obtain a factor set of lower dimensions, improve the accuracy, and reduce the complexity of the model.

Multiple Logistic regression analysis was applied to the training dataset to generate the DR prediction model. The variables included in the Logistic regression analysis were determined based on the feature selection results and clinical utility. The Hosmer-Lemeshow test was used to assess the goodness of fit of the model. Since the model was developed to screen patients requiring further fundus examination, we focused on the recall ratio [true positive/(true positive+false negative)] of the internal validation dataset when optimizing the parameters, and evaluated the overall performance of the model by accuracy, F-measure, and the areas under the receiver operator characteristic curves (ROC-AUC).

External ValidationExternal validation was performed to test the generalizability of the model. First, we conducted a statistical analysis of the differences between the model inclusion variables in the external validation and model development sets. The method was the same as the baseline analysis described above. Then, the trained model was applied to the external validation set to predict the occurrence of DR.The performance assessment indices were the same as those for internal validation.

WeChat Mini Program DevelopmentTo facilitate the predictive function in clinical practice, we designed and developed a DR risk prediction calculator for the above model.This calculator, developed through WeChat Developer Tools using JavaScript, was embedded in the WeChat Mini Program.The general schema for the prediction model building and the WeChat Mini Program development is shown in Figure 1.

Figure 1 The general schema for the prediction model building and the WeChat Mini Program development DR: Diabetic retinopathy.

Table 1 Baseline analysis results of 28 variables of 778 patients with T2DM mean±SD

RESULTS

Baseline Analysis of the Development DatasetThe data of 778 inpatients with T2DM, including 413 patients with DR and 365 patients without DR, and 28 variables were extracted.The baseline analysis of the development datasets is presented in Table 1. The patients with and without DR were similar in terms of sex, smoking history, drinking history, diet control,atherosclerosis, coronary heart disease, and levels of blood glucose, total cholesterol, and triglycerides. Diabetes duration was shorter in the non-DR group than that in the DR group [8.00(3.00, 14.00)vs15.00 (10.00, 20.00),P＜0.01]. The DR group had a higher prevalence of hypertension, diabetic nephropathy,and diabetic nephropathy than the non-DR group.

Table 2 Baseline comparison of development dataset and external dataset mean±SD

Only 9.2% of the DR group underwent DR screening annually;this proportion was significantly lower than that in the non-DR group (26.8%,P＜0.01). The first DR screening was performed earlier in the non-DR group than that in the DR group[4.00 (1.00, 10.00)vs11.00 (5.00, 16.50),P＜0.01]. Almost half of the patients with diabetes did not undergo a fundus examination after diagnosis, indicating that 45.8% of those in the DR group underwent their first DR screening during this hospitalization and were found to have retinal lesions.

Baseline Comparison of the Development Dataset and External DatasetAccording to the results of the Lasso regression, the following ten variables were selected: duration of diabetes, diabetic retinopathy, diabetic nephropathy,hyperlipidemia, annual DR screening, first DR screening,systolic blood pressure, and levels of postprandial blood glucose, serum creatinine, and triglycerides. Considering that few primary medical institutions will check for diabetic neuropathy during routine physical examinations, most users of this DR screening tool may not know whether they have diabetic neuropathy; therefore, we deleted this item.Meanwhile, we added two basic information: sex and age.Finally, 11 variables were selected to build the prediction model. The results of the baseline comparison of the development and external datasets are presented in Table 2.

Model TrainingA Logistic regression model was built on the training dataset, which consisted of 270 patients with DR and 247 patients without DR. The Hosmer-Lemeshow test showedgood model fitness (P=0.157). The odds ratios (OR) for each variable are presented in Table 3. Diabetes duration (OR=1.05,95%CI: 1.01-1.10,P＜0.01), diabetic nephropathy (OR=5.83,95%CI: 3.24-10.97,P＜0.01), and serum creatinine level(OR=1.00, 95%CI:1.00-1.01,P=0.02) were independent risk factors for DR, and annual DR screening (OR=0.34, 95%CI:0.17-0.65,P＜0.01) and hyperlipidemia (OR=0.62, 95%CI:0.39-0.97,P=0.04) were independent protective factors.

Table 3 Results of multivariable Logistic regression analysis

Figure 2 ROC curve of validation set.

Internal and External ValidationFor internal validation,the AUC reached a value of 0.82 (Figure 2). Since this is a screening model, we hoped to identify as many patients with DR as possible without significantly affecting the overall prediction accuracy, that is, to have a high recall ratio. When the probability of DR was ＞0.3, the model judged the sample as a DR sample. External validation was done to assess the generalizability of the model which attained the AUC value of 0.70 (Figure 2). The results of the performance assessment accuracy, recall ratio, F-measure, and AUC are detailed in Table 4.

WeChat Mini ProgramIn this study, we developed a WeChat Mini Program named “Risk calculator for diabetic retinopathy”. The WeChat Mini Program for clinical practice can be widely used by patients with T2DM and medical staff(Figure 3). Users can visit the program on WeChat and use the prediction model by entering 11 clinical indicators. If the calculator determines that the user has DR, it will respond with a message saying, “High risk! Please visit the Ophthalmology Department as early as possible for a fundus examination.” If otherwise, it will feedback with a message that says, “Please follow your doctor’s instructions for regular DR screening.”Meanwhile, we have designed a health education page on DR to improve awareness of DR screening and prevention for patients with diabetes.

DISCUSSION

We developed an innovative and practical WeChat Mini Program for DR screening. This tool required 11 items of easily accessible case information, independent of fundus images, that could be used by patients with diabetes and primary medical institutions without ophthalmologists to determine whether a patient with diabetes should be referred to the ophthalmology department.

Several risk factors were identified in the development of the Logistic regression prediction model: duration of diabetes,diabetic nephropathy, and serum creatinine level, which are consistent with the findings of other studies[10-12]. Here, we included variables related to DR screening. Our results suggest that performing screening regularly has a positive effect on preventing DR occurrence (OR=0.34,P＜0.01). This may bedue to more background knowledge of the disease and better awareness of health management in these patients, as well as greater access to medical advice during the screening.According to the baseline analysis of the development dataset,only 17.48% of the 778 patients with T2DM underwent regular DR screening annually. Before hospitalization, almost half of the patients with diabetes did not undergo a fundus examination after having been diagnosed with T2DM [194(53.2%) patients without DR and 189 (45.8%) patients with DR], thereby failing to meet the ADA guidelines for DR screening[2]. Given the low DR screening rate, doctors have much work to do. Regardless of whether in central hospitals or primary medical institutions, when non-ophthalmologist doctors receive patients with diabetes, it may be a simple and practical way to apply our DR screening tool to conduct a DR risk assessment and provide recommendations for referral in combination with the popular science page of DR.

Table 4 Performance of prediction model in the internal and external validation dataset

One of the main intended use scenarios of this DR prediction tool is prior to a patient’s visit to the ophthalmology department,with the goal of maximizing the screening rate for DR. We assumed that the loss caused by misdiagnosis of DR by this model is less than that by a missed diagnosis when patients do not receive an ophthalmologic evaluation after being diagnosed with diabetes. Therefore, when setting the prediction parameters of the model, we attempted to increase the recall ratio without significantly affecting the overall prediction effect; that is, we attempted to screen as many patients with DR as possible (recall ratio=0.92, AUC=0.82). Nevertheless,our model will still miss some patients with DR, so we have added a disease science page to enhance patients’ awareness of preventing and treating DR. For patients who are predicted to be non-DR by the model, the reminder is “Please follow your doctor’s instructions for regular DR screening.” to increase the DR screening rate further.

Figure 3 The WeChat Mini Program of “Risk calculator for diabetic retinopathy” A: The blank calculator page; B: An example for a 75-year-old female user who suffered from type 2 diabetes for 16y. When she fills in the blanks and clicks Submit, a red message appears at the bottom of the page.

Remarkably, in the external validation dataset where baselines varied significantly, the prediction performance of the model achieved a recall rate of 0.89 and an AUC of 0.70. We believe that the generalization ability is acceptable. However, we advocate that every medical institution should be able to develop its own predictive model. The development of disease prediction models is not a one-time task. Even when applied only to the medical institution where the model is being developed, parameters need to be tuned regularly. Over time,the model should be assessed by updating it[13]. Since changes in Medicare policy, therapy, the comprehensive strength of these hospitals, follow-up schedule, the COVID-19 pandemic, among others, could affect the patient population and further differently affect the weight of each risk factor. Thus, the validation of a prediction model does not provide an absolute result, and it should be a dynamic process evolving over time[14].

A number of previous studies have combined DR prediction models with practical clinical tools. Histograms and scoring scales were common (Table 5)[15-19]. Users need to manually calculate the risk of disease, which may not be easy for patients to master using the method, with a probability of error and low efficiency. The offline or paper scale and histogram have a low promotion efficiency and are not easy to update in sync with the model. Online clinical tools, such as a study on the prediction of progression of chronic kidney disease(CKD)[20], can address these questions. Researchers designed and developed a CKD prediction system embedded in a web tool for the model they trained. Clinicians can visit the system website and enter the features from patients with CKD to obtain feedback of “mild” or “moderate/severe”[20]. The online tool can be accessed by anyone with an Internet connection and is user-friendly with its simple operation patterns. The computer calculates the results directly, with low error rates.Similarly, we developed a DR prediction program using the WeChat as the carrier. Patients and clinicians can access and bookmark this mini program while using this mainstream social software, which makes it more convenient to use and promote.

Table 5 Comparison with previous clinical tools

To our knowledge, this is the first study on the development of DR screening tool using social software. The main strength of this study is its practical way of having developed a DR prediction system to facilitate patient self-health management and assist non-ophthalmologists in referral-making decisions.Additionally, the factors used in the prediction model are all routine items in the health examination, which are easy to popularize in primary medical institutions. Most importantly,we collected data from another hospital and performed external validation to evaluate the tool’s generalization ability.

Admittedly, this study has some limitations. The sample size of the model development dataset was not sufficiently large for a better prediction performance, and the parameters of the prediction model need to be further optimized. Moreover,it is difficult to collect external verification data, so multiple external validations were not performed.

In Conclusion, annual DR screening has a positive effect on preventing DR occurrence. The online DR prediction clinical tool, which integrates disease education, risk prediction, and medical advice functions, could assist in making medical decisions and provide a reference for studies on patient management.

ACKNOWLEDGEMENTS

Conflicts of Interest:Li WY,None;Yang M,None;Song YN,None;Luo L,None;Nie C,None;Zhang MN,None.