Validity and reproducibility of ophthalmologist photo grading of diabetic retinopathy and glaucoma

Screening for diabetic retinopathy (DR) is an important component of diabetes management to prevent vision loss, yet many diabetic patients do not receive screening eye examinations. Tele-ophthalmology programs administering fundus photographs offer a potential solution, and technological advances have made automated photo-interpretation a real possibility. Automated algorithms are typically trained on images whose disease status has been classified by a human, so the validity of human grading is important. And yet, although the variability of ophthalmologist grading for glaucoma has been described previously, limited information exists for DR.^{,  ,}  This study attempts to fill that gap in order to better inform optimal methods of human grading for use in automated algorithms.

In this cross-sectional diagnostic accuracy study, a convenience sample of diabetic patients were recruited from the ophthalmology clinic at Chiang Mai University, Thailand. After pupillary dilation, the fundus of each eye was examined by an ophthalmologist, who noted the presence of DR according to the International Clinical Diabetic Retinopathy Severity Scale as well as the vertical cup-to-disk ratio (VCDR). Each eye was then photographed (TRC-NW6S, Topcon, Tokyo, Japan; manufacturer's software used to create an 85° montage image from photographs captured at 9 fixed locations using a repositionable internal fixation light). The mosaic image was graded for the presence of DR according to the International Clinical Diabetic Retinopathy Severity Scale and for the VCDR by 5 ophthalmologists who were masked to each other's grades and to all participant identifiers. Although mosaic images are not considered the gold standard for DR or glaucoma screening, they have been shown to have sensitivities equivalent to in-person ophthalmology examination. Glaucoma suspect was defined as a VCDR ≥0.6. Cohen's kappa statistic was used to assess agreement between photo grades and clinical examination and also agreement between the 5 photo graders, using bootstrapped 95% confidence intervals (CIs) resampled at the participant level to account for correlation of eyes from the same person (N = 999 replications). Ethics approval was obtained from University of California, San Francisco, and Chiang Mai University.

In total, 235 eyes from 119 participants completed an ophthalmologist examination and fundus photography. Of these, 222 photo montages were judged to have adequate image clarity and coverage of the optic disk and macula by a majority of photo graders and were included in the analysis. On the reference standard ophthalmologist examination, 19 of 222 eyes were assessed as having a VCDR ≥0.6, and 97 eyes were diagnosed with DR—43 (19%) with nonproliferative DR and 54 (24%) with proliferative DR. The validity of photo grading is summarized in Table 1, which shows that agreement between each photo grader and the reference standard was greater for DR than VCDR, and that creating a majority consensus grade improved the agreement much more for VCDR than for DR assessment. The precision of photo grading is summarized in Table 2, which shows that inter-rater reproducibility between the 5 photo graders was significantly higher for DR (κ = 0.75, 95% CI 0.70–0.82) than glaucoma (κ = 0.40, 95% CI 0.28–0.52) (difference 0.35 higher for DR, 95% CI 0.21–0.50).

In this study, inter-rater reproducibility was higher for DR assessment compared with VCDR assessment, and photo grades for DR were a more valid indicator of disease (i.e., better agreement with reference standard). The study was conducted using a single digital camera, and so it is unclear whether these results are generalizable to lower-resolution imaging systems such as the ultra-widefield scanning ophthalmoscope. Nonetheless, these results suggest that automated algorithms for DR could be trained on fundus photographs graded by even a single person, whereas algorithms for VCDR would benefit from grading based on a consensus of multiple graders.