Advertisement

Validation of a microsurgical video-based scoring system for evaluating corneal suturing performance

Published:September 23, 2022DOI:https://doi.org/10.1016/j.jcjo.2022.08.014
      When the SARS-CoV-2 pandemic began in 2020, there was major disruption of elective surgical procedures.
      • Felfeli T
      • Ximenes R
      • Naimark DMJ
      • et al.
      The ophthalmic surgical backlog associated with the COVID-19 pandemic: a population-based and microsimulation modelling study.
      As surgical volume decreased and personnel allowed in the operating room were limited, procedural learning opportunities for ophthalmology trainees were similarly restricted.
      • Pasricha ND
      • Haq Z
      • Ahmad TR
      • et al.
      Remote corneal suturing wet lab: microsurgical education during the COVID-19 pandemic.
      Previous studies have used surgical simulation to augment traditional surgical education, ameliorating the effects of reduced case volume on resident surgical skills.
      • Lee AG
      • Greenlee E
      • Oetting TA
      • et al.
      The Iowa Ophthalmology Wet Laboratory curriculum for teaching and assessing cataract surgical competency.
      Simulation also could allow for remote assessment, input from ophthalmologists in other centres, and reduced costs.
      • Pasricha ND
      • Haq Z
      • Ahmad TR
      • et al.
      Remote corneal suturing wet lab: microsurgical education during the COVID-19 pandemic.
      In this prospective cohort pilot study, our objective was to validate the use of a video-based Objective Structured Assessment of Technical Skills (OSATS)–style rubric for grading corneal suturing performance. The OSATS rubric is a validated checklist used for the evaluation of technical skills.
      • Hatala R
      • Cook DA
      • Brydges R
      • Hawkins R.
      Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence.
      A version was specifically adapted for this study, in which 3 evaluation parameters that were less relevant for the corneal suturing task were removed
      • Hatala R
      • Cook DA
      • Brydges R
      • Hawkins R.
      Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence.
      (Table 1).
      Table 1Objective Structured Assessment of Technical Skills (OSATS)–style rubric.
      Global rating scale of surgical performance

      Please circle the number in each row/category corresponding to the participant's performance (video recording).
      Respect for tissue1. Frequently used unnecessary force on tissue or caused damage by inappropriate use of instruments.2.3. Careful handling of tissue but occasionally caused inadvertent damage.4.5. Consistently handled tissue appropriately with minimal damage.
      Time and motion1. Many unnecessary moves.2.3. Efficient time/motion but some unnecessary moves.4.5. Fluid moves with instruments and no awkwardness.
      Instrument handling1. Repeatedly makes tentative or awkward moves with instruments.2.3. Competent use of instruments, although occasionally appeared stiff or awkward.4.5. Fluid moves with instruments and no awkwardness.
      Flow of operation and forward planning1. Frequently stopped operating or needed to pause before next move.2.3. Demonstrated ability for forward planning with steady progression of operative procedure.4.5. Obviously planned course of operation with effortless flow from one move to the next.
      TOTAL: /20
      Institutional ethics approval was obtained from the Health Sciences and Affiliated Teaching Hospitals Research Ethics Board at Queen's University (ID no. 6030891). Ophthalmology residents from all postgraduate years (1–5) were recruited, each with varying levels of surgical experience. Participants completed a surgical skill test consisting of a corneal suturing task.
      • Pasricha ND
      • Haq Z
      • Ahmad TR
      • et al.
      Remote corneal suturing wet lab: microsurgical education during the COVID-19 pandemic.
      ,
      • Lee WB
      • Mannis MJ.
      Ophthalmic microsurgical suturing techniques.
      Each resident was evaluated over 2 separate attempts that were 3 weeks apart. Assessment at 2 time points allowed for demonstration of surgical skill improvement using this evaluation technique to increase our total sample size. The sessions were scheduled to ensure that there were no additional surgical simulation skills practice or teaching in between. Participants watched a brief instructional video (Appendix A, available online) outlining the task and assessment criteria prior to task execution. Residents were provided a precut centred penetrating keratoplasty graft from a cadaver eye and were instructed to place 4 cardinal sutures. All corneal tissue samples were prepared by the same research team member (R.C.) to ensure consistency. Sessions were recorded in the surgical simulation wet lab using an Echo360 device (Echo360, Reston, Va.) connected to a surgical microscope. Each anonymized video was cut to eliminate dead space, randomly assigned a label, and submitted to each grader for scoring. The videos were independently and blindly marked by 2 faculty ophthalmologists, a cornea specialist and a noncornea specialist, using the OSATS-style rubric. Graders compared rating calibration after the first 5 videos to identify any possible differences in rubric usage; none were identified. IBM SPSS Statistics 27.0 (IBM, Armonk, NY) was used to analyze the OSATS scores using paired t tests that were validated with the Wilcoxon signed-rank test for statistical comparisons within the group, whereas interrater reliability for the OSATS-style rubric was assessed using intraclass correlations (ICCs).
      Eight ophthalmology residents completed the corneal suturing task twice, yielding 16 video sessions. The overall ICC for the 2 independent graders was 0.916 (95% CI, 0.701–0.979; p < 0.001) for the first session and 0.870 (95% CI, 0.242–0.952; p = 0.006) for the second session. An ICC from 0.75 to 0.9 is considered good, whereas an ICC >0.9 is considered excellent.
      • Koo TK
      • Li MY.
      A guideline of selecting and reporting intraclass correlation coefficients for reliability research.
      Each individual OSATS rubric field was within these ranges, and most fields were >0.9.
      • Koo TK
      • Li MY.
      A guideline of selecting and reporting intraclass correlation coefficients for reliability research.
      Notably, all the mean surgical scores increased from the first to second session (Fig. 1). After the second session, respect for tissue score improved by 1.00 ± 2.00 (p = 0.200), time and motion score improved by 0.75 ± 1.2 (p = 0.119), instrument handling score improved by 0.87 ± 1.22 (p = 0.082), flow of operation and forward planning scored improved by 0.44 ± 1.44 (p = 0.500), and total surgical score improved by 3.06 ± 5.91 (p = 0.186).
      Fig 1
      Fig. 1Mean surgical skill grades as assessed by Objective Structured Assessment of Technical Skills (OSATS)–style rubric after sessions 1 and 2.
      In conclusion, we achieved our primary objective of showing an adapted OSATS rubric for video-based assessment as a reliable tool for evaluating ophthalmology resident corneal suturing skills with excellent interrater reliability. Interestingly, we also showed that even over 2 time points, surgical skills improvement can be captured using video rubric grading. Modified rubric grading of video-based simulation surgery may be applied to other microsurgical skills with the advantage of remote evaluation and multiple expert evaluators.

      Footnotes and Disclosure

      The authors have no proprietary or commercial interest in any materials discussed in this article.

      Appendix. Supplementary materials

      References

        • Felfeli T
        • Ximenes R
        • Naimark DMJ
        • et al.
        The ophthalmic surgical backlog associated with the COVID-19 pandemic: a population-based and microsimulation modelling study.
        CMAJ Open. 2021; 9: E1063-E1072
        • Pasricha ND
        • Haq Z
        • Ahmad TR
        • et al.
        Remote corneal suturing wet lab: microsurgical education during the COVID-19 pandemic.
        J Cataract Refract Surg. 2020; 46: 1667-1673
        • Lee AG
        • Greenlee E
        • Oetting TA
        • et al.
        The Iowa Ophthalmology Wet Laboratory curriculum for teaching and assessing cataract surgical competency.
        Ophthalmology. 2007; 114: e21-e26
        • Hatala R
        • Cook DA
        • Brydges R
        • Hawkins R.
        Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence.
        Adv Heal Sci Educ. 2015; 20: 1149-1175
        • Lee WB
        • Mannis MJ.
        Ophthalmic microsurgical suturing techniques.
        Springer, Berlin2007
        • Koo TK
        • Li MY.
        A guideline of selecting and reporting intraclass correlation coefficients for reliability research.
        J Chiropr Med. 2016; 15: 155-163