key: cord-0700399-r9gz9rhj
authors: Radi, Imad; Tellez, Juan C.; Alterio, Rodrigo E.; Scott, Daniel J.; Sankaranarayanan, Ganesh; Nagaraj, Madhuri B.; Hogg, Melissa E.; Zeh, Herbert J.; Polanco, Patricio M.
title: Feasibility, effectiveness and transferability of a novel mastery-based virtual reality robotic training platform for general surgery residents
date: 2022-02-22
journal: Surg Endosc
DOI: 10.1007/s00464-022-09106-z
sha: ee9c472fa1da9bf5c20babad7319a04f19cf8405
doc_id: 700399
cord_uid: r9gz9rhj

BACKGROUND: The annual number of robotic surgical procedures is on the rise. Robotic surgery requires unique skills compared to other surgical approaches. Simulation allows basic robot skill acquisition and enhances patient safety. The purpose of this study was to evaluate the feasibility, effectiveness, and transferability of a mastery-based curriculum using a new virtual reality (VR) robotic simulator for surgery resident training. METHODS: Nineteen PGY2s and 22 PGY4s were enrolled. Residents completed a pretest and posttest consisting of five VR and three previously validated inanimate tasks. Training included practicing 33 VR tasks until a total score ≥ 90% (“mastery”) was achieved using automated metrics (time, economy of motion). Inanimate performance was evaluated by two trained, blinded raters using video review metrics (time, errors, and modified OSATS). Outcomes were defined as: curriculum feasibility (completion rate, training time, repetitions), training effectiveness (pre/post training skill improvement), and skill transferability (skill transfer to validated inanimate drills). Wilcoxon signed-rank and Mann–Whitney U tests were used; median (IQR) reported. RESULTS: Thirty-four of 41 residents (83%) achieved mastery on all 33 VR tasks; median training time was 7 h (IQR: 5′26″–8′52″). Pretest vs. post-test performance improved (all p < 0.001) according to all VR and Inanimate metrics for both PGY2 and PGY4 residents. Significant pretest performance differences were observed between PGY2 and PGY4 residents for VR but not inanimate tasks; no PGY2 vs. PGY4 posttest performance differences were observed for both VR and inanimate tasks. CONCLUSION: This mastery-based VR curriculum was associated with a high completion rate and excellent feasibility. Significant performance improvements were noted for both the VR and inanimate tasks, supporting training effectiveness and skill transferability. Additional studies examining validity evidence may help further refine this curriculum.

training [16] [17] [18] [19] . Recently, Intuitive Surgical introduced a new VR robotic simulator called SimNow® (Intuitive Surgical Operations, Inc.; Sunnyvale, California) which uses the da Vinci Xi surgeon console and computer software to foster basic skill acquisition on 33 drills. Features include modular curriculum design, automated performance metrics, and customizable benchmarks for each drill. Data supporting the use of this new platform are limited and published studies are lacking. Using the SimNow simulator, our team designed a mastery-based curriculum for surgery resident training and implemented training over two academic years. The purpose of this study was to evaluate outcomes including curriculum feasibility, training effectiveness, and skill transferability.

The new VR curriculum was implemented over two academic years (2019-2021) at the University of Texas Southwestern (UTSW) Medical Center for training surgery residents in basic robotic skills. A total of 41 residents were enrolled, including PGY2 (n = 19) and PGY4 (n = 22) residents. Data were collected prospectively and analyzed retrospectively under an exempt study approved by the UTSW Institutional Review Board. Training and testing were performed in the UTSW Simulation Center using the SimNow® simulation software accessed through a Da Vinci Xi surgeon console (VR tasks) and a Da Vinci Xi robotic system (inanimate tasks).

The UTSW curriculum was based on a previous curriculum implemented for surgical oncology fellows at the University of Pittsburgh Medical Center (UPMC) by Hogg, et al. that used a similar but older VR simulator platform (Backpack®, Intuitive Surgical Operations, Inc.; Sunnyvale, California) [15] . The UPMC curriculum also included three inanimate drills to evaluate transferability of skills acquired on the VR simulator to a real environment. As previously described, these include: Ring Rollercoaster (RRC), Around the World Needle Driving (ATW), and Interrupted Suture (IS) (Fig. 1) . Similar to the UPMC curriculum, the UTSW curriculum used a combination of VR and inanimate tasks and a pretest, training, and posttest design. Pretest and posttest consisted of five SimNow® VR and three UPMC inanimate tasks. The VR tasks (Fig. 2) were Around the World Needle Driving (vATW), Big Dipper Needle Driving (BD), Ring Roller Coaster 4 (RRC4), Knot Tying (KT), and Three-Arm Relay 3 (TAR3). The first 4 tasks were chosen due to similarity with the inanimate tasks. RRC4 and vATW are identical to RRC and ATW, respectively, and when taken together, BD and KT incorporate the movements required to complete IS. TAR3 was chosen because it was perceived to be a complex task that would likely discriminate improvement between pretest and posttest. Training included practicing all 33 SimNow® VR tasks until a composite score equal to or exceeding 90 (out of 100 maximum) was achieved (defined as mastery). Although residents were encouraged to reach mastery for all 33 tasks, they were not required to do so in order to posttest. The order of task completion was not stipulated.

Protected time was made available during a month-long rotation with relatively light clinical duties for PGY2 and PGY4 residents. Residents were encouraged to complete the curriculum at their own pace within that month. This design was intended to stagger trainees to maximize access to the VR simulator and facilitate completion of the curriculum. Before beginning the curriculum, each resident completed an online training module through the da Vinci Surgery Community website designed to orient trainees to the robotic console and a pre-curriculum survey which captured basic demographic information, past robotic experience, and attitudes toward RAS.

Performance on the VR tasks was evaluated using metrics automatically generated by the SimNow® software; these metrics included composite score, completion time, and economy of motion (the distance the instruments traveled to complete the task). Performance data were downloaded from the da Vinci Surgery Community website. Inanimate tasks were video recorded to facilitate subsequent assessment by video review. Performance was evaluated according to completion time, errors, and a modified Objective Structured Assessment of Technical Skills (OSATS) similar to the UPMC methodology [15] . This tool has been used in the robotic inanimate setting by some of the authors and has been validated in different fashions previously [15, [20] [21] [22] . Moreover, the OSATS confers the advantage of evaluating the time component as well as the economy of motion and instrument handling. These metrics could be similarly assessed in the VR training as well as in the OR. Consequently, in order to be consistent in our work and to allow future comparison between different training and practicing environments, we decided to use the same grading tool that has been effectively used in the past [23] . The OSATS scale involved a combination of 5-point Likert scales in six categories for a maximum of 30 points; categories included respect for tissue, time and motion, instrument handling, knowledge of instruments, use of assistance, and knowledge of procedure. These metrics were assessed by two blinded graders who received specific training in OSATS evaluation. At the end of their training both graders independently scored 10 inanimate drills and achieved high interrater reliability (Spearman: ρ = 0.84, p < 0.001).

For each of the 33 tasks in the curriculum, descriptive statistics were used to analyze attempts and time to mastery. The VR task "KT" was removed from the analysis of the pretest and posttest due to the likelihood of committing a critical error that ended the drill early, resulting on pretest in an appropriately low score but misleadingly low time and economy of motion. Pretest and posttest performance metrics were examined for normality by constructing histograms and performing Wilks-Shapiro test. Improvement between pretest and posttest was analyzed by summating each metric, across the four remaining VR tasks, and across the 3 inanimate tasks, and comparing metric totals for the pretest and posttest (total score, total time, total economy of motion, total errors). Paired t test was used to analyze improvement of the entire cohort between pretest and posttest. Wilcoxon matched signed-rank test was used to separately analyze improvement of PGY2s and PGY4s between pretest and posttest. Comparisons of performance between PGY2s and PGY4s at pretest and posttest were made using Mann-Whitney U test. Interrater reliability for the graders reviewing the inanimate drills was assessed with Spearman Rank correlation. All tests employed 2 tails and p < 0.05 was considered to be significant. 

From the 41 residents enrolled, pre-curriculum surveys were collected from 32 (78%). Twenty-seven residents (84%), including 14 PGY2s and 13 PGY4s, reported prior robotic simulation experience with VR or inanimate drills with a median of 3.75 h of prior practice. Self-reported simulation experience did not correlate with performance in any metric on the VR or inanimate portions of the pretest or the time to achieve mastery on the 33 tasks in the curriculum.

Thirty-four residents (83%) achieved mastery on all 33 VR tasks. The number of days required to complete the curriculum, defined as the time between the pretest and posttest, ranged from 14 days to 10 months with a median of 1 month (IQR: 0.7-4.8 months). Thirteen residents (32%) required longer than 4 months to complete the curriculum. The console time required to achieve mastery on each task ranged from 2 h and 26 min to 13 h and 24 min, with a median of 6 h and 55 min (IQR: 5′26″-8′52″). Table 1 shows a breakdown of the curriculum by task with the percentage of residents that achieved mastery on each task along with the associated number of repetitions and training time. Figure 3 shows a comparison of pretest and posttest performance on the SimNow® tasks for the entire group, the PGY2s alone, and the PGY4s alone. Statistically significant differences were observed on the pretest between PGY2s and PGY4s in total score (p = 0.0036) and total time (p = 0.0027). On the posttest, however, PGY2 performance was not significantly different from PGY4 performance for both total score (p = 0.535) and total time (p = 0.562). Lower median total economy of motion was observed in the PGY4s relative to the PGY2s on the pretest (p = 0.0538), but this difference did not reach statistical significance. No difference was found in median total economy of motion between PGY2s and PGY4s on the posttest (p = 0.1461). Figure 4 shows a comparison of PGY2 and PGY4 performance on the SimNow® tasks at the pretest and posttest.

High interrater reliability was observed between the two graders in both errors (Spearman: ρ = 0.642, p < 0.001) and OSATS (Spearman: ρ = 0.614, p < 0.001), so averages of the two graders' scores were used for analysis of the inanimate drills. Similar to the VR tasks, improved performance was observed on the posttest on each inanimate task according to all metrics when all trainees were considered as a group (all p < 0.005). When considering only the PGY2s, improved performance was observed on each inanimate task according to all metrics (all p < 0.043). For the PGY2s, median total time decreased from 17.5 min (IQR: 14.9-19.8 min) to 12.6 min (IQR: 11.0-13.8 min) (p < 0.001), median total errors decreased from 5.5 (IQR: 3.5-7.5) to 2 (IQR: 1.5-2.5) (p < 0.001), and median total OSATS increased from 61 (IQR: 53-66) to 75 (IQR: 70.5-80.5) (p < 0.001). When considering only the PGY4s, improved performance was observed on each inanimate task according to Figure 5 shows a comparison of pretest and posttest performance on the inanimate tasks for the entire group, the PGY2s alone, and the PGY4s alone. Statistically significant differences were observed between the PGY2s and PGY4s on the pretest for median total time (p = 0.006), but no differences were observed between the two groups on the posttest. There were no significant differences observed between the PGY2s and PGY4s on either the pretest or the posttest for total errors (pretest: p = 0.531; posttest: p = 0.433) or total OSATS (pretest: p = 0.433; posttest: p = 0.637). Figure 6 shows a comparison of PGY2 and PGY4 performance on the inanimate tasks at pretest and posttest.

Moving the initial phases of robotic surgical skills training from the operating room to the simulation center is beneficial to shorten the learning curve, minimize OR times and cost, and enhance patient safety. Our simulation center is equipped with a dual-console Xi System and the SimNow® VR platform. We designed our curriculum to use the VR system in an effort to allow surgery residents to acquire basic robotic skills without needing to use actual robotic equipment, which is associated with additional costs for instruments and supplies. Publications describing curricula using the SimNow® system are lacking; we therefore, opted to use methodology previously described for a curriculum using a similar VR system and inanimate transferability tasks [15] . Our study demonstrated that PGY2 and PGY4 residents with minimal RAS exposure can effectively train on the SimNow® VR platform to improve their skills in both the virtual and inanimate environments in a reasonable amount of time. This transferability of skills from a virtual to a real environment confirms the positive implications of the use of VR simulation as a major initial component of robotic surgical training.

Eighty-three percent of the residents were able to achieve mastery on every task in the curriculum. This is a relatively high completion rate. Hogg et al. enrolled 17 surgical oncology fellows to undergo a similar VR training curriculum with 94% of completion rate, yet only 24% were able to achieve mastery on every task. These results were attributed to the fact that achieving proficiency was not mandatory in order to proceed with the post-test [15] . Another trial by Kiely et al. enrolled 14 residents and attendings to complete a dVss VR training curriculum but had a completion rate of 36%. This low completion rate was attributed to the short training times available for some of the participants [24] . Several factors likely were responsible for our high curriculum completion rate. Having dedicated training equipment that was accessible 24/7 in the simulation center made access readily available without using clinical equipment. Having protected training time as well as a structured curriculum with clear expectations seemed pivotal as well. Indeed, most residents completed the curriculum in approximately one month, which corresponded to the duration of the clinical rotation we selected for scheduling of this training due to its relatively light clinical demands. However, 31% of residents required more than 4 months to complete the curriculum; but many of these residents either had their training interrupted by a mandatory shutdown of the simulation center due to the COVID-19 pandemic or elected to begin the curriculum before their designated rotation with protected time. Median completion time was 7 h. Considering all these aspects, our data support feasibility of this VR curriculum.

Our data showed that the SimNow® training platform was effective at improving performance in the VR environment. Statistically significant improvements from baseline were observed at the posttest in all metrics on each task. Median total score on all 4 VR drills increased by more than 4 times (83 to 353), median total time halved (26 to 13 min), and median total economy of motion considerably decreased (2250 to 1400 cm). These results support the use of a proficiency-based curriculum that challenges trainees to reach a certain level of expertise rather than to simply perform a pre-determined number of repetitions. Another advantage of this platform is that it gives residents the opportunity to practice using the actual da Vinci Xi® surgeon's console, introducing them to the unique ergonomics of this robotic system, and preparing them to use the same system in real environments.

Importantly, we found that skills acquired through VR training were transferable to the inanimate environment. This finding has been previously described for prior robotic VR platforms [25, 26] but no studies had examined transferability for the SimNow® platform. After performing VR training, residents showed major improvements in total time, total errors, and total OSATS on the inanimate drills; these improvements were demonstrated when analyzed as one cohort as well as when analyzed as individual classes. While the PGY4s had significantly lower completion time compared to PGY2s on the pretest, both groups demonstrated improvement, and the PGY difference was not present on the posttest. Specific analysis of PGY2 performance showed significant improvement for all metrics on all three inanimate tasks. Similar results were found for the PGY4s, except for a nonsignificant improvement in time for RRC. The fact that both junior and senior residents had a comparable performance at completion of the VR curriculum suggests that skill acquisition was independent of the previous level of clinical training. These data further This study has several limitations. First, this study was a retrospective review of data collected for quality improvement purposes and did not follow a prospective experimental design. Nevertheless, our study involved a larger number of participants than previously published studies and represents new data for this simulator; additionally, many of our findings were consistent with results associated with other VR systems [15, [27] [28] [29] [30] . Second, our curriculum used automated metrics produced by the VR system but these metrics have not yet been investigated for validity evidence. Given the positive results of our initial experience, we intend to pursue such studies. Moreover, the passing score used was a pre-defined benchmark of overall score > 90%, established by the manufacturer, Sim-Now® (Intuitive Surgical Operations, Inc.; Sunnyvale, California). There isn't previously reported data or publicly available information regarding these passing thresholds. This value does not seem to be equivalent and consistent for all tasks. Consequently, our team will be working in the future to define the content and construct validity of each individual VR drill. Third, transferability of skills was studied on the inanimate environment and might not correlate to the real operating room environment. Future endeavors in this direction must be taken to understand how VR simulation improves trainees' skills while operating. Lastly, this was a single-institution study and replication of these results at other institutions will be important to establish generalizability of our findings.

In conclusion, this is the first study to evaluate feasibility, effectiveness, and transferability of the new SimNow® VR platform. Our findings documented that completion of the mastery-based VR training curriculum using this platform was feasible for a large majority of learners in a reasonable amount of time and was effective in significantly improving robotic skills that were transferable to a real robotic environment. Additional validation studies may allow further refinement of this VR robotic curriculum.

Trends in the adoption of robotic surgery for common surgical procedures

General surgery residents' perception of robot-assisted procedures during surgical training

Systematic review of learning curves in robot-assisted surgery

robotic pancreatic resections: safety and feasibility

Laparoscopic skills training and assessment

Practicing on the advanced training in laparoscopic suturing curriculum (ATLAS): is mastery learning in residency feasible to achieve expert-level performance in laparoscopic suturing?

Introduction of a comprehensive training curriculum in laparoscopic surgery for medical students: a randomized trial

Intensive laparoscopic training course for surgical residents: program description, initial results, and requirements

Certification pass rate of 100% for fundamentals of laparoscopic surgery skills after proficiency-based training

Initial laparoscopic basic skills training shortens the learning curve of laparoscopic suturing and is cost-effective

Comprehensive proficiency-based inanimate training for robotic surgery: reliability, feasibility, and educational benefit

Content and face validity of a comprehensive robotic skills training program for general surgery, urology, and gynecology

Proficiency-based training for robotic surgery: construct validity, workload, and expert levels for nine inanimate exercises

Developing a comprehensive, proficiencybased training program for robotic surgery

Mastery-Based Virtual Reality Robotic Simulation Curriculum: The First Step Toward Operative Robotic Proficiency

Face, content and construct validity of a virtual reality simulator for robotic surgery

Headto-Head Comparison of Three Virtual-Reality Robotic Surgery Simulators

Validation study of a virtual reality robotic simulator-role as an assessment tool?

Assessment of validity evidence for the RobotiX robot assisted surgery simulator on advanced suturing tasks

Robotic pancreatoduodenectomy biotissue curriculum has validity and improves technical performance for surgical oncology fellows

Validity and reliability of the robotic objective structured assessment of technical skills

Objective structured assessment of technical skill (OSATS) for surgical residents

Grading of surgeon technical performance predicts postoperative pancreatic fistula for pancreaticoduodenectomy independent of patient-related variables

Virtual reality robotic surgery simulation curriculum to teach robotic suturing: a randomized controlled trial

Correlation of virtual reality simulation and dry lab robotic technical skills

Utilization and surgical skill transferability of the simulator robot to the clinical robot for urology surgery

Virtual reality training improves da Vinci performance: a prospective trial

Robotic surgery basic skills training: evaluation of a pilot multidisciplinary simulation-based curriculum

Development and testing of a robotic surgical training curriculum for novice surgeons

Transferability of virtual reality, simulation-based, robotic suturing skills to a live porcine model in novice surgeons: a single-blind randomized controlled trial

The authors gratefully acknowledge support provided by the UT Southwestern Simulation Center.

Disclosures Drs. Imad Radi, Rodrigo Alterio, Daniel Scott, Ganesh Sankaranarayanan, Madhuri Nagaraj, Melissa Hogg, Herbert Zeh, Patricio Polanco, and Mr. Juan Tellez have no conflict of interest or financial ties to disclose.