"Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has."

Margaret Mead
Review article
peer-reviewed

## Skill Acquisition, Assessment, and Simulation in Minimal Access Surgery: An Evolution of Technical Training in Surgery

### Abstract

Diminishing resources and expanding technologies, such as minimal access surgery, have complicated the acquisition and assessment of technical skills in surgical training programs. However, these challenges have been met with both innovation and an evolution in our understanding of how learners develop technical competence and how to better measure it. As these skills continue to grow in breadth and complexity, so too must the surgical education systems’ ability. This literature review examines and describes the pressures placed on surgical education programs and the development of methods to ameliorate them with a focus on surgical simulation.

### Introduction & Background

Many pressures have complicated the teaching and assessment of technical skills in postgraduate surgery over the previous decade. A major contributor was the introduction and development of advanced minimal access surgery (MAS). MAS combined with reduced resident work hours, increased costs of operating room time, increased public awareness and the ethics of learning basic skills about patients have further compounded these challenges. These challenges have led to a re-evaluation of how surgical skills are taught and assessed in trainees. Certainly the Halstedian [1] approach to surgical teaching of, “see one, do one, teach one” is no longer favored. This has fueled investigation into the use of skills laboratories and surgical simulation as methods of training and assessing residents.

The aviation industry has a long history of utilizing simulation for both training and assessment purposes [2-3]. Krummel defined simulation as “a device or exercise that enables the participant to reproduce or represent, under test conditions, phenomena that are likely to occur in actual performance” [4]. Simulation represents a simplified reality that need not include every possible detail or reproduce anatomy with high fidelity (i.e., true to real life) to be effective. Simulated training and assessment environments allow for practice in a realistic setting without the inherent risk of harming others or oneself. This is particularly important when the target performance involves high-stake potentially life-threatening situations, such as with surgery.

Indeed, there are several aspects and evolving challenges that surgical educators must be familiar with in order to facilitate an effective learning environment for technical skills acquisition and assessment in postgraduate surgical education.

### Review

#### Minimal access surgery

With the introduction of minimal access surgery (MAS) in the 1980s, patients experienced the benefits of smaller incisions, shorter hospital stays, and decreased postoperative pain. These techniques are now common in many surgical specialties such as orthopedics (arthroscopy), thoracics (thoracoscopy), and in general, gynecologic surgery, and urologic surgery (laparoscopy). However, MAS is not without risk and potential serious complications. The skills required to perform this surgery are unique and different than those of traditional ‘open’ surgery. MAS uses optical systems that provide monocular vision. This eliminates depth perception so surgeons must depend on other cues such as light and shading to recreate a ‘stereoscopic’ environment [5]. Additionally, the video image is magnified and projected onto a monitor that is not aligned precisely with the surgical target.

Minimal access surgery also uses long instruments and trocars; these amplify movements and tremor, and are more difficult to control than traditional instruments. As the trocars are fixed to the body wall, there is a decreased range of motion. Passing instruments through trocars also result in the fulcrum effect, whereby the surgeon’s hand must be moved 180° opposite the direction of the desired movement of the tip of the instrument [6]. The added length of the instruments also significantly dampens tactile sensation. Surgeons rely heavily on their sense of touch to define tissue planes, pathology, and the resistance required to secure knots. Cues such as touch and the interaction of specific instruments with tissue must be learned to help maintain the necessary tactile feedback [6].

The potential for devastating complications combined with the unique skill set required to perform MAS has prompted the surgical community to reconsider the approach to training and assessment of these skills [7]. The need for the demonstration of technical competence by both surgical trainees and practicing surgeons has been highlighted. Societies and regulatory bodies such as the Society of Gastrointestinal and Endoscopic Surgeons (SAGES) and the European Association of Endoscopic Surgeons (EAES) stipulate minimum requirements for those learning MAS with an emphasis on training both in the operating theatre and in the skills courses [7-9].

#### Work hour restrictions

In 2003, the Accreditation Council for Graduate Medical Education (ACGME), the professional body responsible for accrediting residency-training programs in the USA, implemented an 80-hour resident workweek restriction [10]. In Canada, while there are currently no pan-Canadian limitations on resident duty hours, there are restrictions within each province. Manitoba has a weekly duty hour limit of 89 hours per week. In Quebec, residents’ duty hours are limited to 72 hours per week with a limit of 16 consecutive hours per day [11]. This may be implemented in other jurisdictions in the next few years. These reduced hours combined with the challenging skill set afforded to MAS, make training of skills and demonstration of competency outside of the operating room even more important.

#### Laboratory simulation-based training and assessment

Laboratory simulation-based skills training and assessment for MAS surgical skills include lower fidelity physical box/video trainers and higher fidelity virtual reality trainers. The physical box trainers use actual laparoscopic instruments and an optical system that allows the trainee to perform tasks under videoscopic guidance [3]. Virtual reality trainers use computer-generated instruments to perform computer-generated tasks. There is a spectrum of machines available now with many of them incorporating haptic feedback.

Significant research has been performed in studying the validity and reliability of simulation-based training for MAS skills over the last two decades. Training in lab-based settings results in improvement of skills both in the simulation setting and in the operating room. Simulation is now established as an effective method of skill acquisition [12-23]

Video trainers have been incorporated into the assessment of MAS skills via the fundamentals of laparoscopic surgery (FLS) program. FLS is a simulation-based assessment and certification program developed by SAGES [24]. It has been thoroughly validated [5, 12, 24-27] and is a requirement to be eligible for certification by the American Board of Surgery [24, 28]

Virtual simulators have also been shown to be capable of assessing the psychomotor skills necessary for MAS [19, 29-30], although there is less research in this area than with video trainers.

#### Current assessment of technical skills in surgical programs

A trainee’s technical skills are constantly evaluated throughout residency training [31]. This is most often a subjective assessment performed by practicing staff surgeons as a resident rotates through a particular surgical service [27]. At periodic intervals throughout the rotation, the staff surgeon fills out an in-training evaluation report (ITER) that is used to assess all areas of the trainee’s surgical competence. The ITER is composed of a list of rating scales with sections such as knowledge, professionalism, technical skill, teamwork, and communication skills that the evaluator completes to evaluate the candidate’s clinical competency [3]. The purpose of the ITER is to provide an accurate assessment of the trainee’s performance abilities and useful feedback to both the trainee and the program [32].

The ITER has some advantages as an assessment tool. It can encompass aspects of competency such as professionalism, teamwork, and communication skills that may not otherwise be evaluated. It is inexpensive and no special equipment is required. In addition it allows for multiple observations over time [3].

However, there are criticisms of the ITER as a way to evaluate technical skills. ITERs have been found to have poor validity and limited inter- and intra-rater reliability [3, 31-32]. According to Turnbull et al. this is primarily due to two major factors [32]. First, faculty evaluators often do not receive training in the evaluation process and correct use of the ITER, which results in a number of rater errors. Errors of distribution include a range restriction or central tendency error and a leniency/severity or dove/hawk error. In central tendency errors, evaluators use the mid-range average values and fail to use the whole scale, while in leniency/severity error there is a tendency to assign predominantly low or high scores [33]. Additionally, ITERS may be subject to the halo effect, a correlation error that commonly occurs where the raters’ overall impression of the trainee influences the assessment of each individual component of the evaluation [33]. Second, ITERs are not usually filled out on a daily basis, but rather in a retrospective fashion at the middle or end point of a rotation. Consequently, they are based on a recall of events and are generally not sufficiently detailed or accurate.

Procedural logs record the number and type of procedure that a resident completes throughout their training and are often used as a marker of technical competence [31]. These are a requirement for specialty certification by the American Board of Surgery, where an applicant must demonstrate their participation in a minimum number of cases in the various areas of general surgery [3]. A problem with this is that the number of cases required to achieve competency can vary substantially between individuals and participation in a procedure does not ensure competency [34]. In addition, the reliability of the data recorded can potentially be questionable. Trainees are often in a rush while logging procedures, or do so retrospectively at a time remote from the operation.

Currently, many surgical programs use a combination of ITERs and procedural logs to assess whether or not a trainee’s technical skills are sufficient to progress to the next level in their training.

In-training evaluation reports are summative as opposed to formative measures of assessment. They do not provide residents with a concrete understanding of the particular aspects of their technical skills that need improvement [35]. On the other hand, global rating scales are reliable, have high inter-rater reliability and construct validity [36-37]. Residency programs across the country are trying to incorporate the use of these instruments into the evaluation of their trainee’s technical skills. However, this does not appear to be done routinely in the majority of programs.

#### Objective Structured Assessment of Technical Skills

Martin et al. developed objective structured assessment of technical skills (OSATS) in 1997 as a novel way to more objectively evaluate the technical skills of surgical residents [38]. It is a performance-based examination designed to assess the technical competency of surgical residents that consists of eight 15-minute bench model stations. Each model simulates portions of relevant operative procedures. Examples of these stations include closure of a skin incision, insertion of a T-tube, hand-sewn bowel anastomosis, and control of an inferior vena caval hemorrhage [39]. Two or more observers evaluate each task using a global rating scale with seven dimensions that are each related to some aspect of operative performance. The dimensions assessed include respect for tissue, time and motion, instrument handling, knowledge of instruments, use of assistants, flow of operation, and forward planning and knowledge of specific procedure [38]. Each dimension is graded on a five-point scale with the middle and extreme points anchored by specific descriptors to help clarify scoring [39]. There is also an overall pass/fail judgment for the examiner to make based on their perception of overall performance [38]. In this original study and a subsequent follow up of larger size, the global rating scale was compared to task specific checklists. Both studies demonstrated stronger construct validity and greater reliability for the global rating scale compared to the checklists, and the checklists are no longer used as part of the OSATS scoring assessment.

The OSATS (Figure 1) has been extensively validated and shown to be an effective method for assessment of surgical skills [40]. It does, however, require significant resources and time on the part of faculty evaluators [31]. Over the past 15 years, it has moved from use in the laboratory setting to use in the operating theatre [21, 36-37]. OSATS is validated and used for the evaluation of both open and laparoscopic surgical skills. It is viewed as the standard for skills assessment [39].

#### Global Operative Assessment of Laparoscopic Skills

Originally developed by Vassiliou et al., global operative assessment of laparoscopic skills (GOALS) is a global assessment tool used to assess intraoperative skills in laparoscopy [35]. It is composed of a five-item global rating scale and two visual analog scales (VAS) for overall competence and case difficulty. The five domains examined are depth perception, bimanual dexterity, efficiency, tissue handling, and autonomy [35]. Each domain is scored on a five-point scale for dissection of the vesicular bed during laparoscopic cholecystectomy, with the middle and extreme points anchored by specific descriptors to help clarify scoring [41]. The two VAS are 10 cm lines with descriptive anchors at each end. The first VAS measures how technically difficult the operation is, and the second measures the operator’s overall competence. The evaluator marks the appropriate spot on the VAS with an X [35]

Vassiliou et al. demonstrated construct validity not only for the overall score, but also for each of the five GOALS items individually [35]. Excellent internal consistency and inter-rater reliability were also demonstrated for the five-item global rating scale. Although the VAS did demonstrate construct validity, inter-rater reliability was weak and we feel that it does not add benefit to the five-item global rating scale. It is still included at the bottom of the rating scale; however, most studies published since the original in 2004 do not utilize the VAS.

Fried et al. further validated GOALS as an assessment tool by demonstrating that it has concurrent validity by comparing GOALS scores with assessment of skills using a simulator [12]. In 2007, Gumbs et al. demonstrated construct validity for GOALS for assessment of a complete laparoscopic cholecystectomy and laparoscopic appendectomy [41]. This has added further evidence to the value and effectiveness of GOALS as a global assessment tool for laparoscopic skills.

To date GOALS is the only global rating scale that has been developed and validated specifically for the assessment of laparoscopic surgery (Figure 2).

#### McGill Inanimate System for Training and Evaluation of Laparoscopic Skills

In 1998, Derossis et al. set about developing a method to quantitatively assess technical skills in laparoscopic surgery by measuring performance through a series of exercises performed on a surgical simulator [5]. The result was the McGill inanimate system for training and evaluation of laparoscopic skills (MISTELS) simulation system. The original program consisted of seven standardized tasks performed in a trainer box under videoscopic guidance. The tasks were derived by a review of videotapes of basic and advanced laparoscopic procedures by expert laparoscopic surgeons. The face and content validity of the tasks was confirmed through questioning a large group of 44 expert laparoscopic surgeons [12]. The original seven tasks ranged in difficulty and the skills they aimed at developing. Some of the exercises were chosen to develop basic dexterity skills, whereas others emphasized the use of specialized instruments and particular laparoscopic techniques. The tasks consisted of peg transfer, pattern cutting, clip application, placement of a ligating loop, mesh placement over a defect and intracorporeal and extracorporeal suturing [5]. Since the original publication by Derossis et al., the clipping and mesh placement tasks have been eliminated secondary to a lack of validity and limitations in feasibility and cost [3].

Performance of each task is graded objectively, taking into account both precision of performance and speed. The score for an individual task is calculated by subtracting the time required to complete the task, in seconds, from a preset cutoff time [27]. If the time to complete the task surpassed the preset cutoff time, a score of 0 was assigned. A penalty score for inaccuracy is also deducted according to a predetermined system [5].

The MISTELS system has been further validated since the original study and has been shown to be a highly reliable and valid system [3, 6, 12, 18, 27]. In 2004, Fried et al. demonstrated predictive validity of the MISTELS model by showing that its scores correlate highly with intraoperative measurements of technical skill in laparoscopy (r = 0.81; p < 0.001) [12]. For this, they evaluated a laparoscopic cholecystectomy using a modified version of Vassiliou et al.’s GOALS rating scale [12]. The SAGES have made MISTELS the manual skills component of its FLS program. These skills are the current gold standard for the assessment of laparoscopic skills outside of the operating room.

#### Fundamentals of Laparoscopic Surgery

Fundamentals of laparoscopic surgery is an educational program developed by SAGES. This program was launched in 2004 [25]. The overall goal of the FLS program is to teach and assess the basic cognitive and psychomotor skills required to competently perform laparoscopic surgery [25]

The FLS program consists of a didactic module and manual skills training practicum for education, and an examination component to assess competency. The cognitive portion of the educational program addresses four broad content areas: preoperative, intraoperative, and postoperative considerations and basic laparoscopic procedures [26]. The manual skills training portion of the program is based on the MISTELS program originally developed by Derossis et al. at the McGill University. Competence is assessed through a two-part examination, consisting of a cognitive computer based test and a proctored technical skills component.

The FLS program has been thoroughly tested and validated as a teaching and assessment tool for laparoscopic skills [5, 12, 24-27].  It meets the standards for high stakes examination. As of 2010, the American Board of Surgery requires its candidates to be FLS certified [24, 28].

#### Imperial College Surgical Assessment Device

The Imperial College surgical assessment device (ICSAD) is another method of assessing surgical skills that utilize hand motion analysis.  Sensors are placed on the dorsal surface of the trainee’s hands while they perform a task and the hand motion is tracked. Their movement is translated into a computerized tracing of hand motion by the sensors. This provides an index of technical skill in both laparoscopic and open procedures that has been shown to have good concordance with OSATS scores [42-43]. Although the ICSAD has shown to be a valid indicator of operative skill, the evaluation metrics that are produced do not provide the trainee with feedback that can be used effectively for educational purposes [12, 31].

#### Patients

While there has been a move away from teaching basic skills in the operating theatre, patients remain an integral part of surgical training. An operation can be reduced into its component parts that can be learned and mastered over a number of operations on separate patients. This is best done under the direct guidance and supervision of a staff surgeon [44]. However, there are a multitude of ethical considerations when allowing trainees to operate on patients. This, combined with fiscal restraints, work hour reductions and restricted operating room time, make it important to evaluate other models for their role in assessment of surgical skills.

#### Cadavers

Cadavers have traditionally been used for medical and surgical anatomy sessions throughout training programs [40]. However, preserved cadaveric tissue does not resemble living tissue and it is not a good model for laparoscopic surgery. The tissues are much stiffer, less malleable, and difficult to work with. Depending on the preserving agent, it is extremely difficult or impossible to create a pneumoperitoneum.  Additionally, cadavers are expensive and in limited supply [44-45].

#### Animals

Animal models have been used extensively in medical training in the past. There are moral and ethical issues surrounding the use of animal models. Issues such as: is it absolutely necessary to use the animal model, or could these skills be learned in another way? Are the animals undergoing any harm or suffering as a result? Do the benefits to the learner justify the use of the animal? In addition, the high cost and significant resources required to use these animal models have led many universities and programs away from their use [40].

#### Video trainers

These refer to the low-fidelity physical box trainers or video trainers. These physical box trainers are composed of frames supporting traditional laparoscopic video monitors, light sources, and camera systems [20]. Trainees use actual laparoscopic instruments to perform tasks under videoscopic guidance [3].

#### Comparison of video trainers to virtual simulators

Training in lab-based settings with either virtual reality simulators or video trainers results in improvement of skills both in the simulation setting and in the operating room. Simulation is now established as an effective method of skill acquisition [12-23]

In the first study to validate the transfer of training skills on a virtual reality simulator to the operating room, Seymour et al. demonstrated improved operative performance during laparoscopic cholecystectomy for residents who had trained on the MIST VR simulator [14]. They showed that residents who had trained on the virtual reality simulator made fewer errors and were faster at completing the cholecystectomy than the control group [14]. They did not include a video training group in their study.

In 2000, Scott et al. demonstrated that 30 minutes of daily video training for 10 days improved video-eye hand skills and translated into improved operative performance for junior surgical residents [21]. Second- and third-year general surgery residents were prospectively randomized to either a video training group or a control group. Both groups were evaluated performing a pre- and posttest laparoscopic cholecystectomy using a validated global operative assessment tool by three evaluators blinded to the resident’s randomization status. All residents were also evaluated performing five standardized video trainer tasks at the beginning and the end of the study. The trained group achieved significantly greater improvement in video trainer scores on all five video trainer tasks, and significantly improved on four out of eight criteria on the global assessment scores for the laparoscopic cholecystectomy [21].

The benefit of one simulation method over the other for training purposes has not been clearly established. Hamilton et al. randomized junior general surgery residents to either a virtual reality or a video trainer structured training program, and assessed their baseline skills compared to their post-training skills [20]. There was no statistically significant improvement difference between the two groups, with the virtual reality group improving their post-test score by 54%, and the video trainer improving theirs by 55%. They also looked at the effect of practice on operative performance by assessing all second-year residents on their operative performance during a laparoscopic cholecystectomy before and after skills training. Operative performance improved only in the virtual reality group (p < 0.05) [20]

In another study comparing virtual reality training to video training, Munz et al. randomized 24 novices to a either a control group, video training, or a training on the LapSim virtual reality simulator [13]. The 16 participants assigned to one of the training groups each performed 30 min training sessions once per week under direct supervision. Analysis of pre and post-test scores showed significant improvement in both trained groups, but no significant difference in the improvement between the two groups [13].

A recent Cochrane systematic review concluded that virtual reality training was at least as effective as video training in supplementing standard laparoscopic training, but no clear advantage was demonstrated [15].

Both video trainers and virtual reality simulators have demonstrated significant correlations between operative performance and psychomotor performance in lab-based settings and can be used to assess operative laparoscopic skills [5, 12, 19, 24, 29-30]. There is, however, a significantly larger body of evidence supporting this for the video trainer than the virtual reality simulator, and video trainers are currently used for the technical skills assessment component in the FLS program.

In 2003, Gallagher et al. demonstrated construct validity of the MIST VR virtual reality simulator, and showed that it was capable of evaluating the psychomotor skills required for the performance of laparoscopic surgery [30].

A study published by Kundhal and Grantcharov in 2008 showed validity of the LapSim virtual reality simulator for assessing laparoscopic skill [19]. They recruited 10 surgical residents with limited laparoscopic experience and had them perform three repetitions of seven basic skills tasks on the LapSim trainer, and then one laparoscopic cholecystectomy in the operating room. Time, error, and economy of motion parameters were measured on the LapSim. Operative performance was video recorded and blindly assessed by two independent observers using a modified OSATS rating scale. The correlation between time, economy of motion, and error parameters during the simulated tasks on the LapSim and the laparoscopic cholecystectomy were statistically assessed. Significant correlations between operative performance and simulator performance were demonstrated [19].

A larger prospective study by Langelotz et al. out of Germany established construct validity of the LapSim virtual reality simulator and showed that it could be used to assess operative laparoscopic skill [29].

There are few studies that directly compare video trainer and virtual reality simulation for assessment of operative laparoscopic skills. In Steigerwald’s study of general surgery residents it was found that construct and predictive forms validity were more thoroughly demonstrated for FLS and the low-fidelity video trainer than the LapVR virtual reality simulator when assessing performance against a live in vivo human cholecystectomy. They concluded there may not be enough evidence to recommend utilizing the LapVR virtual reality simulator for assessment of laparoscopic skills [48]. This is a significant consideration for educators who must balance fiscal responsibility when allocating resources for surgical training.

### Conclusions

Diminishing resources and expanding technologies, such as MAS, have complicated the acquisition and assessment of technical skills in surgical training programs. However, these challenges have been met with both innovation and an evolution in our understanding of how learners develop technical competence and how to better measure it. As these skills continue to grow in breadth and complexity, so too must the surgical education systems ability.

### References

1. Sroka G, Feldman LS, Vassiliou MC, Kaneva PA, Fayez R, Fried GM: Fundamentals of laparoscopic surgery simulator training to proficiency improves laparoscopic performance in the operating room-a randomized controlled trial. Am J Surg. 2010, 199:115-120. 10.1016/j.amjsurg.2009.07.035
2. Tedesco MM, Pak JJ, Harris EJ, Jr., Krummel TM, Dalman RL, Lee JT: Simulation-based endovascular skills assessment: the future of credentialing?. J Vasc Surg. 2008, 47:1008-1014. 10.1016/j.jvs.2008.01.007
3. Feldman LS, Sherman V, Fried GM: Using simulators to assess laparoscopic competence: ready for widespread use?. Surgery. 2004, 135:28-42. 10.1016/S0039
4. Krummel TM: Surgical simulation and virtual reality: the coming revolution. Ann Surg. 1998, 228:635-637.
5. Derossis AM, Fried GM, Abrahamowicz M, Sigman HH, Barkun JS, Meakins JL: Development of a model for training and evaluation of laparoscopic skills. Am J Surg. 1998, 175:482-487. 10.1016/S0002-9610(98)00080-4
6. Fried GM: Lessons from the surgical experience with simulators: incorporation into training and utilization in determining competency. Gastrointest Endosc Clin N Am. 2006, 16:425-434. 10.1016/j.giec.2006.03.009
7. Aggarwal R, Moorthy K, Darzi A: Laparoscopic skills training and assessment. Br J Surg. 2004, 91:1549-1558. 10.1002/bjs.4816
8. European Association of Endoscopic Surgeons: Training and assessment of competence. Surg Endosc. 1994, 8:721-722.
9. Society of American Gastrointestinal Endoscopic Surgeons: Granting of privileges for laparoscopic general surgery. Am J Surg. 1991, 161:324-325. 10.1016/0002-9610(91)90588-5
10. Philibert I, Friedmann P, Williams WT: New requirements for resident duty hours. J Am Med Assoc. 2002, 288:1112-1114. doi:10.1001/jama.288.9.1112
11. Canadian patient and physician safety and wellbeing: resident duty hours. (2002). Accessed: May 6, 2018: https://residentdoctors.ca/wp-content/uploads/2017/08/POSITION-PAPER-Resident-Duty-Hours-EN.pdf.
12. Fried GM, Feldman LS, Vassiliou MC, et al.: Proving the value of simulation in laparoscopic surgery. Ann Surg. 2004, 240:518-525. 10.1097/01.sla.0000136941.46529.56
13. Munz Y, Kumar BD, Moorthy K, Bann S, Darzi A: Laparoscopic virtual reality and box trainers: is one superior to the other?. Surg Endosc. 2004, 18:485-494. 10.1007/s00464-003-9043-7
14. Seymour NE, Gallagher AG, Roman SA, et al.: Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002, 236:458-463. 10.1097/01.SLA.0000028969.51489.B4
15. Gurusamy KS, Aggarwal R, Palanivelu L, Davidson BR: Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev. 2009, 21:CD006575. 10.1002/14651858.CD006575.pub2
16. Grantcharov TP, Kristiansen VB, Bendix J, Bardram L, Rosenberg J, Funch-Jensen P: Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg. 2004, 91:146-150. 10.1002/bjs.4407
17. Fairhurst K, Strickland A, Maddern G: The LapSim virtual reality simulator: promising but not yet proven. Surg Endosc. 2011, 25:343-355. 10.1007/s00464-010-1181-0
18. Beyer L, Troyer JD, Mancini J, Bladou F, Berdah SV, Karsenty G: Impact of laparoscopy simulator training on the technical skills of future surgeons in the operating room: a prospective study. Am J Surg. 2011, 202:265-272. 10.1016/j.amjsurg.2010.11.008
19. Kundhal PS Grantcharov TP: Psychomotor performance measured in a virtual environment correlates with technical skills in the operating room. Surg Endosc. 2009, 91:645-649. 10.1007/s00464-008-0043-5
20. Hamilton EC, Scott DJ, Fleming JB, et al.: Comparison of video trainer and virtual reality training systems on acquisition of laparoscopic skills. Surg Endosc. 2002, 16:406-411. 10.1007/s00464-001-8149-z
21. Scott DJ, Bergen PC, Rege RV, et al.: Laparoscopic training on bench models: better and more cost effective than operating room experience?. J Am Coll Surg. 2000, 191:272-283. 10.1016/S1072-7515(00)00339-2
22. Crochet P, Aggarwal R, Dubb SS, et al.: Deliberate practice on a virtual reality laparoscopic simulator enhances the quality of surgical technical skills. Ann Surg. 2011, 253:1216-1222. 10.1097/SLA.0b013e3182197016
23. Palter VN, Orzech N, Reznick RK, Grantcharov TP: Validation of a structured training and assessment curriculum for technical skill acquisition in minimally invasive surgery: a randomized controlled trial. Ann Surg. 2013, 257:224-230. 10.1097/SLA.0b013e31827051cd
24. Vassiliou MC, Feldman LS: Objective assessment selection, and certification in surgery. Surg Oncol. 2011, 20:140-145. 10.1016/j.suronc.2010.09.004
25. Peters JH, Fried GM, Swanstrom LL, et al.: Development and validation of a comprehensive program of education and assessment of the basic fundamentals of laparoscopic surgery. Surgery. 2004, 135:21-27. 10.1016/S0039
26. Soper NJ, Fried GM: The fundamentals of laparoscopic surgery: its time has come. Bull Am Coll Surg. 2008, 93:30-32.
27. Vassiliou MC, Ghitulescu GA, Feldman LS, et al.: The MISTELS program to measure technical skill in laparoscopic surgery : evidence for reliability. Surg Endosc. 2006, 20:744-747. 10.1007/s00464-005-3008-y
28. Edelman DA, Mattos MA, Bouwman DL: FLS skill retention (learning) in first year surgery residents. J Surg Res. 2010, 163:24-28. 10.1016/j.jss.2010.03.057
29. Langelotz C, Kilian M, Paul C, Schwenk W: LapSim virtual reality laparoscopic simulator reflects clinical experience in German surgeons. Langenbecks Arch Surg. 2005, 390:534-537. 10.1007/s00423-005-0571-6
30. Gallagher AG, Satava RM: Virtual reality as a metric for the assessment of laparoscopic psychomotor skills. Learning curves and reliability measures. Surg Endosc. 2002, 16:1746-1752. 10.1007/s00464-001-8215-6
31. Fried GM, Feldman LS: Objective assessment of technical performance. World J Surg. 2008, 32:156-160. 10.1007/s00268-007-9143-y
32. Turnbull J, Gray J, MacFadyen J: Improving in-training evaluation programs. J Gen Intern Med. 1998, 13:317-323. 10.1046/j.1525-1497.1998.00097.x
33. Gray JD: Global rating scales in residency education. Acad Med. 1996, 71:55-63.
34. Beard JD: Assessment of surgical skills of trainees in the UK. Ann R Coll Surg Engl. 2008, 90:282-285. 10.1308/003588408X286017
35. Vassiliou MC, Feldman LS, Andrew CG, et al.: A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg. 2005, 190:107-113. 10.1016/j.amjsurg.2005.04.004
36. Larsen CR, Grantcharov T, Schouenborg L, Ottosen C, Soerensen JL, Ottesen B: Objective assessment of surgical competence in gynaecological laparoscopy: development and validation of a procedure-specific rating scale. Br J Obs Gynaecol. 2008, 115:908-916. 10.1111/j.1471-0528.2008.01732.x
37. Scott DJ, Rege RV, Bergen PC, et al.: Measuring operative performance after laparoscopic skills training: edited videotape versus direct observation. J Laparoendosc Adv Surg Tech A. 2000, 10:183-190. 10.1089/109264200421559
38. Martin JA, Regehr G, Reznick R, et al.: Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997, 84:273-278. 10.1046/j.1365-2168.1997.02502.x
39. MacRae H, Regehr G, Leadbetter W, Reznick RK: A comprehensive examination for senior surgical residents. Am J Surg. 2000, 179:190-193. 10.1016/S0002-9610(00)00304-4
40. Reznick R, Regehr G, MacRae H, Martin J, McCulloch W: Testing technical skill via an innovative "bench station" examination. Am J Surg. 1997, 173:226-230. 10.1016/S0002-9610(97)89597-9
41. Gumbs AA, Hogle NJ, Fowler DL: Evaluation of resident laparoscopic performance using global operative assessment of laparoscopic skills. J Am Coll Surg. 2007, 204:308-313. 10.1016/j.jamcollsurg.2006.11.010
42. Reznick RK, MacRae H: Teaching surgical skills--changes in the wind. N Engl J Med. 2006, 355:2664-2669. 10.1056/NEJMra054785
43. Darzi A, Mackay S: Assessment of surgical competence. Qual Health Care. 2001, 10:64-69. 10.1136/qhc.0100064
44. Hammond I, Karthigasu K: Training assessment and competency in gynaecologic surgery. Best Pract Res Clin Obstet Gynaecol. 2006, 20:173-187. 10.1016/j.bpobgyn.2005.09.006
45. Undre S, Darzi A: Laparoscopy simulators. J Endourol. 2007, 21:274-279. 10.1089/end.2007.9980
46. Rassweiler J, Klein J, Teber D, Schulze M, Frede T: Mechanical simulators for training for laparoscopic surgery in urology. J Endourol. 2007, 21:252-262. 10.1089/end.2007.9983
47. van der Meijden OAJ, Schijven MP: The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review. Surg Endosc. 2009, 23:1180-1190. 10.1007/s00464-008-0298-x
48. Steigerwald S, Park J, Hardy K, Gillman L, Vergis A: Does laparoscopic simulation predict intraoperative performance? A comparison between the fundamentals of laparoscopic surgery and LapVR evaluation metrics. Am J Surg. 2015, 209:34-39. 10.1016/j.amjsurg.2014.08.031
Review article
peer-reviewed

### Author Information

###### Ethics Statement and Conflict of Interest Disclosures

Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.

Review article
peer-reviewed