Virtual Reality Simulation for Laparoscopic Cholecystectomy

Virtual Reality Simulation forLaparoscopic Cholecystectomy

Marlies P. Schijven

Virtual Reality Sim

ulation for Laparoscopic Cholecystectom

y Marlies P. Schijven

ISBN: 90-9019048-1

the process of validation and implementation in thesurgical curriculum outlined

Virtual Reality Simulation for Laparoscopic Cholecystectomy:

the process of validation and implementation in the surgical training curriculum outlined

Virtual Reality Simulation for Laparoscopic Cholecystectomy: the process of validation and implementation in the surgical training curriculum outlined Virtual Reality Simulatie in de laparoscopische chirurgie: het proces van validatie en implementatie in het chirurgische opleidingscurriculum uiteengezet

Thesis, University of Leiden, with summary in Dutch Printed by: Optima Grafische Communicatie, Rotterdam ISBN: 90-9019048-1 © Schijven, M.P. 2005 All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronical or mechanical, including photography, recording or by any other information storage and / or retrieval system without written permission from the author.

Virtual Reality Simulation for laparoscopic cholecystectomy: the process of validation and implementation in the surgical training

curriculum outlined

Virtual Reality Simulatie in de laparoscopische chirurgie: het proces van validatie en implementatie in het chirurgische opleidingscurriculum

uiteengezet

Proefschrift

ter verkrijging van de graad van doctor aan de Universiteit Leiden,

op gezag van de Rector Magnificus Dr. D.D. Breimer, hoogleraar in de faculteit der Wiskunde en Natuurwetenschappen

en die der Geneeskunde, volgens besluit van het College voor Promoties

te verdedigen op donderdag 10 maart 2005 klokke 16:15 uur

door

Marie-Elisabeth Paula Schijven geboren te Eindhoven

in 1969

Promotiecommissie Promotoren: Prof. dr. O.T. Terpstra Prof. dr. A. Cuschieri, Ninewells Hospital and Medical School, University of Dundee, Scotland Co-promotor: Dr. J.J. Jakimowicz, Catharina Ziekenhuis Eindhoven Referent: Prof. dr. H.J. Bonjer, Erasmus Universiteit Rotterdam Overige Leden: Prof. dr. J.M. van Baalen Dr. I.A.M.J. Broeders, Universitair Medisch Centrum Utrecht Dr. F.W. Jansen Prof. dr. A.J.J.A. Scherpbier, Universiteit Maastricht Dr. C. Terleth Prof. dr. C.J.H. van de Velde

´There are some patients whom we cannot help… there are none whom we cannot harm´ Arthur L. Bloomfield 1888-1962

To my parents †: true love never dies

Contents Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 6 Chapter 7 Chapter 8 Chapter 9 Chapter 10

Introduction and outline of thesis Minimal access surgery training in The Netherlands A survey among residents-in-training for general surgery The Advanced Dundee Endoscopic Psychomotor Tester (ADEPT): Objectifying subjective psychomotor test performance Face-, expert- and referent validity of the Xitact LS500 laparoscopy simulator Construct validity: Experts and novices performing on the Xitact LS500 laparoscopy simulator How to select aspirant laparoscopic surgical trainees: Establishing concurrent validity comparing Xitact LS500 index performance scores with standardized psychomotor aptitude test battery scores The Intercollegiate Basic Surgical Skills Course: Laparoscopic skill assessment using the Xitact LS500 laparoscopy simulator The learning curve on the Xitact LS500 laparoscopy simulator: Profiles of performance The Eindhoven Virtual Reality laparoscopic cholecystectomy training course: improving operating room performance Results from the first EAES accredited Virtual Reality trainings curriculum Virtual Reality surgical laparoscopic simulators: How to choose Summary and Conclusions Samenvatting en Conclusies Glossary Curriculum Vitae

9

13

33

47

63

79

95

111

129

143

161

167

175

179

Introduction and outline of thesis Introduction and outline of thesis The popularity of laparoscopy can, in fact, be explained by the increasing demand of the public. This phenomenon is founded in mounting evidence that for certain surgical procedures Minimal Access Surgery (MAS) compares favourably to its conventional counterpart technique in terms of post-operative pain, shortening of hospital stay and cosmesis. Probably the best example and most performed laparoscopic surgical procedure nowadays is the laparoscopic cholecystectomy, being treatment-of-choice in uncomplicated cholelithiasis. MAS poses certain specific demands on the surgeon and hospital setting that must be taken into consideration. As with any new technique a surgeon has to master, there is a learning curve associated with MAS. Transfer of skill, building on techniques already learned for open surgery is not appropriate, as skills needed to perform MAS adequately are quite different in nature. In performing MAS, the surgeon must be able to transfer a two-dimensional monitor image into a three-dimensional mind setting. Also, ergonomics, instrument handling and haptic sensations vary extensively. Specific MAS training is therefore a necessity in the current surgical training curriculum. In- and outside the more or less controlled environment of a surgical skills laboratory, different models have been developed to train surgical residents in MAS. These models, although heterogeneous in concept, have shown to be helpful in mastering basic MAS skill, such as pick-and-place, translocation and laparoscopic navigation. Animal (porcine) models, focussing more on acquiring procedural skill than laparoscopic bench models do, have the benefit of true tissue characteristics in anatomy and tissue manipulation. However, they are costly, require a specific infrastructure and are usually not repetitive in nature. Furthermore, these models cannot be standardized or tailored in terms of specific patient scenarios and do not provide characteristics for objective evaluation. Virtual Reality (VR) simulation is a novel development in surgical skills training. Reports using VR simulation settings in MAS training are appearing in literature, and first outcomes are promising. Providing real-time graphics, repetitive scenery and force-feedback tissue handling combined with objective assessment, VR simulation seems to be a promising asset in constructing a validated skills training program.

9

Introduction and outline of thesis The aim of this thesis is to outline the process of step-wise, structured validation and implementation in the surgical training curriculum of a Virtual Reality laparoscopic cholecystectomy simulator platform, the Xitact LS500. In Chapter 1, the state of surgical training and the current shortcomings in Minimal Access Surgery among Dutch surgical residents are described. Chapter 2 describes the value of the Advanced Dundee Psychomotor Tester as a scorings system correcting for subjective assessor opinion on endoscopic task performance. Chapter 3 focuses on the issues of face, expert and referent validity of the Xitact LS500 laparoscopy simulator. Chapter 4 studies the concept of construct validity, by comparing and analyzing test outcome scores of an experienced group of laparoscopic surgeons and an inexperienced group, on the ´clip and cut´ task of the Xitact LS500 laparoscopy simulator. Chapter 5 addresses the issue of concurrent validity, comparing Xitact LS500 index performance scores with scores on a battery of standardized psychomotor aptitude tests. Chapter 6 describes residents´ progress in Minimal Access Surgery after attending the Intercollegiate Basic Surgical Skills Course, using the Xitact LS500 laparoscopy simulator. Chapter 7 highlights the concept of the learning curve, displaying patterns of individual variation in MAS performance and addresses the amount of training necessary for a given VR simulation task. In Chapter 8, the Xitact LS500 laparoscopy simulator is clinically validated by showing improved operating room performance of candidates who participated in the Eindhoven Virtual Reality laparoscopy training course. Finally, in Chapter 9 gives an overview of Virtual Reality simulation systems currently available and their characteristics. In Chapter 10, the summary and conclusions of this thesis in both English and Dutch are given.

10

Minimal Access Surgery in The Netherlands

Chapter 1

Minimal access surgery training in The Netherlands A survey among residents-in-training for general surgery M. Schijven, J. Berlage, J. Jakimowicz Surg Endosc, 2004. 18: 1805-1814

13

Chapter 1 Abstract Background: The purpose of this study was to assess the state of surgical training and its possible shortcomings in minimal-access surgery (MAS) among Dutch surgical residents. Methods: A pre-tested questionnaire was distributed to all residents-in-training for general surgery in The Netherlands. Results: The questionnaire was sent to 407 surgical residents. The response percentage was 65%. Overall, 87.7% of all the responders were highly interested in the autonomous performance of laparoscopic surgery. Residents interested in gastrointestinal (GI) or oncologic surgery (n=137) are significantly more interested than residents not interested in GI or oncologic surgery (n=98). All the residents (100%) thought it is important to be able to perform the three basic MAS procedures (diagnostic laparoscopy, laparoscopic cholecystectomy, and laparoscopic appendectomy) autonomously at the end of their surgical training. Other MAS procedures were considered to be advanced procedures. Of all residents, 17.8% expected to be adequately prepared to perform these procedures at the end of their surgical training. Gastrointestinal/oncologic interested residents were most interested in performing advanced MAS. Most residents had the opportunity to attend MAS skills education. Irrespective of the format or training method, only 26.9% of residents stated their MAS skills training was objectively evaluated. The residents thought every surgical hospital department in The Netherlands should have a surgeon specialized in laparoscopic surgery (86.9%). Conclusions: The current study showed that Dutch residents believe it is very important to perform basic MAS autonomously. Of the GI/oncologic-interested residents, the majority wants to be able to perform advanced MAS, but expects to be unable to do so at the end of their training. They attribute this discrepancy to “not having enough chance to be the first operator” and “lack of volume of procedures in the hospital”. Specific and properly implemented, monitored and evaluated MAS skills training programs in skills laboratory settings could offer a promising environment in overcoming this problem.

14

Minimal Access Surgery in The Netherlands Introduction Numerous surgical procedures across a broad spectrum of clinical specialties have become adapted to minimal-access surgery (MAS). Probably the best example is the laparoscopic cholecystectomy, currently the procedure of choice over its open counterpart. In MAS, progress has been made regarding consensus in laparoscopic techniques and development of surgical equipment. Unfortunately, the same cannot be said for the current state of training in MAS procedures. Transfer of skills that builds on techniques learned performing open surgery is neither appropriate nor effective because the skills needed to perform MAS tend to be quite different [7]. Specific MAS skills training is thus a necessity for the adequate performance of MAS procedures. Studies have shown training in MAS in fact, to be inadequate [11, 14]. Different MAS training programs are likely to be necessary, depending on the skill or specific procedure to be trained, the resident’s competence level, and the residents MAS interest. There is much controversy about the amount of training necessary (e.g., the learning curve associated with safe performance of advanced laparoscopic surgery for the individual surgeon) [16, 21]. However, there is consensus on the need for a national curriculum and for training guidelines [5, 14]. The first step in overcoming above mentioned problems and creating accessible, tailored, validated, and cost-effective skills training programs, must be to make an inventory of the different types and shortcomings of current MAS education. Therefore, it is important to quantify the interest in, and current format of education in MAS among surgical residents. Only then will it be possible to identify possible gaps between the factual and desired level of MAS, and to address its discrepancies properly. This survey focuses on the state of the art MAS training and its shortcomings in The Netherlands.

15

Chapter 1 Materials and Methods Questionnaire The questionnaire was adapted from a previously validated questionnaire developed by a Canadian research group [2]. Each resident-in-training for surgery in The Netherlands received an individually addressed package containing the questionnaire, an introduction letter and a self-addressed, stamped return envelope. The questionnaire was distributed with approval of the Dutch Society for Endoscopic Surgery. An introduction letter, printed in the house style on posting paper of the Society accompanied the package. In this letter, the concept of anonymity for responders was stated. To maximize response ratings, a reminder package was sent to non-responders three weeks after the initial posting date of the package. A second reminder by email was sent to non-responders six weeks after the initial posting date of the package. Scope of questionnaire The questionnaire (see Attachment 1) addressed basic demographic issues, year of training, future perspective, presumed diffentiation in surgery, interest in performance of laparoscopic surgery, current clinical laparoscopic trainings situation, and relevant MAS skills training issues. Basic and advanced laparoscopic procedures were selected as specified by the Society of American Gastrointestinal Endoscopic Surgeons (SAGES), because no Dutch consensus was available [19]. All procedures other than the laparoscopic cholecystectomy, diagnostic laparoscopy and laparoscopic appendectomy (the so-called “core” procedures) were considered to be advanced procedures [18]. Subjects The package was sent to the cohort of Dutch residents-in-training for surgery, as specified in the particular section of the Yearbook 2003 from the Dutch Society for Surgery. According to the Dutch surgical curriculum, these are residents in their first four years of general surgery, as well as residents in their fifth or sixth (last) year of surgical training, differentiating in a particular subspecialty of surgery. Surgeons, who have completed their formal training but still are in training in the area of their differentiation or working as a fellow, were also included. Statistical analysis Data were collected and analysed using the Statistical Package for the Social Sciences (SpSS) version 9.0 (SpSS, Chicago, Il, USA).

16

Minimal Access Surgery in The Netherlands Results Demographics The questionnaire was sent to 407 surgical residents. Of these residents, thirty-one were no longer in training for surgery nor working as a fellow, or did in fact not receive the questionnaire (sent back as undeliverable). These residents were considered as random dropouts. In all, 245 questionnaires were returned. The response rate therefore was 65%. The residents average age was 32.4 years (range 26 - 40 years) and followed a normal distribution curve. The respondent group was 67.6% was male and 32.4% female. Whereas 40.7% worked in an academic hospital, 59.3% worked in a periphery teaching hospital. All eight surgical trainings regions were more or less equally represented (Fig. 1). Figure 2 shows the distribution according to year of training, and Fig. 3 the respondents future career perspective. Fig. 1. Distribution by surgical trainings region Fig. 2. Year of training

16

20

Fig. 3. Future perspective

UMC Radboud UMCULUMC AMC

VU

Perc

ent

14 12 10 8 6 4 2 0

Erasmus MC AZG AZM Fellow 6 54321

10

Perc

ent

60

50

40

30

20

10

undecided periphery/ruralgeneral

large periphery general

large periphery differentiation

academic

0

Perc

ent

differentiation

17

Chapter 1 In Fig. 4, the differentiation in a surgical subspecialty is depicted. The residents were split based of their presumed differentiation into two groups: the non-gastrointestinal (GI)/oncology-interested residents (n = 98) and the GI/oncology-interested residents (n = 137). This because analysis showed that GI/oncologic interested residents are significantly more interested in MAS compared to non GI/oncologic residents (Mann-Whitney U test, p = 0.001). Figure 5 shows that more than twice as many females (55 versus 21) opt for a career in GI/oncology (Fisher’s Exact Test, two-sided: p = 0.002); as compared with an evenly distributed male population. According to T- and Chi-Square statistics, there were no significant differences between the two groups in age distribution, region, academic/periphery hospital setting, year-of-training, or future career perspective. Fig. 4. Differentiation Fig. 5. Differentiation by sex

Interest in autonomous performance of laparoscopic surgery Overall, 87.7% of the responders were highly interested in the autonomous performance of laparoscopic surgery (score 4 or 5 on the Likert 5-point questioning scale). When clustered, the groups differed significantly in the extent of their interest (Fig. 6). Fig. 6. Interest in autonomous performance of laparoscopic surgery

GI / oncologyno GI / oncology

Coun

t

100

80

60

40

20

0

interest

2

4 5

5

Coun

t

120

40

0

Sex

80

160 40

no GI / oncology GI / onocology GI /oncologyoncology

GI

30

20

10

Perc

ent

fe male

male 0

pediatric

trauma pulmonary

vascular other

18


19

Skills L atory for Minimal Access Surgery Most residents had the opportunity to attend MAS skills education (85.7%), and the majority in fact did so (82%). The 35 respondents who did not (yet?) have the opportunity, indicated it to be of high importance indeed (score 4 or 5, 88.9%). These 35 residents (13 who were not interested in GI/oncologic differentiation and 21 who were, with one undecided) did not differ in opinion concerning the importance of MAS training (Mann-Whitney U test, p = 0.800). e residents who did attend MAS skills education most often received skills-box (organic and/or anorganic) training, and laboratory animal skills training (using anaesthetized pigs). Remarkably, GI/oncologic- interested residents w posed to Virtual Reality train almost twice as often, and to a lesser extent to laboratory animal skills training (Fig. 7). The format, by which the MAS skills were taught varied from video training to hands-on procedural skills training (Fig. 8). Irrespective of the format or the method by which residents were trained, only 26.9% of the residents stated that their skills training program was, in fact, objectively evaluated. Fig. 7. Method of MAS skills training Fig. 8. Format of MAS skills training

abor

Th

ere xeing curricula

0

20

40

60

80

100

120

no GI / oncology GI / oncology

skills box anorganicskills box organicVR simulation modellive animal modelotherunknown

0

20

40

60

80

100

120

no GI / oncology GI / oncology

videosdiscussionbasic skillssuturing / knot tyingproceduralother

Fig. 9. Adequately prepared for basic laparoscopic procedures

noyes

Coun

t

200

100

0

skills training yes

no

Chapter 1 The residents consider it important (76.1% scoring ‘4’ or ‘5’ on the five-point scale of

portance) to have the opportunity for requiring MAS skills in a skills-laboratory, in

able 1. Basic procedures in MAS (in percentages)

earson’s Chi-Square statistic

p = 1.000). All the re nts (100%) considered it is important to very important (score ‘4’ or ‘5’) that residents are able to perform basic MAS procedures autonomously. Nevertheless, about 1 in every 10 residents (11.5%) expected to be inadequately prepared for performing this basic laparoscopic surgery after becoming a qualified surgeon. Apparently, completion of skills training programs is not the only factor determining a resident’s opinion about adequate preparation (Fig. 9, Pearson’s Chi-Square p = 0.431). According to the consensus established by the Society of American Gastrointestinal Surgeons (Table 2), seventeen procedures were defined as advanced laparoscopic procedures. Table 2 shows that, for all procedures, GI /oncologic-interested residents are more often interested, although hesitant in their expectation to perform these procedures autonomously at the end of their surgical training. Only 17.8% of residents believed they would be adequately prepared for the aprocedures they wish to perform, once qualified.

imorder for proper preparation for the operating theatre. However, residents who did not attend MAS skills education did not feel inadequately prepared for basic MAS (Fig. 9.) Nevertheless, residents agreed about the importance for MAS skills education (Mann-Whitney U test, P = 0.702). Procedures in Minimal Access Surgery (MAS) Three procedures were defined as basic laparoscopic procedures: diagnostic laparoscopy, laparoscopic cholecystectomy and laparoscopic appendectomy (Table 1). The groups did not differ in their expectation of performing these procedures autonomously after completion of their surgical training program (Mann-Whitney U test; p = 0.774). The residents believed strongly that educators are obliged to offer a validated skills training program for basic laparoscopic procedures during their residency (97.1%). T

No GI / oncology GI/oncology Basic

*P

procedures MAS

autonomous

(%)

would like to perform

autonomous

(%)

not interested

(%)

autonomous

(%)

would like to perform

autonomous

(%)

not interested

(%)

P-value*

Diagnostic laparoscopy

82.7

17.3

76.6

22.6

0.7

0.266

Cholecystectomy 84.7 15.3 75.2 24.1 0.7 0.066 Appendectomy 67.3 29.6 3.1 66.2 31.6 2.2 0.964

The groups were fully agreed about this obligation (Pearson’s Chi-Square

side

dvanced

20

Minimal Access Surgery in The Netherlands The groups did not differ in this expectation (Pearson’s Chi-square p = 0.593). Nevertheless, 58.4% of GI/oncologic-interested residents considered it important to very important (score ‘4’ or ‘5’) for them to be able to perform these advanced MAS

e advanced laparoscopic rocedures (no significant difference between groups).

able 2. Advanced Procedures in MAS (in percentages)

BD: co

tant reasons given were as follows: residents feel they did not have sufficient opportunity to be the first operator for these procedures and the procedures are not (often) performed in their hospital. These reasons are likely to be interdependent.

procedures autonomously. Of the non-GI/oncologic- interested surgeons, 40.9% do so (Mann-Whitney U test, p = 0.014). Of all the residents, 53.3% believed educators are obliged to offer a validated skills training program for thp T

No GI/oncology GI/oncology Advanced procedures

MAS auto-

nomous

would like to

perform auto-

not interested

autonomous

would like to

perform auto-

not interested

nomous

(%)

nomous

(%)

(%)

P-

value* (%) (%) (%)

Nissen 18.1 44.7 37.2 fundo

8.9 77.2 13.8 plication

0.000*

He 0.376 ller myotomy 4.9 47.6 47.6 3.1 57.7 39.2 gastric resection 1.3 36.7 62.0 2.8 62.6 34.6 0.001* gastro- my .1 47. 17.jejunosto 6 46.3 6 6.3 76.6 1 0.000* cyst-gastrostomy 3.8 59. 66. 32.37.2 0 2.0 0 0 0.001* bariat ery 69. 38. 53. 0.093 ric surg 5.1 25.6 2 8.6 1 3 ( c r

71. 51partial) hepatiesection

1.3 27.3 4 1.0 48.0 .0 0.021*

C 34. 17BD** exploration 5.9 60.0 1 11.5 70.8 .7 0.020* c 6 29holedocho- 1.3 38.0 0.8 4.0 66.3 .7 0.000* jejunostomy distal pancreatectomy 1.3 32.1 66.7 1.1 48.9 50.0 0.081

* P Chi-Square statistic ** C mmon bile duct When there was an expectation of inadequate preparation for the advanced procedures, GI/oncologic-interested residents tended to attribute this significantly more to a presumed lack of interest of the educator than non-GI/oncologic-interested residents do (Table 3). There were no significant differences between the groups for the other possible explanations. The most impor

earson’s

splenectomy 8.0 69.3 22.7 13.9 72.2 13.9 0.149 adrenalectomy 7.2 42.2 50.6 6.7 64.4 28.8 0.007* right hemicolectomy 14.3 64.8 20.9 16.0 76.8 7.2 0.013* sigmoid resection 10.3 64.4 25.3 13.3 76.7 10.0 0.013* rectal surgery 3.7 43.9 52.4 6.5 73.1 20.4 0.000* inguinal hernia repair 31.9 53.2 14.9 27.2 63.2 9.6 0.274 ventral hernia repair 26.1 54.3 19.6 21.1 65.1 13.8 0.284

21

Chapter 1 Table 3. Reasons for inadequate mastering of Advanced Laparoscopic Procedures in MAS (in percentages)

No GI / oncology

GI / oncology Total P-value*

not of influence / highly of influence

(%)

not of influence/ highly of influence

(%)

Reasons

Median

1 2 3 4 5 1 2 3 4 5

Lm

ack of interest 47.7 22.7 18.2 8.0 3.4 58.9 21.8 9.7

*Pearsons Chi-Square statis

in T etherl

ted that ry sur hosp epartm in T ethe s spec d in rosco urger .9% ct, % of the

ted the as such a surgeo . A m ity o resere should a sp diffe ation lapar pic s ry

). Among the ents g a eon ializery eir cli 2.2% eved resen f thi geo

ion of basic laparoscopic skills. For advanced laparoscopic hed 88.3%.

tic Laparoscopic Surgery he N ands Residents sta eve gical ital d ent he N rland should have a surgeon ialize lapa pic s y (86 ). In fa 72residents repor re w n in their clinic inor f the idents believed th be ecific renti for osco urge in the surgical curriculum (43% resid havin surg spec ed in laparoscopic surg in th nic, 8 beli the p ce o s sur n was

yself 6.5 3.2 1 0.381

Lacmyeducato

22.7 1 11.5 26.0 27. k of interest of surgical

r(s)

0.2 27.3 31.8 8.0 5 23.7 11.5 3 0.013*

Lack of ap riate patients

13. 23.6 .6 15. 14. 33.3 0 0.801 prop 5 614. 32 7 .39 0 9.52 14. 3

Not enough opportunity to be the first operating surgeon during

res

6.9 10.3 11. 2 39.1 7.6 4.5 16.7 39. 8 4 0.276

procedu

5 32. 4 31.

Not enough opportunity to be the

12.8 1 20.9 3 20.9 7 22.1 2 26.7 0.160

assisting surgeon during procedures

22. 23. 13. 28. 9.2 3

Lack of operating time to perform these procedures

9.5 .6 29.8 16.7 24.8 8 27. 0.22 21.4 9.6 28. 2 9.6 3 591

In my hospital, thprocedures are not

ese 17.6 .3 17.6 30.8 15.2 6 25. 3 0.

performed

14 19.8 15.2 13. 8 30. 4 808

important for the educatskills, this estimate reac Discussion The results of the current survey, with a response rate of 65% and displaying an even distribution by region and by year of training, support a true reflection of the surgical resident population in The Netherlands. In conformity with results from a large Canadian study, Dutch residents (100 % !) feel it is very important to perform basic MAS procedures autonomously.

22

Minimal Access Surgery in The Netherlands Residents feel strongly about the need for validated basic laparoscopic procedural

um [2]. Most Dutch residents had the opportunity to S skills education indeed. Of the resi ry more than ) rated such education

highly in relation to th rmance. Most residents (88.5%) also expected that they would be able ic MAS pr eing This is promising, although educators should try to ide e the ‘one-out-of-ten’ resident who fe sec e perf ming a laparo cholecystectomy,

y, or nostic laparoscopessfu of ocedural training is a

p i e n S r n 7 of ur s their current MAS training to be adequate for the advanced MAS procedures

rf . s n Ca dia . r US r s,

a need f d r advanced l rge Ind e be global, as a Belgian study under the auspices of the Belgian

doscopic Surgery showed that 2 out of every 3 surgical trainees consider l training in laparoscopy to be inadequate, with virtually no opportunity to

vanced r d t hin f a d dures –as well as the basic procedures- can, and in fact should, be

to the surgical residency [11, 13]. ost imp n n e s ween the t

ce ent e Canadian ot enough chance of being the first operating surgeon” and “lack of

e h t a f im ies e ese procedures must be overcome somehow crossing the

rgical curriculum. For the MAS interested resident, who is cologic-interested resident, a skills training program with

ning could offer possibilities for overcoming this ew, the Society of American Gastrointestinal

ronment and actual surgical performance [2, 17]. his might be attributed to the fact that for surgical outcome, underlying decision

es and sequels of errors leading to complications cannot be trained non-animal training models, and only partly with the use of animal

training in their surgical curriculattend MAeducation,

dents attending MAS skill the benefit of

s laborato skill two-t (76.1%

eir clinical perfohirds

to perform bas ocedures once b qualified.ntify a educatnd

els in ur or scopic appendectom Succ

diagl integration

y. anced laparoscopic pr adv

logical next steestimated

n th educatio for MA . St ikingly, o ly 1 .8% o resident

they wish to peRattner et al. [16] and others [11, 13] re

orm Thi percentage isvea

ideled a simila

tical to thr p

at inroblem fo

the na n studyesident

perceiving problem seems toGroup for Entheir practica

or a ditional t aining in aparoscopic su ry. eed, th

perform adlaparoscopic proceincorporated in

lapa oscopic procedures [12]. It is state tha teac g o dvance

The msurgical curriculum and clinicstudy, were: “n

orta t real practi

asons me, in both the cu

tion d for thirr

gap stud

bety and th

curren

volume of these learning curve for thboundarie

proc dures in the ospi al” (medi n o 4). This pl that th

s of the current suost likely to be a GI/onm

emphasis on specific MAS skills trairoblem. In keeping with this vip

Endoscopic Surgeons (SAGES) supports the creation of skills laboratories, suggesting guidelines for faculty training, and is supporting (postgraduate) MAS fellowships [18]. Inside and outside the more or less controlled context of a skills laboratory, different models have been developed to train residents in MAS skill. These models, heterogeneous in concept, have proved to be helpful in the learning of basic MAS skills, such as pick-and-place, translocation and navigation [6]. So far there is some, but little, evidence to suggest a positive relationship between

erformance in the simulated envipTmaking processwith the use ofsupported ones. Park and Witzke earlier assessed current training in MAS to be inadequate [14].

23

Chapter 1 This phenomenon can be explained by the fact that current MAS training programs are neither widespread nor standardized. In fact, there is little evidence concerning what a MAS training program needs to be effective. It is suggested that a multimedia training program, incorporating interactive and various training methods is both attractive and beneficial in adoption of new MAS skills [15]. Training programs offered to residents should therefore be carefully evaluated, selecting and separating the most promising elements from the less promising ones. Our study showed that skills lab program evaluation in The Netherlands is, in fact, a scarce phenomenon. Virtual Reality (VR) procedural simulation is a novel development in surgical skills training. Reports of advances in MAS training using VR simulation are appearing in the literature, and first outcomes are promising [1, 9, 10, 23]. Providing repetitive scenery combined with objective assessment, VR surgical simulation offers excellent and in nature repetitive training opportunities for basic MAS skills. VR skills training introduces the trainee to the enlarged and two-dimensional monitor image of the three-dimensional workspace, to the fulcrum effect* inherent to laparoscopic surgery, and to limitations in vision and movements. Recent developments in VR simulation focus on haptic feedback, combining realistic anatomy graphics with realistic tissue deformation upon manipulation. Progress has been made in creating multiple MAS tasks, incorporating validated assessment modules and intelligent processing of error sequels [20]. Learning curves, inherent to the hurdles of MAS mastery, are known to

ecialized surgeon is very important for the acquisition of basic, and moreover,

impose a great deal of practice time on the surgical novice in achieving proficiency and competency. Indeed, there is controversy about the amount of training necessary for the safe performance of laparoscopic surgery [16, 21]. This is to result from a combination of factors: the resident’s ability for MAS, the type of MAS procedure and, of course, the clinical variation in the patient’s anatomy. There is no doubt that in time, objective, validated and reproducible VR procedural simulation will become highly important in surgical training and evaluation of surgical competence [3, 4, 20, 21, 22]. Another finding of our study is the fact that residents do believe that every clinic should have a MAS-specialized surgeon. According to the residents, not every teaching hospital (28%) has employed such a surgeon. Residents believe the presence of a MAS-spadvanced MAS skill. Fowler and Hogle [8] showed that the impact of an experienced MAS surgeon in the clinic raised the number of laparoscopic surgical procedures in which residents participated by more than 100%. Residents´ involvement in laparoscopic training sessions and MAS research projects also increased measurably [8]. The fact that experienced MAS surgeons have once been residents themselves enhances the arguments for starting good skills training early, as a means of increasing the potential of future trainers. * fulcrum effect: the effector end of the laparoscopic instrument moves in the opposite direction of the surgeon’s hand movement.

24

Minimal Access Surgery in The Netherlands Conclusion Dutch residents are interested in the autonomous performance of MAS once they become qualified surgeons, GI/oncologic-interested residents having even greater interest. For the basic MAS procedures, e.g. the laparoscopic cholecystectomy, laparoscopic appendectomy and diagnostic laparoscopy, residents are quite confident they will be able to perform them autonomously indeed (88.5%). As for the advanced procedures, residents are far more doubtful (17.8%) whether they will be adequately

repared once becoming qualified surgeons. Most GI/oncologic-interested residents

rograms, valid MAS training guidelines and assessment protocols can be designed.

pbelieve it is important to master advanced MAS procedures at the end of their surgical training. Skills laboratory offering MAS skills training programs could be helpful in shortening learning curves in MAS outside the operating room. Most residents had the opportunity to attend such courses, which primarily emphasized on acquiring basic laparoscopic skills. Procedural skills are taught in skills courses as well, but in a lesser extent. This seems plausible because procedural skills can be taught only with the use of animal models, and therefore not repetitively, or with the use of the relatively new VR procedural MAS simulation. A minority of residents report objective evaluation after participating in a MAS skills course (26.9%) Residents believe that every surgical hospital department in The Netherlands should employ a surgeon specialized in MAS, and they believe the presence of such a surgeon is important for the acquisition of MAS skill. The results of our study indicate there is a definite need for validated education in (advanced) MAS skill in The Netherlands. In the design of such a curriculum, a multimedia approach is advisable and procedural skills training should be taught. Because of its standardized scenery, repetitive possibilities, and unbiased assessment modalities, VR simulation is undoubtedly going to be important in future training programs. The skills laboratory seems to be the ideal place to set up, teach and evaluate such curricula. Only by proper and multi-centre evaluation of skills trainingpTherefore, a discussion under the auspices of the Dutch Society of Endoscopic Surgery with regard to method, amount, location, and regularity of MAS training modules, as well as its controlled implementation, needs to be conducted.

25

Chapter 1 References 1. Ali MR et al. (1999) "Training the novice in laparoscopy: more challenge is better" Surg Endosc 16: 1732-1736 2. Chiasson, PM et al. (2003) "Minimally invasive surgery training in Canada: a survey of general surgery" Surg Endosc 17: 371-377 3. Coleman J, Nduka, CC, Darzi, A (1994) "Virtual reality and laparoscopic surgery" Br J Surg 81: 1709-1711 4. Cosman, PH et al. (2002) "Virtual reality simulators: current status in acquisition and assessment of surgical skills" Aust NZ J Surg 72: 30-34 5. Cuschieri A et al. (2003) "Reflections on surgical training" Surg Endosc 7: 73-74 6. Derossis, AM, Antoniuk M, Fried, GM (1999) "Evaluation of laparoscopic skills: a 2-year follow-up during residency training" Can J Surg 42: 293-296 7. Figert, PL et al. (2001) "Transfer for training in acquiring laparoscopic skills" J Am Coll Surg 193: 533-537 8. Fowler DL, Hogle N (2000) "The impact of a full-time director of minimally invasive surgery: clinical practice, education and research" Surg Endosc 14: 444-447 9. Gallagher AG et al. (1999) "Virtual reality training in laparoscopic surgery: a preliminary assessment of minimally invasive surgical trainer virtual reality (“MIST VR”) Endoscopy 31: 310-313 10 Jordan JA et al. (2001) "Virtual reality training leads to faster adaptation to the novel

psychomotor restrictions encountered by laparoscopic surgeons" Surg Endosc 15: 1080-1084 11. Liberman MA, Greason K (1999) "Residency training in advanced laparoscopic surgery: how are

we doing?" Surg Laparosc Endosc Percutan Tech 9: 87-90 12. Navez B, Penninckx, F (1999) "Laparoscopic training: results of a Belgian survey in trainees.

Belgian Group for Endoscopic Surgery" Acta Chir Belg 99: 53-58 13. Nussbaum MS (2002) "Surgical endoscopy training is integral to general surgery residency and

should be integrated into residency and fellowships abandoned" Semin Laparosc Surg 9: 212-215 14. Park A, Witzke DB (2002) "Training and educational approaches to minimally invasive surgery:

state of the art" Semin Laparosc Surg 9: 198-205 15. Ramshaw BJ et al. (2001) "The role of multimedia interactive programs in training for laparoscopic procedures" Surg Endosc 15: 21-27 16. Rattner DW, Apelgren, KN, Eubanks WS (2001) "The need for training opportunities in advanced laparoscopic surgery" Surg Endosc 15: 1066-1070 17. Rogers DA, Elstein AS, Bordage G (2001) "Improving continuing medical education for surgical techniques: applying the lessons learned in the first decade of minimal access surgery" Ann Surg 233: 159-166 18. SAGES (1998) "Integrating advanced laparoscopy into surgical residency training" Surg Endosc 12: 374-376

19. SAGES, SAGES position statement on advanced laparoscopic training 20. Schijven M, Jakimowicz J (2003) "Construct validity: experts and residents performing on the Xitact LS500 laparoscopy simulator" Surg Endosc 17: 803-810 21. Schijven MP, Jackimowicz JJ (2003) "The learning curve on the Xitact LS500 laparoscopy

simulator: profiles of performance" Surg Endosc 18: 121-127 22. Seymour NE et al. (1992) "Virtual reality improves operating room performance: results of randomized double-blind study" Ann Surg 236: 458-463 23. Strom P et al. (2003) "Validation and learning in the Procedicus KSA virtual reality surgical simulator: implementing a new safety culture in medical school" Surg Endosc 17: 227-231

26


27

Chapter 1 Attachment 1: Questionnaire -translated from Dutch format- Personal Characteristics Age: _________year Sex: M/F Hospital: _______________in _________(city) Academic region-of-training:______________ Year of training: 1 / 2 / 3 / 4 / 5 / 6 / or 7 (=surgeon in advanced training or fellow) Future Career Perspective 0 Plans to work in an academic medical centre, predominantly in a surgical differentiation / subspecialty 0 Plans to work in a large periphery medical centre, predominantly in a surgical differentiation / subspecialty 0 Plans to work in a large periphery medical centre, predominantly as a generalist 0 Plans to work in a small (rural) periphery medical centre, predominantly as a generalist 0 Undecided Desired diffentiation / subspecialty: 0 vascular 0 trauma 0 gastro-intestinal 0 pulmonary 0 children 0 oncology Please indicate how interested you are in the autonomous performance of laparoscopic surgery, once being a qualified surgeon: 1 2 3 4 5 not interested very interested Basic Skills Minimal Access Surgery (MAS). 1. Are you / have you been in the opportunity to attend a skills training course or –education program for MAS? 0 yes, in 19__ in __________(location) -> go to question 1c 0 yes, I have been in the opportunity but I did not attend due to__________________please specify reason -> go to question 2 0 no -> go to question 1b 1b. If not, on a scale of 1 to 5, encircle the number that best indicates how important you feel it is that MAS skills are being thought to you in a skills laboratory during your surgery training: 1 2 3 4 5 not important very important 1c. If yes, please answer the following questions: - what kind of MAS skills will be / were thought to you? (multiple answers possible) 0 laparoscopic skills box (abdominal box model, abdominal mannequin, no organic substrate) 0 laparoscopic skills box with organic substrate (pig liver/ gall bladder, intestine etc) 0 Virtual Reality laparoscopic simulation model (i.e. MIST-VR, Xitact, LapSim) 0 live animal model (e.g. anaesthesized pig) 0 other, namely ______________________ 0 do not know -Are / were the MAS skills thought to you evaluated in an objective manner (standardised observation list, video, other method)? 0 yes 0 no 0 do not know

28

Minimal Access Surgery in The Netherlands -What kind of MAS skills instruction will be/ was given to you? (multiple an observation of instruction videos of standard MAS procedures (laparosco

swers possible) pic cholecystectomy, Nissen

ndoplication, laparoscopic hernia repair, laparoscopic appendectomy) t-up (stack, electronic gear, camera, insufflator, monitor, material and

S skills (navigation, pick-and-place, transfer, translocation, cutting, diathermy)

section of Calot’s triangle in gallbladder surgery)

e number that best indicates how important it is / was to you to

very important

0fu0 discussions about laparoscopic seinstruments) 0 basic MA0 intra-corporal suturing techniques 0 procedural MAS skills (e.g. clip-and-cut and dis0 other, namely_________________________ -On a scale of 1 to 5, please encircle thacquire MAS skills in a skills laboratory setting, referring to your MAS skills in the actual operating theatre:

5 1 2 3 4 not important Basic Procedures Minimal Access Surgery (MAS). 2. The following procedures are defined as basic laparoscopic procedures. Which of these do you expect to

copy 0 0 0 aparoscopic cholecystectomy 0 0 0

a. Do you expect your current surgical trainings program to prepare you sufficiently for the autonomous erformance of above mentioned basic

perform once being a qualified surgeon? performs autonomous would like to perform autonomous not interested -Diagnostic laparos-L-Laparoscopic appendectomy 0 0 0 2p laparoscopic procedures?

ned asic

0 yes no, because_______________________ (reason) 0

2b. Do you think it is a mandatory trainings obligation of the Surgical Society to offer you a validated skills training programme for the above mentio b laparoscopic pro dures?

important do you feel it is to be able to perform the above mentioned basic

ce0 yes

no 0 c. How 2 laparoscopic

procedures autonomously once being a qualified surgeon? (please encircle): 2 3 4 1 5

nt not important very importa Advanced Procedures Minimal Access Surgery (MAS). 3. The following procedures are defined as advanced laparoscopic procedures. Which of these do you expect

nomous not interested 0 0 0 0

n 0 0 0 astro-jejunostomy 0 0 0

0 0 Hepatic resection 0 0 0

exploration 0 0 0 nostomy 0 0 0

istal pancreatectomy 0 0 0 -Splenectomy 0 0 0

to perform, once being a qualified surgeon? performs autonomous would like to perform auto-Nissen fundoplication 0 -Heller myotomy 0 -Gastric resectio-G-Cyst-gastrostomy 0 0 0 -Bariatric surgery 0 -(partial)-CBD-Choledochojeju-D

29

Chapter 1 3. –continued- performs autonomous would like to perform autonomous not interested -Adrenalectomy 0 0 0

ction 0 0 0 0

0 0 0

a. Do you expect your current surgical trainings program to prepare you sufficiently for the autonomous

-Right hemicolectomy 0 0 0 -Sigmoid rese-Rectal surgery 0 0 -Inguinal hernia repair 0 -Ventral hernia repair 0 0 3performance of above mentioned advanced laparoscopic procedures? 0 yes 0 no, because_______________________(reason)

aining program for the advanced

3b. Do you think it is a mandatory trainings obligation of the Surgical Society to offer you a validated skills tr laparoscopic procedures?

yes 0 0 no 3c. In case you expect not to be skilled enough to perform above mentioned advanced laparoscopic procedures once being a qualified surgeon, what do you feel is the cause? (multiple answers possible. Please indicate the strength of influence scaled 1 to 5) not of much influence highly of influence

2 3 4 5

4 5

ow important do you feel it is to be able to perform the above mentioned advanced

-Lack of interest myself 1 2 3 4 5 -Lack of interest of my surgical educator(s) 1 2 3 4 5 Lack of appropriate patients 1 2 3 4 5 -

-Not enough opportunity to be the first operating surgeon during procedures 1 2 3 4 5

nough opportunity to be the assisting -Not e surgeon during procedures 1 Lack of theatre operating time to perform -

these procedures 1 2 3 4 5 -In my hospital, these procedures are not

rmed 1 2 3 perfo d. H laparoscopic 3

procedures autonomously once being a qualified surgeon? (please encircle): 1 2 3 4 5 not important very important

30

Minimal Access Surgery in The Netherlands Laparoscopic Surgery in The Netherlands 4. Do you feel every surgical training hospital should have a specialized laparoscopic surgeon? 0 yes 0 no 4a. Do you feel there is a need for a subspecialty / differentiation in laparoscopic surgery? 0 yes 0 no 4b. Is there /are there surgeon(s) working in your hospita0 yes

l who are specialized in MAS?

cialized surgeon in a surgical trainings hospital the teaching/training of basic

0 no 4c. How important, according to you, is it to have a MAS spe for MAS skills to surgical residents?

rtant

nt

hank you !

1 2 3 4 5 not important very impo

v al 4d. How import, according to you, is it to ha e a MAS specialized surgeon in a surgical trainings hospitkills to su ical resid ts? for the teaching/training of advanced MAS s rg en

3 4 5 1 2

not important very importa Remarks: T

31

The Advanced Dundee Endoscopic Psychomotor Tester Chapter 2 The Advanced Dundee Endoscopic Psychomotor Tester (ADEPT) Objectifying subjective psychomotor test performance M. Schijven, J. Jakimowicz, C. Schot Surg Endosc, 2002. 16: 943-948

33

Chapter 2 Abstract Background: This study was undertaken to establish the value of the Advanced Dundee Psychomotor Tester (ADEPT) as an objective real-time scoring system, correcting for subjective assessor opinion on endoscopic task performance. The main research questions were as follows:

rs of their own performance on ADEPT? EPT to be a valid instrument for measuring

ect innate psychomotor ability? Methods: Each of 45 surgeons completed two runs on ADEPT. The runs comprised five standardized tasks. A post-test visual analogue scaled (VAS) questionnaire measuring attitude toward skills testing in general, validation, and performance on ADEPT was used. Subjective responses were compared with objective scores generated through performance on ADEPT. Results: Surgeons emphasize the importance of using a variety of training methods for surgical residents during their residency, including laparoscopic Virtual Reality simulators. Monitoring of residents' endoscopic progress seemed to be a key issue. Surgeons themselves underestimate their individual performance on ADEPT (mean subjective score of 6.1 versus mean objective score of 6.6). Self-reported performance on ADEPT is unreliable because confidence intervals between the VAS score and the ADEPT score overlap. Surgeons disagree on the validity of ADEPT. The mean score for validity was 5.8, ranging from 0 to 10 with almost equal distribution over the scale. Innate ability is established as surgeons' scores express high concordance between test run and true run, with 72.7% of the participants' true run score within one distance from the test run. Conclusion: Surgeons cannot correctly predict their standardized individual test result on ADEPT. Performance on ADEPT reflects innate psychomotor ability along with improvement over runs. Surgeons are ambivalent in assessing the validity of ADEPT, irrespective of personal performance.

• Are surgeons good estimato• Do surgeons perceive AD

laparoscopic skills? • Does performance on ADEPT refl

34

The Advanced Dundee Endoscopic Psychomotor Tester Introduction

the University of Dundee [9]. In addition to evaluating end o innate psyc mresear

aterials and Methods

To improve and evaluate endoscopic task performance, attempts have been made to establish objective means for assessment of performance. The Advanced Dundee Endoscopic Psychomotor Tester (ADEPT) was developed for this purpose in 1997 in the Ninewells Hospital at

osc pic task performance, this device can be of value in evaluating ho otor ability and training methods for minimal access surgery [15]. The

ch questions for our study were as follows:

• Are surgeons good estimators of their own performance on ADEPT? • Do surgeons perceive ADEPT to be a valid instrument in measuring

laparoscopic skills? • Does performance on ADEPT reflect innate psychomotor ability?

M Fig.1. ADEPT system

1. Base plate

2. Probe mounts

3. Scope mount

4. Task Box

Equipment A computer-controlled device, ADEPT was developed for objective evaluation of endoscopic task performance (Fig. 1). Its hardware consists of a dual gimbal mechanism that accepts 4.8-mm standard endoscopic instruments for bilateral manipulation in a defined three-dimensional workspace.

5. Interface unit

35

Chapter 2 The device has three ports: one to mount a standard endoscope and two to mount

anipulation instruments.

ontact between one of the results in a so-called ¨probe

plate resu

and o program as n a st session. The maximum time allocated to each task was 60 sec. Elapsed time during

completion of task; horizontal, vertical, and ruments; plate- and probe errors are recorded through

interface unit, and subsequently translated into data by the computer. ADEPT was set up to achieve optimal angles for endoscopic

manipulation [10, 13]. The elevation angle of the endoscope was chosen so that the optical angle was perpendicular to the task box (task box horizontal plane, 25°; endoscope mount, 65°; distance to task box, 10 cm). Left and right instruments were each mounted at 35° so that elevation angles were 60°. The azimuth angle (between each grasper and endoscope) was 30°, and the manipulation angle (between left and right graspers) was 60°. The base-plate angle (distance between the two probe mounts) was 75°. A standard Storz cold light fountain 450-V light source with a standard Storz Endovision 9050-PB single-chip camera, a Sony high-resolution monitor, and a 0° forward-viewing 10-mm endoscope 33 cm in length were used (Karl Storz, Tuttlingen, Germany). Participants The subjects participating in this study were 45 congress participants in the 9th

International Congress of the European Association of Endoscopic Surgery, held 13±16 June 2001, in Maastricht, The Netherlands. Their endoscopic cal xperience varied from that of a surgical resident with no laparoscopic experience to

ly trained consultant (attending) surgeon.

time, successful completion, total plate error time, and total probe error time per task were recorded. The number of tasks completed in one run without any plate error, without probe error, and without either plate or probe error was recorded as perfect plate task run, perfect probe task run, and perfect task run, respectively.

m On the iso-centre of the device, a task box is placed. This box comprises five

different target tasks based on the main actions involved in endoscopic manipulation. The tasks include movements such as grasping, moving, and positioning an object, for example, manipulating a top plate with one instrument while negotiating the underlying task with the other instrument. The five target tasks in the task box are overlaid by a transparent spring-mounted top plate with access apertures. The task box itself is mounted on a spring-mounted base plate. C

struments and the lined edges of any access aperture inerror¨ simulating overload in tension, while overt pressure anywhere on the base

lt in a ¨plate error¨, simulating overload in pressure. The five task targets in the box are a flick-switch, a rotation dial, a joystick,

tw slider tasks positioned at different angles. A self-running computerbee developed, which randomly assigns any number of specified tasks duringh

teperformance of target tasks; success in rotational movements of the instan

For our study,

surgiethat of a ful Outcome measures Two runs of five randomly assigned tasks, each with a maximum allocated time period of 60 sec, were performed by each participant. Execution

36

The Advanced Dundee Endoscopic Psychomotor Tester Subjective assessment The parti

ndpoints cm

eaching

cipants' opinion was asked on a variety of subjects using a questionnaire with a l0-cm continuous response scale, a visual analogue scale (VAS). Statements on thenecessity for standardized laparoscopic and Virtual Reality training courses for surgical residents were proposed, as well as questions for performance on ADEPT. Eon the VAS for the statements ranged from 0 cm (complete disagreement) to 10(complete agreement). An escape answer “do not know'' was available. Other statements related to personal performance on ADEPT, its validity for tpurposes and assessment of laparoscopic skills. Statistics

Frequency tables and box plots were constructed for visualization of data. Student's paired T-test was performed. The Statistical Package for the Social Sciences, version 9.0, was used. Results Demographics Fig. 2. Occupation Fig. 3. Number of laparoscopic procedures

% %

5060708090100 30

20

010203040

> 100> 100<100<75<50< 25<100

10

The 45 participants originated from 16 countries. Their ages ranged from 27 to 61 ears (mean 42.3 years). Of these participants,

Surgeon Resident Number of laparoscopic procedures

yle

80% were right-handed, 10% ambidextrous. Their level of experience varied from that of

f surgeon (Fig. 2), with 15% working as residents, 85% as surgeons. fthanded, and 10%sident to that ore

In terms of laparoscopic surgical experience, 19.5% of the participants performed fewer than 50 laparoscopic operations, 14.6% fewer than 100, and 65.9% more than 100 laparoscopic procedures per year (Fig. 3). Only one surgeon performed fewer than 100 laparoscopic procedures per year, and no resident performed more than 100 laparoscopic procedures per year.

37

Chapter 2 Questionnaire Fig. 4.a. Importance of training by Fig. 4.b. Importance of training Virtual Reality trough Basic Surgical Skills Course

10,00,00

%30

20

10

10,02.0

10

0

%30

20

ig. 5. Importance of monitoring laparoscopic skills F

VAS scale 0-10 VAS scale 0-10

VAS scale 0-10

Almost all participants agreed with the statement that it is necessary for surgical residents-in-training to participate in a (basic) laparoscopic skills course before oper-ating on patients (mean 9.3 ± 1.34). Most of the participants thought it was important to train esidents-in-training on laparoscop s such s

IST-VR and Xitact before they operated on patients (mean 7.9 ± 3.04; Fig. 4). Fur-surgical r ic Virtual Reality simulator a

Mthermore, most of them considered an objective assessment method for monitoring the progress of residents' laparoscopic skills to be a valuable asset (mean 9.16 ± 1.12; Fig 5).

38

The Advanced Dundee Endoscopic Psychomotor Tester Surgeons' perception of performance on ADEPT

Fig. 6. Composition of ''SUM''-score Fig.7. Distribution of SUM-score

Components:

-score 2 (successful tasks second run): 0-5-time run < 150 seconds equals “6”: 1-number of “perfect tasks” ≥ 1: 1-number of “perfect probe tasks” ≥ 1: 1-number of “perfect plate tasks” ≥ 1: 1-test run ≠ 0 (at least one task successful): 1

10 10,09,08,07,06,05,04,03,0

12

10

8

6

4

2

0

%

ig. 8. Correlation between score and Fig. 9. Is the visual analogue scale ther components of SUM-score a reliable estimator of objective

ance?

Score Fo perform

SUM10,009,008,007,006,005,004,003,00

scor

e ru

n 2

5,5

5,0

4,5

4,0

3,5

3,0

2,5

2,0

1,5

SUM10,009,008,007,006,005,004,003,00

VAS-

scor

e

12

10

8

6

4

2

0

-2

For each participant, self-perceived performance, as reported on the VAS scale, was compared with the participant's score on ADEPT. A positive correlation between the two variables indicated that the s reliable assessor of performance. Participants' score on ADEPT was refl e variable ''SUM'' (Fig. 6). This ariable was computed as follows: successful tasks (0 to 5 points per run), total

tionnaire (Fig. 9). The mean VAS for the test run was 5.4 ± 3.8, and the mean AS for the true (second) run was 6.1 ± 3.4. The mean score for ``overall

performance on ADEPT'' was 6.1 ± 2.2, and the mean score on SUM was 6.6 ± 1.64. Thus, surgeons tend to underestimate performance on ADEPT slightly, but not significantly (p= 0.25, paired T-test). Nonetheless, the VAS does not seem to be a reliable predictor of objective performance because the confidence intervals for the scores of SUM all seem to overlap. In other words, surgeons' SUM score cannot be predicted by knowing his or her self-reported score for overall task performance.

urgeon was a ected through th

vexecution time of less than 150 seconds (1 point), number of ``perfect tasks'' (1 point), number of ``perfect probe tasks'' (1 point), number of ``perfect plate tasks'' (1 point), and score on test run (1 point for at least one task successful). The result from this calculation is a maximum SUM score of 10. The SUM distribution for the participants is normal in shape (Fig. 7), and the elements in SUM were in concordance (Fig. 8). Therefore, SUM seems to be a valid objective estimate of performance. The SUM score was plotted against reported performance scores resulting from the quesV

39

Chapter 2 Surgeons' perception of ADEPT as a valid instrument for measuring laparoscopic skills

Fig. 10. Is ADEPT a valid instrument, Fig. 11. Construct Validity corrected for individual performance?

SUM10,00

VA 0

-2

S Sc

ore

12

10

8

6

4

2

9,008,007,006,005,004,003,00 Numberof laparoscopicprocedures>100>100<100<100<50<50

SUM

1212

1010

88

66

44

22

Most of the surgeons (78.6%) were not familiar with ADEPT. The participants who were acquainted with ADEPT (21.4%) had either heard of it or read published work on if. None of the participants had been exposed previously to ADEPT. The participants can thus be considered unbiased because no one had ever worked with ADEPT previously, and most had never heard of it. Scores for validity given by the participants averaged 5.8, with a range of 0 to 10 distributed almost equally on the VAS scale. It is interesting to see that opinion on validity, corrected for individual performance by SUM, was not uniform (Fig. 10). Individuals performing well on ADEPT did not rate its validity any higher than individuals performing poorly. Interestingly, inexperienced surgeons (who had performed fewer than laparoscopic procedures) were not worse in terms of SUM score than experienced surgeons (who had performed more than 100 laparoscopic procedures; Fig. 11). Therefore, ADEPT seems capable of excluding experience in laparoscopic surgery as a determinant of importance in assessing psychomotor ability. Comparison of performance between test and true runs Fig. 12. Innate ability

Test- run (run 1)5,004,003,002,001,00,00

True

run

(run

2)

5,0

4,0

3,0

2,0

40

The Advanced Dundee Endoscopic Psychomotor Tester The participants showed some improvement in total time needed for runs (test run

ean time of 129 ± 54.4 sec versus true run mean time of 100 ± 39.3 sec). Also, the umber for successful task performances on the true run was higher than for the test

r, ecause the cordance between test run and true run. More

specifically, 72.7% of the participants expressed a true run score within one scoring distance from the test run. On paired-sample Student's T-test, both variables time and score, differed significantly (p = 0.001 for both variables), indicating some improvement or training effect on ADEPT. When the data are plotted, (Fig. 12), it can be seen that, in general, the participants who performed poorly on the test run did not perform much better on their true run. Also, the participants who initially performed well on the test run were likely to perform well on the true run. Only one participant had a score of 0 on the test run and a score of 5 on the true run. This participant obviously used the test run as an exercise to get fully acquainted with ADEPT.

iscussion

ial prerequisite for safe y other important factors influencing count. Factors, such as knowledge,

ersona room staff surrounding the surgeon, resulting from the actual performance of

or visual-spatial and perceptual abilities, ADEPT may be considered an important and feasible device for monitoring endoscopic skills assessment. Our study used ADEPT to focus on concordance between subjective and objective assessments of psychomotor skills performance. Attempts have been made to develop an objective instrument for estimating surgical skills performance. Structured questionnaires for observation of surgical performance are used [18], as well more objective outcome denominators such as precision and speed for laparoscopic task performance [1]. However, there still is a subjective component in these scores because they are derived from human observation. Also, important aspects such as pressure and range of motion are not taken into account.

mnrun (2.4 ± 1.3 versus 3.3. 1). Howeve innate ability was established bsurgeons' scores expressed a high con

D The usual format for measuring surgical endoscopic skills uses subjective assessment methods of performance, with or without the use of structured rating scales. More objective methods of assessment are important for the validation of performance. Furthermore, objective measurement methods may identify a person's innate ability for endoscopic task performance [12, 15]. Finally, objective measurement methods can be useful in establishing concurrent validity for a variety of laparoscopic training settings, which can vary from the well-known and cheap endoscopic training box or mannequin to more expensive laparoscopic skills training settings using animal material. Recently, skills training methods simulating laparoscopic procedures such as MIST-VR, LapSim, and Xitact, all computer controlled laparoscopic Virtual Reality trainers, have become the object of study [5, 14, 16]. Important abilities in endoscopic surgery are controlled precision, two-hand coordination, steadiness, and aiming [2, 11]. Psychomotor skill ac-quisition for laparoscopic surgery is a difficult but essentsurgery. It must be noted, however, that mansurgical performance must be taken into acp lity traits, skills, fatigue, operatinglaparoscopic instrumentation, and stress surgery [4]. By measuring psychomot

41

Chapter 2 In contrast, ADEPT, the device of our study, does take these aspects into account, along with elapsed time and successful task performance. Face validity and concurrent validity of ADEPT was ensured previously by Francis et al. [3] and Macmillan and Cuschieri [15].

Our first research question was: Are surgeons good estimators of their own performance on ADEPT? Most studies compare observatory results with the outcome of task performance. Surgeons' estimate of performance seldom is compared with an objective outcome parameter of task performance. Our results show that, in fact, little trust can be placed in participants' own estimate of performance. Because estimation of performance seems to be so difficult, the need for an objective scoring system to evaluate endoscopic task performance is obvious.

Our second research question was: Do surgeons perceive ADEPT to be a valid instrument for measuring laparoscopic skills? This question refers primarily to the external validity of ADEPT, comparing task performance with the clinical laparoscopic

tting. Before external validity can be assessed, internal validity of the instrument ust be secured. Our study shows ADEPT's internal validity to be high. Participants

e performing a standardized test procedure. By establishing the stable d reliable indexed performance score SUM, as shown by this study, ADEPT can be

gs, virtual training programs such as MIST-VR and Xitact have only

semwere assessed alikanregarded as a powerful objective scoring system for endoscopic psychomotor testing. For external validity, ADEPT has previously been shown to express a strong correlation with clinical competence [15]. However, in our study, surgeons did not rate validity of ADEPT highly. Moreover, the surgeons did not express uniformity in their opinion about ADEPT'S validity. Experienced surgeons did not perform worse or better on ADEPT than inexperienced surgeons. It is a fact that the endoscopic setting of ADEPT is not similar to the clinical laparoscopic situation. More precisely, ADEPT is a psychomotor tester, focusing on only one aspect of a complex area of interacting determinants (cognitions, skills, abilities) influencing laparoscopic surgery. This might explain why experienced surgeons do not perform any better. No surgeon had any previous practical experience with ADEPT.

Therefore, ADEPT indeed measures pure psychomotor ability without any of the additional determinants that make someone a good or experienced surgeon. The true purpose of ADEPT must be stressed before performance. Otherwise, frustration about performance may occur. Therefore, the nature of the question proposed to the participants actually is incorrect. The external validity of ADEPT would be estimated better by evaluating responses to the statement: I believe ADEPT is a valid instrument in measuring psychomotor skills.

As for using ADEPT to establish concurrent validity of laparoscopic (virtual) training settin

cently re been developed, and currently are being further refined and studied. Because surgeons must inevitably go through a learning curve in performing operations, in current practice, patients still have to pay the costs, not only literally speaking. The introduction of varied laparoscopic procedures calls for a more structured approach toward attainment of technical and ultimately clinical competence [8]. Computer-based Virtual Reality training programs, once validated, have the potential to solve many of the economic, educational, ethical, and safety issues related to the process of becoming a surgeon [6, 7].

42

The Advanced Dundee Endoscopic Psychomotor Tester The extent to which skills can be transferred from a skills training environment to the true clinical environment is likely to be dependent on the similarity of the setting. Advocates of surgical Virtual Reality simulator technology predict a revolution in surgical education. Although considerable enthusiasm for this concept exists, the developm

performance on ADEPT reflect innate ab

ent and implementation of simulation technology in surgical training has been limited by the absence of skills assessment devices [17]. The process and progress of teaching residents in a surgical skills laboratory environment for laparoscopic skills can be monitored by using ADEPT. For a variety of laparoscopic teaching programs, ADEPT may act as a reliability and validity check in addition to and in comparison with standard rating scales for performance.

Our third research question was: Doesility? Macmillan and Cuschieri [15] have shown ADEPT to be a system iden-

tifying aspects of performance that do not improve with practice. Indeed, in our small study analyzing two runs of five tasks, both runs had a high concordance of 72.7%. As discussed earlier and visualized by Figs. 11 and 12, scores on ADEPT seem independent of laparoscopic experience, and scores are stable over runs. However, some training effect from the use of ADEPT itself seems to play a significant role because both time and number of successful tasks are significantly higher among the second run statistics.

43

Chapter 2 References 1. Derossis AM, et al. (1998) "Development of a model for training and evaluation of laparoscopic

skills" Am J Surg 175: 482-487 2. Fleishman E (1996) Acquisition of skill, Academic Press, New York pp 147-167 3. Francis N, Hanna GB, Cuschieri A (2001) "Reliability of the Advanced Dundee Endoscopic Psychomotor Tester for bimanual tasks" Arch Surg 136: 40-43 4. Gallagher A, Crothers I, Satava R (2001) "Comprehensive objective assessment of fundamental

skills for laparoscopic surgery" In 9th International Congress of the European Association for Endoscopic Surgery, Maastricht, The Netherlands

5. Gantert WA, et al. (2001) "The “virtual patient”: a VR simulator for training in laparoscopic surgery: initial experience in a training course" In: 9th International Congress of the European

T, Clinch P (1996) "A microprocessor-controlled psychomotor tester for gery" Surg Endosc 10: 965-969

12. Hanna GB, Drew T, Cuschieri A (1997) "Technology for psychomotor skills testing in endoscopic surgery" Semin Laparosc Surg 4: 120-124

13. Hanna GB, Shimi S, Cuschieri A (1997) "Influence of direction of view, target-to-endoscope distance, and manipulation angle on endoscopic knot tying" Br J Surg 84 1460-1464

14. Hyltander A, Rhodin P, Liljegren E (2001) "Virtual reality training skills can be transferred to the operating room: a prospective and randomized study" In: 9th International Congress of the European Associating for Endoscopic Surgery, Maastricht, The Netherlands

15. Macmillan AI, Cuschieri A (1999) "Assessment of innate ability and skills for endoscopic manipulations by the Advanced Dundee Endoscopic Psychomotor Tester: predictive and concurrent validity" Am J Surg 177: 274-277

16. Satava M, Gallagher A, McGuigan J (2001) "Reliability and validity of the minimally invasive surgical trainer virtual reality system" In: 9th International Congress of the European Association for Endoscopic Surgery, Maastricht, The Netherlands

17. Smith CD, Stubbs J, HD (1998) "Simulation technology in surgical education: can we assess manipulative skills and what does it mean to the learner" In: Medicine meets virtual reality: art, science, technology: healthcare (R) Evolution

In: Westwood JD, et al. IOS Press and Ohmsa pp 379-380 18. Szalay D et al. (2000) "Using operative outcome to assess technical skill" Am J Surg 180: 234-237

Association for Endoscopic Surgery, Maastricht, The Netherlands 6. Grantcharov TP, et al. (2001) "Virtual reality computer simulation: an objective method for the

evaluation of laparoscopic surgical skills" Surg Endosc 15: 242-244 7. Haluck R, K, TM (2000) "Computers and virtual reality for surgical education in 21st Century" Arch Surg 135: 786-792 8. Hamdorf JM, Hall JC (2001) "Acquiring surgical skills" Br J Surg 87: 28-37 9. Hanna, GB,et al. (1998) "Computer-controlled endoscopic performances assessment system" Surg Endosc 12: 997-1000 10. Hanna GB, Cuschieri A (1999) "Influence of the optical axis-to-target view angle on endoscopic

task performance" Surg Endosc. 13: 371-375 11. Hanna GB, Drew

minimal access sur

44

Face-, expert and referent validity Chapter 3 Face-, expert- and referent validity of the Xitact LS500 laparoscopy simulator M. Schijven, J. Jakimowicz

Surg Endosc, 2002. 16: 1764-1770

47

Chapter 3 Abstract Background: This study was undertaken to establish face-, expert- and referent validity of the Xitact LS500; a Virtual Reality laparoscopic cholecystectomy simulator. Methods: A four-page, 20-item structured questionnaire was presented to 120 surgeons

Participants received an instructed hands-on ‘‘tour’’ on ere analyzed according to the level of experience of the

surgeon, esulting in an ‘‘expert group opinion’’ of 87 surgeons and a ‘‘referent group

Results: The majority of respondents believe Xitact has the potential to become a useful tool in teaching (93.1%) and measuring performance assessment (79.3%) in lap-aroscopic cholecystectomy. Expert and referent group opinion does not differ significantly on any of the presented statements. The opinion regarding the realism of the virtual laparoscopic cholecystectomy environment is favourable among both groups, although it is considered not yet perfect. The ‘‘haptic feedback’’ sensation of the Xitact is a parameter that needs further development. Conclusions: Both expert-and referent surgeons value Xitact to be an important and useful tool in the laparoscopic teaching setting. Further studies need to be performed to establish the construct validity of the simulator (e.g., to what extent is the simulator logically encompassed into a theoretical framework of acquiring skills, needed for the laparoscopic cholecystectomy) to measure shortening of learning curves on the laparo-scopic cholecystectomy procedure, and ultimately to justify its use in the surgical curriculum.

attending a surgical convention.the Xitact simulator. Data w

ropinion’’ of 33 surgeons.

48

Face-, expert and referent validity Introduction

rm paroscopic cholecystectomy correctly cannot be extrapolated from skills acquired

. However, there are no existing standards that must be met y a surgeon to practice this technique safely [13]. Moreover, there is no agreement

the end of the individual arning curve. In current medical practice, it is not only a challenge but merely a

nd effective teaching setting. The surgical skills laboratory may play an important role in the acquisition of skills in minimal access surgery, and surgical curricula should contain a laparoscopic skills training program [8, 12]. Such a training setting requires a stimulating, tension-free environment that must be highly similar to the actual working environment, and, most importantly, the surgical community itself must be willing to adopt its concept.

New technologies, such as Virtual Reality surgical simulators and objective methods of assessment, are powerful instruments that could improve a physician’s competence and overall quality of patient care if validated and implemented properly. Ongoing improvements in this technology suggest an important role for Virtual Reality and simulation in surgical education and training [6]. Virtual Reality simulators ideally present a computer-generated ‘‘natural’’ representation of an environment allowing sensory (sound, sight, and touch) interaction. Other high-risk professions, such as in the field of aviation, aerospace, maritime, military, and nuclear energy, have been using such simulators for the purpose of training difficult and demanding tasks. In doing so, these industries have reduced errors to nearly zero. Since 1955, examination on a flight simulator has been required by the aviation industry and military as a component for certification and annual re-certification. Currently, all military and commercial pilots must train and be certified in their technical skills on a flight simulator specific for the aircraft they will fly [11]. Although simulation and objective assessment methods for medical and surgical procedures are in their infancy, new technologies are challenging and gaining rapid interest in the surgical community.

For some time, laparoscopic cholecystectomy has replaced the open cholecystectomy as treatment-of-choice for symptomatic cholelithiasis. Skills needed to perfolafrom performing the open procedure, since the laparoscopic technique requires distinct psychomotor abilities, hand–eye coordination, and different skills than those needed for the open procedure. At present, residents-in-training are introduced to the procedure mainly by classical surgical apprenticeship, that is, by guided hands-on operating room experiencebwhatsoever on the method or the means with which to measure laparoscopic performance objectively.

It is inevitable for surgeons learning a certain surgical procedure to go through a learning curve. Acquiring laparoscopic surgical skills involves initial learning and further refinement of cognitive and psychomotor skills [2]. Related to the acquisition of such skills are performance errors. These errors will occur less often only when surgeons become more skilled, i.e., toward lenecessity to shorten learning curves safely in order to diminish avoidable errors in the clinical setting. Practice on patients is no longer considered acceptable. The Senate of Surgery of Great Britain and Ireland have declared that no surgeon should undertake any operative procedure unless competent to do so [9]. This statement is in dispute with the classical surgical apprenticeship model. One way to shortcut the learning curve safely outside the clinical setting is by means of creating an alternative, equally informative a

49

Chapter 3 Virtual Reality computer simulation of laparoscopic procedures is an era that is eveloping fast. The very nature of laparoscopic surgery makes it likely to benefit from

dy (expert opinion) as well as the potential trainee (referent

ddevelopments in Virtual Reality and tele-presence technology [3].

Logically, the more closely a Virtual Reality training instrument mimics the realism of the actual procedure in the operating room, the more valid it will be for the acquisition of surgical skill. Any useful surgical Virtual Reality simulator will need accurate, realistic depiction of detail combined with a high level of sensory interaction. Previous studies have shown potential and interest for Virtual Reality trainers in the field of laparoscopic surgery in terms of tutoring, training, and assessing skills and performance [2, 4, 5, 7, 9, 10, 14–18]. The Xitact LS500 is the Virtual Reality laparoscopic simulator under study. The value of a teaching instrument such as the Xitact LS500 can only be assigned if both surgeons who are beyond the learning curve of the procedure under stuopinion) believe in it. An important step in establishing the validity of any new technological equipment is the concept of face validity. Face validity addresses the question: to what extent does the instrument simulate what it is supposed to represent, e.g., the procedure of the laparoscopic cholecystectomy. The current study focuses on expert- and referent face-validity of the Xitact LS500 laparoscopic cholecystectomy simulator.

50

Face-, expert and referent validity Materials and Methods Subjects One hundred and twenty surgeons and surgical residents with no previous knowledge or exposure to the apparatus were introduced to the Xitact LS500. Eighty-seven participants were questioned at the 87th Annual Clinical Congress of the American College of Surgeons, held 7–12 October in New Orleans, LA. Thirty-three participants were questioned during the International Symposium of Laparoscopic Surgery, held 25–27 October in Paris, France. All participants were given an instructed ‘‘tour’’ to familiarize them with Xitact LS500, its features, and the laparoscopic cholecystectomy simulation. The ‘‘tour’’ incorporated a hands-on instruction of approximately half an hour by trained instructors. Afterwards, participants were asked to complete a questionnaire on virtual skills training and -testing, and on the experience with the Xitact LS500 laparoscopic cholecystectomy simulation more specifically. Questionnaire

cipant’s oncerning the

isual scene, haptic feedback, evaluation of location, deformation, movement, surgical techniques, design, and ergonomics of the simulator were presented on a 5-point ordinal answering scale. Value 1was assigned to ‘‘not realistic/good/useful,’’ value 5 to ‘‘very realistic/good/useful.’’ The usefulness of the simulator in terms of training/teaching capacities was questioned. Also, several open-ended questions concerning missing elements in Xitact, expectations toward Virtual Reality simulating settings and need for Virtual Reality surgical training procedures were proposed. Finally, eight statements concerning Xitact and the need for training by Virtual Reality were proposed.

The questionnaire consisted of a four-page, 20-item survey. Next to partidemographics and surgical laparoscopic experience, nine questions cv

51

Chapter 3 Apparatus Fig. 1. Xitact LS500 Simulator

52

Face-, expert and referent validity The Xitact LS500 laparoscopy simulator (Fig. 1) is a modular Virtual Reality training

rogram developed for training and education of a variety of laparoscopic skills. It is a physical object (the OpTable, or ‘‘virtual abdomen’’)

ith a computer program providing the visual image and haptic feedback. The program featured and under study is the laparoscopic cholecystectomy simulation. The Xitact LS500 is developed and registered by Xitact SA, Morges, Switzerland. Hardware The hardware of the Xitact LS500 is a modular system containing:

• MTower (Monitor Tower). The MTower (0.6 m. x 1.0 m. x 0.4 m., 20 kg) is an independent, freely movable tower that supports a flat panel 18-inch TFT monitor with a resolution of 1280 · 1024 pixels.

• OpTable (Operation Table). The OpTable-console (0.6 m. x 0.7 m x. 0.7 m.,

50 kg) houses the ‘‘abdomen’’ of the virtual patient, the endoscopic instruments, and the endoscopic camera. Two instruments are mounted in the OpTable, with the possibility of using a virtual third instrument by using a ‘‘freeze’’ software-generated option. The instruments have high-performance Force Feedback with five degrees of freedom, featuring performance, precision, and force feedback on original endoscopic instruments. One endoscope is mounted, with four degrees of freedom, offering several optical axis angles featuring a freeze picture switch combined with adjustable positioning possibilities. The console is topped by an exchangeable port-positioning matrix, enabling various trocar placement possibilities for different procedures. The OpTable’s height is adjustable according to the operator’s height, there is a Trendelenburg capacity, and the OpTable has a connected two-pedal foot switch for simulation of electrosurgical coagulation.

• Ebox (Electronic Box). The Ebox (0.6 m. x 0.7 m. x 0.6 m., 25 kg) houses the

electronics of the system and is connected to the Op-Table. The box includes an industrial personal computer, system memory 512 MB RAM, 20 GB Hard disk. Operating system Microsoft Windows 2000 Professional, CPU Intel Pentium III 1GHz; video: GeForce II GTS. The electric characteristics of the system include a mains voltage of 100–249 V, 50–60 Hz and a maximum power consumption of 750 W for the entire system.

phybrid simulator, combining a w

53

Chapter 3 Fig. 2. Xitact scenery

lowing scenes are featured: exposure of the abdominal cavity and the er region, dissection of Calot’s triangle, clipping and cutting of the cystic d duct. The system incorporates a combination of multimedia elements text, surgical video clips with voice-over, simulation recordings, and example ases. The tasks of the laparoscopic cholecystectomy procedure (insufflation, acement, exploration, clipping, cutting, separation, extraction, and closure) are rated (Fig. 2). The various aspects of the procedure can be assigned and in any desired combination. Different anatomical varieties of Calot’s triangle ntly under construction.

Software The folgallbladdartery anincludingpatient ctrocar pldemonstrepeated

curre

Computer Data recincorrectassessmeRecordinonstruc

tatistics

Data were analyzed using the Statistical Package for the Social Sciences, version 9.0.

are

orded by the computer include time per task and error rates (loss of and placement of clips). Real-time analysis of hand motions to give continuous nt of skills performance is possible as tool trajectories are recorded. gs of ‘‘economy of movement’’ for the performer are currently under tion. c

S

54

Face-, expert and referent validity Results Demographics One hundred and twenty surgeons participated in this study, originating from 19 different countries. Fifty percent of them are inhabitants of the United States, and 17% are inhabitants of France. The mean age of the participants is 44 years, ranging from 24 to 88 years of age. There are 13 females in the study (10.8%) and 107 males (89.2%). Dexterity results in 7.6% left-handedness, 76.3% right-handedness, and 16.1% ambidexterity. Of all participants, 96.6% are working as general surgeons, 0.8% as gynaecologists, 0.8% as thoracic surgeons, and 1.7% are from other occupations. Of all participants, 65.8% are qualified specialists, 10.3% are in their sixth year of training, 9.4% are in the fifth year of training, 4.3% are in the fourth year of training, 7.7% are in the third year of training, and 2.6% are in the second year of training.

Figure 3 represents the number of laparoscopic cholecystectomies performed by the participants. Thirty-eight respondents have performed fewer than or equal to 100 laparoscopic cholecystectomies. This group of respondents is considered to be the ‘‘referent group.’’ Eighty respondents have performed more than 100 laparoscopic holecystectomies, and this group is considered to be the ‘‘expert group’’.

ig. 3. Number of laparoscopic cholecystectomies

≤ 100 laparoscopic cholecystectomies, N=38) (> 100 laparoscopic cholecystectomies, N=80)

c F

Referent group (

100

80

60

40

>10050-100<500

20

0

P

erce

nt

□■ Expert group

55

Chapter 3 Table 1. Ratings

Total Referent group ≤ 100

Expert group > 100

RATINGS

realistic to 5: very realistic Ρ-

value* 1: not

Mean Mean SD Mean SD

N=33 /

N=87 Global realism of the virtual laparoscopic cholecystectomy environment in Xitact®

3.42 3.47 0.83 3.38 0.99 1.00

Visual realism 3.59 3.66 0.75 3.56 0.76 0.878 Reality of haptic feedback (tactile sensation) 3.07 2.98 1.00 3.09 0.87 0.680 Position / location of organs/structures 3.93 4.03 0.73 3.87 0.78 0.928 Reality of deformation organs/structures upon grasping or manipulating

3.38 3.34 0.84 3.41 0.73 0.922

Reality of movement of organs/structures upon grasping or manipulating

3.49 3.50 0.93 3.48 0.85 1.000

Reality of exposure using laparoscopic instruments

3.88 4.09 0.74 3.78 0.87 0.713

Reality of clipping 3.92 3.97 0.84 3.91 0.83 1.000 Reality of cutting 3.98 4.03 0.81 3.94 0.78 1.000 Reality of dissection 3.30 3.29 1.15 3.30 1.25 1.000 Reality of diathermia 3.62 3.53 1.06 3.67 1.16 0.908 Design of simulator 4.00 4.03 0.65 4.00 0.66 1.000 Overall ergonomics 3.87 4.03 0.63 3.81 0.77 0.688 Trocar port placement 3.93 4.00 0.78 3.91 0.86 1.000 Freedom of movement of instruments 3.87 4.00 0.77 3.81 0.88 0.836 Force feedback while inte cting with organs

3.33 3.42 1.20 3.27 1.06 0.995 ra

*Kolmogorov-Smirnov test, two-tailed

ace validity

able 1 refers to the similarity of the Xitact laparoscopic cholecystectomy environment surgeons’ actual experience with or perception toward clinical laparoscopic

holecystectomy. Expert opinion is represented by surgeons having performed more an 100 laparo ic cholecystectomies, referent opinion by surgeons having

erformed fewer than 100. Compared to the overall mean (‘‘Total mean’’), the expert pinion tends to ore conservative. Nevertheless, overall opinion about Xitact is avourable, with the item’s v ual rea , pos locat realism of exposure,

The exterior of the apparatus re-h scores approaching 4.0 (design of

simulator, overall ergonomics, trocar port placement, freedom of movement of instruments). The reality of the haptic feedback, as represented by ‘‘reality of haptic feedback’’ and ‘‘force feedback while interacting with organs,’’ is considered to be not very realistic. Referent and expert group are uniform in their opinion on presented questions.

F Ttocth sco

be

ppo mf is lism ition/ ion, clipping, cutting, and diathermia scoring above 3.5. sembles the actual working situation quite well, wit

56

Face-, expert and referent validity Table 2. Statements

NT D e

FET*

STATEME isagree Agre Do not

knowI believe it is necessary for surgical residents-in-training to participate in a basic laparoscopic skills course before operating on patients

5.1 %83.8 % E

T % R

96.2 % E8 %

T .3 % R

94.0 % 88.9

0.9 %2.

T R

0.369

I believe it is important to train surgical residents-in-training using laparoscopic virtual reality

Xitact® before operating on pati

11.17.1 % R 8.

T 74.3 % R

E

.2 %8.6 % R .8 %

settings such as ents

2 % T

9 % E

83.6 %

87.3 %

5 T

3 E

0.199

I believe it is important to monitor surgical residents-in-training in their progress on laparoscopic skills throughout their residency

g

11.16.9.

T R E

.7 %

.1 %

.4 %

by using a laparoscopic virtual reality settin

1 % T 2 % R

0 % E

81.2 %75.7 %84.6 %

7 T 86

R E

0.340

I believe Xitact® is an useful instrument for measuring performance assessment in laparosco

12. T 13.13.0 % E

T R

66.2 % E

8 % 2

20.8 % E pic

procedures

9 % 5 % R

67.2 % 70.3 %

19.16.

T % R

1.000

I believe Xitact® could become a useful insfor measuring p

trumeerformance assessment in

ures

9. 8.1 % R

10.

T 75.7 % R

E

2 16.2 % R

.1 %

nt

laparoscopic proced

5 % T

4 % E

79.3 %

80.5 %

11. % T

9 E

1.000

I believe Xitact® provides a user-friendly laparoscopic skills

4.

6.

T R E

.1 %

.1 %

.3 %

environment for training

3 % T

6 % E

89.6 %91.9 %88.2 %

6 T 85

R E

0.174

I believe Xitact® could become a useful tool in cholecystectomy

0.

1.

T R E

.0 %

.7 %

.8 %

teaching laparoscopic

9 % T

3 % E

93.1 %97.3 %90.9 %

6 T 2 R 7 E

1.000

I believe Xitact has potsimulator for laparoscop

® ential to be a cost-effectic cholecystectomy

8. 8. 9.

T R E

3 9 2

ive

8 % T6 % R2 % E

54.9 %48.6 %56.6 %

36.42.

% T % R

34. % E

1.000

T 7, R rent N= ishe Test, two-

spondents believe it is necessary to participate in a basic laparoscopic

omy.

=total group N=120, E= expert group N=8 =refe group 33. *F r Exact tailed

Training capacities Almost all reskills course before operating on patients (94%); in fact, experts feel this even more strongly (96.2%, Table 2). Respondents feel it is important to train surgical residents using laparoscopic Virtual Reality settings such as Xitact (83.6%). Again, experts agree even more so (87.3%), but groups do not differ significantly in their opinion (FET 0.199). High ratings are obtained for the statement that Xitact could become a useful tool in teaching laparoscopic cholecystectomy (93.1% agree), and the referent group is particularly enthusiastic (97.3% agree). Respondents also believe Xitact provides a user-friendly environment for training of laparoscopic skills (89.6%). Many respondent feel Xitact is already useful (67.2%), but more respondents believe it could become a useful tool (79.3%) in measuring performance assessment in laparoscopic procedures. Related to this, is the potential of Xitact to monitor individual progress. Respondents believe Xitact to be an important instrument (81.2%) to monitor progress.

In summary, the majority of respondents believe and agree Xitact could become a useful tool in teaching and monitoring progress in laparoscopic cholecystect

57


Total

t gr≤ 100

pe>

Referen oup Ex rt group100

R1: not useful to 5: very useful

Mean ean

Ρ

N=87

ATINGS

Mean SD M SD

-value*

N=33 /

M 3.97 3.91 1.10 0.889 edical student training / education 4.12 1.02 S 4.32 8 0urgical residents training / education 4.40 0.74 4.2 0.79 .983 Se

6 0.937 urgical specialist training / continuing 3.93 ducation

4.09 1.11 3.8 1.15

O 3.86 8 0verall laparoscopic error reduction 4.03 1.04 3.7 1.12 .985 O 3.96 9 0.999 verall laparoscopic skills enhancement 4.09 0.98 3.8 1.11

* S um X es rg ul herX to be most useful for s si trai ecially by t Table 3). X terms of u skills enhancement. Discussion Results o and u p ong erent and

ic holecystectomy as presented by Xitact. The term validity means no less and no more

representation and extrapolation of the results from the research opulation to the target population under study (internal validity). In our study, there

First, th

ion. This so-called ‘‘Hawthorne effect’’ is partially accounted for by iving the same amount of attention to both respondent groups, so uniformity of the

overall opinion is not troubled by differences in attention.

Kolmogorov-Smirnov test, two-tailed

uitability of Xitact related to the surgical curricul

itact is considered to be of use in all stag of the su ical curric um and t eafter. itact is considered urgical re dents-in- ning, esp

he referent group themselves (4.40,itact is considered to be of value in

error red ction and

f this study show a favourable niform o inion am both refexpert groups regarding the face validity of the Virtual Reality laparoscopcthan an accurate pare two target populations, that is, expert (qualified surgeons experienced in laparoscopic cholecystectomy) and referents (surgical trainees). The term face validity refers to the resemblance of a test task to the actual clinical task, in other words: does the task on the apparatus resemble what it claims to resemble? In our study, this means the extent to which the Virtual Reality laparoscopy on Xitact mimics the reality of the clinical laparoscopic cholecystectomy. In order to make reliable extrapolations, results must be as free as possible from systemic, non-random sources of error or bias. This opinion, and thus the validity of this study, may be influenced by several systemic sources of bias or adverse evaluation effects [1]. Threats to validity

e opinion may be influenced by the individual attention given to the-respondent during his or her performance, and favourable responses may occur be-cause of this attentg

58

Face-, expert and referent validity Furthermore, there is a social desirability effect apparent by the 100% response rate.

o respondent refused to complete the questionnaire. To what extent referents are feel obliged to fill in a qu n e t

icult to measure. gain, the ctors disp no diff e in attention to either group. By gu ei ny s f res , researchers aimed for psychological freedom in answering.

e mentio is ‘Py on ct me ancient k olo rv s ure ve with an to this set it might b

nt’s opinion was influenced by the mere enthusiasm of Xitact’s developers, giving the on ion. la res

e nove th ara ein rac bec he referent group may be particularly susceptible to

is phenomenon, having being around for less time in the working field of surgery g display of apparatus by the industry.

able 1 does show a slightly more favourable attitude among referents.

ed themselves by the classical surgical

ems not to be the case with Xitact. The eagerness of referents to train with it and the nthusiasm and need for ex-vivo training modules as expressed by the experts seem to

setting for the implementation of this new type of technology. n important prerequisite for the face validity of Xitact is its haptic potential. Grasping

plementation

00 simulator will be dependent on a variety of factors.

Njust polite or estion aire in exchang for a chance o ‘‘play’’ with the simulator is diff A instru layed er c

po esen

arante ng ano mou use o the ns

Another effect that must b ned the ‘ gmali effe ,’’ na d afterPygmalion, a king figure from Gree myth gy, ca ing a culpt out ofstone so beautifully he fell in loresponde

it. Tr slated ting, e that a

present at the exhibition and dem strat And stly, ponses may befavourably influenced by the mer lty of e app tus, b g att tive ause ofits premiere at the exhibitions. Tthand relatively unprotected against the temptinT

Nevertheless, even if these phenomena are of influence, they are not reflected in the outcomes of the study as expressed in the firm uniformity of opinion among both groups. One might expect experts, trainapprenticeship model, to be conservative in their opinion on novelties. However, this seeprovide an ideal Aobjects without weight, shape, or texture makes a virtual environment ghostly and insubstantial [3]. Touch, however, is one of the most difficult of sensations to mimic in a Virtual Reality setting. Xitact, providing a haptic sensation and feedback, has a potential that exceeds that of other Virtual Reality laparoscopic surgery simulators currently available. Not considered very realistic to the clinical laparoscopic situation by the respondents, it is beyond a rudimentary sensation, and this is precisely why Xitact’s technology must be considered promising. However, this aspect must be further refined.

The training capacities of Xitact are vast since it is a computer based technology. Xitact, therefore, not only is a skills trainer but can also be seen as a rich source of data which can be used to objectively assess individual improvement in performance and compare performance across peer groups. Hence, its data representation capacity combines a modular training schedule with an underlying basis for constructive feedback and options for a personalized, rational training schedule. Im

The success of the Xitact LS5The scenery needs further refinement, especially with respect to the reality of dissection and haptic feedback. Content and construct validity need to be established and are currently under study. As with any training tool, there will be a simulator learning curve that has to be taken into account.

59

Chapter 3 The length of this curve is dependent on the quality of the interface for the human–computer interaction and will be determined by the initial learning rate. In order to reflect these rates, the economy of movement as measured by the simulator will be a parameter of utmost importance.

For widespread acceptance of the Xitact by the surgical community, it must be demonstrated that skills acquired via the Xitact are transferable to the operating room environment. This so-called ‘‘Virtual Reality to operating room study’’ can only validly be initiated after face-, construct-, and content validity of Xitact are established. Retail costs of the simulator are about $150,000. A major advantage of the apparatus is the potential to train residents adequately, while reducing the need for (live) animal material with its associated costs and infrastructure. Ultimately, there is justification for purchase when shorter learning curves can be demonstrated by a valid ‘‘VR to OR’’ study. Next, the more surgical modules available (laparoscopic Nissen fundoplication, laparoscopic inguinal hernia, laparoscopic appendectomy, laparoscopic colon surgery, bariatric procedures, and laparoscopic gynaecology operations), the more interesting the apparatus will be.

60

Face-, expert and referent validity References 1. Bouter LM, van Dongen MCJM (1988) Epidemiologisch onderzoek: opzet en interpretatie (Epidemiologic Research: Desig2.

n and Interpretation) Bohn, Scheltema & Holkema, Utrecht

scopic surgery" Ann Chir Gynaecol 83: 137–141 l. (2000) "A comparison between randomly alternating imaging, normal

laparoscopic imaging, and virtual reality training in laparoscopic psychomotor skill acquisition" Am J Surg 180: 208–211

10. McNatt SS, Smith SGT (2001) "A computer-based laparoscopic skills assessment device differentiates experienced from novice surgeons" Surg Endosc 15: 1085–1089

11. Satava RM (2001) "Accomplishments and challenges of surgical simulation. Dawning of the next- generation surgical education" Surg Endosc 15: 232–241 12. Scott DJ et al. (2000) "Laparoscopic training on bench models: better and more cost effective than operating room experience?" J Am Coll Surg 191: 272–283 13. Shaper NJ, Harrison M, Bates T (1996) "Impact of laparoscopic cholecystectomy on surgical training" Am R Coll Surg Engl 78: 39–42 14. Summers AN et al. (1999) "Acquisition of surgical skills: a randomized trail of didactic, videotape, and computer-based training" Surgery 126: 330–336 15. Taffinder N et al. (1988) "Validation of virtual reality to teach and assess psychomotor skills in

laparoscopic surgery: Results from randomized controlled studies using the MIST VR laparoscopic simulator" In: Westwood JD (ed.) Technology and Informatics 50: Proceedings of Medicine Meets Virtual Reality, Jan 28–31, 1988. IOS Press and Ohmsha

Amsterdam, San Diego, pp 124–130 16. Torkington J et al. (2001) "Skill transfer from virtual reality to a real laparoscopic task" Surg Endosc 15: 1076–1079 17. Wilson MS et al. (1997) "MIST VR: a virtual reality trainer for laparoscopic surgery assesses performance" Am R Coll Surg Engl 79: 403–404 18. Witzke DB et al. (2001) "Immersive virtual reality used as a platform for perioperative training for surgical residents" Stud Health Technol Inform 81: 577–583

Chaudhry A et al. (1999) "Learning rate for laparoscopic surgical skills on MIST VR, a virtual reality simulator: quality of human– computer interface" Ann R Coll Surg Engl 81: 281–286

3. Coleman J, Nduka CC, Darzi A (1994) "Virtual reality and laparoscopic surgery" Br J Surg 81: 1709–1711 4. Derossis AM, Antoniuk M, Fried GM (1999) "Evaluation of laparoscopic skills: a 2-year follow-up during residency training" Can J Surg 42: 293–296 5. Gallagher AG et al. (1999) "Virtual reality training in laparoscopic surgery: A preliminary

assessment of minimally invasive surgical trainer virtual reality (MIST VR)" Endoscopy 31: 310–313 6. Gorman PJ, Meijer AH, Krummel TM (1999) "Simulation and virtual reality in surgical education: real or unreal?" Arch Surg 134: 1203–1208 7. Grantcharov TP et al. (2001) "Virtual reality computer simulation: an objective method for the

evaluation of laparoscopic surgical skills" Surg Endosc 15: 242–244 8. Jakimowicz JJ (1994) "The European Association for Endoscopic Surgery recommendations for

training in laparo9. Jordan J-A et a

61

Construct validity Chapter 4 Construct validity: Experts and novices performing on the Xitact LS500 laparoscopy simulator M. Schijven, J. Jakimowicz Surg Endosc, 2003. 17: 803-810

63

Chapter 4 Abstract Background: This study was undertaken to establish construct validity of the Xitact LS500, a Virtual Reality laparoscopic cholecystectomy simulator. The primary research statement is: ¨The clip-and-cut task on the Xitact LS500 Virtual Reality laparoscopic holecystectomy simulator mimics the surgical procedure of the clipping and cutting of

ring the laparoscopic cholecystectomy adequately.’’

ethods: According to the level of experience of the surgeon, an ‘‘expert group from 37 surgeons having performed over 100

paroscopic cholecystectomies, and a ‘‘novice group opinion’’ was formed resulting of 37 surgeons having no experience at all with performing laparoscopic cholecystectomy. Participants received an instructed hands-on ‘‘tour’’ on the Xitact simulator and performed three formal simulation runs. Results: The ‘‘novice group’’ is younger and more surgeons are female. Performance scores in the ‘‘expert’’ group are significantly higher on the second (p-value 0.011) and third (p-value 0.005) run, compared to the novices’ scores. Experts are significantly faster on completion of all three runs. There is an increase in score over runs in both groups, which is statistically significant in the ‘‘expert group’’. Less than one-third of surgeons in either group are able to correctly predict their performance score as generated by Xitact. Both ‘‘experts’’ and ‘‘novices’’ feel it is useful to train with Xitact LS500 in the surgical curriculum. Conclusions: Three hypotheses, formulated to operationalize the primary research statement, could be answered affirmatively. Although further validation studies are needed, the Xitact LS500 simulator seems to be able to discriminate between expert and novice surgeons in this research setting, and thus the construct for this setting is considered to be valid.

cthe cystic duct and artery du Mopinion’’ was formed resultingla

64

Construct validity Introduction

in the lassic surgical curriculum [21]. Shortened working weeks, tight surgical curricula, and,

more, it may satisfy the need for accurate and bjective assessment of technical process and skill acquisition, as for the moment,

to mimic visual–spatial and real-time characteristics f the procedure and to provide realistic haptic feedback. Besides, it must be able to

alistic anatomical scenery with haptic feedback, may be an effective tool for teaching and testing laparoscopic surgical skills.

One of the most important research lines in the development of the Xitact is the attention to the psychometric properties of the instrument, that is, to its validity and reliability. The concept of validity addresses the question, ‘‘Do we measure what we intend to measure?’’ [16]. Basically, valid innovations are innovations with low non-random (systematic) errors. Validity can be categorized into different types: face, content, construct, criterion-related and concurrent validity. The type of validity referred to must relate to the purpose of the concept of interest [29]. The concept of construct validity is often regarded as the central theme in validation studies [5]. Ultimately, a new Virtual Reality simulator should pass multiple aspects of validity to become a reliable skills trainer and predictor of performance. Previous studies with the Xitact LS500 laparoscopy simulator successfully addressed the issues of face and content validity of the laparoscopic cholecystectomy scene.

The face validity study addressed the question: ¨To what extent does the Xitact LS500 simulate what it is supposed to simulate, e.g., the procedure of the lapa-roscopic cholecystectomy?’’ [24]. Although not a formal validity concept, it refers to the subjective opinion about a test, e.g., about its appropriateness for its intended use of purpose within the target population. Face validity must be considered extremely

Surgical skills, and in particular, complex psychomotor skills as needed in laparoscopic surgery, are in part innate, and can in part be learned from extensive, repetitive practice [25]. Although many skills and traits are needed to be a competent surgeon, the element of technical competence is eminent. However, the teaching and testing of technical skill is known to be of the least systematic or standardized components cmore recently, medico-legal considerations stress the acquisition of surgical skill. This dilemma, combined with the fast evolving capacities and implementation of the computer in medicine, makes Virtual Reality in surgical training and education a phenomenon gaining rapid interest of the surgical community. The greatest power of Virtual Reality medical simulation is the opportunity to try and fail without consequence for the patient [23]. Furtherothere is in fact none [19]. The nature of laparoscopic surgery makes it likely to benefit from developments in Virtual Reality [3]. Previous studies have shown potential and interest for Virtual Reality simulators in the field of laparoscopic surgery in terms of tutoring, training, and assessment of skill and performance [2, 4, 6, 7, 10, 11, 13, 15, 26, 28, 30].

A valid Virtual Reality simulator provides an environment that closely approximates the characteristics of the environment in which the task eventually will be performed [20]. It must be ableoevaluate the performance of the procedure under study objectively. Only then will the non-stressful environment of the virtual training setting enhance both level of skill and level of confidence of the trainee and lead him or her gently into a more advanced state of performance. The Xitact LS500, being one of the latest Virtual Reality training simulators combining re

65

Chapter 4 important for a test’s practical utility and success of implementation [14].

he content validation of the clipping-and-cutting task of the Xitact LS500 addressed

ng of the cystic duct and artery during the laparoscopic cholecystectomy adequate

Tthe question, ‘‘Does the simulation measure all relevant dimensions of the construct under study?’’ Content validation was obtained by thorough search of the literature and by using results of a focus-group meeting held among expert laparoscopic surgeons.

The next step in the validation of the Xitact LS500 laparoscopic simulator is the establishment of the construct validity of the system. Construct validity refers to the concept that a novelty actually mimics what it intends to mimic, by direct or indirect objective standards. It is satisfied when test performance is logical and con-sistent with parameters of interest [14]. Fundamentally, it is concerned with explaining individual differences in scores among subjects by relating the various outcomes with anticipated ones. A valid system should be able to differentiate between different levels of skill. One way to establish content validity for the Xitact LS500 is comparing scores within and between experts and novices in laparoscopic cholecystectomy.

The current study focuses on the concept of construct validity of the Xitact LS500 laparoscopic cholecystectomy simulator. Translated to our setting, the primary research statement is: ¨The clip-cut task on the Xitact LS500 Virtual Reality laparoscopic cholecystectomy simulator mimics the surgical procedure of the clipping and cutti

ly.’’

66

Construct validity Fig. 1. Xitact LS500 Simulator

Three hypotheses were formulated to explore the primary research statement. First hypothesis: Performance scores derived from the Xitact cholecystectomy clip-and-cut task by experts in clinical laparoscopic cholecystectomy are significantly higher than the performance score derived by novices in clinical laparoscopic cholecystectomy. Second hypothesis: Performance scores derived from the Xitact cholecystectomy clip-and-cut task are related to the clinical laparoscopic cholecystectomy experience of the participant. Third hypothesis: Performance scores of experts and novices in clinical laparoscopic cholecystectomy improve over runs of the Xitact cholecystectomy clip-and-cut task.

67

Chapter 4 Materials and Methods Subjects Fifty-seven attendants of the 8th World Congress of Endoscopic Surgery, held 13–16 March 2002 in New York, participated. Also, 34 surgical residents attending the Basic Surgical Skills Courses in the Academic Hospital of Leiden or at the Catharina Hospital in Eindhoven, The Netherlands, held May 2002, participated. All participants were given an instructed one-on-one ‘‘tour’’ to familiarize them with Xitact LS500, its features, and the laparoscopic cholecystectomy simulation. The ‘‘tour’’ also featured a voice-over instruction video of the task under study, e.g., the clipping and cutting of the cystic artery and cystic duct. Trained instructors gave a detailed explanation of possible errors in the procedure, as well as hands-on instruction in the task under study. Three runs were performed by each participant, as earlier studies with laparoscopic surgical Virtual Reality trainers showed an early familiarization with the scenery within three runs [2]. Afterward, participants were asked to complete a questionnaire on virtual skills training and testing. For the purpose of this study, only surgeons having performed over 100 clinical laparoscopic cholecystectomies were selected for the ‘‘expert’’ group (n = 37), and only residents/interns with no experience in laparoscopic cholecystectomy were selected for the ‘‘novice’’ group (n = 37). Apparatus The Xitact LS500 Laparoscopy Simulator (Fig. 1) is a modular Virtual Reality training program developed for training and education of a variety of laparoscopic skills. It is a hybrid simulator, combining a physical object (the OpTable, or ‘‘virtual abdomen’’) with a computer program providing the visual image and haptic feedback. The program featured and under study is the clipping-and-cutting (´clip-cut´) task of the laparoscopic cholecystectomy simulation. The Xitact LS500 is developed and registered by Xitact SA, Morges, Switzerland. Composition of sum-score

sum-score was computed to estimate the ‘‘end result’’ of participants’ p rformance. sed and lap for

e for the Social Sciences, version 9.0.

A eItems of interest were: time needed for completion of run; number of clips u

laced on cystic artery or duct; number of clips lost; partial closure and overpartery and duct; clipping on clips; clipping multiple or in-adverse structures; correct placement of first medial, second medial, and lateral clips on artery and duct; correct location of cut for cystic artery and duct; cutting on clips; and finally, the cutting of multiple or in-adverse structures. The sum-score was computed by subtracting weighted and expert-rated error scores for each possible outcome of the procedure, of a maximum score of 100%. Most severely weighted error—resulting in a subtraction of 100%—was clipping or cutting of the common bile duct. Clipping or cutting of the hepatic artery resulted in a subtraction of 80%. The scale was checked for internal consistency using Cronbach’s alpha, resulting in an overall score of 0.47 with no negative correlations for the 15 items. Data were analyzed using the Statistical Packag

68

Construct validity Questionnaire A questionnaire consisting of a two-page survey was given to each participant. Next to

e participant’s demographics and surgical laparoscopic experience, questions

vice’’ roup, the majority (89.2%) was Dutch. The mean age of the participants in the

oup is 44 years, ranging from 28 to 61years of age. The mean age of the articipants in the ‘‘novice’’ group is 29 years, ranging from 23 to 58 years. Thus, there

ral surgeons and 2.8% as paediatric rgeons. Of the novices, 17% are interns, 21.6% are in training to be general

training to be emergency medicine specialists, 8.1% are in aining to be orthopaedic surgeons, 2.7% are in training to be thoracic surgeons, and

thconcerning the global realism, visual scene, haptic feedback, movement, and surgical techniques of the ´clip-cut´ task on the simulator were presented. A 5-point ordinal Likert answering scale was constructed. Value 1was assigned to ‘‘not realistic/good/useful,’’ and value 5 to ‘‘very realistic/good/useful.’’ The usefulness of the simulator in terms of training/teaching capacities and error reduction was questioned. Also, participants were asked to score themselves for performance on Xitact. Results Demographics Seventy-four surgeons and surgical residents/interns were selected for in this study, originating from 17 different countries. Of the ‘‘expert’’ group, the majority (32%) were inhabitants of the United States; 10.8% were UK residents. Of the ‘‘nog‘‘expert’’ grpis a significant difference in age distribution among groups (p-value Students´ T-test: 0.000). There are 20 females and 54 males participating. In the ‘‘expert’’ group, the per-centage of females is 10.8%; in the ‘‘novice’’ group, 43%. Therefore, sex is unequally represented among groups (p-value Pearson’s chi-square: 0.002). Dexterity is more or less equally distributed among groups, with 86% of right-handedness in the ‘‘expert’’ group and 73% of right-handedness in the ‘‘novice’’ group.

Of all experts, 97.2% work as genesusurgeons, 10.8% are intr2.7% are in training to become urologists. There are three qualified specialists in this group: a gynaecologist, a general surgeon, and a thoracic surgeon.

69


Total

Novices

Experts

RATINGS 1: not realistic to 5: very realistic


Ρ-value*

N=37 /

N=37 Global realism of the clipping-and-cutting task in Xitact

3.72 3.74 0.51 3.70 0.70 1.000

Visual realism of the clipping-and-cutting task in Xitact

3.62 3.55 0.73 3.68 0.75 1.000

Reality of haptic feedback (tactile sensation) of the clip

3.42 3.67 0.76 3.17 1.08 0.124 ping-and-cutting task in Xitact

lity of movement of organs/structures in ing-and-cutting task in Xitact upon

anipulating

3.63

3.65

0.88

3.62

0.79

0.988

Reathe clippgrasping or mReality of exposure 3.92 3.93 0.61 3.92 0.77 0.947 Reality of clipping 3.92 3.91 0.67 3.92 0.72 1.000 Reality of cutting 3.87 3.82 0.72 3.92 0.86 0.997

*Kolmogorov-Smirnov test, two-tailed Face validity Table 1 refers to the similarity of the Xitact laparoscopic cholecystectomy environment to surgeon’s actual experience with or perception toward clinical laparoscopic cholecystectomy. There is no significant difference in opinion regarding the perception of Xitacts´ reality. Table 2. Comparing novices and experts over runs between groups

RUN 1 RUN 2 RUN 3 Sum-score

Cat Label

Novices Experts Novices Experts Novices Experts

0–20 I Very inadequate performance

21.6 0.0 8.1 5.4 5.4 0.0

21-40 II Inadequate performance

0.0 2.7 8.1 5.4 5.4 2.7

41-60 III Questionable performance

10.8 21.6 16.2 5.4 21.6 2.7

61-80 IV Adequate performance

24.3 21.6 32.4 16.2 16.2 18.9

81 -100

V Highly adequate performance

43.2 54.1 35.1 67.6 48.6 75.7

SUM 1 SUM 2 SUM 3 Kruskall–Wallis test (categorized sum-score) 0.180 0.011* 0.005*

*Asymp. Sig. (2-tailed)

70

Construct validity Fig. 2. Time Fig. 3. Sum-scores

T ng novices and experts o un iable 3. Compari ver r s with n group

Category ovMean rank

ertMean rank

s

Run N ices, Exp s,

1 Sum 1 1.99 1.78

2 Sum 2 1.92 2.00 3 Sum 3 2.09 2.22

Friedman test (categorized sum-score ) 0.648 0.041* *Asymp. Sig. (2-tailed) Construct validity

igure 2 refers to the time needed for completion of the clip-cut task. Comparing m u Stud -test, gro significan needed f tion o run (p-value of 0.000, F 3 r r e end-para r ‘‘sco which was calcu ftethe clip-an -c ean sc e novice group were 64.6 for run 1, 67.6 for run 2, and 71 Mean scores of the expert group were 77.6 for run 1, 77.6 for run 2, an run Scores re clu in r to ruct five c ie T ). O ns, s s betw novi nd ex s see e ly, a e statistically significant for the second (p-val0.011) and thi 5) run. Wit roup e is an increas es the sequence s (Table 3). In the expert group, because of smaller standard de on hi e is stati ly sig t (p 0.04

F

eansf each

sing ents T ups differ00, and 0.

tly in timectively).

or comple0.0 000, respe

igure efed

s to thut task. M.8 for run 3.

meteores of th

re,’’ lated a r completion of

d 86.4 for 3. we stered orde constategor

m to s ofdiff

performance (r increasing

able 2nd ar

ver ru core een ces a pertue

rd (p-value 0.00of three run

hin g s, ther e in scor over

viati s, t s increas stical nifican -value 1).

ExpertNovice

600 Sec 700

400

500

300 200 100

0

Run 1

Run 2

Run 3ExpNovice

e 120

100

60

-

ert

Scor

80

40

20

0

20

Sum 1

Sum 2

Sum 3

71

Chapter 4 Table 4. Us for skills training/efulness of Xitact’s clip-and-cut scene error reduction

Total Novices Experts RA GS 1: not eful to 5: very useful


P-value*

N=37

/ N=37

TIN us

Medical student training/education 3.9 4.05 1.12 3.91 1.22 0.994 9 Surg residents training/education 4.4 4.65 0.55 4.27 0.90 0.399 ical 4 Surgic pecialist training/education 3.7 3.93 0.89 3.65 1.16 0.997 al s 8 Overall laparo rror reduc 3.91 4.03 0.67 3.81 0.94 0.765 scopic e tion Overal laparoscopic skills enhancement 3.98 4.20 0.66 3.81 1.04 0.884 l

*Kolmogorov–Sminov test, two-tailed Table 5. Comparing self-reported scores with categorized sum-sore

Novices Experts N=35 % N=34 %

17 8 10

48.6 22.9 28.6

21 10 3

61.8 29.4 8.8

Within groups Self-re rted end score categorized < sum 3 zed Self-re rted end score categorized = sum 3 ized Self- rted end score categorized > sum 3 ized Wilcoxon Sig , Asymp. Sig. (2-tailed)

0.214

popo

categoricategor

repo categor

ned Ranks Test 0.001*

Between groups Mann-Whitney U Test

0.170 * Asymp. Sig. (2-tailed) Questionnaire Both novices and feel the clip-cut task of the Xitact roscopic cho tectomy set ery useful for surgical residents’ training and education. To a e simulato r training m udents

ists. Xitact is considered to be a useful instrument for laparoscopic t (Table 4). Experts and novices are uniform in

there seems to be no significant difference in responses. Asked for

experts lapalecys ting is v

somewhat lesser extent, they feel th r is useful fo edical stand surgical specialerror reduction and skills enhancementheir opinion, astheir own performance on the clip-and-cut task, the self-reported and categorized score in the novice group does not seem to differ significantly from the Xitact generated categorized score (Table 5). In the expert group, there is a difference, e.g., experts tend to underestimate their own performance. For both groups, less than one-third were able to predict their score accurately. Novice and expert groups do not differ in their predictive abilities for the Xitact score.

72

Construct validity Discussion Evaluating validity is an important, but often neglected part of the development of innovations. It is very important to realize that we cannot study well what we cannot measure well. To date, limited data exist concerning the assessment of surgical simulators in the teaching environment. Before surgical simulators can be used to train and assess surgical skill formally, they must be clearly shown to be both reliable and valid [19]. Fortunately, more and more studies are becoming available concerning the validation of new Virtual Reality and simulation system in teaching and training of surgical (laparoscopic) skills [11, 19]. Unique to the concept of Xitact’s development is that the software is being build and refined not only by software engineers. Assumptions and boundaries of the simulation are determined by a board of experts within the laparoscopic surgical community. Validation of the simulation is being assessed per scene (inspection, clip-and-cut, dissection), so that important adjustments can be made before there is a fixed end product. Constructing a sum-score for the purpose of this study and for evaluation purposes has been an arbitrary procedure in itself. The clip-cut exercise is a multidimensional procedure, as it demands multiple skills and insights of the participants. The internal consistency of the items making up the sum score, as measured by Cronbach’s alpha, is therefore relatively low. Deletion of items out of the scale would have increased alpha, but items were considered to be essential for the scale and contributing to the end score.

The current study compares the results of an expert group of laparoscopic surgeons with a novice group, performing the clip-cut task on the Xitact laparoscopic cholecystectomy simulator. Age and sex were unequally distributed among groups, but this is more or less inevitable as novices tend to be younger and nowadays, more females are in their surgical residences than ever before. Briefly, the face validity of the clip-cut task was assessed. A face validity study concerning the overall realism of the Xitact laparoscopic cholecystectomy scenery has been performed, but authors felt it would be appropriate for the subtask of the clip-and-cut to look into this concept even further [24]. Again, there is a favourable opinion on the realism of the scene, and ratings for the clip-and-cut task are even higher (3.72 for global realism of clip-cut versus 3.42 for global realism of the visual laparoscopic cholecystectomy scene). The parameter of haptic feedback is improved (3.42 for clip-cut in this study versus 3.07 in the previous study). Respondents of both groups value Xitact’s use in the surgical resident training and education curriculum highly. The main focus of this study was the concept of the content validity. For this, the hypotheses-of-research generated beforehand were as follows:

• First hypothesis: Performance scores derived from the Xitact cholecystectomy clip-and-cut task by experts in clinical laparoscopic cholecystectomy are significantly higher than the performance score derived by novices in clinical laparoscopic cholecystectomy. In other words, the question is: do experts score higher on the Xitact runs than novices do? There is indeed a difference in the summative score between experts and residents. Figure 3 and Table 2 provide evidence that on runs 2 and 3, groups differ significantly.

73

Chapter 4

It can be seen that the confidence intervals in both groups are wide, especially in the novice group. This phenomenon is to be expected, as in the novice

, surgeons are inexp d wi parosc ironmen As dge and instrumental familiarization was set at t

level for both groups by one-on-one training and a familiarization run, the difference in confidence in s a ean re e de combination of abilities and experien

• Second hypothesis: Performance scores derived from Xitact cholecys relate th ical laparo ic cys ipanale s al r nts re p t vice )

ore t lde m urg (m pre a exp ce t linic par marily be the explanation for the higher scores. The

e significant differences in time needed to complete

Third hypothesis: Performance scores of experts and novices in clinical opic cholecystectomy improv runs o tact tectomy clip-and-cut task.

Table 3 shows an increase in scores over qu e s, for both This means alt gh there has been ups e learning e associated with

lf probably weighs heavily on the novice group.

The three hypotheses, generated for indirect operationalization of the primary research s p-cut task on the Xitact LS500 Virtual Reality laparoscopic

ulator mimics the surgical procedure of the clipping and cutting of e cystic duct and artery during the laparoscopic cholecystectomy adequately,’’ can be

rmatively. Therefore, the assumption must be that the Xitact LS500 paroscopic simulator is in fact mimicking the laparoscopic cholecystectomy.

Therefore, although better than o means for structured evaluation at all, it is unjust to state firm conclusions regarding surgical resident’s performance based solely on a questionnaire.

group erience th the la opic env t. he sameprocedural knowle

terval nd mce.

sco must b ma up by a

tectomyclip-and-cut task are d to e clin scop chole tectomyexperience of the partic t. Unless younger or fem urgic eside (mo resen in no grouplack native abilities m han o r and ale s eons ore sent the experienced group), the ctual erien with he c al la oscopiccholecystectomy must prilatter is supported by ththe runs, as is shown in Fig. 2.

•laparosccholecys

e over f the Xi

the se enc of three runthe expert and the novice groups. e expert group shows statistically significant improvement over runs. Th that, houa familiarization protocol for both gro , th c vurthe laparoscopic environment itse

tatement ‘‘The clicholecystectomy simthanswered affila The need for accurate and appropriate assessment for surgical trainees is well recognized [18]. Technical skill assessment in a residency program is mainly based upon subjective judgments [21], and is therefore in itself inconsistent. Summative assessments in terms of paper examinations tend to focus on theoretical knowledge. Self-reported progress in skill is extremely susceptible to bias. Our study shows that less than one-third of either population estimates performance correctly. Question-naires, as a form of formative assessment, are more structured, but most of them do not seem to assess all areas in surgical performance, particular not in psychomotor skills. In general, questionnaires are seldom properly evaluated. The questionnaire that is tested for reliability and validity still has the subjective component of a surgeon that is ranking the score [22]. In general, because of the subjective character of a questionnaire, it is prone to many threats to validity. na

74

Construct validity Xitact, being a computer, is able to assess a resident’s performance objectively as it is

latively free of subjective assessment and non-random error.

ct LS500, one application can be onsidered valid when another one is not.

In conclmen ncholecyssimulato

reCriterion-related validation studies, address the presence of evidence of a

relationship between scores on a research instrument, e.g., the Xitact LS500, and re-lated concepts which can be considered the criterion, such as the MIST-VR. So far, the MIST-VR is the only laparoscopic Virtual Reality trainer that can act as a criterion, because it is the only surgical Virtual Reality system which has been reasonably validated. A study by Taffinder et al. established construct validity of the simulator by showing significant differences in score comparing experienced laparoscopic surgeons with trainee surgeons and non-surgeons [27]. In other settings, similar conclusions were stated [2, 8]. Another study presenting a basic laparoscopic skills trainer called LapSim, has recently shown criterion validity in a small-scale study [12]. Although simulators are not fully alike (both MIST-VR and LapSim are simulators lacking haptic feedback and anatomical representation of a laparoscopic procedure) and probably more pure psychomotor trainers than the Xitact LS500, a study addressing criterion validity would be interesting. Virtual Reality simulation can become a promising and potent tool for surgical skills training and testing, but only when property validated [20]. The end goal of the validation process refers to the concept of instructional effectiveness, e.g., its concurrent validity. Practice on a VR module must ultimately improve user’s perfor-mance of the actual skill. Recent studies with the MIST-VR both failed [19] and succeeded in showing concurrent validity of the simulator [9]. Concurrent validation of the VR system requires correlation of performance in the Xitact LS500 with skill in vivo, that is, the performance of the laparoscopic cholecystectomy in the operating room. Validity itself is not an all-or-nothing statement, but merely reflects a gradual judgment, depending on the purpose of the measurement and the proper interpretation of the results. A common mistake is the assumption that validity is a characteristic of the system. The term ‘‘validity,’’ in fact, refers to the proper interpretation and use of the measurement results of the system. A single instrument may be used for many different purposes, and resulting scores may be more valid for one purpose than for another [5]. For the Xitact LS500, this means that validity state-ments based on the evaluation of one task can, and probably will be, different on another task. Indeed, it is fundamental for creating a useful VR teaching environment to recognize that a surgical procedure has to be divided into series of steps that can be trained and measured separately [11]. It is unlikely, though, that within one systematically developed system such as the XitacIt is important to remember that evidence derived from validation studies is never stable; it will in fact vary between settings.

usion, repetitive sampling in various settings exploring the multiple di-sio s of validity for the various scenes of the Xitact LS500 laparoscopic

tectomy simulator is needed for firmer positioning of this Virtual Reality r in the surgical curriculum.

75

Chapter 4 Refere 1. 2.

3. 4. 5. 6.

7.

8. ld J Surg 25 (11): 1478-83.

9.

isboa, Portugal 10.

11.

12. room" Surg Endosc, 2002. 16: 1324-1328.

13. rdan J-A, et al. (2000) "A comparison between randomly alternating imaging, normal

nces

Ault G, et al. (2001) "Exporting a technical skills evaluation technology to other sites" Am J Surg 182: 254-256 Chaudhry A, et al. (1999) "Learning rate for laparoscopic surgical skills on MIST VR, a virtual reality simulator: quality of human–computer interface"Ann R Coll Surg Engl 81: 281– 286 Coleman J, Nduka CC, Darzi A (1994) "Virtual reality and laparoscopic surgery" Br J Surg 81: 1709–1711 Derossis AM, Antoniuk M, Fried GM (1999) "Evaluation of laparoscopic skills: a 2-year follow- up during residency training" Can J Surg 42: 293–296 Gaberson KB (1997) "Measurement reliability and validity" AORN J 66: 1092–1094 Gallagher AG, et al. (1999) "Virtual reality training in laparoscopic surgery: a preliminary assessment of Minimally Invasive Surgical Trainer Virtual Reality (MIST VR)" Endoscopy 31: 310–313 Gallagher A, Crothers I, Satava R (2001)"Comprehensive objective assessment of fundamental skills for laparoscopic surgery." In: 19th International Congress of the European Association for Endoscopic Surgery, Maastricht, The Netherlands Gallagher AG, et al. (2001) "Objective psychomotor skills assessment of experienced, junior, and novice laparoscopists with virtual reality"WorGallagher AG, et al. (2002) "From VR to OR: a prospective, randomized, double-blind controlled trial of virtual reality (VR) training for operating room (OR) performance of laparoscopic cholecystectomy" In 10th International Congress of the European Association for Endoscopic Surgery, Centro de Congresses, LGrantcharov TP, et al. (2001) "Virtual reality computer simulation: an objective method for the evaluation of laparoscopic surgical skills" Surg Endosc 15: 242–244 Grantcharov TP (2002) "Teaching and testing surgical skills on a VR laparoscopy simulator: learning curves and effect of previous operative experience on performance" In: 10th International Congress of the European Association for Endoscopic Surgery, Centro de Congresses, Lisboa, Portugal Hyltander A et al. (2002) "The transfer of basic skills learned in a laparoscopic simulator to the operating Jolaparoscopic imaging, and virtual reality training in laparoscopic psychomotor skill acquisition" Am J Surg 180: 208–211

14. Karras D (1997) Statistical methodology: II. Reliability and validity assessment in study design, part B. Acad Emerg Med 4: 144-147 15. McNatt SS, Smith SGT (2001) "A computer-based laparoscopic skills assessment device

differentiates experienced from novice surgeons" Surg Endosc 15: 1085–1089 16. Nelson AA (1980) "Research design: measurement, reliability and validity" Am J Hosp Pharm 37: 851–857 17. Ota D, et al. (1995) "Virtual reality in surgical education" Comput Biol Med 25: 127–137 18. Paisly AM, Baldwin P, Paterson-Brown S (2001) "Feasibility, reliability and validity of new

assessment form for use with basic surgical trainees"Am J Surg 182: 24–29 19. Paisly AM, Baldwin P, Paterson-Brown S (2001) "Validity of surgical simulation for the assessment of operative skill" Br J Surg 88: 1525–1532 20. Prystowski JB, et al. (1999) "A virtual reality module for intravenous catheter placement" Am J Surg 177: 171-175 21. Reznick RK (1993) "Teaching and testing technical skills"Am J Surg 165: 358–361 22. Reznick R, et al. (1997) "Testing technical skills via an innovative ‘‘Bench Examination"" Am J Surg 173: 226–230 23. Satava RM (2001) "Accomplishments and challenges of surgical simulation. Dawning of the next- generation surgical education" Surg Endosc 15: 232–241 24. Schijven MP, Jakimowicz J (2002) "Face, expert, and referent validity of the Xitact LS500 Laparoscopy Simulator" Surg Endosc 16: 1764-1770 25. Schijven MP, Jakimowicz J, Schot C (2002) "The Advanced Dundee Endoscopic Psychomotor

Tester (ADEPT) objectifying subjective psychomotor test performance" Surg Endosc 16: 943-948

76

Construct validity 26. Taffinder N, et al. (1998) "Validation of virtual reality to teach and assess psychomotor skills in

laparoscopic surgery: results from randomized controlled studies using the MIST VR laparoscopic simulator" In: Westwood JD, et al. (2000) (ed) Technology and Informatics 50: Proceedings of Medicine Meets Virtual Reality, 28– 31 January 1998 IOS Press and Ohmsha Amsterdam, San Diego

27. Taffinder NJ, et al. (1998) "An objective assessment of surgeons’ psychomotor skills: validation of the MIST-VR laparoscopic simulator" Br J Surg 85 (Suppl 1): 75

28. Torkington J, et al. (2001) "Skill transfer from virtual reality to a real laparoscopic task" Surg Endosc 15: 1076–1079 29. Whitney JD (1999) "Notes on methodology. Validity and reliability: basic research concepts" J WOCN 26: 240–249 30. Wilson MS, et al. (1997) "MIST VR: a virtual reality trainer for laparoscopic surgery assesses performance" Ann R Coll Surg Engl 79: 403–404

77

Concurrent validity / Selection of laparoscopic trainees Chapter 5

ical trainees: How to

battery cores

. Schijv J Surg Research, 2004. 121/1: 112-119

select aspirant laparoscopic surgEstablishing concurrent validity comparing Xitact LS500 index performance scores with standardized psychomotor aptitude test s M en, J. Jakimowicz, F. Carter

79

Chapter 5 Abstract Background: Although a controversial topic in medical education, the selection of aspirant surgical trainees is a subject that needs to be addressed. In the view of preventing surgical trainee drop-outs and of appropriate allocation of limited resources, it is an issue critical to the profession. Traditional methods of selection are often subjective and do not seem to correlate with skill needed for surgery. Standardized neuro-psychometric test batteries may be useful in helping to select aspirant –laparoscopic- surgeons. Our study attempts to link surgical novices psychometric ability test battery data with actual performance outcome on an objective, validated and reproducible surgical laparoscopic task using Virtual Reality Simulation. Methods: Thirty-three novices with no laparoscopic surgical experience participated. Each participant performed the Xitact LS500 Virtual Reality cholecystectomy clip-and-cut module thirty times. Individual learning curves were computed and patterns assessed. Participants were examined using the aptitude test battery including the Abstract Reasoning test, the Space relations test, the Gibson Spiral Maze test and the Crawford Small Parts Dexterity tester. Results: Over 900 Virtual Reality simulation tasks were generated and assessed. Of the participants, 93.3% was able to complete both the Virtual Reality simulation 30 times and all elements of our aptitude test battery. The abstract reasoning test is the only test correlating significantly to Xitact test outcome. This test is highly correlated to the space relations test. Both tests have discriminative power, comparing groups of performers. Conclusions: The present study addresses the concurrent validity in aptitude testing, comparing scores of surgical novices on the Xitact LS500 laparoscopic cholecystectomy Virtual Reality simulation with performance scores on a battery of standardized psychometric aptitude tests. The abstract reasoning- and the space relation test have predictive and selective value, identifying individuals who have good laparoscopic surgical Virtual Reality performance. Aspiring –laparoscopic- surgeons who score below 25 on either test, that is an expected 36%, would have to be further assessed using Xitact surgical task performance. The group of participants scoring above 35 on the Abstract Reasoning test and above 45 on the Space Relations test, that is an expected 18% of the population, is unlikely to mal-perform on Xitact. The other 46% could very well benefit from Xitact simulation and assessment when the opportunity is present.

80

Concurrent validity / Selection of laparoscopic trainees Introduction

safety. A pilot needs

ents of cognition, psychomotor tests are valid selection tools and have rognostic significance in pilot training programmes [1, 2]. In parallel, surgical skill is learly linked to the clinical outcome of surgical procedures. Preliminary studies have

sycho-motor test systems used in aviation and the ndoscopic performance of surgical trainees [3].

at needs to be addressed in the view of the selection of surgical trainees. Medical

educators tend to pay much attention to the design of the curriculum, but little to the selection of students [4]. Effective and fair selection of aspirant surgeons is under heated debate, but nevertheless, it is critical to the profession. Traditional measures such as Medical College Admission tests, National Board Scores and CGE –levels do not correlate [5] or even correlate negatively [6] with subsequent surgical performance ratings. Ratings, which are subjective in themselves because they are derived from a senior surgeons’ subjective assessment, using unstructured observational methods. However, this is still the predominantly used method of assessment of surgical skill. Neuropsychologic and –psychometric tests, assessing visuo-spatial perception, stress- tolerance and psychomotor ability have been reported to be valid and consistent predictors of future surgical performance in multiple studies. [6-9] Nevertheless, attempts to establish a standardized battery of aptitude measures has proved unsuccessful, indicating inconsistency and counterintuitive findings between scores on different tests[10, 11]. The main factors accounting for the variability in those studies were the lack of objectivity in assessing participants’ surgical task performance and the use of only one surgical procedure for the to assessment of task performance, that is, no longitudinal studies have assessed surgical task performance data. It is clear that a proper identification of predictive psychomotor abilities is needed before the issue of selection of surgical trainees can even be discussed. Thus, one needs to establish a valid aptitude and psychomotor ability test battery that has potential for predicting future surgical performance. One way of doing so is by linking surgical novices’ psychometric ability test data on various aptitude tests to their outcomes on a validated, reproducible surgical task. Nowadays, Virtual Reality medical simulation offers, next to excellent standardized training opportunities, the possibility of unbiased, structured, validated and repetitive assessment of surgical task performance. The Xitact LS500 is such a Virtual Reality laparoscopic simulator. This simulator mimics, among others, the cholecystectomy clip-and-cut procedure. The simulation is thought to integrate psychomotor, visuo-spatial as well as cognitive elements associated with the clip-and-cut procedure of the laparoscopic cholecystectomy. The present study addresses the concurrent validity in aptitude testing, comparing scores of surgical novices on the Xitact LS500 laparoscopic cholecystectomy Virtual Reality simulation with performance scores on a battery of standardized psychometric aptitude tests.

In aviation, pilot skills have, obviously, a large impact on flightcertain basic (innate) abilities which cannot be trained, but which are assessed in the aspirant pilots. Selected pilots thus possess a certain set of abilities that act as the framework for aviation skills acquisition through training. Studies show that, next to measurempcindeed shown a correlation between pe

Although controversial, an issue thappropriate allocation of limited resources is

81

Chapter 5 Materials and Methods

g of live video lips, a colour-guided teaching approach with instruction on common errors and

formers. One month later, the participants took battery of standard aptitude tests.

It is a hybrid simulator, combining a physical object (The OpTable, or virtual abdomen”) with a computer software simulation providing the visual image

pants (63.3%) were able to achieve laparoscopic task proficiency over thirty simulation runs. Nevertheless, 20% of the participants seemed to be lacking innate manipulative abilities to such an extent that they could not achieve a stable task performance. [12] Based on these results, different Xitact performance group profiles were constructed.

Subjects Only surgical novices were selected as participants in order to measure pure abilities and to exclude individuals with skills derived from previous surgical experience. Thirty-three hospital residents and final-year interns -unconditioned for laparoscopy- participated in the study. Participants received a one-hour familiarization protocol on the Xitact simulator, introducing them to the laparoscopic cholecystectomy clip-and-cut scenery.

Participants followed a step-by-step teaching schedule, consistincproblems and a free-format clip-and-cut exercise. Feedback through Xitact’s assessment sheet and the instructor were given after this one-hour familiarization. Each participant then performed the clip-and-cut exercise thirty times, that is, ten times per session for three consecutive days. Learning curves were then computed for each participant [12]. Based on the individual learning curve pattern, participants were classified into one of four groups of pera Simulator The Xitact® LS500 laparoscopic cholecystectomy simulator is a modular Virtual Reality training platform, developed for training and education of a variety of laparoscopic skills (Fig. 1).“and tactile feedback. The Xitact incorporates Basic Surgical Skills, the Clip-and-Cut task of the laparoscopic cholecystectomy, a Camera Navigation module and a Peritoneal Dissection module for dissection of Calot’s triangle. The module used for determining learning curves is the Clip-and-Cut task. This module has been previously validated [13, 14]. The Xitact LS500 is developed and registered by Xitact SA, Morges, Switzerland. Xitact Performance Groups Previous research on the Xitact LS500, assessing over 900 simulation task runs of 30 laparoscopic novices, revealed that 16.7% of participants are naturally gifted and did not need much simulation training to achieve adequate task proficiency. The majority of partici

82

Concurrent validity / Selection of laparoscopic trainees • Group 1 consists of performers with a high level of innate abilities, gaining little

extra improvement through VR training (16.7% of total group) • Group 2 consists of performers with a moderate level of innate abilities, gaining

improvement and stability through VR training (30% of total group) • Group 3 consists of performers with a moderate level of innate abilities, gaining

unstable improvement through VR training (33.3% of total group) • Group 4 consists of performers with a low level of innate abilities, not gaining

improvement through VR training (20% of total group). Aptitude Tests Battery The Aptitude Test Battery included the Abstract Reasoning test and the Space Relations test, which are sub-tests of the Technical Abilities Battery of the Differential Aptitude Test (Psychological Corporation Ltd, London, UK). Also, the Gibson Spiral Maze was included in the Battery (Gibson HB. 1961: Hodder and Stoughton, London, UK), as well as the Crawford Small Parts Dexterity Tester (Psychological Corporation Ltd, London, England). Each test is thought to be indicative for a certain psychometric trait or ability. The Abstract Reasoning test investigates individual’s non-verbal reasoning ability and is related to IQ. Subjects must reason with geometric designs. They have to try to complete 40 tasks of abstract reasoning, choosing the right sequel option following a series of logical, abstract patterns within 20 minutes. The Space Relations test investigates individuals´ visuo-spatial ability. Subjects must mentally reconstruct a 3D object from a 2D pattern and rotate this object in mind in space. They have to complete 50 tasks within 25 minutes. A multiple-choice standardized answering sheet for analysis of results is used for both tests. The Gibson Spiral Maze test measures eye-hand coordination. Subjects must trace a line through a printed paper maze, avoiding obstacles. Standardized vocal stress enhancing triggers are administered at interval bases during the test. Execution time and error score are endpoint parameters. The Crawford Small Parts Dexterity tester measures eye-hand coordination and manual dexterity. Part one involves placing small pins into holes on a metal plate, followed by fitting collars over the pins. Execution time is the endpoint parameter. Statistics The Statistical Package for the Social Sciences (SpSS) version 9.0 was used for statistical calculations. A correlation matrix using either Pearson’s or Kendall-tau’s b correlation coefficient –whenever appropriate- was constructed. Also, groups were clustered and compared for significant differences in test outcome using the Mann-Whitney U test with the significance level set at 0.05.

83

Chapter 5 Results Demographics

fully complete the required thirty runs. The learning curves were thus calculate

s. The Abstract Reasoning Test in itself is highly lated to the Space Relations Test, revealing that abstract reasoning and visuo-spatial

highly related concepts. Also, Gibson’s Spiral Maze time and error are orrelated, depicted in Correlation Matrix 2 (Table 2).

Mean age of the participants was 28 years, ranging from 21 to 35 years. There were 18 males and 15 females in the study, 19 were right-handed, 2 left-handed and 2 ambidextrous. Eleven participants were interns, 2 residents were in training for emergency medicine, 6 residents for radiology, 3 for urology, 1 for cardiology, 3 for pulmonology, 2 for anaesthesiology and 5 for internal medicine. Three participants could not

d for 30 participants. Of these participants, 93.3 % was able to take part in the Aptitude Test Battery. Two participants could not take part, and they were therefore omitted from further analysis. Correlation Matrices and Scatter Plots The Aptitude Test Battery was compared with Xitact Group Classification in Correlation Matrix 1 (Table 1). Only the Abstract Reasoning Test seems to correlate with the Xitact Performance Groupreability arec

84

Concurrent validity / Selection of laparoscopic trainees Table 1. Correlation Matrix 1

GROUP Crawford Small Parts

Gibson Spiral

Gibson Spiral

Abstract Reasoning

Space Relations

Aptitude Tests Dexterity

Tester

Maze time

Maze error

Test Test Correlation

Matrix

N 28 27 28 28 28 28 GROUP

Corr. 1.000 . 0.983 b 0.364 b 0.852 b

0.003 0.134 -0.028 -0.310* Coeff.

Sig. 0.038 b 0.085 b

-0.260

Crawford Small Parts

Dexterity

Corr. Coeff.

Sig.

0.003 0.983 b

1.000 .

0.096 0.633 a

0.484* 0.011 a

-0.216 0.278 a

-0.229 0.251 a

Tester

Gibson Spiral Maze

Corr. Coeff.

Sig.

0.134 0.364 b

0.096 0.633 a

1.000 .

-0.515** 0.005 a

-0.310 0.108 a

-0.065 0.742 a

time

Gibson Corr. -0.028 Spiral Maze error

Coeff.Sig.

0.852 b 0.484* 0.011 a

-0.515** 0.005 a

1.000 . 0.507 a 0.910 a

0.131 -0.022

Abstract Reasoning

Test

Corr. Coeff.

Sig.

-0.310* 0038 b

-0.216 0.278 a

-0.310 0.108 a

0.131 0.507 a

1.000 .

0.845** 0.000 a

Space Relations

Test

Corr. Coeff.Sig..

-0.260 0.085 b

-0.229 0.251 a

-0.065 0.742 a

-0.022 0.910 a

0.845** 0.000 a

1.000

a: Correlation coefficient used is Pearson b: Correlation coefficient used is Kendall ‘s tau-b; both variables were ranked * Correlation is significant at the .05 level (2-tailed) ** Correlation is significant at the .01 level (2-tailed)

85

Chapter 5 Table 2. Correlation Matrix 2

Space Relations Test

Pearson Correlation 0.845*

asoning Abstract Re

Sig. (2-tailed) 0.000

Gibson's Spiral Maze error

Pearson Correlation

-0.515*

Gibson's Spiral Maze time

Sig. (2-tailed) 0.005 *Correlation (Pearson) is significant at the 0.01 level (2-tailed) Plots were constructed in order to visualize the correlations. Fig.1 shows a negative orrelation between error and time for the Spiral Maze, indicating that the longer a

maze, the less error is likely to occur. Plot 2 shows a ositive correlation between the Abstract Reasoning Test and the Space Relation Test,

pace relation test versus Abstract Reasoning Test

cparticipant takes to draw the spiralpindicating that individuals are consistent in scoring either highly or low on both tests. Fig.1. Scatterplots Plot 1. Gibson Spiral Maze Plot 2. S

Gibson's Spiral Maze time

706050403020 10

Gib

son'

s Spi

ral M

aze

erro

r

30

20

10

0


60 5040302010

Abs

tract

Rea

soni

ng T

est

40

30

20

10

0

86

Concurrent validity / Selection of laparoscopic trainees Discriminative power of the Aptitude Test Battery

Tab e te : discriminativ

A Grou red Rank

–Whitney U test

Asymp. Sig. (2-tailed)

le 3. Aptitud st battery e power

pt stsitude te ps cluste N Mean Mann

Crawford Small Parts D es

group 1 + group 2 group 3 + group 4

12 15

12.33 15.33 70.000 0.3exterity T ter 29

Gibson Spiral time

group 1 + group 2group 3 + group 4

04 .59 78.50 0

Maze 12 13.16 15 0 .423

Gibson Spira ze error group 1 + group 2

group 3 + group 412 16

14.46 14.53 95.500 0.982

l Ma

A e g Tbstract R asonin est 0

2roup 3 roup 4

.42

.56 00group 1 + groupg + g

12 16

1811 49. 0 .029*

group 1 + group 2group 3 + group 4

12 16

S tions Test pace Rela Total 28

18.25 11.69 51.000 0.037*

For al e rfo n Xitac (group 2) pa the who rmed badly (groups 3 and 4) (Table 3). On comparison of the means of these two clusters, only the Abstract Reasoning Test and the Space Relations Test proved to be discriminative. Con val Tab examining concurrent validity

this an y hsis, t perfo

better pe rmers o t s 1 and were com red withones

current idity

le 4. Distribution plot

60

act R

easo

ning

T

30

10

est

40

20

GROUP

p 4 0.79

p 3 Rsq = 0.70

Group 2 Rsq = 0.40

Group 1 Rsq = 0.34

302010 Abs

tr 05040

GrouRsq =

Grou


87

Chapter 5 Table 4 shows a distribution plot of the concurrent validity between Xitact

erformance Groups and the Aptitude Test Battery. A logical dispersion of groups is dividuals a e regression inst the

background of the Abstra sus Space Relation ig 1, plot 2). In other words, individuals of gr t score below 25 on either one of the aptitude tests.

Discussion

has been directed towards the use of titude tests [15]. Laparoscopic surgery, assuming a high degree of psychomotor

, formation management and manual dexterity. All individuals possess such abilities, ut individuals have different patterns of strengths for their abilities, as defined by their

Now what is meant by skills? Skills describe an individual’s developed with, and is modified by training and

skill is ased on a specific combination of the relativ mental abilities. hus, innate abilities underlie and are the limiting factor of the individual’s performance

of a rtain task. [16]. Different psychometric tests have been constructed for the identification of

the level of an individual’s particular ability. It stressed that the psychometric tes r test battery were included beca f their alleged capacity to assess a specific type of psychometric or –motor a ty. Inevitably, test outcomes are related since the abilities they test are not mutually sive. There are no clear-cut groupings of psychometric or -motor abilities. Id metric and –motor ties that underpin optimal endoscopic su al performance would be most helpfu r the selection of surgical trainees. Our st ery included four aptitude tests.

erity Te cted because it has been report ion accuracy, when u easures of selection for

ental School students [17]. In the present study, this test seemed not to be significantly linked to Xitact test performance. It does seem to be significantly linked, however, to the Gibson Spiral Maze error test outcome. Although both tests are not able to explain Xitact test performance outcome, they cannot be regarded useless for surgical assessment as they may discriminate for conventional psychomotor skill, not specifically addressed in this study. The Gibson Spiral Maze test was chosen because of its reported contribution in assessing hand-eye coordination [18]. Earlier research showed faster times in executing time, comparing scores of surgical with psychiatrist trainees. On the other hand, psychiatrist trainees had less error scoring. Thus, it has been difficult to assess and interpret the Spiral Maze’s value in the testing for surgical aptitude.

Pshown, plotting the in nd th lines of the groups aga

ct Reasoning ver plot (Foup 1 do no

Recent interest in the selection of surgical traineesapabilities could benefit from specifically oriented ability tests in order to aid predicting career appropriateness for an aspirant laparoscopic surgeon. It is important to realize what is meant by the term abilities itself. Abilities are innate, stable and enduring aptitudes that an individual brings to the performance of tasks. Multiple perceptual motor abilities are identified, such as spatial orientationinbgenetic make-up.

roficiency at a particular task that has ppractice. An individual may acquire many skills through life, but each particular b ely small number of fundaT

ce

must be ts in ou

entification of psychorgicudy batt

use obiliex

abl fo

clu

ili

The Crawford Small Par

ed to improve selectts Dext ster was sele

sed with other mD

88

Concurrent validity / Selection of laparoscopic trainees In our study, execution time and -error are negatively correlated (the faster, the higher hance on errors). Since both parameters time and error are linked to the same concept,

oncepts are in itself not correlated Xitact test outcome. Therefore, in this setting, authors feel the Gibson Spiral Maze

test not to be of high value. tial abilit i ocedures. The Space Relations test, assessing atial ability

pre or delaying graduation for of 1392 s ]. Recent research in g ly-

l ability is related to competence level nd rs con that vis ial

ability testing can be used in the selection of surgical residents [19]. In our study, the y e outcomes of the Abstract Reasoning test.

e plot combining the resu a e test e Space ns test outcomes seems to fol ti of ovices’ performance of Xitact accurately. of concurrent validity with Xitact surgical performance test

The current study a wo e four tests, includ he e., t a ing Test and the Space Relation Test

ve a predictive and selec yin viduals will ach d aroscopic surgical performanc e Xi sim r. The Abstract Reasoning test

y aptitude test which co dire with Xitact performance outcome. The

th tests with good results. As a group, individuals in indeed show intermediate test results. Thus, it is unlikely that persons sychomotor abilities are performing poorly on a combination of both

nts, all surgical novices, were pooled fro a heterogeneous group of hospital interns, (possibly interested in a surgical specialty) and non-surgical hospital residents (urology, emergency medicine, radiology, pulmonology, cardiology, anaesthesiology and internal medicine). It may be so that these part ipants will have lower scores on the Xitact simulator than, in fact, true novice surgical trainees. It is possible that ´self-selection´; based on a persons´ own perception of his or hers surgical abilities; leads to interest and a career in another medical sp cialty. But it may also be a non-ability related choice, and merel(lack of) surgical career interest. In fact, surgeons themselves see quite ´incapable´ of estimating their own performance in surgical simulation [Ne thel s, performance scores on Xitact were objectively assessed. tudied ind ua omparing his or her Xitact score with his or her psychomo tude test batt come is therefore valid, and in our study, logic dis mong per man groups is displayed. Extern validity, in terms of extra of our res to s a group of aspiring surgic trainees, can be biased by rather an underestim than an overestimate of innate abilities -as evaluated by Xitact outcome.

cnot much can be said studying only one parameter. Cto

Visual-spaspecific surgical pr

y is thought to be an m aport nt predictor of com inpetence visuo-sp

proved to be a strong dictor for students resigning dentistry in a study dental student [5 learnin spatialcomplex surgical skill show

d outcome after compleed that visual spatia

aan x surgery. Wanzel othe clude ual-spat

Space Relations test is highl correlated to thTh lts of the Abstr

low the distribuct on

R asoning the n

with th Relatio

This is indicative scores.

thus shows th t t of th ed in tAptitude Test Battery, i. he Abstract Re sonha tive value in identif g indi who ieve goolap e on th

rrelatestactctly

ulatois the onltest itself is highly correlated to the Space Relations test. There is a fan-like distribution of regression lines on the bi-variate scatter plot of both aptitude tests. No outliers of performance are present among members of group 1. All members of this group score above 75% in correct answers on the abstract reasoning test. In contrast, members of roup 4 are less able to perform bog

group 2 and 3 doith high innate pw

the abstract reasoning test and the space relations test. It must be said, however, that the study participa

m

ic

e y a matter of m to be20].

verividery outforults

esl, c

For the stor aptipersion a

cewa

polationrd

ate al

89

Chapter 5 Our test results must, therefore, be considered to be a prudent estimate of aspiring laparoscopic surgical trainee outcome. Bearing the above in mind, our study results lead to the following preliminary step-wise selection recommendations, considering economic use of our study resources (e.g. the Aptitude Test Battery and Xitact simulator). Aspirant and novice –laparoscopic-

rgeons can be tested using the Abstract Reasoning test and the Space Relations test.

articipants scoring below 25 on either test, that is an expected 36% of e population, would have to be monitored on Xitact to assess surgical task

opic- surgeon to compensate for a deficiency in specific psychom

the combination of a subset of aptitude tests; a

e offered the possibility to undergo aptitude testing b

su The group of pthperformance. The group of participants scoring above 35 on the Abstract Reasoning test and above 45 on the Space Relations test, that is an expected 18% of the population, is unlikely to mal-perform on Xitact. The other 46% could very well benefit from Xitact simulation and assessment when the opportunity is present. Conclusion The use of aptitude testing for aspirant surgical trainees is not without debate. It is important to realize that a ‘good’ surgeon is not a merely a product of a persons’ knowledge and psychomotor abilities. Personality traits, such as interest, endurance, empathy, stress-resistance and decision-making abilities are important and equally necessary to make up an all-round good surgeon. It is in the combination of these skills and personality variables the ‘ideal’ surgeon must be sought. Nevertheless, it is hard for an aspirant –laparosc

otor abilities. Previous studies show that surgeons are not good in estimating their own performance, [20] and authors feel someone should at least have the opportunity to objectively assess these important features before deciding on a surgical –laparoscopic- apprenticeship. Such an opportunity could be offered by aptitude testing.

Our study shows that it is in surplus value can be seen. Therefore, it is surprising that studies focussing on aptitude test battery outcome have not regularly studied their test interaction and /or (cor-) relation. Aspirant surgical trainees should b

efore embarking on a surgical traineeship. The advantages for both trainee and educator are evident. Information resulting from such tests can help the trainee to make an appropriate career decision, preparing him or her for future skills training or, in fact, revising initial aspirations. In any case, it is best to acknowledge aptitude problems early on in the selection, so that a trainee with a sub-optimal set of innate abilities for a laparoscopic surgical career has basically two options. He or she can either choose to pursue a career in –laparoscopic- surgery nevertheless, knowing there will be heavier emphasis on skills training compared to his or her peers in order to attempt to achieve the same surgical end-quality. Or, it may be a better suitable option for that person to opt for one of the alternative careers in surgery or other area of medicine. For the educator, aptitude testing helps in tailoring specific skills training programs, offering the right amount of training to the individual.

90

Concurrent validity / Selection of laparoscopic trainees The above is only possible once aptitude testing is considered to be valid and reliable. The emerging era of medical Virtual Reality simulation offers excellent opportunities for repetitive, reliable and objective assessment, and can thus double as a trainings- as well as a selection instrument once properly validated. The Xitact LS500 laparoscopy Virtual Reality simulator under study significantly, and in expected orderly fashion, correlates to the Abstract Reasoning Test of our Aptitude Test Battery. Our study established the concurrent validity of the Xitact LS500 with the combination of the Space Relations- and Abstract Reasoning test measuring individual’s visuo-spatial abilities.

91

Chapter 5 References 1. Muller, MH (1999) "How does aviation find the ideal pilot ? Suitability testing: applicability to

Price, M (2002) "Selection of medical students" BMJ 324: 1170-1171

Risucci DA (2002) "Visual spatial perception and surgical competence" Am J Surg 184(3): 291-295

, Chambers DW (1992) "The search for a profile of aptitudes that characterize dentists" J uc 56(5): 317-21

. Francis NK et al (2001) "The performance of master surgeons on standard aptitude testing."

those of different medical specialties" Br J Surg 81(3): 382-3 19.

surgery? Methods for determing basic occupational suitability" Zentralbl Chir 124(10): 889-894 2. Carretta TR, Ree MJ (1996) "U.S. Air Force pilot selection tests: what is measured and what is

predictive?" Aviat Space Environ Med 67(3): 279-283 3. Dashfield AK, Smith, JE (1998) "Correlating fibreoptic naso-tracheal endoscopy performance and

psychomotor aptitude" Br J Anaesth 81: 687-91 4.5. Smith, BG (1989) "A longitudial study of the value of a spatial relations test in selecting dental

students" Br Dent J 167(9): 305-8 6. Schueneman, AL et al (1984) "Neuropsychologic predictors of operative skill among general

surgery residents" Surgery 96(2): 288-295 7. Deary IJ, Graham KS, Maran AS (1992) "Relationships between surgical ability ratings and spatial

abilities and personality" J R Coll Surg Edinb 37(2): 74-79 8. Wanzel KR, Ward M, Reznick RK (2002) "Teaching the surgical craft: from selection to

certification" Curr Probl Surg 39(6): 574-659 9. 10. Simon JF

Dent Ed11 Am J Surg 182(1): 30-33 12. Schijven MP, Jackimowicz JJ (2003) "The learning curve on the Xitact LS500® laparoscopy

simulator: profiles of performance" Surg Endosc 18: 121-127 13. Schijven M, Jakimowicz J (2003) "Construct validity: experts and residents performing on the

Xitact LS500 laparoscopy simulator" Surg Endosc 17: 803-810 14. Schijven M., Jakimowicz J (2002) "Face-, expert- and referent validity of the Xitact® LS500

Laparoscopy Simulator". Surg Endosc 16: 1764-1770 15. Graham KS, Daery IJ (1991) "A role for aptitude testing in surgery?" J R Coll Surg Edinb 36(2): 70-74 16. Dashfield AK et al (2001) "Correlation between psychometric test scores and learning tying of

surgical reef knots" Ann R Coll Surg Engl 83: 139-143 17. Boyle A.M, Santinelli JC (1986) "Assessing psychomotor skills: the role of the Crawford Small

Parts Dexterity Test as a screening instrument" J Dent Educ 50(3): 176-9 18. Harris CJ, Herbert M, Steele RJ (1994) "Psychomotor skills of surgical trainees compared with

Wanzel KR et al (2002) "Effect of visual-spatial ability on learning of spatially-complex surgical skills" Lancet 359(9302): 230-231

20. Schijven MP, Jakimowicz J, Schot C (2002) "The Advanced Dundee Endoscopic Psychomotor Tester (ADEPT) objectifying subjective psychomotor test performance"

Surg Endosc 16: 943-948

92

The Intercollegiate Basic Surgical Skills Course Chapter 6 The Intercollegiate Basic Surgical Skills Course: Laparoscopic skill assessment using the Xitact LS500 laparoscopy simulator M. Schijven, R. Klaassen, J. Jakimowicz, O. Terpstra Surg Endosc, 2003. 17: 1978-1984

95

Chapter 6 Abstract Objective: This study was undertaken to establish residents’ progress in minimal access surgery (MAS) after attending the Intercollegiate Basic Surgical Skills Course (BSSC) by means of the Xitact LS500 laparoscopy simulator assessment program. Methods: Twenty-five surgical residents attended the BSSC in Leiden and Eindhoven, The Netherlands. Before and after the course, participants performed three ‘‘runs’’ on the Xitact LS500, featuring a standardised laparoscopic cholecystectomy clip-and-cut task. A control group of 25 interns not attending the course also performed two sessions of three runs. Parameters of interest were ‘‘score’’ and ‘‘time for completion of task’’. Results: No significant differences were found within the resident group for the parameters ‘‘time’’ and ‘‘score’’ when comparing outcomes pre-and post-BSSC. No significant differences were found comparing time and score between residents and interns on each of the six runs, except for time in run 2. Over six runs, both residents and interns became significantly faster. Conclusions: The Xitact LS500 cholecystectomy simulator did not detect significant improvement in MAS performance among a group of surgical residents attending the BSSC.

96

The Intercollegiate Basic Surgical Skills Course Introduction

en cholecystectomy as the

pic surgical techniques for various surgical procedures have been eveloped [11, 14]. Today, laparoscopic surgery is an integral part of general surgery 7]. The term minimal access surgery (MAS) refers to this field of surgery [24].

s needed to perform MAS, are in art innate and can be learned in part from extensive, repetitive practice [27]. Although

be a competent surgeon, the element of technical ompetence is eminent. However, the teaching and testing of technical skill are known

to be one of the least systematic or standardised components in the classical surgical curriculum [22]. Skills needed to perform MAS correctly cannot be directly extrapolated from skills acquired from performing open surgery. MAS requires distinct psychomotor abilities and different skills [25]. It has been recognised that psychomotor skill acquisition is distinct from other types of learning [13]. The incorporation of MAS in the modern surgical curriculum has created a need for specific training and education [23, 28]. Training courses, adopted by surgical colleges and implemented by pro-fessional organisations, have addressed this need. Currently, some surgical skills courses focus on knowledge and skills needed for performing MAS safely. The Intercollegiate Basic Surgical Skills Course (BSSC) is one of them. Since August 1996, all four surgical Royal Colleges in the United Kingdom have demanded that senior house officers entering basic surgical training have to have participated in the BSSC successfully. It is a mandatory requirement for the membership examination of the colleges. Since its introduction in 1996, the BSSC has trained more than 2,000 surgical trainees at 42 centres in the United Kingdom. The success of the course led to its adoption and implementation in other centres in Europe, including two surgical centres in The Netherlands. The main areas covered by the course are basic surgical techniques for open surgery, trauma and orthopaedics, and minimal access surgery [31]. The effectiveness of hands-on training courses, and the BSSC in particular, is rarely documented [19]. It has been shown that clinical laparoscopy experience is related to outcome on training tasks in a simulated environment [5]. Studies have shown that there is a link between performance and outcome parameters such as rating, score, or time [5, 23]. For the Virtual Reality laparoscopy simulator used to objectively assess participants in this study, the Xitact LS500, a recent construct-validity study showed that experts indeed performed better than novices on the laparoscopic basic clip-and-cut task [26]. This suggests that the reverse might be true as well—that is, that clinical performance might improve with practice resulting from training tasks. The score as generated by the Virtual Reality laparoscopy simulator may be an estimate of clinical improvement. Such a score may satisfy the need for accurate and objective assessment of technical process and skill acquisition since in fact, there currently is none [21]. Previous studies have shown potential and interest for Virtual Reality simulators in the field of laparoscopic surgery in terms of tutoring, training, and assessment of skill and performance [2, 4, 7–10, 12, 16, 33, 35].

Laparoscopic cholecystectomy has largely replaced optechnique-of-choice for uncomplicated symptomatic cholelithiasis [30]. Since the general introduction of the laparoscopic cholecystectomy in the early 1990s, a variety of laparoscod[1Surgical skills, particularly complex psychomotor skillpmany skills and traits are needed toc

97

Chapter 6 In this study, the Xitact LS500 Virtual Reality laparoscopy simulator was used for

bjective assessment of the impact of the laparoscopic module of the BSSC on

aterials and Methods

he Netherlands, and 14 residents and 14 interns were tested at the BSSC in indhoven, The Netherlands.

ry). In advance of the course, an instruction ook and video were sent to each participating resident. The MAS module, given on

itor). Video and hands-on instructions are provided. Next, trainees perform the exercise of safe introduction and maintenance of the pneumo-peritoneum. There is an introduction to laparoscopic instruments, and simple grasping, cutting, and manipu-lations skills are practised for approximately 2½ h. Then, advanced dexterity skills, such as clipping and loop ligation, are practised for approximately 1 h, and the use of diathermy in MAS is discussed and practised. Introduction to the Xitact LS500 laparoscopy simulator All subjects were given an instructed one-on-one ‘‘tour’’ to familiarise them with the Xitact LS500, its features, and the Virtual Reality interface of the laparoscopic cholecystectomy clip-and-cut scene. The tour featured a voiced-over instruction video of the clip-and-cut task showing the correct way to place clips and perform a trans-section of the cystic artery and cystic duct. Trained instructors gave a detailed explanation of possible errors of the procedure as well as a hands-on instruction on the apparatus. All subjects underwent baseline testing (run 1). On the first day of the course, three runs were performed by each subject because earlier studies with high-interface laparoscopic surgical Virtual Reality trainers have shown that familiarisation occurs within three runs [2]. Next, the group of residents attended the BSSC. After completion of the course, following the laparoscopic training module, the second series of three runs was performed by the residents. As a control group, interns also performed a baseline test, three runs on day 1 and three runs on day 3, without attending the BSSC.

oparticipants´ performance. M Subjects Twenty-five residents (the intervention group) and 25 interns (the control group) were selected for the study. Eleven residents and 11 interns were tested at the BSSC in Leiden, TE The Intercollegiate Basic Surgical Skills Course The BSSC is 3-day course, with 1 day for each module (open surgery, trauma and orthopaedics, and minimal access surgebday 3 of the course, starts with an introduction to MAS, Participants are introduced to the laparoscopic stack (rapid flow insufflator, light source, video camera, laparoscope, and mon

98

The Intercollegiate Basic Surgical Skills Course Fig. 1. Xitact LS500 simulator.

Apparatus The Xitact LS500 laparoscopy simulator (Fig. 1) is a modular Virtual Reality training program developed for training and objective assessment of a variety of laparoscopic skills. It is a hybrid simulator, combining a physical object (the OpTable or ‘‘virtual abdomen’’) with a computer program providing the visual image and haptic feedback. The program featured and under study is the clip-and-cut task of the laparoscopic cholecystectomy simulation. Face-referent and expert validity, as well as construct validity on this module, has recently been established [25, 26]. This module not only represents the exercise of the clipping and cutting of the cystic artery and duct but also records subject’s performance and computes a trainee’s performance in terms

f time ao nd errors made. A specific performance outcome sheet for each resident is linked to each performance run and thus provides interesting properties for subjects’ assessment. The Xitact LS500 is developed and registered by Xitact SA (Morges, Switzerland). Questionnaire A questionnaire consisting of a 2-page survey was given to the participant. In addition to participants´ demographics and previous surgical laparoscopic experience, questions concerning the experience with the BSSC and the experience with the Xitact simulator were asked. Subjects’ opinion on the usefulness of the simulator in terms of training/ teaching capacities and error reduction was also questioned. Data were analysed using the Statistical Package for the Social Sciences version 9.0. A p-value ≤ 0.05 was considered statistically significant.

99

Chapter 6 Results Demographics Mean age of the residents was 30.7 years (SD = 3.5), and the mean age of the interns

ps differed in age distribution (p-value 0.000, Mann–hitney U-test). There were 22 women and 28 men in the total population: there were

(36%) residents and 13 women (52%) interns. Groups had an equal sex istribution (p = 0.259; Mann–Whitney U-test). Regarding dexterity, 4% of the

ould not be met. Of the residents, 36% were specialising in general surgery, 4% in gynaecology,

in orthopaedics, and 24% in emergency edicine. Seventy-two percent were in their first year of training, 8% in their second,

ig. 2 shows the amount of time in seconds needed for completion of the clip-and-cut erns. There was a trend over the six

ns for completion times to become shorter for both the residents and the interns.

was 24.3 years (SD = 1.5). GrouW9 womendresidents and 16% of the interns were left-handed, 92% of the residents and 72% of the interns were right-handed. Four percent of the residents and 12% of the interns considered themselves to be ambidextrous. Chi-square testing could not be performed because conditions for the test c 16% in urology, 4% in thoracic surgery, 16%m4% in their third, 4% in their fourth, and 12% had not yet started their formal specialist training. Six residents had previous experience with the clinical laparoscopic cholecystectomy. Of these residents, 4 were in their first year of training, 1 was in his third year, and 1 was in his fourth year. None of them had performed more than 25 laparoscopic cholecystectomies. In the intern group, no one had any laparoscopic experience. By definition, interns were not yet in training for a speciality. Assessment of performance Time Ftask per run on Xitact for both residents and intru Fig. 2. Boxplot time

internresident 0

700

600

500

400

300

200

100

run 1

run 2

run 3

run 4

run 5

run 6 econ

ds

S

100

The Intercollegiate Basic Surgical Skills Course Also, standard deviations of both groups tended to become narrower over runs.

lthough individuals in both groups became significantly faster over the six runs (Table 1: Friedman test for time, interns, p = 0.000; residents, p = 0.026), between groups there was no difference except for run 2 (Table 2). Table 1. Comparison of interns and residents over runs within groups

Intern Resident

A

Run

Sum

(mean rank)

Time

(mean rank)

Sum

(mean rank)

Time

(mean rank) 1 3.02 4.83 3.36 4.57

2 3.38 4.15 3.20 3.39 3 3.48 3.79 3.04 3.57 4 3.58 3.33 3.54 3.70 5 3.50 2.65 3.72 2.91 6 4.04 2.25 4.14 2.87 Friedman test Friedman test Time 0.000* 0.026* Sum-score 0.332 0.175

Wilcoxon signed-ranks test Wilcoxon signed-ranks test Sum6–Sum3 0.563 0.064 Sum6–Sum1 0.025* 0.128 *p ≤ 0.05 Table 2. Comparison of interns and residents over runs between groups

aMann–Whitney U-test, bKruskall–Wallis test *p ≤ 0.05 Score

Variable Run 1 Run 2 Run 3 Run 4 Run 5 Run 6

Timea 0.059 0.003* 0.052 0.227 0.056 0.068

Sumscoreb 0.430 0.631 0.677 0.862 0.396 0.385

A sum score was computed according to previously established and validated

arameters [26]. Table 3 shows the distribution of this score into performance catego-

n test for score; interns, es

pries. There was progress over runs in both groups: 16% in both groups had a very inadequate to inadequate performance for run 1, whereas 8% of interns and 4% of residents had such a performance for run 6. Although individual scores for both groups tend to increase over the six runs, this increase seems to be of no significance (Table 1; Friedmap = 0.332, residents, p = 0.175). Between groups, there is no difference in sum-scorfor each run (Table 2).

101

Chapter 6 Table 3. Comparison of interns and residents over runs between groups: distribution

f sum

Intern run (%) Resident run (%)

o

Score Label Category

1 2 3 4 5 6 1 2 3 4 5 6

0–20 I Very inadequate performance

12 8 0 12 4 8 16 4 8 4 4 0

21–40 II Inadequate performance

4 8 8 4 8 0 0 4 0 8 4 4

41–60 III Question- able

24 4 20 4 16 8 4 20 20 12 12 12

performance 61–80 IV Adequate

performance 16 28 20 20 20 16 28 28 28 20 16 4

81 –100

V Highly adequate performance

44 52 52 60 52 68 52 44 44 56 64 80

Fig. 3 shows a visual representation of the un-clustered performance scores. As can be seen, there is a large standard deviation for both groups for this parameter, narrowing over runs. Residents did not seem to be consistent in improvement of scores over uns, and no consistent ‘‘jump’’ in score after the BSSC was established. r

Table 1 shows he second series (after t

no significant improvement for score between the last run of he BSSC) and the last run of the first series.

. Boxplot sumscore

t

ig. 3F

120

100

80

internresident

60

40

20

0

-20

SUM 1 SUM 2 SUM 3 SUM 4 SUM 5 SUM 6

102

The Intercollegiate Basic Surgical Skills Course Questionnaire Table 4 shows the frequencies of the scores on the questionnaire presented to each participant. In general, residents believed that they performed better in the second

ries of Xitact. Also, they were more secure after performing one series of exercise on

Ta Questionnaire for

sagree, 5: ag )

seXitact.

ble 4. residents

1: di ree (%Statement

31 2 4 5

The first series of exerci the Xitact (befor aparoscopic rt of the Basic Surgica Training) went w

29.2 20.8 12.5 ses on e the lpa l Skills ell

4.2 33.3

During the first series rcises on the Xitact (before the paroscopic part of the Basic Surgical Skills Tra elt secure

29.2 25.0 4.2 of exela ining) I f

4.2 37.5

The second series of ises on the X (before the paroscopic part of the Basic Surgical Skills Tra ent well

4.2 29.2 12.5 exerc itact la ining) w

4.0 50.0

During the second series of e Xitact (before the scopic part of the Basic Sur raining) I felt secure

0.0 62.5 16.7 xercises on the laparo gical Skills T

4.2 16.7

I feel my performance on the sec of exercises on the Xitac is better than m rst series of

12.5 8.3 8 29.2 ond seriest y performance on the fi

exercises

29.2 20.

I feel the lapaan essential pa

roscopic part of th ical Skills Course is rt of the course

0.0 8.3 5 62.5 e Basic Surg 16.7 12.

I learned a lot from the laparoscopic part of the Basic Surgical 0.0 14.2 8.3 41.7 45.8 Skills Course I learned a lot from the exercises on the Xitact laparoscopy 8.3 20.8 20.8 simulator

25 25

I think I will feel m in m clinic opicperformance after participating in the Basic Surgical Skills Course

8.3 45ore secure y future al laparosc 0.0 16.7 .8 29.2

I t eel mo re in clinica scopic performance after practising on the Xitact laparoscopy simulator

16.7 7 hink I will f re secu my future l laparo 12.5 20.8 41. 8.3

I feel the Xitacof the training i

t lapar simulat ld be a al part n the Basic Surgical S urse

20.8 2 29.2 oscopy or shoukills Co

n integr 4.2 16.7 29.

However, approximately one-fifth believed that they did worse the second time on

itact compared to their first series of runs. More than 70% agreed with the statement e laparoscopic part of the BSSC is an essential part of the course; approximately

0% believed that the Xitact laparoscopy simulator should be an integral part of the

Xthat th6course. More than 80% indicated that they learned a lot from the laparoscopic part of the BSSC, and 50% believed they that learned a lot from the exercise on Xitact.

103

Chapter 6 Table 5. Usefulness of Xitacts’ clip-and-cut scene for skills training/error reduction

Intern1 Resident1 RATINGS ve

Total

Mean

Ρ-e* 1: not useful, 5: ry useful

Mean SD Mean SD

valu

M tude ini cation 3. 4.5 65 3 1.2 00edical s nt tra ng /edu 90 6 0. .24 3 0. 6 S reside trai on 4. 4.4 63 4 0.7 000 urgical nts ning/educati 45 3 0. .45 2 1.Surgical specialist trai 3.87 3.95 0.85 3.83 1.09 0.998 ning/educationO r c 3. 4.0 59 3 0.8 000 verall lapa oscopi error reduction 92 7 0. .83 2 1.O r c en 4. 4.0 80 4 0.6 993 verall lapa oscopi skills enhancem t 12 7 0. .17 4 0.

1R not ul, *Kolmogorov-Smirnov test, two-tailed T sho tha ern d si t re o h e X ct inand educating sur ts, specialists, overall error reduction and laparoscopic sk nhan en ies. I c si ed b u ul r gi re nby both groups. Interns believed it was also useful for medical students; surgical

sidents did not value this purpose of the simulator equally, nor did they seem to

l practice within a ontroll t. Furthermore, it aims to teach, assess, and certify

ques. Until recently, it was unclear if the course was able to meet this objective. Formal course assessment was performed using one assessment form for all three modules. Participants are scored on a 3-point ordinal scale, and overall assessment is divided into the categories of satisfactory/not satisfactory. The criteria by which as-sessments are performed are poorly defined and subjective [34]. Furthermore, parti e same surgeons who are their mentors during the course. It is obvious that many flaws are inherent in this subjective means of assessment of score. Technical competence as judged by the mentor will always be subjective [20]. The lack of validated, reliable objective tests of surgical technical MAS comp problem in the assessment of MAS trainees. So far the Minimally Invasive Surgical Trainer Virtual Reality (MIST-VR), another vali C-based laparoscopic surgical trainer, is the only system used to assess performance objectively before and after the BSSC. In one study, the scores of a gro 12 residents’ for time to complete the MIST-VR task and the number of er ere significantly improved compared to those of a control group of non-BSSC-trained medical students [32]. However, another study showed no significant dif in per mance between MIST-VR and non-MIST-VR trained students on a simulated appendectomy in a mal model [1].

ating: 1: usef 5: very useful

able 5 ws t both int s an re den s ag ed n t e us of ita for train g gical residen

ills e cem t capacit t is on der to e most sef fo sur cal side ts

reagree on this (p = 0.006). Discussion The BSSC was designed to introduce surgical trainees to safe surgicac ed workshop environmentrainees’ ability to use safe surgical techni

cipants are scored by th

etence has long been a dated P

up ofrors w

ferencefor

live ani

104

The Intercollegiate Basic Surgical Skills Course In our study, there was no significant difference in score and time between the 25

sidents and 25 interns before the BSSC. This seems plausible because both residents

row confidence intervals of both parameters, indicative of a growing quaintance with the Virtual Reality interface itself. This is why the study was set up

Xitacts’ contribution to the outcome of the arameters of interest.

r research setting, the BSSC did not seem to re generated by Xitact. There was a decrease in time over the six runs f s, but there did not seem to be significant decrease in ti e af the SC for r b ed es si on in the BSSC does not s e t e e p e control group of interns. A er aining is st n established [18]. Concern has b surgical n [3 e is essential and a 2-or 3-day m f o p ti ive and validated means of a u idf is merely a basic skills oriented course and it provides an i a sc im to st The cen eat is m t rin e rse im g c e. tanding of and acquaintance w c s ty th sid b s orted by th es of o b a t r sip the need for structured, repetitive training.

herefore, the BSSC may be seen as an introductory course in an ongoing need for

hus at the beginning of the learning curve. Also, inexperienced surgeons are usually eager for any means that can improve their performance and susceptible to novelties, as indicated by Table 5. It is also likely to increase confidence and involvement in MAS. Learning curves, as well as ‘‘performance curves’’ described in the general motor skills literature, tend to be steep during the early part of the performance. A relatively small increase in the amount of practice provided during a course could substantially improve technical performance.

reand interns have virtually no laparoscopic experience. It is logical that over runs, standard deviations of both groups for ‘‘time’’ and ‘‘score’’ become narrower. Part of this phenomenon must be explained by the fact that the use of Xitact for multiple runs will naracusing a control group to account forp

In ou interfere with time or scoor both group

m ter BS the esident group. Also, there was no difference in time onterns for the individual runs apart from run 2. Participati

serv between r ident and

eem to increase residents’ scores significantly comparost- BSSC scores) and compared with the scores of th

d to hems lves (pre- v rsus

lthough there is little dispute that special laparoscopiche optimal format and training contents has not bee

surg y tr nece sary,

een expressed about short courses on medical education for 6]. Gadacz and others note that hands-on experienc

tech iques 3, 6,

ini-course is essential but not sufficient. Unfortunatelyng programs that claim improvement have object

, very ew w rksho s or rain-

ssessment. It must be stated that the BSSC is not a coor a specific procedure, it

rse aimed at training res ents

ntroduction to the MAS techniques. Also, the Xitact lest the exact

paro opy s ula r doe not same tasks that are taught in the course.

ore complex and incorporates multiple skills taugh s

due f

g thured

cou probabl (a

ying,

rasping, translocating, cutting, and clipping) in one proThe BSSC is likely to contribute to the unders

edur

ith the most common principles of MAS, thus reduefore his or her first clinical MAS experience. This is

ing inupp

ecuri for e outco

e rem

ent

ur questionnaire. Another positive effect of either articipating in the BSSC is the focus on

eing men or o a re dent

Ttraining and educational courses for new MAS techniques. Basic skills must be mastered prior to attempting more complex tasks, and faulty habits learned early are difficult to correct. It is suggested that the intern year is the most appropriate time to teach such skills [15], but this might be too early since only a few interns will become surgical trainees. Inexperienced surgeons may benefit the most from skills training [29] because they are learning a new technique and t

105

Chapter 6 No studies have compared simulated or clinical outcome following various schedules

f training within the BSSC. No formal task analysis has been performed in which the training tasks presented were chosen no ective iptio f a ped by which participants can be assessed. Onlspecific goals and objectives for each module of the BSSC are defined, accurate assessment and refinement of the cour ro C T simu did det gni t im em

resi ts who attended thr there was no significant improvement to detect, or

itact did not measure the basic psychomotor skills that are trained in the BSSC.

aparoscopic Virtual Reality training systems must be a etectable increase in performance in the laparoscopic operating room. Therefore, a alid Virtual Reality training system must mimic integration of various laparoscopic

at the BSSC does not increase such skills in novices in only one Further research on integral laparoscopic skills training sessions using

o. Also, clear obj s and descr n o

y when ssessment have been develo

se be p vided.

onclusion

he Xitact LS500 cholecystectomy lator not ect si fican prov ent inMAS performance among a group of surgical den e BSSC. Two explanations are possible. EitheXProbably, it is a combination of both.

It is unlikely that within the short, half-day time period for training laparoscopic basic skills during the BSSC a definite, stable improvement in laparoscopic skill is achieved. Also, Xitact tests task performance presumes a certain degree of integration of various laparoscopic skills. The end point of research in per-formance studies using ldvskills. It is likely thtraining session. Virtual Reality simulation for training and assessment is necessary.

106

The Intercollegiate Basic Surgical Skills Course References 1. Ahlberg G, et al. (2002) "Does training in a virtual reality simulator improve surgical performance?" Surg Endosc 16: 126–129 2. Chaudhry A, et al. (1999) "Learning rate for laparoscopic surgical skills on MIST VR, a virtual

reality simulator: quality of human– computer interface" Ann R Coll Surg England 81: 281–286 3. Dent TL (1991) "Training, credentialling and granting of clinical privileges for laparoscopic general surgery" Am J Surg 161: 399– 403 4. Derossis AM, Antoniuk M, Fried GM (1999) "Evaluation of laparoscopic skills: a 2-year follow- up during residency training"Can J Surg 42: 293–296 5. Derossis AM, Fried GM, Abrahamowicz M (1988) "The effect of practice on performance in a laparoscopic simulator" Surg Endosc 12: 1117–1120 6. Gadacz TR, Talamini MA (1991) "Traditional versus laparoscopic cholecystectomy" Am J Surg 161: 336–338 7. Gallagher AG, et al. (1999)"Virtual reality training in laparoscopic surgery: a preliminary

assessment of minimally invasive surgical trainer virtual reality (MIST VR)" Endoscopy 31: 310–313 8. Gallagher A, Crothers I, Satava R (2001) "Comprehensive objective assessment of fundamental

skills for laparoscopic surgery" 9th International Congress of the European Association for Endoscopic Surgery. Maastricht, The Netherlands

9. Grantcharov TP (2002) "Teaching and testing surgical skills on a VR laparoscopy simulator— learning curves and effect of previous operative experience on performance" 10th International Congress of the European Association for Endoscopic Surgery Lisboa, Portugal 10. Grantcharov TP, et al. (2001) "Virtual reality computer simulation: an objective method for the evaluation of laparoscopic surgical skills" Surg Endosc 15: 242–244 11. Jakimowicz JJ (1994) "The European Association for Endoscopic Surgery recommendations for training in laparoscopic surgery" Ann Chir Gynaecol 83: 137–141 12. Jordan J-A, et al. (2000) "A comparison between randomly alternating imaging, normal

laparoscopic imaging, and virtual reality training in laparoscopic psychomotor skill acquisition" Am J Surg 180: 208–211 13. Lewis CE (1998) "Continuing medical education: past, present and future" Western J Med 168: 334–340 14. Lytinski GS (1999) "Mouret, Dubois and Perissat: the laparoscopic breakthrough in Europe (1987–1988)" JSLS 3: 163–167 5. 1 McMahon DJ, Chen S, MacLellan DG (1995) "Formal teaching of basic surgical skills"

Aust N Z J Surg 65: 607–609 16. McNatt SS, Smith SGT (2001) "A computer-based laparoscopic skills assessment device differentiates experienced from novice surgeons" Surg Endosc 15: 1085–1089 17. Melvin SW, Johnson JA, Ellison CE (1996) "Laparoscopic skills enhancement" Am J Surg 172: 377–379 18. Mori T, et al. (1998) "Significance of ‘‘hands-on training’’ in laparoscopic surgery" Surg Endosc 12: 256–260 19. Olinger A, et al. (1999) "Effectiveness of a hands-on training course for laparoscopic spine surgery in a porcine model" Surg Endosc 13: 118–122 20. Ota D, et al. (1995) "Virtual reality in surgical education" Comput Biol Med 25: 127–137 21. Paisly AM, Baldwin P, Paterson-Brown S (2001) "Validity of surgical simulation for the assessment of operative skill" Br J Surg 88: 1525–1532 22. Reznick RK (1993) "Teaching and testing technical skills" Am J Surg 165: 358–361 23. Rogers DA, Elstein AS, Bordage G (2001) "Improving continuing medical education for surgical

techniques: applying the lessons learned in the first decade of minimal access surgery" Ann Surg 233: 159–166 24. Sackier JM (1998) "Evaluation of technical surgical skills. Lessons from minimal access surgery" Surg Endosc 12: 1109–1110 25. Schijven M, Jakimowicz J (2002) "Face-, expert-and referent validity of the Xitact LS500

Laparoscopy Simulator" Surg Endosc 16: 1764–1770 26. Schijven M, Jakimowicz J (2002) "Construct validity: experts and residents performing on the Xitact LS500 laparoscopy simulator" Surg Endosc 17: 803-810

107

Chapter 6 27. Schijven MP, Jakimowicz J, Schot C (2002) "The Advanced Dundee Endoscopic Psychomotor

Tester (ADEPT) objectifying subjective psychomotor test performance" Surg Endosc 16: 943–948 28. Scott DJ, et al. (2000) "Laparoscopic training on bench models: better and more cost effective than operating room experience? " J Am Coll Surg 191: 272–283 29. Scott DJ, et al. (2001) "Laparoscopic skills training" Am J Surg 182: 137–142 30. Soper N, et al. (1992) "Laparoscopic cholecystectomy: the new gold standard?"

Arch Surg 127: 917–923 31. Thomas B (1998) The Intercollegiate Basic Surgical Skills Course. Participant handbook.

lege of Surgeons of England, London, p 76 et al. (2001) "The role of the Basic Surgical Skills Course in the acquisition and

36.

Royal Col. Torkington J, 32

retention of laparoscopic skill" Surg Endosc 15: 1071–1075 33. Torkington J, et al. (2001) "Skill transfer from virtual reality to a real laparoscopic task" Surg Endosc 15: 1076–1079 34. Tyherleigh MG, et al. (2001) "The assessment of surgical skills and a simple knot-tying exercise" Ann R Coll Surg England 83: 69– 73 35. Wilson MS, et al. (1997) "MIST VR: a virtual reality trainer for laparoscopic surgery assesses performance" Am R Coll Surg England 79: 403–404

Zucker KA (1992) "Training issues [Editorial]" Surg Laparosc Endosc 2: 187

108

The learning curve

Chapter 7 The learning curve on the Xitact LS500 laparoscopy simulator: Profiles of performance M. Schijven, J. Jakimowicz Surg Endosc, 2004. 18: 121-127

111

Chapter 7 Abstract Background: This study assesses the acquisition of skill and the learning curve associated with the performance of the clip-and-cut task on the Xitact LS 500 Virtual Reality (VR) simulator in laparoscopic cholecystectomy. Methods: A group of 33 residents and interns with no previous laparoscopic experience participated in the study. All participants received a 1 hour familiarisation tour on the simulator. Thirty participants completed a full course of 30 simulation runs over 3 days (10 runs per day). The outcome parameters were a previously validated sum-score and time to complete performance. Results: Group demographics were similar. Of the participants who completed the full study, 16.7% appeared to have such a high level of innate psychomotor abilities that they were considered proficient in the task immediately after the initial familiarisation tour. Most participants (63.3%) had a moderate level of innate abilities, and their performance improved through repetitive VR training. In our study, 20% of the participants had such a low level of innate abilities that they were unable to achieve an acceptable performance in our minimal-access surgery (MAS) simulation. Conclusions: Learning curves cannot be assessed by examining repetitive training outcomes of only one person. There seem to be four different performance profiles, reflecting the fact that some people are more adept than others to be trained by MAS procedural VR simulation. For participants receptive to training -63.3% in this study- proficiency in the task occurs after approximately 25 simulative runs.

112

The learning curve Introduction

rgery [2]. ents and

surgical educators to question whether the operating eater should be the primary teaching environment for the initial acquisition of

the need for supplementary and innovative surgical skills outside the operating theater. Currently, a 3-

mandatory part of training in surgery in ctors of many surgical residency programs now

ecognise the need for additional training programs to supplement the more traditional ticeship graded responsibility model of surgical training.

The actual performance of surgical tasks is known to improve with experience—that is, through standardised repetition. Improvement tends to be most rapid at first; subsequently, it tails off over time until a steady state of performance is reached [14]. The term ‘‘learning curve’’ is often used to describe this phenomenon. In fact, the learning curve is the graphic representation of the relationship between experience with a procedure and an outcome variable, such as operative time or complication rate [21]. Learning curves have long been recognised in areas other than health care assessment, such as psychology, manufacturing, and aviation. The term was first used in health care in the 1970s and came to greater prominence in the 1980s after the introduction of MAS.

MAS places specific strains on the surgeon, requiring particular psychomotor abilities and skill to overcome difficulties imposed by the videoscopic surgical interface. Studies show that laparoscopic surgery is associated with a higher rate of complications than open surgery [5, 13]. With the widespread application of laparoscopic techniques in the early 1990s, an associated two-to three-fold increase in bile duct injuries was documented among certified surgeons. The risk of complications is known to be highest during the early part of the surgeon’s MAS experience [19]. Historical estimates of the number of cases needed to master the procedure of laparoscopic cholecystectomy range from eight-to 40. A study by the Southern Surgeons Club that used multivariate regression to analyse >8,000 procedures showed that >90% of all bile duct injuries occur within the first 30 cases performed by an individual surgeon [12].

When performing MAS, trainee surgeons cannot easily mimic their mentor’s actions or manoeuvres without actually manipulating the laparoscopic instruments in an initially disorienting, two-dimensional environment [11]. It is known that surgical skills—and in particular the complex psychomotor skills needed for endoscopic surgery—are in part innate and in part learned through extensive repetitive practice of a procedure [18]. Recent advances in Virtual Reality (VR) technology have led to the development of VR surgical skills simulators. These novelties are promising assets for the training and assessment of surgical skills [1, 7]. First, VR simulators provide an opportunity for repetitive practice, allowing for trial and error in the acquisition of new skills without the pressures or consequences of clinical reality. Second, simulators offer flexibility and independence, because training does not depend on the presence of an instructor.

Improvement in the performance of surgery depends, in fact, on practising su

ules, shortened training curriculum for residTight operating room schedmedico-legal issues have led thsurgical skill [8, 17]. This dilemma points tomethods for the teaching of day hands-on basic surgical skills course is a the U.K. [10]. Indeed, the direrappren

113

Chapter 7 Third, simulators offer excellent opportunities for the assessment of surgical skills.

nd fourth, the very nature of laparoscopic surgery, with its videoscopic interface,

eading to proficiency in the in vivo arning curve

ncorporates ve video clips of the simulated in vivo and the VR clip-and-cut procedure, a colour-

s given after the

1-h familiarisation. Each participant performed the procedure 30 times—that is, 10 consecutive procedures per session over 3 consecutive days. Simulator The Xitact LS500 laparoscopic cholecystectomy simulator is a modular VR training platform that was developed for training and education in a variety of laparoscopic skills. It is a hybrid simulator that combines a physical object (the OpTable, or ‘‘virtual abdomen’’) with a computer software simulation providing the visual image and tactile feedback. The Xitact incorporates basic surgical skills, the clip-and-cut task for the laparoscopic cholecystectomy, and a peritoneal dissection module for opening Calot’s triangle. The module used in this study for determining learning curves was the clip-and-cut task, because this module has been previously validated [16, 17]. The Xitact LS500 was developed and is registered by Xitact SA (Morges, Switzerland).

Amakes it likely to benefit from developments in VR [3]. Ultimately, learning curves for laparoscopic surgery could be shortened by achieving a good and stable performance on the simulator, l

edure. Our study focuses on the acquisition of skill and the leprocassociated with the task-oriented clip-and-cut scenario of the laparoscopic cholecystectomy, as represented by the Xitact LS 500 cholecystectomy VR simulator. Materials and Methods Subjects Thirty-three hospital residents and last-year interns, all without any laparoscopic experience, participated in the study. Participants received a 1-h familiarisation protocol on the simulator, introducing them to the laparoscopic cholecystectomy clip-and-cut scenario. Participants followed a step-by-step teaching schedule for the laparoscopic cholecystectomy clip-and-cut task. This teaching schedule iliguided teaching approach showing the exact area and preferred sequence for the placement of the clips on the virtual cystic duct and artery, specific instruction on what are regarded as common faults and/or resulting problems, and finally a free-form exercise without colour guidance. Feedback, through an assessment sheet, showing theend result of the procedure, and via comments from the instructor, wa

114

The learning curve Fig. 1. The Xitact LS500 laparoscopic cholecystectomy simulator

Statistical analysis The measurement of the performance of a new task—e.g., the learning curve—poses specific difficulties. Identification of the correct parameter to measure performance is one of them. In MAS, one could opt for the most obvious measurements, such as complications (bleeding or leakage) or ultimately conversion. Nevertheless, they may be too infrequent or not useful (because they tend to be dichotomous) for statistical analysis. Time is considered to be a parameter of importance, although not necessarily a reflection of proper outcome. In our view, statistical methods exploring the learning curve should address three basic aspects of the performance: the aspect of learning itself (was there change in performance over time?), the aspect of proficiency (when is there no further change in performance; e.g., what is the asymptote of the learning curve?), and the aspect of stability (is the change in performance stable?). Parameters of interest must therefore be continuous in outcome. Performance in this study was expressed through a previously validated Xitact-specific performance score [17].

Curve estimation, fitting a line to individual performance data, offering least squares regression was used for trend analysis. Lines reflecting the observed performance outcome and the best-fitting statistical model were plotted. For comparison of the demographics, the Kruskal-Wallis test was chosen. The Statistical Package for the Social Sciences (SPSS) version 9.0 (SPSS Inc., Chicago, IL, USA) was used for statistical calculations.

115

Chapter 7 Results Demographics The mean age of the participants was 28 years (range: 21–35). There were 18 men and 15 women in the study. Nineteen were right-handed, two were left-handed, and two were ambidextrous. Eleven participants were interns. Two were residents in emergency medicine, six were residents in radiology, three were residents in urology, one was a

sident in cardiology, three were residents in pulmonology, two were residents in aesthesiology, and five were residents in internal medicine. Three participants could

s. They were therefore excluded from further analysis. hen Kruskal-Wallis nonparametric tests were used for comparison of the four

ere no significant differences (Table 1).

reannot complete the required 30 runWgroups, there w Table 1. Group demographics versus test statistics

Sex Age Dexterity Speciality Year of training Chi-square 3.805 1.347 6.429 1.804 5.067

Asymp. sig. 0.283 0.718 0.093 0.614 0.167 Kruskal-Wallis test, Grouping variable: GROUP Plots 1 All groups. When scores and time to complete each run for each participant were plotted in one overall graph, there seemed to be no correlation at all (Fig. 2). Scores are dispersed evenly throughout the three sessions of 10 runs. Time needed to complete

also distributed in an incoherent manner, with no apparent decrease in time 0 runs. This lack of coherence could only be explained by assuming that

the runs is over the 3there were different ‘‘sets’’ of profiles that made interpretation of the results impossible when they were displayed together in one graphic plot. Therefore, individual curve estimations were created to identify the different sets of profiles. Curve estimation. There appeared to be four different types of curves. Different models (linear, logarithmic, power, and S) were used to estimate the best-fitting curve. The S model—Y = e**[b0 + [b1/t]]—appeared to have the best overall fit. For each group, a representative curve of one of the participants was chosen as an example.

116

The learning curve Fig. 2. Scores and times to complete each run for each participant (all groups).

core versus run Time (seconds) versus run

Table 2. Classification of groups

Group Profile Participants %

S

run

30 20100

450

400

350

300

250

200

150

100

50

0

run

3020100

120

100

80

60

40

20

0

1 High level of innate abilities, gaining little extra improvement through VR training

• No score <50 in any run • No score <80 during the last series of 10 runs • Standard deviation over all runs ≤ 10

16, 19, 21, 22, 26

16.7

• Overall mean ≥90 2 Moderate level of innate abilities, gaining improvement

and stability through VR training • No scores of 0 in the last two series of 10 runs • No scores <50 in the last series of 10 runs • Standard deviation over all runs 10 ≤ 30

4, 5, 7, 8, 9, 17, 24, 31, 33

30.0

3 Moderate level of innate abilities, gaining unstable improvement through VR training

• No scores of 0 in the last series of 10 runs • Standard deviations over all runs 30 ≤ 40

1, 3, 6, 11, 15, 20, 23, 28, 29, 32

33.3

4 Low level of innate abilities, not gaining improvement through VR training

• Scores of 0 in last series of 10 runs • Standard deviations over all runs >40

10, 12, 13, 14, 27, 30

20.0

117

Chapter 7 Group 1 was labelled ‘‘High level of innate abilities, gaining little extra improvement

rough VR training’’ (Fig. 3 and Table 2). Neither the linear model nor the S curve plain the variance in the scores in this group, because there is little

ariation. For this participant (no. 26), Rsq linear is 0.006, Rsq S is 0.000 (NS).

thwas able to exv Fig. 3. Group 1. High level of innate abilities, gaining little extra improvement through Virtual Reality training

Score

120

100

80

60

302010

40

20

00

run

ObservedS

Participant nr. 26

Participant nr. 4

Group 2 was labelled ‘‘Moderate level of innate abilities, gaining improvement and stability and through VR training’’ (Fig. 4 and Table 2). The S-curve can be used to explain the variance in the scores in this group, because there is a definite learning curve. For this participant (no. 4) Rsq S is 0.694, Sigf is 0.000. That is, 69% of the observed variance is explained by the model.

Fig. 4. Group 2. Moderate level of innate abilities, gaining improvement and stability through Virtual Reality training

Score

run

3020100

60

40

20

0

120

100

80

Observed

S

118

The learning curve Group 3 was labelled ‘‘Moderate level of innate abilities, gaining unstable improvement

rough VR training’’ (Fig. 5 and Table 2). The S curve cannot be used explain the runs, and

o better in their second series of runs than in their third series. For this participant (no. 11), Rsq S is 0.038 (NS). Fig. Group 3. Moderate level of innate gaining unstable improvement through Virtual Reality training

Score

Participant nr. 11

Group 4 wa not gaining hrVR training’ cannot explain the variance in the scores in this gro . throughout the runs, and participants are u ab hout the runs. For this participant (no. 10), Rsq S is 4 Fig. 6. Gro ng ent through Virtual Reality tra ng

Participant nr. 10

thvariance in scores in this group. Much variation is present throughout the some participants d

5. abilities,

run

s labelled ‘‘Low level of inna’ (Fi 6

te abilities, improvement t ough g. and Table 2). The S curve up Much and large variations are presentnst le in their performance throug 0.0 3 (NS).

up 4 Lo. w level of innate abilities, not gaini improvemini

Score

3020100

120

100

80

60

40

20

Observed

S

run

3020100

40

20

0

120

100

80

60

ObservedS

119

Chapter 7 Plots 2 When score, time to complete each run and the ´score versus time´ where plotted for the individuals in group 1, there appeared to be little dispersion (Fig. 7). There seems

be a strong relation between time and score, indicating that the runs with the lower

igh scores. Fig. 7. Group 1. High level of innate abilities, gaining little extra improvement through Virtual Reality training

Score Time

toscores actually take longer to complete. Also, participants become faster over runs while preserving their h

run

3020100

scor

e

120

100

80

60

40

20

0

302010

cond

s

45040035030025020015010050

0

se 0

run

Score versus Time

score

1201008060400

seco

nds

450

350300250

200150100500

20

400

120

The learning curve Fig. 8. Group 2. Moderate level of innate abilities, gaining improvement and stability through Virtual Reality training

Score Time

run

3020100

scor

e

120

100

80

60

40

20

0

run

3020100

s 0econ

ds

450400350300250

20015010050

Score versus Time

score

450

100806040200

seco

nds

0300250200150100500

120

40035

121

Chapter 7 Fig. 9. Group 3. Moderate leve , gaining unstable improvement through Virtual Reality training

Score Time

l of innate abilities

run

3020100

scor

e

120

100

80

60

40

20

0

run

3020100se

cond

s

450400350300250200150100500

Score versus Time

score

120100806040200

seco

nds

450400350300250200150100500

122

The learning curve Fig. 10. Group 4. Low level of innate abilities, not gaining improvement through

irtual Reality training

Time

V

Score

run

3020100

scor

e

120

10

run

3020100

seco

nds

450400

0

803503002502150100500

60

20

0

00

Score versus Time

40

score

120100806040200

seco

nds

450400350300250200150100500

123

Chapter 7 When score, time to complete each run and the ´score versus time´ where plotted for

e individuals in group 2, there appeared to be some dispersion, especially in the first ries of runs (Fig. 8). There seems to be a relation between time and score, indicating at the runs with the lower scores actually take longer to complete. However, perfect ns sometimes take more time to complete, whereas non-perfect runs may take less

me. When score, time to complete each run and the ´score versus time´ where lotted for the individuals in group 3, there appeared to be large dispersions roughout runs (Fig. 9). There seems to be little relation between time and score. hen score, time to complete each run and the ´score versus time´ where plotted for

time and score. If we look specifically at the curves of e participants in groups 2 and 3, the average number of runs needed to be in the

‘‘safe zor incurring other troublesome errors—seems to be around 25 runs. Discussion Surgical competence has two major components. The first one is cognitive competence, which requires cognitions mainly based on surgical, anatomical, and medical knowledge. The second one is technical skill in surgery, which is the result of a person’s innate abilities to perform a specific surgical task combined with repetitive training in that procedure. There are numerous theories that attempt to explain the acquisition of technical skill. In combination, they emphasise the importance of modelling, repetitive practice, and formative edback. Obviously, both components are essential for a surgeon to become comp tent. Assessment tools for measuring cognitive competence are widely available t i the technical aspect that suffers from poor and subjective assessment studies, so far have lacked the power to measure fundamental issues in tor skills adequately [6]. In response to the growing need for better methods to assess surgical competence, an initial framework has recently been devised to standardise definitions, measurements, and criteria for objective metrical assessment [15]. There is general agreement that, as with any method of teaching, VR simulators should be evaluated repeatedly before they widespread implementation into surgical education [4]. No doubt, this principle is even more important when assessing surgical technical skill using VR simulators. The essential measures of a psychometrically sound test are its reliability and its validity. The validity of a test must be considered proportional to the realism of the simulation. Few studies have focused on the validation of VR training tools. So far, only a few research groups have attempted to validate a VR simulator in a setting that goes beyond the context of a basic psychomotor skills trainer. No studies have attempted to assess the learning curve for laparoscopic task-oriented procedural VR settings. It must be stressed that the term ‘‘learning curve’’ is, in fact, a misnomer. Learning is a parameter that cannot be measured in itself. It is usually an extrapolation from changes in performance over time. It is important to realise that the outcome measures used in this study—performance time and sum-score—are fixed due to time restraints and the finite endpoint of score.

thsethrutipthWthe individuals in group 4, there appeared to be very large dispersions (Fig. 10). There seems to be no relation betweenth

one’’ that is, to complete the procedure without harming the common bile duct

fees; i

strategies [21]. VR the learning of mo

124

The learning curve There are four major implications of our study that result from the assessment of the learning curves for our VR procedural surgical simulation.

irst, the data show that is not opportune to assess a VR system’s learning curve using only one test subject who performs a simulation task repeatedly. In fact, there appear to be ry different profiles in the spectrum of performers. We identified four basic profiles of performers along the spectrum. Performers in groups 1 and 4 have profiles that dphenomenon is, in essence, very different for the two groups. Group 1 appears to have such strong innate abilities that after the 1-h hands-on familiarisation protocol, there is little more to be gained from repetitive training. This group comprises 16.7% of the total group. In contrast, in group 4 (20% of the total group), native abilities seem to be lacking, so there is not much of an innate psychomotor framework to build upon. Second, the study shows that most performers are indeed responsive to training. These are the individuals in group 2 and 3 (together, 63.3% of the total group). Performers in these groups displayed curves with a definite asymptote, although performance is less stable in group 3 than in group 2. An equilibrium occurs after approximately 25 runs, after which no major errors are incurred. From the plots, we can conclude that the parameter of time, in itself, is a fairly untrustworthy basis for assessing surgical competence. Third, the study shows that it is, in fact, possible to identify differ t profiles of performers using VR procedural simulation. This finding justifies the use of VR procedural simulators such as the Xitact LS500 to train most surgical residents in the performance of the tasks ess c cholecystectomy. And finally, it s dural simulators can be used for the future guidance and selection of surgical trainees in MAS. The dynamics of acquiring laparoscopic proficiency through VR simulation are complex. In MAS, learning is likely to be affected by a variety of factors, including previous experience with the specific procedure or similar procedure, the experience of the supporting surgical team, the type of equipment, and of course the nature of the clinical case itself. Many of these factors are not of influence whilst training with the Xitact LS500. None of the participants in this study had any hands-on experience in MAS. No anatomical variations were encountered in the repetitive simulations, in sharp contrast to clinical practice with real patients. The instruments did not vary during the simulation, nor did the place of ‘‘operation.’’ Excluding these factors, there is little statistical noise—e.g., validation bias—to be expected in our study. Nevertheless, these factors are definitively present in real-time operations.

We believe that it is important to proceed carefully, taking a stepwise approach in assessing VR simulative training through validation studies. Only by repeated validation in different settings, a solid and optimal framework for the use of VR simulation can be built. Therefore, further research should be aimed at the elaboration of heretofore undefined, and therefore critical parameters, para eters such as the ideal time interval for initial training in VR simulators during the learning curve and the establishment of optimal training schedules thereafter.

The ultimate purpose of VR simulations of laparoscopic cholecystectomy is not to ain residents to operate safely on simulators; it is to train residents to operate

fely on patients. Ultimately, the endpoint of a valid and stepwise VR validation, and learning skill studies in general, is to address the question of whether the skill

acquired on the simulator in fact translates to the clinical setting.

F

ve

o not seem to improve through repetitive training. The explanation for this

en

ential for MAuggests that VR proce

S procedures, such as the laparoscopi

m

trsaof

125

Chapter 7 Advances in medical science and technology are likely to be accompanied by dramatic changes in the way that surgery is taught. For many procedures, the surgical ommunity is now moving from the open surgical approach to the minimally invasive

one, where different sensory feedback properties are eminent and specific psycho otor skills are important. Such a paradigm shift will have profound implication for the way surgical training programs are developed, surgeons are selected, and th

c

m

e policy for the (re)-certification of surgeons is established.

126

The learning curve References 1. Ali MR, Mowery Y, Kaplan B, De Maria EJ (2002) "Training the novice in laparoscopy: more challenge is better" Surg Endosc 16: 1723-1736 2. Bevan PG (1986) "Crafts workshops in surgery" Br J Surg 73: 1-2 3. Coleman J, NDuka CC, Darzi A (1994) "Virtual reality and laparoscopic surgery" Br J Surg 81: 1709-1711 4. Cosman PH, Cregan PC, Martin CJ, Cartmill JA (2002) "Virtual reality simulators: current status

in acquisition and of surgical skills" Aust AN Z J Surg 72: 30-34 5. Deziel D, Milikan KW, Economon SG, Doolas A, Ko ST, Airan MC (1993) "Complications of

laparoscopic cholecystectomy: a national survey of 4,292 hospitals and an analysis of 77,604 cases" Am J Surg 165: 9-14

6. Farrell A (2002) "Can virtual reality be used to measure and train surgical skills?" Ergonomics 45: 362-379 7. Gallagher A, Satava M (2002) "Virtual reality as a metric for the assessment of laparoscopic

psychomotor skills" Surg Endosc 16: 1746-1752 8. Grantcharov TP (2002) "Teaching and testing surgical skills on a VR laparoscopy simulator:

learning curves and effect of previous operative experience on performance" 10th International Congress of the European Association for Endoscopic Surgery (EAES), Lisbon, Portugal

rov TP, Baruran L, Fuch-Jensen P, Rosenberg J (2003) "Learning curves and impact of operative experience on performing on a virtual reality simulator to test laparoscopic

surgical skills" Am J Surg 185: 146-149

rgical

9. Grantchaprevious

10. Hamdorf JM, Hall JC (2000) "Acquiring surgical skills" Br J Surg 87: 28-37 11. Melvin SW, Johnson JA, Ellison CE (1996) "Laparoscopic skills enhancement" Am J Surg 172: 377-379 12. Moore MJ, Bennett CL (1995) "The learning curve for laparoscopic cholecystectomy" Am J Surg 170: 55-59 13. Peters JH, Ellison EC, Innes JT (1991) "Safety and efficacy of laparoscopic cholecystectomy" Ann Surg 312: 3-12 14. Ramsay CR, et al. (2001) "Statistical assessment of the learning curves of health technologies" Health Technol Assess 5: 1-98 15. Satava MR, Cuschieri A, Hamdorf J (2003) "Metrics for objective assessment: preliminary

summary of the Surgical Skills Workshop" Surg Endosc 17: 220–2226 16. Schijven M, Jakimowicz J (2002) "Face, expert, and referent validity of the Xitact LS 500

laparoscopy simulator" Surg Endosc 16: 1764-1770 17. Schijven M, Jakimowicz J (2002) "Construct validity: experts and residents performing on the

Xitact LS 500 laparoscopy simulator" Surg Endosc 17: 803-810 18. Schijven MP, Jakimowicz J, Schot C (2002) "The Advanced Dundee Endoscopic Psychomotor

Tester (ADEPT) objectifying subjective psychomotor test performance" Surg Endosc 16: 943-948 19. Southern Surgeons Club (1995) "The learning curve for laparoscopic cholecystectomy" Am J Surg 170: 55-59 20. Strorn P, et al. (2003) "Validation and learning in the Procedicus KSA virtual reality su simulator: implementing a new safety culture in medical school" Surg Endosc 17: 227–231 21. Wanzel KR, Ward M, Reznick RK (2002) "Teaching the surgical craft: from selection to

certification" Curr Probl Surg 39: 574-659

127

The Eindhoven laparoscopic cholecystectomy training course Chapter 8 The Eindhoven laparoscopic cholecystectomy training course: improving operating room performance using Virtual Reality Training Results from the first EAES accredited virtual reality trainings curriculum M. Schijven, J. Jakimowicz, I. Broeders, L. Tseng Surg Endosc, in press

129

Chapter 8 Abstract Background: This study was undertaken to investigate operating room performance of sur residents, after participating in the Eindhoven virtgical ual reality laparoscopic

olecys cal resident inings ills training

olecys

Method: Tety oup ´ktured,m pe

twelve ca num

of the ´clip-and-cut´ procedure of the lapaicipa observation es, w

ults: R er in in

ur o

tsrove

roveed b

ch tectomy training course. This course is the first formal surgia complementary Virtual Reality (VR) sktr course, using a variety of

simulation in order to prepare surgical residents for their first laparoscopic ch tectomy. The course was granted EAES certification.

he four-day course is based on a multi-media and –modality approach. A vari f increasingly difficult simulation training sessions, next to intimate focus-gro nowledge sessions´, are included. Both basic- and procedural VR simulation is fea using MIST-VR and the Xitacts´ LapChol simulation software. The operating roo rformance of twelve surgical residents who participated in the course and

se-control counterparts were compared. The case-control group was matched for ber of clinical laparoscopic cholecystectomy performances (maximum of 4 procedures). Two observers analyzed a randomly mixed videotape, featuring the part

roscopic cholecystectomy, being blinded for part nts´ group status. Structured questionnaires, including multiple cal formance. s ere used to assess per

es the experimental and control group did not diffR esidents of both

demographic parameters, except for number of laparoscopic cholecystectomies favo f the control group (p-value 0.008). Both observers judge the experimental group to perform significantly better (p-value 0.004 and 0.013). Experimental group residen valued their course highly in terms of their laparoscopic surgical skills

ment and the use of VR simulators in the surgical curriculuimp m. Conclusion: The Eindhoven Virtual Reality laparoscopic cholecystectomy training course imp s surgical skill in the operating room above the level of residents who were train y a variety of other training methods.

130

The Eindhoven laparoscopic cholecystectomy training course Introduction For over a decade now, laparoscopic cholecystectomy has become the treatment-of-choice for uncomplicated cholelithiasis. The introduction of laparoscopic cholecystectomy and rapid evolvement of minimal access surgery (MAS) challenge the classical apprenticeship model for surgical training. In fact, it is known that with the introduction of MAS, complications are bound to arise during surgeons´ early experience with the particular procedure [1]. Nowadays, due to shortened surgical curricula, stress on working hours and medico-legal issues, MAS training needs to be intensified in order to ensure safe and high-quality patient care. The traditional apprenticeship model for teaching is under debate, as it is costly in terms of time and resources and unlikely to adequately train skills needed [2, 3]. In fact, such skills cannot be extrapolated directly from those acquired in open surgery, as MAS requires quite distinct psychomotor abilities, hand-eye coordination and different skills. Moreover, the surgeon engaging into MAS has to overcome the hurdles of the two-dimensional videoscopic surgical interface, the restricted degrees of freedom of movement of the MAS instruments and different haptic sensations.

Virtual Reality (VR) surgical simulators provide excellent opportunities for enhancing psychomotor skill and training MAS procedures in a safe environment. Such systems allow repeated, unbiased practice of a standardized task, provided they are well validated in order to be accepted by the surgical community. In the literature, few studies have assessed the transfer of skill using Virtual Reality simulation to actual operation room performance. Of these studies, the majority indeed showed improved performance, [4-7] although some failed to do so. [8] These studies all feature basic psychomotor skills trainers such as the MIST-VR (Mentice Medical Simulation, Gothenburg, Sweden, and LapSim (Surgical Science Ltd, Gothenburg, Sweden). Recent developments in VR simulation have now put simulators beyond the level of basic computerized psychomotor skills trainers. The newer generation of VR simulators and -simulations are in fact able to mimic a MAS surgical procedure convincingly. No studies have investigated operating room performance using such a second generation VR simulation. The aim of the present study was to investigate operation performance of surgical residents, after participating in the Eindhoven Virtual Reality laparoscopic cholecystectomy training course. This course uses the Xitact LS500 as a second generation open platform VR simulator. The procedure of the laparoscopic cholecystectomy is featured. Participants´ operating room performance was compared to the operating room performance of a matched control-group. Materials and Methods Inclusion The study was performed including surgeons-in-training, working in one of eleven participating teaching hospitals. Participants were recruited between April 2003 and March 2004, all surgeons-in-training and novices in laparoscopic cholecystectomy.

131

Chapter 8 Twelve surgeonroup, and twelve oth

s participating in the Eindhoven course constituted the experimental er surgeons-in-training constituted the case-control group. All

p as restricted to participants with a performance status of at most four laparoscopic

d ASA class , with a medical history of uncomplicated cholelithiasis and without previous dominal complaints or surgery.

ourse

he surgeons in the experimental group participated in the four-day Virtual Reality training course at the Catharina Hospital Eindhoven,

The Netherlands. Participants were insured as active surgical residents for the duration of the course. The course was set up incorporating a variety of teaching elements, relevant to the procedure of study. Videos featuring the laparoscopic cholecystectomy in uncomplicated, as well as complicated cases were discussed, alongside a variety of oral presentations. Inter-active ‘transfer of laparoscopic cholecystectomy’- knowledge table sessions were incorporated, and laparoscopic instruments and stack needed for performance of laparoscopic cholecystectomy were explained and displayed. Core-elements of the course were repetitive training sessions on a variety of VR software simulations, using the open Xitact LS500 laparoscopy simulator platform (Xitact SA, Morges, Switzerland). An increasingly difficult level of both basic psychomotor VR simulation (MIST-VR) and procedural laparoscopic cholecystectomy simulation, including the clip-and-cut, navigation and dissection modules (Xitact) were featured. Thus, an integrated, procedure-specific, multimodality VR trainings curriculum was presented to the participants. On day two and three of the course, residents attended the operation room (OR) next to their VR training sessions, to act as either assistant surgeon or camera assistant assisting a laparoscopic cholecystectomy being performed by an expert laparoscopic surgeon. At the fourth day of the course, course participants performed a full laparoscopic cholecystectomy themselves under the close supervision of the expert surgeon. The procedure was videotaped. The course was supervised by a faculty member assigned by the Scientific, Educational and Programme Committee of the European Association for Endoscopic Surgery (EAES). Subsequently, the course accredited formal EAES certification. Assessment Participants in both research groups performed the procedure of the laparoscopic cholecystectomy under expert surgical supervision. Only the ´clip-and-cut´ part of the laparoscopic cholecystectomy, e.g. the clipping and cutting of the cystic artery and cystic duct, was object of study. This part was chosen for multiple reasons. First, because this is likely to be one of the most essential stages of the laparoscopic cholecystectomy procedure, that has to be performed safely in order to avoid possible damage to the common bile duct.

gparticipants, both in the experimental and the case-control group, attended a Basic Surgical Skills Course before participating in this study. None of the participants engaged in a specific cholecystectomy skills training course. Inclusion to either grouwcholecystectomies. Patients selected for the course were patients considere1ab C Tlaparoscopic cholecystectomy

132

The Eindhoven laparoscopic cholecystectomy training course Second, by selective assessment of the ćlip-and-cut´ scenery, technical skills can be

sessed quite independently of other factors influencing outcome variation. Third, it is

ors were removed from e operative field. Video fragments from both the experimental and control group

-point Likert rating-scale (ranging from 0, ompletely disagree to 5, completely agree) was used for assessment. Parameters of

as a amework for the assessment of Xitact’s clip-and-cut simulation [3].

he final outcome parameter ´Judgement´ was operationalized by the question: ¨For the procedure of clipping and cutting of the cystic duct and cystic artery by this surgeon, I would grade him or her a __¨ (number, range 1-10, one decimal). Time to complete the clip-and-cut procedure was recorded. Results Demographics. Mean age of participants in both groups was 31 years. All participants were right-handed. In the experimental group there were 4 females, whereas there were 2 females in the control group. Mean year-of-training was 1.8 for both groups.

aslikely to be a good predictor for over-all performance and finally, by monitoring this specific part of the procedure assessment is facilitated as the outcome assessment scale can be properly standardized. The procedure was assessed starting from the moment the laparoscopic clip applier was introduced, ending at the moment the laparoscopic scissthresidents were evaluated by two reviewers, both laparoscopic engaged surgeons, from different academic training hospitals, independent of each other. Participant’s video fragments were mixed in random order before being copied to the reviewers´ videotape. In both the experimental- and the control group, two recordings were excluded from analysis as a result of technical recording failure. Statistic analysis assessing the data of the remaining 20 procedures was performed using the SpSS version 10.0 software package (SpSS, Chicago, Illinois, USA). Outcome parameters A structured questionnaire using a fivecinterest were: ´Fluency´, which was operationalized by the following statement: ‘Pattern of movement is fluent, precise and efficient (few únnecessary´/random movements)’ and Ćarefulness´, which was operationalized by the following statement: ‘The tissue is treated with respect during the procedure, without visible excessive force, traction or resulting in injury’. A specific rating scale, named ´Sumscore´ was designed to focus on the phase of the cholecystectomy clip-and-cut procedure. In this scale, performance is judged based on an integration of psychomotor skills, procedural knowledge of anatomy and decision making. This sumscore was constructed much according to the metrics usedfr T

133

Chapter 8 All participants in both the experimental and the control group were either in training

r general surgery, or in the first two mandatory general surgery training years while in fotraining for another surgical subspecialty. Fifty percent of participants in the experimental group were in training to become (general) surgeons, in contrast to 70% of participants in the control group. Two participants in the control group had engaged into an animal gallbladder training course, one in a non-animal one. Participants in the experimental group were not previously engaged in specific cholecystectomy trainings courses. Table 1. Demographics by group (experimental versus control)

Number of laparoscopic

cholecystectomies

Sex Age Year of

Training

Specialty

performed assisted

Mann-Whitney U 40,000 43,500 34,000 31,500 14,500 24,500


0,342 0,900 0,317 0,201 0,008* 0,083

*p ≤ 0.05 Of the demographic variables, only the number of laparoscopic cholecystectomies performed before participation on the study differed significantly (Table 1, p-value Mann-Whitney U test significance: 0.008). Mean number of laparoscopic cholecystectomies performed by the experimental group was 0.3 (range 0-1), whereas the mean number of procedures performed in the control group was 1.8 (range 0-3). Course satisfaction Figure 1 depicts the course satisfaction 95% confidence intervals, according to course participants´ opinion, referring to statements as presented in Table 2. Fig. 1. Course Satisfaction 95% C.I. Table 2. Satisfaction statements

1. I feel this VR-to-OR course was a valuable course for me

2. I feel this course adequately highlights the most important aspects of the laparoscopic cholecystectomy

3. I believe I will feel much more secure performing my first laparoscopic cholecystectomy on the OR having followed this course

4. I think this VR-to-OR course is a highly valuable asset in the modern surgical curriculum

5. I believe in the use of VR-simulators to train surgical residents

6. I believe in the use of VR-simulators to monitor surgical residents' progress

7. I believe in the use of VR-simulators for selection of surgical residents

my

8. I think my laparoscopic surgical skills have improved significantly after following this course

9. I felt comfortable when I did my exercises on the VR-simulator

10. I would recommend this VR-to-OR course to colleagues

0: higly dissatisfied, 10: highly satisfied)

10987654321

12

10

8

6

0

4

2

134

The Eindhoven laparoscopic cholecystectomy training course Fig. 2. Virtual Reality Simulation Ratings (0-10 point scale)

Outcome Parameters Normal distribution of the primary outcome parameter ´judgement´ and secondary outcome parameters ´fluency´ and ´carefulness´ and was confirmed by Q-Q plots. Agreement on performance outcome Cohen’s kappa could not be computed using the original outcome scale, as it was not dichotomous or nominal, nor did judgement outcomes fit in similar number of classes. Compression of data into fewer classes to compute kappa as a measurement of inter-observer agreement was not chosen, because too much information would be lost. An alternative approach was chosen, e.g. by comparing the scoring results of the two observers using non-parametrical testing (Table 3). Observers do not seem to differ in their opinion on surgeon’s performance in either category. Table 3. Agreement on performance outcome parameters ´fluency´, ´carefulness´ and ´judgement´

Group Movement fluency

Movement carefulness

Judgement 2 versus Judgement 1

Z*

-,447 0,000 -1,509 Experimental

Asymp. 0,655 1,000 0,131

Sig. (2-tailed) Z*

-1,134 -0,707 -0,640 Control


0,257 0,480 0,522

*Wilcoxon Singed Ranks test

12

10

8

6

4

N =

Xitact navigation Xitact dissection

Xitact clip-and-cutMIST-VR difficult

MIST-VR moderateMIST-VR easy

7

96

7373

2

0

-2

135

Chapter 8 The parameter sumscore, estimator for clinical outcome, was related to final judgement. Indeed, scores are significantly and highly correlated (Table 4). Table 4. Correlations between ´sumscore´ and ´judgement´

GROUP Sumscore Judgement

–mean- –mean-

Correlation Coefficient

1,000 0.711 Sumscore –mean-

Sig. (2-tailed) , 0,000*

Correlation coefficient

0,711 1,000 Judgement –mean-

Sig. (2-tailed) 0,000* ,

*Kendall´s tau_b. Correlation is significant at the .01 level (2-tailed Asymp. Sig.)

ance outcome between experimental and control groups

both observers and ´Fluency´ by observer 1.

Observer 1 Observer 2

Differences in perform Fig. 3a and 3b graphically represent clinical outcome as judged by both observers. Table 5 depicts the null hypothesis and its rejection for outcome parameter ´Judgement´ by Fig. 3a. Clinical outcome (observer 1) Fig. 3b. Clinical outcome (observer 2)

Table 5. H0: Groups do not differ in performance status

Fluency Careful Judgement Fluency Careful JudgementTime

Ua 24,000 37,500 12,000 29,000 31,500 18,000 33,500

Asymp. Sig. (2-tailed)*

0,0037 0,214 0,004* 0,077 0,108 0,013* 0,212

aMann-Whitney U test

ControlExperimental

10

8

6

4

2

0

FluencyCarefulness

1

8

N =10 Judgement

N =10 0

10

8

6

4

2

ControlExperimental

Fluency Carefulness Judgement

49

8

16

1812

15

136

The Eindhoven laparoscopic cholecystectomy training course Fig. 4. Plot of judgement versus ´time´ (seconds)

10

9

8

Discussion Published evidence on the ability of transferring skill using VR simulation to the operating room is present, however, it is limited. Few studies have focussed on transfer

f such skill. [4-7] Grantcharov´s study randomized between surgical trainees receiving basic psychomotor VR training and a c

ber of laparoscopic cholecystectomies performed in the experimental group is higher than in the control group. Considering the study design an nit is not surprising that outcome results a porandomized between basic psychomoto Vtraining. Unfortunately, perhaps the mostlaparoscopic cholecystectomy, e.g. the cartery, was not performed by their experimen f the OR model is porcine or human, nor is there informaconducted by Hyltander´s research gronavigation and instrument navigation in a porfact the first study showing improvemen in OR performance in humanMIST-VR training, whereas their v oimprovement. Psychomotor skills improved i grou

show a marked decrease in the time taken to perform the clinical procedure and error reduction for their experimental group, as well as a significantly enhanced economy of movement, and must be regarded indicating predictive validity for transfer of psychomotor laparoscopic skill.

oontrol group of surgical trainees which receivedthis study, the mean numno form of training whatsoever. In

d the on-random bias introduced in this study, re sitive. The study of Seymour and others r R training and standard programmatic

critical phase in decision making in lipping and cutting of the cystic duct and –

tal group. It is not clear ition regarding ´patients´ variability. The study up used a Likert rating scale for camera

cine OR model. Hamilton´s study was in t s through

ide -trained control group showed no n both the VR and video-trained control

p. It seems that, independently of the study design, all above mentioned studies

seconds

700600500400300200100

mea

n 5

4

judg

emen

t

7

6

GROUP

control

experimental

137

Chapter 8 This study investigates the operation room results of novice surgical residents after par the Eindhoven Virtual Reality laparoscopic cholecystectomy training course. During this tailored course, both basic and procedural VR skills training tasks are integrated, resulting in a m roach which is a novelty in procedural teaching. Surgical novice residents were compared with controls who have had none or some (animal) training, but were considered able to start with the procedure by their superiors. In fact, the constitution of our case-control group refers much to the common, unstructured clinical practice of today. As in the traditional apprenticeship mod it is the expert surgeon who decides if a resident is ready to start operating on patients –or not. Groups were demographically comparable except for the parameter ńumber of laparoscopic cholecystectomies performed´ (Table 1). In the case-control group, there is even somewhat more clinical experience present. However, as defined by the selec is is limited m of 3 cl ures. Therefore, participants in both be consid e beginning of the learning

act that among the controls, significantly more clinical experience is seem to contribute much to their performance outcome, as

g a 10]. over-all performance in the experimental group to be

come, e.g. large confidence

rimental group, suggesting more efficient performance. ey enjoyed and valued the course highly, in terms of the

essed. Participants were n the experimental group

were analysed against the ba of a group of controls trained according to ćurrent clinical practice in The Netherlands´, e.g. declared fit to perform the procedure according to the opinion of their own clinical supervisor. As there were no baseline recordings -using VR simulation- available for both the experimental and the control group on the assessed procedure, it cannot be firmly stated tparticipants are equally skilled upon inclusion. Although inclusion criteria restricted clinical experience to a maximum of four procedures, and in fact, controls were mexperienced beforehand, their outcome is significantly worse.

ticipating in

ulti-modality app

el,

tion criteria, th to a maximu inical proced groups must ered well at th

curve. [1, 9]. The fresent, does notp

observers agree (no significant differences between observers on outcome parameters (Table 3). Rating scales were constructed for the observers, as this is regarded as the

ost reliable and valid method for observers to assess performance durinmlaparoscopic cholecystectomy [

Observers judge the clearly superior. One observer also feels the pattern of movement in terms of fluency to be significantly better in this group, compared to the performers in the control group (Table 5). The parameter ´sumscore´, used to estimate performance outcome in earlier validation studies of the Xitact simulation software, seems to be correlate highly to observers´ judgement (Table 4). Indeed, scores are highly correlated and therefore sumscore is thought to be a reliable estimate for observers´ final judgement. As for the parameter ´time´, it cannot be said that there is a significant difference between groups Table 5). This is probably due to the large dispersion in out(

interval. However, as depicted in figure 4, linking observers´ judgement to time needed for completion of the clip-and-cut task, there is less dispersion in procedural time

ithin the expewAs for the participants, thstatements presented (Fig. 1 and Table 2). Participants´ rated the Xitact developed and validated clip-and-cut scenery highest (Fig. 2). [3, 11]. A few limitations to the methods of this study must be addrnot randomly assigned to either group. In fact, residents i

ckground

ćlip-and-cut´ hat

ore

138

The Eindhoven laparoscopic cholecystectomy training course The better performance of residents in the experimental group following the VR

hat the control group ad a lower level of skills to begin with. Also, by assessing only the ćlip and cut´ part,

le procedure of the

on-anim ring endoscopic surgical skill are usually limited usua s

elieved that psychomotor skills can bough there is no form of objective asse esides structured observation and

no procedural training can be performed [12-14]. Animal training models provide excellent opportunities for tissue handling

and exposure to anatomy almost identical to the human counterpart. However, animal models ar t be repeated easily, thus providing inadequate possibilities for the resident to reach the plateau phase herent to training of a new laparoscopic procedure. Also, animal training models require a demanding skills laboratory in uct .

Virtual Reality si d o become ´the next big thing´ in surgical tr re specifi tu lation is free of many of boundaries and limitations mentioned above, provided th ystem ell validated

]. Furthermore, VR simulation systems have the asset of being an objective scoring g individual learning curves

]. Earlier studies have shown that VR training results in psychomotor skill acquisition d as, if not better than, programs employing conventional box

trainers [12, 15]. The second generation of VR simulators provide excellent real-time graphics, various patient scenarios and anatomical variations, combined with near-to-real haptic sensations and behavioural characteristics of the simulation. Therefore, it is imported for both surgeons as educators to be aware and oriented in this fast expanding field of educational tools [16]. Precisely so, the Eindhoven course was further on developed featuring aspects of both LapSim and Simbionix LapMentor VR simulation software. Conclusion Procedural VR simulation can no longer be considered fancy éye-candy´. Indeed, it should be regarded upon as se effective teaching instrument for acquiring endoscopic surgical skill. Our grating multiple repetitive, complimentary VR simulations showed that ope ting room performance is significantly better in the experimental group.

Therefore, the Eind tual Reality laparoscopic cholecystetraining course is likely to in rticipants´ level of skill in such a way tshortens the learning curve inherent to the procedure in a safe environment.derived from the course can be brought to bear on to the clinical situation, and wan asset in the curriculum to tr rgeons.

course must be interpreted with care, as it cannot be ruled out thone must be careful in extrapolating study results to the wholaparoscopic cholecystectomy. Nevertheless, it is highly unlikely that residents who are not fluent or careful in motions on the ćlip and cut´ part are technically able to dissect Calot´s triangle properly. N al training models for acquito box trainers. Bench models are safe, lly readily available and inexpensive. It ib e trained in these environments effectively, alth ssment b

e costly and cannoin

frastr uremulation is boun

in MAS. Vir t

aining, and mo cally al Reality simue s

the s are w

[3system and inherently, offer the possibility of constructin[9are at least as goo

rious, cost-study, intera

hoven Vir ctomy crease pa hat it

Skills ill be

ain future su

139

Chapter 8 References

The tectomy" Am J Surg 170: 55-59

2. Bridges M, amond DL (1989) "The financial impact of teaching surgical residents in the operating room" Am J Surg 210: 118-121

3. Schijven M, Jakimowicz J (2003) "Construct validity: experts and residents performing on the Xitact LS500 laparoscopy simulator" Surg Endosc 17: 803-810

4. Grantcharov, TP et al (2004) "Randomized clinical trail of virtual reality simulation for laparoscopic skills training" Br J Surg 91: 146-150

5. Seymour NE t al (2002) "Virtual reality improves operating room performance: results of a randomized, double-blind study" Ann Surg 236(4): 458-463

6. Hylta al (2002) "The transfer of basic skills learned in a laparoscopic simulator to the operat o " Surg Endosc 16: 1324-1328

7. Hamilton EC et al (2002) "Comparison of video trainer and virt ty training systems on acquis o roscopic skill" Surg Endosc 16: 406-411

8. Ahlbe et al (2002) "Does training in a virtual reality s r improve surgical perfor Surg Endosc 16: 126-129

9. Schijven MP, Jackim J 03)" cu on aroscopy simulator: pr of perform " Su ndo 8: 1 7

10. Eubanks TR et a 9) "An objective scoring system for laparoscopic cholecystectomy" J Am Coll Surg 556-574

Jakimowicz J (2002) "Face-, expert- and referent validity of the Xitact® LS500 opy Simulator" Surg Endosc 16: 1764-1770

. Rossner JC, Rossner LE, Savalgi RS (1998)"Objective evaluation of laparoscopic surgical skill

1.

Southern Surgeons Club (1995) "The learning curve for laparoscopic cholecys

Di

e

nder Aing r

ition rg Gmance?

etom

ual realif lapa

imulato"

owicz J (20ance

The learg E

rningsc 1

rve21-12

the Xitact LS500® lapofiles

l (199 189: p.

11. Schijven M,Laparosc

12program for residents and senior surgeons" Arch Surg 133: 657-661

13. Taffinder NJ et al (1998) "An objective assessment of surgeons' psychomotor skills: validation of the MIST-VR laparoscopic simulator" Br J Surg 85 (Suppl 1): 75

14. Scott DJ et al (2000)"Laparoscopic training on bench models: better and more cost effective than operating room experience?" J Am Coll Surg 191(3): 272-283

15. Rossner JC, Rossner LE, Savalgi RS (1997) "Skill acquisition and assessment for laparoscopic surgery" Arch Surg 132: 200-204

16. Schijven M Jakimowicz J (2003) "Virtual reality surgical laparoscopic simulators: how to choose" Surg Endsc 17: 1943-1950

140

Virtual reality surgical laparoscopic simulators: how to choose Chapter 9 V Reality surgical irtual

Schijv

Surg End

laparoscopic simulators: How to choose M. en, J. Jakimowicz

osc, 2003. 17: 1943-1950

143

Chapter 9 Introduction In the past 10 years, minimal access surgery (MAS) has replaced open surgery for a variety of procedures. In general surgery, procedures such as cholecystectomy, anti-reflux surgery, bariatric surgery, colon surgery, and knee and shoulder surgery are performed laparoscopically. Also, laparoscopy is used increasingly for surgical procedures in gynaecology, cardiac surgery, and urology [14]. MAS imposes specific strains on the surgeon, requiring specific psychomotor abilities and skills due to the videoscopic surgical interface. Conventional teaching methods for open surgery, such as the classical apprenticeship with hands-on theater experience, do not emphasize the acquisition of such skills. Furthermore, when performing endoscopic surgery, surgeons in training cannot easily mimic their mentor’s actions or manoeuvres without first manipulating the laparoscopic instruments in an initially disorientating, two-dimensional environment [6].

Due to the nature of laparoscopy, it will likely benefit from developments in Virtual Reality (VR) technology. In fact, elaborating on the successful paradigm of flight-simulator training for pilots, the potential of VR applications for laparoscopic surgical skills training was proposed almost a decade ago [7]. Recent advances in computer technology, combined with the consensus about the need for shortening learning curves in surgery in a non-patient but equally informative teaching settings, have led to the development of laparoscopic VR simulators. Evidence is accumulating that such simulators appear to be valid instruments in the acquisition of surgical laparoscopic skill [2–5, 8–13].

Because there are many ‘‘players’’ in the field of medical VR simulation, and because developments are evolving fast, it is quite difficult and confusing for the potential consumer to be informed about what is on the market and to how make a proper choice among available simulators. In fact, there are several models, some with identical hardware but all featuring different software. Basic and more advanced laparoscopic simulations are available, each with different features, and sometimes different intentions. This review provides an overview of the main simulators available, highlights assets and shortcomings, clarifies underlying similarities, and notes their differences. Not all companies offer both software and hardware simulation environments, and partnerships between several companies have been formed. An attempt has been made to feature the most important, most promising, and/or the ‘‘leading’’ companies in the area of laparoscopic VR simulation.

Care was taken to ensure that information is up-to-date and correct. However, we had to rely largely on information provided by the participating companies. The importance of correct, realistic, and unbiased information was stressed. We cannot fully exclude or be held responsible for errors resulting from faulty information.

144

Virtual reality surgical laparoscopic simulators: how to choose Methods Eight companies, all renowned firms in the field of hardware and/or software laparoscopic Virtual Reality system development, were asked to participate in the study. All companies were sent a similar, open-ended questionnaire asking for a description of their VR simulator and its features, modules, and properties for assessment. Furthermore, primary user target groups were defined as well as the specific simulators’ most important aspects, shortcomings, and technological specifications Results Table 1 gives a comprehensive overview of the major companies in the field of laparoscopic surgical simulation and their simulators. Some companies offer complete simulation solutions. Others emphasize either software or hardware development and are often linked in mutual partnerships. Most companies indicate their intended primary user group to be surgical trainees, specifically physicians (residents) in preparation for their first live MIS operation. Some companies stress the importance of their simulator as an objective assessment tool. For each simulator, the basic and advanced training modules available are listed in table 1 as well as more simulator-specific modules. The presence or absence of force feedback is noted, reflecting the important asset of haptic sensation. In some simulators, force feedback is optional. The presence of a performance playback pos-sibility is noted as well as the presence or absence of ‘‘live’’ surgical or instructional video clips of the procedure. All simulators feature some form of assessment, which is explained in the specific sections in more detail. Finally, table 1 denotes which simulators are and which are not (yet) validated.

145

Chapter 9 Table 1. Overview of simulators

Company

ntice Reachin SimSurgery Me

RLT Procedicus MIST Name of SimulatorB C

VR Anastomosis Trainer Classic Procedicus

Basic Skills navigation/coördination touching grasping stretching/traction translocation -others

● ● ● ● ● -

● ● ● ● ●

Force-Back

○ ●

needle ○

needle -

● ● ● ● ● -

● ● ● ● ● -

Advanced Skills clip applying transsection / cutting dissection diathermia suturing knot tying cholangiography -others

● ● ● ● ○ ○ ○ -

● ● ● ● ○ ○ ● -

○ ○ ○ ○ ● ● ● -

○ ○ ○ ○ ○ ○ ○ -

● ● ○ ● ● ● ○ -

Modules B BC SimLap Sjmcor

SimPlanner

MIST

Core skills 1 & 2

K.S.A. Mist

M-suture M-arthro

Force Feedback ● ● ○ ○ *

Playback Function ○ ○ ○ ● ● Video ○ ● ● ● ● Assessment ● ● ● ● ● Validation ● ● ○ ● ●

Costs (€) 58.900

72.800 No price indication from 16.000 from 25.000

○: not present N.A.: non-applicable ●: present *: optional ●: in preparation or studies not available trough PubMed

146

Virtual reality surgical laparoscopic simulators: how to choose Table 1. Overview of simulators- continued-

ct Select-IT

Vest

Surgical Science Immersion Simbionix Xita

LapSim VSOne Basic Skills Dissection V.L.I. L.S.W LapMentor LS500

● ● ● ● ● -

● ● ● ● ●

● ● ● ●

N.A. N.A. ● ● ● ●

●● ● ●

- -

- - ● ●

○

● ● ● ● ● ● * -

● ● ○ ○ ● ● ○

● ● ● ● ● ● ○

N.A. N.A. ● ● ● ● ○ ○ ●

● ● ● ● ○ ○ ○

ultrasound scissors

ultrasound scissors

-

-

Gyn or Cho;

C.B.T. B.T.T. S.P.T.

Basic Skills Dissection Other simulators: -AccuTouch

-Endovascular & Endoscopy,

-CathSim Vascular (own hard- and

software)

Basic Tasks C.B.T.

Lap. Chol. Full Procedures

Lap.CClip-cDissecNaviga

hol. ut & tion tion

● * * ○ ● ● ●

* ○ ● N.A. N.A. ● ● ● ○ ○ N.A. N.A. ○ ● ● ● ● N.A. N.A. ● ● ○ ● ○ N.A. N.A. ● ●

70.000 / 112.5000

39.000 54.000 incl. Basic skills

6.000 28.000 90.000 110.000

147

Chapter 9 Reachin Reachin Technologies AB was founded in 1997 as a result of collaboration between research institutes in Australia, Singapore, and Sweden. Reachin’s Laparoscopic Trainer (RLT) is their simulator. Two versions are available: the Reachin Laparoscopic Trainer Basic (RLT-B) for basic skills training and the Reachin Laparoscopic Trainer Basic and Cholecystectomy (RLT-BC) for basic skills training and laparoscopic cholecystectomy. Features The RLT-BC module follows a four-step system. Tasks start with an instruction video,

a playback lesson commenting on the video. The third step, called orceback , leads the trainee through the procedure. The trainee merely holds the

of classes, designing courses, remote tutoring, and viewing of results from any remote location). Raw data are stored in XML format for exchangeability and calculation purposes. Most important aspects The Forceback feature is unique. The BC module instructs the student gradually through a procedure. The Web-based Lesson Management System offers easy remote control. Shortcomings The system cannot perform a full laparoscopic procedure from start to finish. There is no playback possibility of the performance of the participant. Validation studies are ongoing. SimSurgery SimSurgery AS was established in 1999 to develop medical simulation software and solutions for virtual training and robotic surgery. Its software has been developed by specialists in mathematics in collaboration with the Norwegian National Hospital.

followed byTM

Finstruments, following the autonomic movements of the instruments as previously recorded through the performance of an expert laparoscopic surgeon. It is possible to incorporate the movements of an expert laparoscopic surgeon of one’s own hospital. Finally, the trainee has a free-format session, making his or her own movements. Six complementary video instruction lessons are available for the RTL-BC. Assessment Movement patterns are recorded and, thus, it is possible to present performance data. Touch-sensitivity thresholds can be set according to the Administrator’s preferences. Bilateral movement patterns can be displayed and compared to the ‘‘ideal’’ route. Data on acceleration of movement and performance errors, such as faulty touch of tissue and incorrect use of diathermia, are available. The Lesson Management Environment is Reachin´s Web-interfaced tutorial management system offering a variety of possibilities to the administrator (i.e., the creation

148

Virtual reality surgical laparoscopic simulators: how to choose SimSurgery developed the VR anastomosis trainer. Basically, their software can be

apted to any surgical simulator or any other system requiring real-time three-dimensional (3D) capacity. Features The VR anastomosis trainer trains laparoscopic continuous suturing. A module has been developed to perform suturing for connecting a simulated artery onto a coronary artery on a beating heart (SimCor module). A surgical planning module (SimPlanner) teaches the trainee the optimal configuration of the surgical robot system and port placements. The simulated robot and operating room scenes can be merged with patient 3D data from CT and MRI scans. Knot-tying is available. Assessment Time to complete the procedure is recorded as part of the simulation. The placement of stitches and needle/ instrument trajectories are part of the evaluation database. Most important aspects SimSurgery SIM3DM is a major software developer, licensing its products to Zeus’ robotic system (Computer Motion), the Procedicus platform (Mentice), and Immersion’s VLI. SimSurgery’s Web site is informative, with links to videos and images of the various systems, articles in the literature, and partners. Shortcomings The anastomosis trainer needs a separate hardware platform to run on. Thus, the software itself cannot mimic force-feedback. Mentice Mentice, a Sweden-based corporation, develops medical simulation software and systems. The Procedicus, featuring Minimally Invasive Surgical Trainer-Virtual Reality (MIST-VR) Core Skills 1 & 2, is suitor to the ‘‘classical’’ MIST-VR surgical simulator launched in the mid-1990s. The Key Surgical Activities (KSA) module and MIST suture module, developed by SimSurgery, and other modules can be incorporated in the Procedicus platform. Features Procedicus is a modular simulation environment. From a functional stand-point, it can be divided into two major groups: the Procedicus MIST ‘‘classic’’ for basic skill acquisition and the Procedicus simulation platform. This platform was originally developed for the virtual arthroscopy shoulder and knee simulation; later, the laparoscopic KSA module was implemented.

ad

149

Chapter 9 Three MIST modules are available: the MIST Core Skills 1 & 2 for basic psychomotor

aining and the MIST suturing module for training of needle handling, suturing, and (forthcoming) knot tying. The original MIST system does not provide force feedback (because it uses Immersion’s VLI) but has a ‘‘colour code’’ for guidance. When running MIST on Procedicus, force feedback is optional. Procedicus has configurable levels of skill/task difficulty for each module. The Core Skills modules cover six basic exercises. The Procedicus KSA module was designed for more advanced laparoscopic training, focusing on procedures such as scope and instrument navigation, pick and place, pick and pass, cutting, suturing, needle passing, and the use of diathermy in a VR abdominal environment. The Procedicus environment can incorporate training modules in arthroscopy, interventional cardiology, and endovascular surgery. Assessment A database contains records of students across trained modules. It keeps track and organizes classes and curricula. The system recognizes where the trainee has left off and will move on to the next scenario, with feedback on performance and a post-test at every level. The instructor can follow and comment on the student’s progress through the Internet. There is a performance analysis and report tool set, and data can be imported exported. For the Core Skills module 1 & 2, performance is measured by time, number of errors, and tool efficiency. Most important aspects Various modules for the Procedicus MIST-VR system are available. In addition to the KSA module, other minimally invasive simulations in arthroscopy, urology, gynaecology, interventional cardiology, and radiology are available. The MIST system is the most extensively validated VR system in surgical VR training. Shortcomings Force feedback with the Procedicus system is optional, based on the Immersion hardware that is used. Mentice acknowledges that there are still limits to the level of its haptic realism. Select-IT VEST systems Select-IT VEST Systems AG was founded in 2000 by a team of experts in endoscopy, surgery, VR, mechatronics, marketing, and sales. Their simulator is the Virtual Endoscopic Surgery Training (VEST) system. Two applications of the VEST system are available: the VSOne Cho trains laparoscopic cholecystectomy, and the VSOne Gyn trains laparoscopic gynaecology.

tr

150

Virtual reality surgical laparoscopic simulators: how to choose Features Both VSOne applications have a basic and more advanced set of surgical training tasks. It is a network-compatible assessment platform. VSOne offers different training programs. Both VSOne Cho and Gyn have peer-reviewed computer-based training (CBT), a basic task training set (BTT), surgical procedure training (SPT), and either a cholecystectomy or a gynaecology (laparoscopic sterilization/ovary manipulation) simulation. BTT familiarizes a trainee with the system and trains basic laparoscopic

h as camera handling, navigation, and dexterity training. It focuses on ma-ipulation and coordination skills with endoscopic camera and tools in a non-

g depends on xercise. In general, the parameters time and errors are used to construct

e diagrams and show task history, and precision of performed task and

g CME credit points. The VSOne a platform for designing and implementing individual

issing force feedback in the grip. This problem is recognized and is

tasks, sucnanatomic, game of skill-like parcourse. SPT modules train more specific surgery procedures, such as suturing, dissection, knot tying, clipping, and cutting. The VSOne Cho and Gyn modules are specific surgical applications. Force feedback is coupled to all procedures. Assessment Optional to the system is an authoring toolkit, which allows the user/trainer to design nd implement customized procedures and trainings. Performance scorina

the kind of eerformancp

economy of operation are measured as path length if applicable. Errors are defined depending on the procedure. In general, errors result from touch in restricted areas or clipping/cutting outside feasible regions, touch with forces higher than pre-defined values (values depend on the procedure), and incorrect use of coagulation. Data are stored in a database for post processing, for example, by Microsoft Excel. Most important aspects The VSOne is an open system. The included CBT with Internet access allows for heoretical training in surgery with the possibility of earnint

authoring tool makesraining. t

Shortcomings VSOne is mcurrently the focus of research and development. The system is not fully validated; studies are ongoing.

151

Chapter 9

Surgical Science

Surgical Sciensimulator f

ce AB was founded in 1999 and is based in Sweden. Its LapSim is a eaturing different training modules for training of minimally invasive

rences can be set . LapSim Dissection aims at practicing both blunt and sharp dissection in the hepatoduodenal ligament. Anatomical variations are present. For both

ing, and suturing, in which the task is to make a stitch and a double a game-like module integrating different skills at different levels,

is a network-evice.

ned and data assessed. Parameters such rs, tissue tension and damage, and movement patterns and trajectories are epending on the module, parameters such as bleeding and number of clips

has a high degree of realism with regard to graphics and tissue–instrument n. Different anatomical settings are available in the dissection program. The

ilable. Validation of the system is limited but ongoing. Force feedback is optional.

surgical skills. The simulation programs are LapSim Basic Skills and LapSim Dissection. Features LapSim Basic Skills 2.0 offers a set of nine training modules. Practice sessions may vary in graphic complexity as well as level of difficulty, and prefeindividuallythe area of programs, curricula can be created or modified by the teacher to fit a trainee’s specific needs or the training requirements. In Basic Skills, nine training modules are featured, ranging from navigation to more advanced skills such as coordination, graspcutting, clip applyingknot. It also features coupled to a scoring system. In Dissection, different anatomical variations are available. Parameters to be set are lens angle, tissue properties, tissue environment, object size and numbers, object movements, time limits, and handedness. A variety of instruments can be used. Exercises in both simulation programs can be performed at different levels of difficulty. Force feedback is optional. LapSimcompatible d Assessment In the assessment mode, curricula can be desigas time, errorecorded. Dused can be measured as well. Results are presented in graphical figures and illustrations. Examinations are provided to validate student’s progress, and requirements for passing an examination can be set by the administrator. Data can be exported. Most important aspects The systeminteractiosystem is flexible and based on a platform/ module principle. There is a simple network connection. Shortcomings There are no complete surgical procedures ava

152

Virtual reality surgical laparoscopic simulators: how to choose Immersion Immersion Corporation was founded in 1993. The company develops hardware and software systems for surgical simulation settings. For laparoscopic simulation, the Virtual Laparoscopic Interface (VLI) and the Laparoscopic Surgical Workstation (LSW) were designed. Both VLI and LSW are systems enabling developers, researchers, and educators to develop specific software for simulating laparoscopic abdominal procedures. The VLI is a system for tracking instrument motions associated with minimally invasive surgical procedures. The LSW is specifically designed for acceleration of learning curves in minimal access surgery, integrating force feedback.

he simulators are licensed to different manufacturers/developers, integrating eir specific medical simulation devices. Surgical Science

ses the VLI as the hardware platform for its LapSim basic skills, as does Mentice in

surgical instruments through a 3D work-ace. Tools have five degrees of freedom, encompassing the motion range of most

res. There is a near real-time representation of the movement of e hands to the image on screen. It allows users to become comfortable with the

ever, a software evelopment kit is provided with the both systems.

imbionix

eloping and producing training simulators for minimally inva-ve surgery since 1997. The LapMentor is their laparoscopic surgical simulation

skills in a non-anatomic setting can be practiced, such as camera navigation at 00 or 300 instrument navigation, object manipulation, and clipping and cutting. More complex skills, including virtual patient cases, allow practice of realistic laparoscopic procedures, such as laparoscopic cholecystectomy.

TImmersion’s technology in thuits classic MIST environment. The LSW is used by Reachin, Surgical Science, and Mentice. Features VLI offers a method for monitoring pitch-and-jaw motions of laparoscopic instruments. It tracks the motion of a pair ofsplaparoscopic proceduthremote manipulation of surgical instruments, accelerating the experience curve for laparoscopic procedures for both physicians and students. LSW offers basically the same features but has TouchSense technology and different sets of handles. No specific surgical modules are supplied with the simulators. Howd S Simbionix is headquartered in Cleveland, Ohio, and has a subsidiary in Israel. The company has been devsiplatform. Features The LapMentor allows for practicing of basic laparoscopic skills using the Basic Task Modules (BTM), as well as for more complex skills mimicking surgical laparoscopic procedures. In BMT, a variety of basic

153

Chapter 9 In the Cholecystectomy Basic Tasks (CBT), a variety of didactic basic tasks in an

atomical setting, such as cystic artery and duct exposure, cannulation tasks, and a

d, and navigation). Didactic parameters such as decision-making tions (‘‘convert to open’’ option and the reason to convert) are included. Scores

time, speed, and navigation parameters as well as accomplishment of task. he score for the virtual patient cases is based on an equation reflecting all the

uments can e mimicked using a number of possible ports. Furthermore, Simbionix is known for

ith building other medical simulators (gastroscopy, urology, ercutaneous, vascular and endoscopic ultrasound).

y to provide realistic tactile sensation in LapMentor is limited. Validation of e system is preliminary.

y simulator providing a modular training environment for the education and assessment of laparoscopic procedural surgical skill.

antutorial cholangiogram can be explored. A virtual instructor guides the trainee and gives intra- simulative feedback. Actual setup of operative port placement can be mimicked and changed during operation. Assessment The trainee receives immediate feedback from the virtual instructor. Different training curricula can be set up. Quantative parameters of performance vary over task and procedure. Among these are total time, efficiency rate for task, safety parameters (percentage safe clipping, percentage safe cutting, minimal length in distances between clips, amount of energy used for diathermia, safe cauterization, percentage of thorough dissection, speeopreflect task Tquantitative parameters and can be fine-tuned according to the training methods used by the administrator. The export data feature enables the administrator to transfer information. Most important aspects The system has high-quality anatomical visual representation, featuring patient cases using CT/MRI images from real patients. Complete and continuous laparoscopic procedures are represented. The physical insertion of the laparoscopic instrbhaving experience wp Shortcomings The abilitth

Xitact Xitact SA was founded in 2000 in Morges, Switzerland. Its product was developed by independent expert laparoscopic surgeons and technical hardware and software engineers. Its product is the Xitact LS500 laparoscopy trainer platform. The Xitact LS500 is a laparoscop

154

Virtual reality surgical laparoscopic simulators: how to choose Features

he Xitact LS500 features three simulation environments, the laparoscopic navigation

ents such as surgical video clips with voice-over, phased simulation ns, a library of performance errors, simulation playback, and assessment s. Two performance modes are incorporated—a free-format mode and a

ile, an assessment sheet registering all important events during the d a playback feature. Data can be exported.

t to complete the procedure. The clip–cut scenery and scoring are odules built by other companies can run on the open Xitact

motor skills, unless licenced and upon request. No patient orporated. The cholecystectomy clip–cut and dissection module are the

ly available.

Tand the clip–cut curriculum / peritoneal dissection curriculum of the laparoscopic cholecystectomy. It combines proprietary force feedback technology with real-time deformation of organs within the simulation environment. Both the clip–cut and the dissection curriculum provide a multidisciplinary stepwise training environment,

cluding elemininstructio

cordingreguided mode with specific instructions for gradually increasing the level of skill needed to perform the complete exercise. Feedback through a performance report is presented at the end of the exercise. Xitact is a flexible, open platform able to feature various software VR simulations. Assessment Administrators can build ‘‘classes’’ and specify ‘‘curricula’’ and thus are able to create and monitor progress on different surgical curricula for individual user accounts. A performance report is available the shows validated general sum score incorporating performance time and the number and nature of errors (clips used, correct distances and placement of clips, cutting errors, bleeding, and leakage). Tool trajectories are also recorded. Improvement over exercises for the individual trainee and relative to trainee’s performance class is displayed. The individual learning curve is plotted over exercises. The administrator has access to all data, including the performance report, a hronologically log fc

excursive, an Most important aspects The Xitact LS500 is an open platform simulating a clinical procedure including 3D interactivity and Xitact developed proprietary force feedback. A clinical procedure with its various surgical steps (e.g., the laparoscopic cholecystectomy) is simulated. The scoring system is based on extrapolated clinical patient outcome assessment for the simulation rather than on more indirect performance parameters. Guided and free-

rmat modes exisfovalidated. Software m

latform. p Shortcomings Xitact does not incorporate standard software for other ‘‘basic skill’’ tasks for building

d evaluation of psychoancases are inc

ttings currentse

155

Chapter 9 Discussion

rovide an environment in which it is possible to learn and perform multaneously and to have the opportunity to repeatedly practice the same procedure

ssible complications under varying conditions and to get tailored feedback.

that training on these types of mulators will enhance laparoscopic performance. However, it is less likely that

ese types of simulators will lead to true and stable surgical proficiency in

tor is, of course, an important consideration. When the budget permits, nce haptics are one of the more costly features embedded in some VR simulators,

a simulation platform with force-feedback incorporated is recommended.

e nt 1,

e public proven to work as intended. Probably the best way to make a de-

cision is by obtaining hands-on experience with the different types of simulators.

The term Virtual Reality refers to a computer-generated representation of an environment allowing sensory interaction, thus giving the impression of true realism [1]. The three ways of ‘‘sensing’’ a virtual world are through sound, sight, and touch. Laparoscopic VR surgical simulators must therefore be able to generate 3D images on a 2D monitor that appear to be ‘‘natural’’ (e.g., moving and behaving in a realistic manner). The simulation must allow a high level of interactivity and be stable and reactive to the surgeon’s actions. Haptic feedback is important, preserving the individual an input of tactile sensation. Thus, VR laparoscopic procedural simulators that are valid representations of reality must present organs that are anatomically correct, with natural real-time deformation properties and resistance when manipulated, preserving natural traits such as bleeding or leakage when treated abusively. Furthermore, a good VR surgical simulator must let the surgical resident internalize a surgical procedure without the risk of harming patients. In order to do so, it must psiand its poUltimately, this process can be translated into an objective examination for certification of that procedure using the same simulator. A prerequisite for a VR simulator that is to be used simultaneously as an assessment tool is a proper validation of the system.

Until recently, most VR training settings focused exclusively on basic surgical psychomotor skills, such as grasping and translocation of virtual objects. Unlike laparoscopic surgery, some of these simulation models did not provide haptic sensations through force feed back. However, force feedback is an indispensable component of a realistic simulation environment. Psychomotor abilities may be trained on the less expensive non-force feedback type of surgical simulators, which teach more basic laparoscopic skills. The literature suggests sitraining on tha clinical laparoscopic procedure, with all its associated variability. The newer types of simulators do indeed acknowledge and emphasize the importance of force feedback and procedural surgical scenery, and they have force feedback integrated in their systems. Unfortunately, these are still very expensive simulators. The cost-benefit ratio of a simulasichoosing Although still expensive, these types of simulators will undoubtedly become less expensive as competition increases and technology becomes shared or morwidespread. For now, it is recommended that one browse through most receliterature in search for evidence of validation of a system. As can be seen in Tablesome simulators have cholangiography, whereas others have suturing modules; some can run multiple procedures on the same platform, and others are less expensive.

Finally, it all comes down to the issue of what is most appealing to thand what has been

156

Virtual reality surgical laparoscopic simulators: how to choose Major surgical and MAS conventions, such as those hosted by the American ColleSurgeons, the European Association for Endoscopic Surgery, and the Society American Gastrointestinal Endoscopic Surgeons, offer an excellent opportunityvisit different manufacturers of VR simulators on exhibit. Most manufactures are vewilling to provide visitors a hands-on expe

ge of of

to ry

rience on their simulator. For all simulators atured in this review, additional and most-up-to-date information can be obtained

nufacturers themselves (see Acknowledgments). fefrom the ma Acknowledgments We thank Dr. A. Fingerhut, chairman of the Scientific and Educational Committee of the European Association of Endoscopic Surgery (EAES). This review was performed under the auspices of the EAES Scientific and Educational Committee.

157

Chapter 9 We thank the following companies for their participation:

elect IT VEST Systems AG select-it.de

WW: http://www.select-it.de

-mail: [email protected] urgical-science.com

Immersion Corporation e-mail: [email protected] WWW: http://www.immersion.com Mentice Medical Simulation e-mail: [email protected]; [email protected] WWW: http://www.mentice.com Reachin Technologies AB e-mail: [email protected] WWW: http://www.reachin.se Se-mail: info@W Simbionix e-mail: [email protected] WWW: http://www.simbionix.com SimSurgery AS e-mail: [email protected] WWW: http://www.simsurgery.no Surgical Science AB eWWW: http://www.s Xitact SA e-mail: [email protected] WWW: http://www.xitact.com

158

Virtual reality surgical laparoscopic simulators: how to choose References 1. Coleman J, Duka CCN, Darzi A (1994) "Virtual reality and laparoscopic surgery" Br J Surg 81: 1709-1711 2. Gallagher A, Satava M (2002) "Virtual reality as a metric for the assessment of laparoscopic

psychomotor skills" Surg Endosc 16: 1746-1752 3. Grantcharov TP, et al. (2001) "Virtual reality computer simulation: an objective method for the

evaluation of laparoscopic surgical skills" Surg Endosc 15: 242-244 4. Grantcharov TP, et al. (2003) "Learning curves and impact of previous operative experience on

performing on a virtual reality simulator to test laparoscopic surgical skills" Am J Surg 185: 146-149 5. Haluck R, TM (2000) "Computers and virtual reality for surgical education in the 21st Century" Arch Surg 135: 786-792 6. Melvin SW, Johnson JA, Ellison CE (1996) "Laparoscopic skills enhancement" Am J Surg 172: 377-379 7. Satava RM (1993) "Virtual reality surgical simulator: The first steps" Surg Endosc 7: 203-205 8. Schijven M, Jakimowicz J (2002) "Face-, expert-and referent validity of the Xitact LS500

laparoscopy simulator" Surg Endosc 16: 1764-1770 9. Schijven M, Jakimowicz J (2003) "Construct validity: experts and residents performing on the

Xitact LS500 laparoscopy simulator" Surg Endosc 17: 803–810 10. Seymour NE, et al. (1992) "Virtual reality improves operating room performance: results of a

randomized, double-blind study" Ann Surg 236: 458-463 11. Strom P, et al. (2003) "Validation and learning in the Procedicus KSA virtual reality surgical

simulator" Surg Endosc 17: p. 227-231. 12. Taffinder N (1988) "Validation of virtual reality to teach and assess psychomotor skills in

laparoscopic surgery: results from randomized controlled studies using the MIST VR laparoscopic simulator" In: Westword JD (Ed.) Technology and informatics 50: proceedings of medicine meets virtual reality. IOS Press, San Diego, pp 124-130

13. Torkington J, et al. (2001) "Skill transfer from virtual reality to a real laparoscopic task" Surg Endosc 15: 1076-1079 14. Versweyveld L (2001) "Montefiore surgical residents receive laparoscopic training in freshly

opened Virtual Reality and Skills" Lab. http://www.hoise.com/vmw/01/articles/vmw/LV-VM-03-01-21.html

159

Summary and Conclusions Chapter 10 Summary and Conclusions /

Samenvatting en Conclusies

161

Chapter 10 Summary and Conclusions

the In he need for cific reasingical sp-wisea Virt

Chapre described.

is stud -of-the art in le

one out ed for these basic procedures

once bei gh chance to the f lthough dents raining,

evaluateddesired advanced laparoscopic procedures at the end of their surgical training. Specific, properly implemented, monitored and evaluated skills training programs could offer a promising environment in overcoming the discrepancy between fact and desire in autonomous performance of advanced laparoscopic surgery. Chapter 2 addresses the value of the Advanced Dundee Psychomotor Tester (ADEPT) as an objective scoring system for psychomotor task performance. Gaining insight in the issue of objective assessment of residents´ endoscopic performance is key issue in building surgical training programs. Therefore, the ADEPT system was used to assess the following three research questions: ´Are surgeons good estimators of their own performance on ADEPT´; ´Do surgeons perceive ADEPT to be a valid instrument for measuring psychomotor skills´ and ´Does performance on ADEPT reflect innate psychomotor ability´. Forty-five surgeons participated in the study, and each surgeon performed 2 runs of five randomly assigned tasks. A sum-score was computed and compared with surgeons´ own estimate of performance. According to the research questions, this chapter concludes that surgeons cannot adequately predict standardized individual test outcome on ADEPT. Also, ADEPT indeed reflects innate psychomotor ability along with improvement in performance over runs. Surgeons are ambivalent in assessing ADEPTs´ validity, irrespective of personal performance.

In tline of thesis, the reader is introduced to ttroduction and ouspe laparoscopic training for surgical residents, against the background of the inc g popularity of laparoscopic surgery. The novelty of Virtual Reality (VR) sur imulation is proposed, stating the aim of this thesis: to outline the process of ste , structured validation and implementation in the surgical training curriculum of ual Reality laparoscopic cholecystectomy simulator, the Xitact LS500. In ter 1, the results of a survey among residents-in-training for general surgery concerning Minimal Access Surgery (MAS) training in The Netherlands a

h order to make an inventory of current stateT y was carried out inMAS training and -performance among residents surgery, and its possibshortcomings. Having a response percentage of 65% (407 residents), this survey shows that Dutch residents (100%) feel it is important to be able to perform the three basic MAS procedures autonomously at the end of their surgical training; e.g. the diagnostic laparoscopy, the laparoscopic cholecystectomy and laparoscopic appendectomy. Still,

of ten residents expects to be inadequately preparhaving completed his or her training. GI/oncologic interested residents are more interested in performing advanced laparoscopic procedures, although the majority expects to be unable to perform advanced laparoscopic procedures autonomously

ng a surgeon. This phenomenon is attributed to ‘not having enoube l’. Aesi e need for validated basic laparoscopic skills t

irst operator’ and ‘lack of volume of procedures in the hospitar feel strongly about thonly a quarter of them indicate that attended skills training programmes are indeed

. In conclusion, only 17.8% of residents expect to be able to perform the

162

Summary and Conclusions This is attributed to the fact that ADEPT is a pure psychomotor tester, not taking into account other laparoscopic surgical parameters that determine clinical procedural outcome. Chapter 3 assesses the issue of face-, expert- and referent validity of the Xitact LS500 laparoscopic cholecystectomy VR simulator. The simulator is depicted and its features are described. The results of a survey, representing the opinion on proposed statements of 120 interviewed surgeons are presented. The results are categorized by the clinical laparoscopic experience of the surgeon, thereby formulating an opinion of ´experts´ (N= 87) and ´referents´ (N=33). The over-all opinion on Xitact is positive, believing it can act as a useful tool in teaching (93.1%) and measuring (79.3%) performance assessment. The study concludes that there is no significant difference in opinion between both groups, thus stating acceptance of the simulator in both the group deciding on funds and methods of teaching (expert group) and the group to be taught (referents). Face validity is thus confirmed by both groups. The study described in Chapter 4 was undertaken to assess the concept of construct validity of the Xitact LS500 laparoscopic cholecystectomy VR simulator. The primary research statement was ¨The clip-and-cut task on the Xitact LS500 Virtual Reality laparoscopic cholecystectomy simulator mimics the surgical procedure of the clipping and cutting of the cystic duct and artery during the laparoscopic cholecystectomy adequately¨. Construct validity is assessed by comparing the scores of an expert group of performers with the scores derived by novices on a novelty simulating the actual clinical performance. A valid system should be able to differentiate between different levels of skill. Three hypotheses were formulated to operationalize the primary research statement. An expert group (N=37, >100 laparoscopic cholecystectomies) and novice group (N=37, 0 laparoscopic cholecystectomies) opinion was formed. After getting familiar with the apparatus, three simulation runs were performed and groups were compared. This study shows that both novices and experts feel the clip-cut task to be very useful in surgical residents´ training. The majority of participants in both groups are not able to adequately predict their test outcome on Xitact. The first hypothesis, stating that performance scores derived from the clip-and-cut task by experts in clinical laparoscopic cholecystectomy are significantly higher than the performance scores derived by novices, could be answered affirmatively. The actual experience with the clinical procedure is thought to be the explanation for higher scores, as is supported by a shorter time-span to complete the simulation in the expert group. Therefore, scores are likely to be related to the number of clinical laparoscopic cholecystectomies, as stated in hypothesis 2. Also, hypothesis 3, stating that scores of both expert and novices should logically improve over runs – if the simulation concept is to be valid- is supported by our research findings. The construct validity of the Xitact LS500 is thus supported by our results. Chapter 5 addresses the issue of selection of laparoscopic surgical trainees, and focuses on establishing concurrent validity of the Xitact LS500 with test results on a standardized battery of psychomotor aptitude tests.

163

Chapter 10 As traditional methods of selection are subjective and do not seem to correlate well to

e scores with a ere previously

urgeons. Thirty-three laparoscopic novices participated in this study. Each participant performed the Xitact clip-and-cut procedure 30 times, in order to assess the individual learning curve and –pattern. The aptitude test battery included the Abstract Reasoning test, the Space Relation test, the Gibson Spiral Maze and the Crawford Small Parts dexterity test. The outcomes of this study are that only the Abstract Reasoning test correlates significantly and directly to Xitact test outcome. The test outcomes on the Abstract Reasoning test are highly related to test outcomes on the Space Relation test. Both tests do have discriminative power in comparing the group of talented trainees with less talented ones. (Chapter 7 further explains the issue of the learning curve and elaborates on performance group allocation). In conclusion, although the use of aptitude selection for surgical trainees must not be without debate, a combination of the Space Relation- and Abstract Reasoning test correlates to Xitact performance outcome. Low test scores on the combination of these two psychometric tests might be indicative for performance assessment using the Xitact LS500. Participants with high test scores on the Space Relations - and Abstract Reasoning test are unlikely to mal-perform on Xitact. Concurrent validity of the Xitact LS500 with the combination of the Space Relation test and Abstract Reasoning test is affirmed by our study. Chapter 6 describes the progress of residents in laparoscopic surgery, after attending the Intercollegiate Basic Surgical Skills Course (BSSC) and as measured by the Xitact simulator. In this course, residents engage in different aspects of laparoscopic surgery for about half-a-day. Laparoscopic (basic) skills are practiced in laparoscopic box trainers for about 3½ hour. Before and after the BSSC, 25 surgical residents performed three simulation runs on Xitact (experimental group). A control-group of 25 interns, not participating in the BSSC, also performed two sessions of three simulation runs. There appeared to be no significant difference in the parameters ´score´ and ´time´ between the experimental group and control group before the BSSC. Although there was a significant decrease in time to complete the simulation over the runs within the groups, there was none (except for run 2) between the two research groups. For the parameter ´score´, there is no difference between the two research groups. The BSSC is not designed to introduce surgical trainees to procedural laparoscopic surgery. Our study shows that the course does not increase laparoscopic procedural task performance in candidates as measured by the Xitact clip-and-cut scenery. Chapter 7 focuses on the learning curve on the Xitact LS500 simulator and assesses the acquisition of skill on the Xitact clip-and-cut laparoscopic cholecystectomy module. Thirty-three residents without clinical laparoscopic experience received a 1 hour familiarization on the apparatus, and completed a series of 10 simulation runs for three consecutive days (total of 30 simulations per participant). Outcome parameters were sum-score and time to complete performance. After analyzing individual curves, it appears that 4 categories of performers can be identified, reflecting simulator task proficiency. Group 1 includes performers with a high level of innate abilities, gaining little extra improvement through Virtual Reality training (16.7%).

clinical practice, an attempt was made to compare Xitact test outcomwstandardized psychometric test battery. These psychometric tests

validated, and could be a promising asset in the selection of aspirant s

164

Summary and Conclusions Group 2 has a moderate level of innate abilities, gaining improvement and stability

rough Virtual Reality training (30%), group 3 has a moderate level of innate abilities,

r guidance and selection of surgical trainees.

the companies featured,

thgaining unstable improvement through Virtual Reality training (33.3%) and group 4 has a low level of innate abilities, not gaining improvement through Virtual Reality training (20%). Our study shows is not opportune to assess or validate a VR system using learning curves derived from only one participant. In fact, it is possible to identify different profiles of performance using VR procedural simulation. Most performers are indeed responsive to training. VR procedural simulation may be used fo Chapter 8 assesses the issue of the predictive validity of the Xitact LS500. A four-day training course, -´The Eindhoven VR laparoscopic cholecystectomy training course´- was constructed, integrating a multi-media and modality teaching approach. A variety of increasingly difficult laparoscopic VR simulation training sessions are incorporated in this course, using MIST-VR and Xitact LapChol software on the Xitact laparoscopy platform. The clip-and-cut performance of twelve participants to the course in the operating room was videotaped. This group constitutes the experimental group of the predictive validity study. A matched case-control group of twelve laparoscopic cholecystectomy novices, not participating in the Eindhoven course, also videotaped their operating room performance. Two observers analyzed the randomly mixed videotape, containing the 24 video-fragments of the clip-and-cut procedure of the laparoscopic cholecystectomy. Structured questionnaires including multiple observation scales were used to assess performance. Both observers judge the experimental, ´Eindhoven´ group to perform significantly better on the laparoscopic cholecystectomy ´clip-and-cut´ procedure in the operating room. The course was granted certification of the European Association for Endoscopic Surgery. Finally, Chapter 9 discusses VR simulation systems, hardware- and software developers. It provides an overview of simulators available, their assets, modules and capacities. As developments are going fast and information is bound to be outdated at

oment information is printed, the end of the article listsmincluding their contact details. This, in order to facilitate information updates for the interested reader. Virtual Reality simulation in surgery is, indeed, a viable concept. In fact, it is a development no longer in its infancy. Provided a VR system is properly validated, it is a novelty compelling structural embedding in the curriculum of the medical specialist-in-training. Like no other educational method or instrument, VR simulation provides an environment for safe, repetitive training of residents, with excellent opportunities for objective assessment of the performed procedure. Virtual Reality simulation modules are available in a variety of formats, designed to train a variety of clinical procedures. Not only (laparoscopic) surgical residents can profit, but also future (vascular) surgeons, cardiologists, pulmonologists, urologists, radiologists, anaesthesiologists, orthopaedic surgeons, opthalmologists, ear, nose and throat specialists, gynaecologists and gastro-enterologists.

165

Chapter 10 Furthermore, other medical professions such as theatre nurses, anaesthesiology assistants and nurse practitioners engaged into invasive clinical medical practice may benefit from VR simulator technology. It is important to be well-informed about

ngoing developments in VR simulation. Methods for proper and cost-effective oimplementation of VR simulators in various medical trainings curricula should be discussed by medical educators. Skills-laboratory are likely to provide an excellent environment and infrastructure for both VR supported skills training and VR simulator maintenance. Such an environment should be accessible for each –surgical- resident in training, and training should preferably be embedded in validated training courses. Following the path preceded by aviation and in pilot training, future surgical trainees will benefit from VR surgical training, and VR simulation is likely to play a role in selection and certification of surgeons.

166

Samenvatting en Conclusies Samenvatting en Conclusies In het hoofdstuk Introductie en overzicht van dit proefschrift wordt de lezer geattendeerd op de noodzaak tot het ontwikkelen van specifieke laparoscopische trainingsprogramma´s voor de huidige opleidingsassistent chirurgie, tegen de achtergrond van de toenemende populariteit van laparoscopische chirurgie in de samenleving. Als innovatie in het arsenaal laparoscopische trainingsmethodieken wordt de Virtuele Realistische (VR) chirurgische simulatie aangemerkt. De doelstelling van dit proefschrift wordt geformuleerd: Het uiteenzetten van het proces van een stapsgewijze, gestructureerde validatie en implementatie in het chirurgische curriculum van de Xitact LS500 VR laparoscopische cholecystectomie-simulator. In hoofdstuk 1 worden de resultaten van een beschrijvend onderzoek, gehouden onder chirurgische opleidings-assistenten in Nederland gepresenteerd. In dit onderzoek wordt naar de mening van assistenten betreffende de ´stand-van-zaken´ voor wat betreft trainingsaspecten in de Minimaal Invasieve Chirurgie (MIC) gevraagd. De studie heeft een responspercentage van 65%, (407 assistenten), en laat zien dat alle assistenten (100%!) het belangrijk vinden dat ze aan het einde van hun chirurgische opleiding de basale laparoscopische procedures kunnen verrichten (de diagnostische laparoscopie, de laparoscopische cholecystectomie en de laparoscopische appendectomie). Eén op de tien assistenten geeft aan dat hij of zij denkt hiertoe niet in

aat zal zijn. Assistenten chirurgie met een bijzondere interesse in de gastro-intestinale

oscopische aardigheden kunnen, mits goed opgezet en geëvalueerd, wellicht een uitkomst bieden.

stof in de oncologische chirurgie zijn vaker geïnteresseerd in het zelfstandig uitvoeren van de meer geavanceerde laparoscopische verrichtingen. Deze groep assistenten verwacht niet dat ze de geambieerde geavanceerde laparoscopische verrichtingen aan het einde van hun opleiding ook daadwerkelijk zelfstandig uit kan voeren. Deze discrepantie wordt door de assistent hoofdzakelijk toegeschreven aan het feit dat men ´niet genoeg kans ziet om als eerste operateur de verrichting uit te voeren´ en ´dat er niet genoeg van deze geavanceerde ingrepen plaatsvinden in het ziekenhuis van opleiding´. Hoewel assistenten de noodzaak tot gevalideerde basale laparoscopische trainingsprogamma´s sterk onderschrijven, geeft slechts een kwart van hen aan dat het door henzelf gevolgde trainingsprogramma ook daadwerkelijk geëvalueerd werd. Geconcludeerd wordt dat slechts 17.8% van de assistenten chirurgie in Nederland verwacht in staat te zijn zelfstandig de door hem of haar geambieerde geavanceerde laparoscopische procedure(s) uit te voeren aan het einde van de chirurgische opleiding. Specifieke laparoscopische trainingsprogramma´s gericht op de chirurgische assistent geïnteresseerd in het zelfstandig uitvoeren van bepaalde geavanceerde laparv Hoofdstuk 2 presenteert een onderzoek naar de waarde van de Advanced Dundee Psychomotor Tester (ADEPT) als objectief scoringssysteem voor psychomotore taakverrichting. Het verkrijgen van inzicht in het objectief beoordelen van endoscopische taakuitvoering is een belangrijk gegeven, wil men chirurgische trainingsprogramma´s adequaat kunnen ontwerpen.

167

Chapter 10 Het ADEPT-systeem werd ingezet om de volgende onderzoeksvragen te beoordelen: ´Zijn chirurgen goede schatters van hun taakuitvoering op ADEPT´?, ´Vinden chirurgen ADEPT een valide instrument om psychomotore vaardigheden te meten´? en ´Geven de resultaten op ADEPT het aangeboren psychomotore talent van de uitvoerende weer´?. Vijfenveertig chirurgen namen deel aan dit onderzoek. Iedere chirurg completeerde 2 rondes van vijf taken, in een willekeurige volgorde. Een objectieve som-score werd berekend. Deze score werd vergeleken met de individuele inschatting van het resultaat van zijn of haar taakuitvoering door de betreffende chirurg. Chirurgen blijken niet in staat zijn hun testuitslag op de ADEPT adequaat te oorspellen. Een tweede conclusie is dat het ADEPT systeem daadwerkelijk in staat is

chirurgen =87, meer dan 100 klinische laparoscopische cholecystectomieën verricht) wordt

blijkt dat er geen verschil in opinie is tussen de éxpert´ en eferente´ groep. Dit geeft aan dat zowel de groep die beslist over de organisatie en

us cysticus en arteria cystica goed eer?`.

vaangeboren talent voor psychomotore verrichtingen in te schatten, en dat door het repeteren van de taken de uitvoering verder verbetert. Chirurgen zijn ambivalent in het beoordelen van de validiteit van het ADEPT systeem, en dit staat los van de individuele score van de chirurg. Dit kan wellicht worden verklaard uit het gegeven dat het ADEPT systeem een psychomotore tester is, welke psychomotore kwaliteiten onderzoekt, maar andere belangrijke factoren voor het adequaat verrichten van laparoscopische chirurgie niet meeweegt. Hoofdstuk 3 reflecteert de bevindingen van een onderzoek naar de face-, expert- en referente validiteit van de Xitact LS500 laparoscopische cholecystectomie VR simulator. De simulator is afgebeeld en wordt beschreven. De resultaten van een onderzoek waarin de mening van chirurgen over de simulator onderzocht werd, worden in dit hoofdstuk weergegeven. Een gestructureerde vragenlijst met diverse stellingen werd voorgelegd aan 120 chirurgen. De resultaten van deze studie zijn gecategoriseerd naar gelang de klinische laparoscopische ervaring van de betreffende chirurg. Een opinie berustend op de meningen van een éxpert´-groep van(Ngeformuleerd en vergeleken met de opinie berustend op de meningen van een ´referente´ groep van chirurgen (N=44, minder dan 100 klinische laparoscopische cholecystectomieën verricht). De algemene opinie betreffende de Xitact LS500 is positief, en de simulator wordt gezien als een bruikbaar instrument voor het aanleren (93.1% van de respondenten) en meten (79.3%) van laparoscopische taakuitvoering. Uit dit onderzoek´ruitvoering van trainingsprogramma´s voor chirurgische assistenten (de éxpert-groep´) als de doelgroep waarop de innovatie als trainingsinstrument gericht is (´referente-groep´) de simulator accepteren. Uit dit onderzoek blijkt dat zowel de expert- als de minder geoefende chirurg gelooft in de toepassing van de simulator als trainingsinstrument voor de procedure van de laparoscopische cholecystectomie. Het onderzoek dat wordt beschreven in hoofdstuk 4 werd verricht om het concept van ćonstruct´- validiteit van de Xitact LS500 laparoscopische cholecystectomie VR simulator te meten. De primaire onderzoeksvraag was: ´ Geeft de clip-en-knip taak op de Xitact LS500 laparoscopische cholecystectomie VR simulator de klinische procedure van het clippen en knippen van de ductw

168

Samenvatting en Conclusies Ćonstruct´ validiteit wordt beoordeeld door het vergelijken van de resultaten van een groep van excellente klinische uitvoerders met de resultaten van een groep van beginnende klinische uitvoerders op dezelfde, gesimuleerde klinische procedure. Logischerwijs zouden excellente klinische uitvoerders hoger moeten scoren dan beginners. Een valide systeem zou dus tussen beide groepen moeten kunnen discrimineren. Drie hypothesen werden geformuleerd om de primaire onderzoeksvraag te operationaliseren. Een groep van expert chirurgen (N=37, > 100 laparoscopische cholecystectomieën) en een groep van beginnende chirurgen (N=37, 0 laparoscopische cholecystectomieën) werden gevormd. Nadat de deelnemende chirurgen een kennismakings protocol op de simulator hadden afgelegd, werd een drietal simulaties doorlopen. Vervolgens werden de resultaten van beide groepen chirurgen vergeleken. Uit dit onderzoek blijkt dat zowel de experts als de beginners de ćlip-en-knip´-simulatie van grote waarde achtten in het trainen van chirurgische assistenten. De meerderheid van de deelnemers in beide onderzoeksgroepen was niet in staat de score van hun simulatie adequaat te voorspellen. De eerste hypothese die geformuleerd werd, stelt dat scores gegenereerd na het uitvoeren van de ´knip-en-clip´-taak op de Xitact LS500 simulator in de expert-group gemiddeld significant hoger zullen zijn dan in de scores in beginnersgroep. Dit is inderdaad het geval. Dit wordt toegeschreven aan het feit dat experts hun hogere score te danken hebben aan de klinische ervaring met de procedure, wat nog ondersteund wordt door het gegeven dat zij gemiddeld minder tijd nodig hebben voor het uitvoeren van de simulatie. Simulatie-scores zijn daarom afhankelijk van de mate van klinische ervaring met de gesimuleerde procedure, zoals gesteld wordt in hypothese 2. Hypothese 3 stelt dat gemiddelde scores van beide groepen logischerwijze hoger worden naarmate de simulatie vaker verricht wordt. Dit werd eveneens gevonden. De construct validiteit van de Xitact LS500 simulator wordt onderbouwd door deze bevindingen. Hoofdstuk 5 behandelt een onderzoek waarin het thema van selectie van de laparoscopische chirurgische assistent aan bod komt. Tevens wordt de ćoncurrente´-validiteit van de Xitact LS500 laparoscopische cholecystectomie VR simulator onderzocht. Deze ćoncurrente´ validiteit werd onderzocht door individuele Xitact resultaatsscores te vergelijken met de resultaten van een testopstelling, welke gericht is op het in kaart brengen iemand´s psychomotore aanleg. Deze al eerder gevalideerde psychomotore test-opstelling zou een betekenisvolle aanwinst kunnen zijn bij de selectie van aankomende chirurgen. Drie-en dertig beginners in laparoscopische chirurgie namen deel aan dit onderzoek. Elke deelnemer verrichtte de Xitact ćlip-en-knip´-procedure 30 keer. Aan de hand van de individuele resultaten werd een leercurve berekend. De psychomotore testopstelling bestond uit de Abstract Reasoning test, de Space Relation test, de Gibson Spiral Maze en de Crawford Small Parts dexterity test. De testresultaten gegenereerd door de Abstract Reasoning test zijn als enige direct gecorreleerd aan de individuele Xitact score. Wel is de Abstract Reasoning test hoog gecorreleerd aan de Space Relation test. Beide testen zijn in staat om te discrimineren tussen de groep van hoog getalenteerde assistenten en de groep van minder getalenteerde assistenten (In hoofdstuk 7 wordt dit gegeven verder belicht).

169

Chapter 10 Hoewel het onderwerp ¨selectie van assistenten¨ verdere discussie behoeft, correleren

e gecombineerde testresultaten van de Space Relation test en Abstract Reasoning test

adat zij deelgenomen hebben aan de Intercollegiate Basic Surgical Skills Course

din deze onderzoekssetting met de individuele test resultaten op de Xitact ćlip-en-knip´- simulatie. Lage scores op een combinatie van deze twee psychometrische testen kunnen een aanwijzing zijn om een assistent ter evaluatie te testen op de Xitact LS500. Kandidaten die zowel de Space Relation- als de Abstract Reasoning test goed afleggen, zullen waarschijnlijk niet slecht presteren op de Xitact. De concurrente validiteit van de Xitact LS500 laparoscopische cholecystectomie VR simulator met de Space Relation test en de Abstract Reasoning test wordt onderschreven door dit onderzoek. Hoofdstuk 6 beschrijft de vorderingen van assistenten in laparoscopische chirurgie, n(BSSC) zoals gemeten door de Xitact LS500 laparoscopische cholecystectomie VR simulator. Gedurende deze cursus maken assistenten tijdens een dag kennis met verschillende aspecten van de laparoscopische chirurgie. Laparoscopische vaardigheden worden geoefend in een pelvi-trainer oefenopstelling gedurende 3½ uur. In dit onderzoek verrichtten assistenten vóór en ná de cursus een drietal simulatierondes van de ćlip-en-knip´ procedure op de simulator. Een controlegroep van 25 co-assistenten, die niet deelnamen aan de BSSC, verrichte ook een tweetal sessies van drie procedures op de simulator. Al hoewel er een significante afname was in tijd nodig voor het verrichten van de simulatie naarmate meer simulaties verricht werden in beide groepen, vond deze afname plaats binnen en niet tussen de onderzoeksgroepen (behalve in run 2). Voor wat betreft de parameter ´score´ is er geen verschil tussen de twee onderzoeksgroepen. Gesteld kan worden dat de BSSC ontworpen is om assistenten kennis te laten nemen van de basisprincipes in de laparoscopische chirurgie, en niet zozeer om assistenten te trainen in de meer procedurele laparoscopische verrichtingen. Daarom is het niet meer dan waarschijnlijk dat er ook geen significante verbetering optreedt in laparoscopische procedurele taakverrichting, zoals gemeten door de Xitact ćlip-en-knip´ simulatie in dit onderzoek. Hoofdstuk 7 richt zich op het beschrijven van de leercurve van de ćlip-en-knip´-simulatie, zoals uitgevoerd kan worden op de Xitact LS500 laparoscopische cholecystectomie VR simulator. Drie-en dertig assistenten, zonder laparoscopische klinische ervaring volgden een één-uurs kennismakingsprotocol, en verrichten vervolgens een serie van 10 simulaties, drie dagen achtereenvolgend (totaal 30 simulaties per assistent). Uitkomstparameters van dit onderzoek waren ´sum-scoreén ´tijd´. Nadat iedere curve afzonderlijk geanalyseerd was, bleek het mogelijk aan de

and hiervan vier elkaar uitsluitende patronen te identificeren, die de taakverrichting en hhet leerproces op de Xitact simulator categoriseren. Categorie 1 betreft de groep van hoog getalenteerde assistenten, die weinig meer verbetert door herhaalde simulatietraining (16.7%). Groep 2 betreft de groep van gemiddeld getalenteerde assistenten, die beter en stabieler in uitvoering wordt door repetitieve simulatietraining (30%).

170

Samenvatting en Conclusies Groep 3 betreft de groep van gemiddeld getalenteerde assistenten, die wel beter wordt door simulatietraining, maar wel moeite heeft om een stabiele prestatie te verkrijgen gedurende de repetities (33.3%) en groep 4 betreft assistenten die laag getalenteerd zijn en ook niet veel verbeteren gedurende repetitieve simulatietraining (20%). Deze studie laat zien dat het niet correct is om leercurves te construeren en extrapoleren, gebaseerd op de verrichtingen van slechts één persoon. Het blijkt mogelijk om profielen van verschillend getalenteerde assistenten te abstraheren met behulp van VR simulatie. De meeste assistenten blijken gevoelig voor verbetering in laparoscopische taakuitvoering met behulp van herhaalde VR simulatie training. Hoofdstuk 8 richt zich op het vraagstuk van predictieve validiteit van de Xitact LS500 laparoscopische cholecystectomie VR simulator. Een vierdaagse cursus, de Éindhoven VR laparoscopic training course´ werd ontwikkeld, waarin een verscheidenheid aan multi-media en –modaliteiten onderwijsinstrumenten gebruikt worden. Een variëteit aan verschillende, en in toenemende mate moeilijkere VR simulaties (met behulp van de programma´s MIST-VR en Xitact LapChol) werden ingezet om deelnemende assistenten te trainen in laparoscopische vaardigheden gericht op de

rocedure van de laparoscopische cholecystectomie. De ´knip-en-clip´ procedure van

t en gebed te worden in het opleidingscurriculum van de toekomstig geneeskundig

ptwaalf deelnemende cursisten, zoals uiteindelijk verricht op de operatiekamer, werd op videoband opgenomen. Deze groep vormde de experimentele groep. Een gematchde case-controle groep, bestaande uit twaalf assistenten die niet deelnamen aan de Eindhovense cursus, verrichte hetzelfde onderdeel van de laparoscopische cholecystectomie. Ook deze assistenten namen hun verrichting op videoband op. Twee observatoren beoordeelden aan de hand van een gestructureerde vragenlijst een videoband met de 24 willekeurig gemixte ćlip-en-knip´ fragmenten. Beide beoordelaars vonden de experimentele, Éindhovense´ groep beter presteren in de operatiekamer dan de assistenten uit de controlegroep. De cursus kreeg officiële erkenning en certificering van de European Association for Endoscopic Surgery. Het afsluitende hoofdstuk van dit proefschrift, Hoofdstuk 9, vergelijkt verschillende VR simulatie systemen en VR hardware en -sofware ontwikkelaars. Gezien de snelle ontwikkelingen is informatie die in druk verschijnt vaak niet meer actueel. Daarom staan aan het einde van het artikel de verschillende fabrikanten vermeld met hun contactgegevens. Op deze wijze kan de geïnteresseerde lezer eenvoudiger de laatste informatie rechtstreeks van de betrokken fabrikant verkrijgen. Virtual Reality-simulatie in de chirurgie is inmiddels een levensvatbaar concept. Feitelijk is het een ontwikkeling die de eerste ´kinderziektes´ voorbij is. Mits gevalideerd, zijn het systemen die erom vragen op een structurele wijze ingezeinspecialist. VR simulatoren bieden, als geen andere onderwijsmethode, de mogelijkheid tot het veilig, herhaald en op gestructureerde wijze trainen van assistenten in een klinische procedure met de mogelijkheid tot een objectieve beoordeling van de uitvoering ervan. Virtual Reality simulatiemodulen zijn er heden-ten-dage in velerlei uitvoeringen, gericht op het trainen van een veelheid aan klinische procedures.

171

Chapter 10 Simulatiemodules zijn niet alleen beschikbaar voor toekomstige (laparoscopische) chirurgen, tevens zijn er modules voor het trainen van toekomstige (vaat)chirurgen, cardiologen, pulmonologen, urologen, radiologen, anaesthesiologen, orthopaeden, oogartsen, keel- neus- en oorartsen, gynaecologen en gastro-enterologen. Ook beroepsgroepen als operatiekamer assistenten, anaesthesie verpleegkundigen en nurse-practitioners, betrokken bij invasieve klinische procedures, kunnen profiteren van deze technologie. Het is belangrijk om overzicht te houden op de ontwikkelingen en te discussiëren over de wijze waarop deze simulatoren op een kosteneffectieve wijze kunnen worden geïmplementeerd in het medisch (specialistisch) onderwijs. Het skills-laboratorium, waarin simulatoren naast elkaar ingezet en onderhouden kunnen worden, elkaar aanvullen en ingepast kunnen worden naast andere vormen van vaardigheidsonderwijs lijkt hiervoor de ideale omgeving. Een dergelijke opstelling zou in principe toegankelijk moeten zijn voor iedere (chirurgische) assistent in opleiding. Bij voorkeur zou toegang verleend moeten worden via vormgegeven, gevalideerd cursorisch onderwijs. In navolging van de ontwikkelingen in de luchtvaartindustrie en de training van piloten, zullen Virtual Reality-simulatoren in de nabije toekomst onmisbaar worden in de opleiding, en wellicht ook behulpzaam bij de selectie en re-certificering, van chirurgen en andere snijdende heelkundige specialismen.

172

Glossary Glossary Concurrent validity / Divergent validity: the degree of concordance of –paired- population test outcomes between the study results using a concept instrument and the study results on a established concept instrument, that is believed to measure the same underlying theoretical construct. Concurrent validity, when referring to the degree of concordance of a concept instruments´ outcome and operation room performance is also referred to as predictive validity. Construct validity: the degree of empirical foundation of a concept instrument, based on theoretical constructs. In practise: often based on the presence of a logical difference in outcome between two research populations (e.g. experienced surgeons perform better than inexperienced surgeons on a certain procedure). Content validity: the degree to which a concept instrument adequately covers the relevant dimensions of the theoretical construct of study (e.g. an instrument measuring psychomotor skills is actually measuring psychomotor skills and not anatomical nowledge). k

Co

riterion-related validity: the degree of concordance of -paired- population test utcomes between the study results using a concept instrument and the study results

on a established instrument, that is considered the criterion or ´gold standard´ in assessing performance on a certain construct. Calot´s triangle: the triangle formed by the cystic artery (CA) superiorly, the cystic duct (CD) inferiorly, and the hepatic duct medially. Cholecystectomy: the surgical procedure of removal of the gallbladder. Endoscope: a hollow, tube-like instrument attached to a light-cable and mounting a camera, containing a distal lens. Used in performing endoscopy. Endoscopy: the procedure of visualisation of a (human or animal) natural cavity (e.g. bronchoscopy, cystoscopy, gastroscopy, laparoscopy). Expert validity: the degree of resemblance between a concept instrument and the actual construct, as judged by experts (e.g. the ones who built and/or established the construct). Face validity: the degree of resemblance between a concept instrument and the actual construct, as judged by a specific (target) population. Force feedback: the real-time, mechanically generated sensation in haptics as a reaction to user interface manipulation in Virtual Reality endoscopic simulation.

175

Glossary Fulcrum effect: the phenomenon of the effector-end of the laparoscopic instrument moving in the opposite direction of the surgeon’s hand movement, due to the fixation of the instrument in the abdominal wall acting as a rotation axis. Haptics: the quality of touch, defined by the quality and variety of sensations inherent when contacting an object (surface texture, temperature, weight, deformity, volume, contour and shape). Innate abilities: natural, ´in-born´, enduring qualities, aptitudes or capabilities (´talents´) that a person brings to a given task. Internal validity: an accurate representation and extrapolation of (study) results from a research population, acting as a true representative group for a certain target population under study. Laparoscopy: a minimally invasive (e.g. minimal access) surgical procedure within the abdominal cavity by means of using an endoscope. Popularly known as ´key-hole´ surgery. Predicitive validity: the extent to which test outcomes on a concept instrument are related to outcome scores for the specific construct in real life (e.g. those who do well according to the concept instruments´ assessment perform well in the operating room). Procedure: a series of well-defined steps, taken to accomplish a certain task. Proficiency: the quality of state of being proficient, e.g. being well advanced or well skilled in a certain procedure. Referent validity: the degree of resemblance between a concept instrument and the actual clinical concept, as judged by referents (e.g. the ones the concept instrument is built for or aimed at). Skill: a proficiency or dexterity in the execution of a given art or craft, developed and modified by training and practice. Task: an assignment or piece of work to be done.

176

Curriculum Vitae Curriculum Vitae

he author of this thesis, Marie-Elisabeth Paula (Marlies) STE

chijven was born in

started her studies in Health Sciences at the University of

e same year, Marlies started to study

ee.

nal head: Dr. J. Jakimowicz). In this hospital, she also started her formal training

nal head: Dr. I. Dawson).

n January

indhoven on the 17th of December in 1969. In 1988 she finished her secondary school education at the Hertog Jan College in Valkenswaard, and in the same year she started at the Academy of Industrial Design in Eindhoven. In 1989 she changed her tudy direction and s

Maastricht. As of 1992, she worked as a student-assistant in the department of Educational Development and Research. She matriculated successfully in 1993, and

as granted the predicate ´cum laude´. In thwmedicine and worked as a research assistant in the Skills Laboratory of the Faculty of Medicine. In August ´96 she matriculated ´cum laude´ at the faculty of Medicine of the same University. Hereafter she started her internships, and in April 1998 Marlies btained her medical degro

In May 1998, Marlies started as a resident surgery –not in training- at the Department of Surgery in the Catharina Hospital Eindhoven, actually the hospital she was born in educatio(

as a resident in general surgery, as of January 2000. In 2004, she moved to Rotterdam to complete her training at the Department of Surgery of the Erasmus Academic Medical Centre (educational head: Prof dr. H. J. Bonjer / Prof. dr. J.N.M. IJzermans) and further on at the Department of Surgery of the IJsselland Hospital in Capelle aan de IJssel educatio(

After receiving her doctorate, Marlies intends to become a qualified surgeon i2006. Address for correspondence: Marlies Schijven Gedempte Zalmhaven 107 3011 BT Rotterdam Email: [email protected]

179

Virtual Reality Simulation forLaparoscopic Cholecystectomy

Marlies P. Schijven

Virtual Reality Sim

ulation for Laparoscopic Cholecystectom

y Marlies P. Schijven

ISBN: 90-9019048-1

the process of validation and implementation in thesurgical curriculum outlined

Virtual Reality Simulation for Laparoscopic Cholecystectomy

Documents

Transcript of Virtual Reality Simulation for Laparoscopic Cholecystectomy