TobiasHuber,TomWunderling,MarkusPaschold,

HaukeLang,WernerKneist,ChristianHansen

Highly-ImmersiveVirtualRealityLaparoscopySimulation:DevelopmentandFutureAspects Pre-print version TobiasHuber,MarkusPaschold,HaukeLang,WernerKneist

DepartmentofGeneral,VisceralandTransplantSurgery,

UniversityMedicineoftheJohannesGutenberg-UniversityMainz,Germany

tobias.huber@unimedizin-mainz.de

TomWunderling,ChristianHansen

FacultyofComputerScience,

Otto-von-GuerickeUniversityMagdeburg,Germany

christian.hansen@ovgu.de

Thisisapre-printofanarticlepublishedintheInternationalJournalofComputerAssisted

RadiologyandSurgery.Thefinalauthenticatedversionisavailableonlineat:

https://doi.org/10.1007/s11548-017-1686-2

2

Abstract

PurposeVirtualReality(VR)applicationswithhead-mounteddisplays(HMDs)havehad

animpactoninformationandmultimediatechnologies.Thecurrentworkaimedto

describetheprocessofdevelopingahighly-immersiveVRsimulationforlaparoscopic

surgery.

MethodsWecombinedaVRlaparoscopysimulator(LapSim)andaVRHMDtocreatea

user-friendlyVRsimulationscenario.Continuousclinicalfeedbackwasanessentialaspect

ofthedevelopmentprocess.WecreatedanartificialVR(AVR)scenariobyintegratingthe

simulatorvideooutputwithVRgamecomponentsoffiguresandequipmentinan

operatingroom.Wealsocreatedahighly-immersiveVRsurrounding(IVR)byintegrating

thesimulatorvideooutputwitha360°videoofastandardlaparoscopyscenariointhe

department'soperatingroom.

ResultsClinicalfeedbackledtooptimizationofthevisualization,synchronization,and

resolutionofthevirtualoperatingrooms(inboththeIVRandtheAVR).Preliminarytesting

resultsrevealedthatindividualsexperiencedahighdegreeeofexhilarationandpresence,

withrareeventsofmotionsickness.Thetechnicalperformanceshowednosignificant

differencecomparedtothatachievedwiththestandardLapSim.

ConclusionOurresultsprovidedaproof-of-conceptforthetechnicalfeasibilityofan

customhighlyimmersiveVR-HMDsetup.Futuretechnicalresearchisneededtoimprove

thevisualization,immersion,andcapabilityofinteractingwithinthevirtualscenario.

Keywords:SurgicalTraining,VirtualReality,LaparoscopicSurgery,Human-Computer-

Interaction,Visualization

3

1Introduction

VirtualReality (VR) iscurrentlyused insurgical training to improvepsychomotor

skills that are important for laparoscopic surgery, suchashand-eye coordination, spatial

orientation,andmanipulationsinthepresenceofafulcrumeffect[1].Nonetheless,theuse

ofVRsimulatorsislimitedindailyclinicalroutine,duetoalackingrealismoftasks,abstract

graphicdesign,andtheawarenessofparticipantstobeinatrainingenvironment[2,3].The

varietyoftaskshasbeenenlargedsincethedevelopmentofVRlaparoscopysimulations,and

currently,abstracttrainingtasksandproceduraloperationsareavailableinVR.Inaddition,

graphic design of simulation tasks and virtual tissue interaction in VR laparoscopy

simulationhaveimproved.However,usersarecontinuouslyawarethattheyareinatraining

environmentandnotarealsurgicalsituation.Realisticsurroundingsthatincreasetheuser’s

senseofpresenceduringaVRsimulationareonlypossiblewhenperformingteamtraining

sessions, which require more time, infrastructure, and human resources [4]. Recent

advancementsininformationandmultimediatechnologyhavemadeitpossibletodevelop

VR applications in combination with head mounted displays (HMDs). These combined

applicationsarecurrentlyusedintheentertainmentindustry,inthemilitary,andinaviation

training. Medical applications include VR-HMD psychological interventions for

posttraumaticstressdisorders,phobias,cognitiverehabilitationandandpaintreatmentfor

burn victims [5-8].Additionally, VRwill enhance the validity of clinical, behavioural and

affective and social neurosciences due to more realistic test scenarios [9]. The rise of

commerciallyavailableVR-HMDsoverthelastyearhasledtothedevelopmentofavirtual

operating room environment to increase the attractiveness and the degree of presence

duringVRlaparoscopysimulations.Thepresentworkaimedtodescribethedevelopment

processofanimmersiveVRlaparoscopysimulationsetup.Thegoalwastocombineexisting

technologies to create a user-friendly simulation scenario with high immersion and

presence. Continuous clinical evaluations and feedback from laparoscopic surgeons

comprisedanessentialpartofthedevelopmentprocess.

4

2RelatedWork

ThecurrentuseofVRinmedicalfieldsismainlyduetothevisualpossibilitiesitoffers,

whichareveryhelpfulforprocessessuchaseducation,simulation,planning,navigation,and

even rehabilitation [10]. VR simulations in surgery are used to teach technical skills,

behavioralskills,andentireprocedurestotraineesandpracticingsurgeonsworldwide[11].

In VR laparoscopy trainers, users perform surgical tasks with standard laparoscopy

instruments [12]. Studies on VR laparoscopy simulations have concluded that the skill

acquisition is equivalent to that acquiredwith laparoscopy box trainer simulations, and

theseskillscanbetransferredtotheoperatingroom(OR)[13,14].Also,abriefpre-surgical

VRwarm-upcanimproveperformanceintheOR[15].

Technical and visual improvements have influenced VR simulators. Three-

dimensionaldisplayswithpolarizationor shutter technologieshavebeen integratedand

investigated with differing results [16,17]. On the other hand, recent developments in

surgery, such as robotic operations, have led to the development of VR robotic surgery

simulators[18].VisualizationforaVRroboticsurgerysimulationcanbeseenasavariantof

anHMDsimulation,performedontheroboticconsole;however,theORsurroundingshave

not yet been considered in this context. A recent study described a VR application that

combinedaVRlaparoscopysimulationandalow-immersionHMD,withonlya45°fieldof

view, to produce a virtual scenario, which consisted of a peg transfer task in a plain,

computer-generatedroom[19].

The latest generation HMDs, which began shipping in 2016, feature several

technological advancements realized in recent years. High pixel density displays from

smartphoneswereusedinearlyprototypestoreducethe‘screen-door’effect.Thesedisplays

operatewith a high refresh rate to reduce latency,which causesmotion sickness.Novel

methods and custom sensors were developed to improve positional tracking [20].

Asynchronousre-projectionwasintroducedtoreducelatencyevenfurther.Thistechnique

introducessmallchangesontoapreviouslyrenderedframe,accordingtothemostrecent

positional tracking data, which lowers the computational requirements of the graphics

processingunit(GPU)[21].Othersoftwaresolutionswererequiredtocorrectfordistortion

andchromaticaberrationsthatarisefromthelenses.Thesesolutionswereneededtomap

theHMDtoawiderfieldofviewandcreateamorecomfortablepointoffocus.

5

Fig. 1: Custom IVR setupwith LapSimSimulatorwith the4D joysticks (Simball, G-coder

Systems,Sweden);thegogglesaretheHTCVive(HMD);andtheheadphonesprovidesound.

Aconventionalmonitorisnotneededinthissetup.

3MaterialsandMethods

VirtualRealityLaparoscopySimulationSystem

The basis of our custom setup was a VR laparoscopy simulator without haptic

feedback(LapSim),purchasedfromSurgicalScienceAB,Goethenburg,Sweden.Itconsisted

ofa27-inchLCDmonitor(AOCInternational,Taiwan),akeyboardandmouse,aWindows7

PC,andSimball™4Djoysticks,withadoublefoot-switch(G-coderSystems).Allhardware

components were mounted on a rolling, height-adjustable array, and they were readily

accessible,duetotheopendesignofthechassis.ForinteractionswiththeVRenvironment,

the simulator provided Simball 4D joysticks (Fig. 1 andFig. 2). Their laser-marked ball

joints,withthreedegreesoffreedom,allowedreal-timecalculationsoftheexact3Dangular

position.Theinputdevicesincludedagrasperinstrumentontheleftandrightsides,anda

camera instrument in the center. During our tests, the camera was not used in any

6

laparoscopic task.Thecomputer featuredproprietarysoftware fromSurgicalScienceAB,

version2015,whichranonWindows7.Thissoftwareallowedtheuser toperformbasic

trainingtasks,likepegtransferorpatterncutting,butalsocomplex,morerealisticscenarios,

like a cholecystectomy or appendectomy simulation. The software logged the user's

performancebyrecordingasetoftask-specificparameters.Thisallowedtheassessmentof

execution quality for each performed task; thus, the individual's improvement in

psychomotorskillscouldbemonitoredovertime.

CustomVirtualRealityHeadMountedDisplaySystem

TwoVR-HMDsolutions fromtheconsumermarketwereconsidered inouraimto

extendtheexistingLapSimexperiencewithadditionalhardwareandcontent:TheOculus

RiftCV1andtheHTCVive.Comparedtotheconventionalbuilt-inmonitor,HMDsprovidea

widerfieldofview.Withthecombinedheadtrackingandstereoscopicdeptheffects,theuser

isimmersedinanall-aroundvisualexperiencethatismuchclosertorealhumanvisionthan

the2DdisplayoftheLapSim.ThehighrefreshrateandlowlatencyoftheOLEDdisplayson

theHMDwereimportantinachievingminimumsimulatorsickness.WechosetheHTCVive

HMDforthisstudy,becausecomparedtotheOculusRiftCV1, it featuredaslightlylarger

fieldofviewof110°comparedto101°fortheOculusRiftCV1andalargetrackingareaof

4.6 by 4.6 meters, which allowed highly-immersive, room-scale VR [22]. A separate

computerwasnecessarytodrivetheHTCViveHMDwithoutinterferingwiththeexisting

LapSimsoftwareandhardware(Fig.2c).AVR-readylaptop(MSIGT72VR-6RE16H51)was

chosenoveracustom-builtdesktopPC forbettermobility.TheMSI laptopwasequipped

withanIntelCorei7processor(6700HQ,16GBRAM),aNvidiaGTX1070graphicscard(8GB

VRAM),andtheportsnecessaryforconnectingtheHTCVivelinkbox.

VideoandAudioSignalTransfer

Tointegratethevideooutputpreviouslydisplayedonthe2DLapSimmonitor(Fig.

2a) intoavirtualenvironmentcreatedwith theseparateHTCViveHMDsystem,a frame

grabberwasused(Fig.2d).ThisUSB3.0HDvideocapturedevice(StartechUSB3HDCAP;

Startech, Northampton, UK) received the HDMI output from the LapSim PC at 1080p

resolutionand60fps,andsent itovertheUSBtothe laptopviatheHTTPlive-streaming

protocol,HLS.AnHDMIsplitter(Fig.2e)wasinsertedupstreamtoallowthevideosignalto

7

beroutedbacktotheLapSimsystemforsimultaneousdisplayonthebuilt-inmonitor.The

simultaneous display allowed other developers to follow the laparoscopy simulation, as

usual,onthe2Dmonitor.Thus,theotherdeveloperscouldprovidetechnicaladministration

andcontroltheLapSimsoftwarewiththekeyboardandmouse,whiletheuserwaswearing

theHMD. Audio feedback generated by the LapSim software (e.g.,whenworkingwith a

virtualelectronicdevice)wastransferredviaananalogcablefromtheLapSimPCheadphone

jacktotheMSIlaptopmicrophonejack(Fig.2f).Thefinalaudiomix-downwasperformed

on the laptop. The user wearing the HMD listened to the audio output with stereo

headphonesconnectedtotheHTCViveHMD.

Fig.2:ComponentdiagramoftheVRlaparoscopysimulationsystemandourcustomizedVR

HMDsystem.a: conventional2Dscreen,b: surgical simulator, c:VR-readyMSI laptop,d:

HDMItoUSBframegrabber,e:HDMIsignalsplitter,f:stereoaudiocable,g:localnetwork

connection(LAN),h:Simballdataloggerauxiliaryprogram.

8

CreationofVirtualRealitySurroundings

The cross-platform game engine, Unity™ (Version 5.4.2), was used to create the

virtualsurroundingsbyintegratingthevirtualmonitorwiththeSimballjoystickmovements.

Thelivevideostreamfromthescreengrabberwasdisplayedonaplane,whichservedasa

virtualmonitor,with theUnity engine, “WebCamTexture class”. The original streamwas

rendered at the standard resolution of 1920´1080 pixels. To optimize performance, the

texturewasdown-sampledto960´540pixelspriortorendering.SoundfromtheLapSim

systemwasintegratedbyplacinganAudioSourceobjectinthescenethatcapturedtheinput

fromthelaptop’smicrophonejack.Theremainingvirtualspacewasfilled,viadraganddrop,

withmodelsofmedicaldevices,furniture,props,andanimatedcharactermodelsofmedical

personnel and the patient. The models could be moved around to create a setting that

mimickedtheappearanceofanORduringlaparoscopicsurgery.

Analternativewaytocreateavirtualsurroundingwastointegratetheplaybackfrom

a360°video.Tocreatethiseffect,thevideoclipcouldbemappedontoaspherewiththe

MovieTexturefeatureinUnity.TheSteamVRcamerarigwasthenplacedinthecenterofthe

sphereandscaleddowntoaverysmallsizetoadjustforthefixedperspectiveofthe360°

videorecording.

ClinicianFeedback

Four members of the surgical department were continuously involved in the

developmentprocess.AllfourmembershadpreviousexperiencewiththeVRlaparoscopy

simulator.

TestingPhase

As part of the first clinical pilot investigations, other members of the surgical

departmentperformedlaparoscopictasksonthesimulator.Hypothetically,adifferencein

performanceduetodistractionsintheimmersivesetupmaybepossible.Furthermore,the

degreeofimmersionintoavirtualworld,theattentiontotheenvironmentandexhilaration

of the participant are important aspects to optimize the surrounding.We thus used the

validated questionnaire by Nichols et al. to quantify these aspects [23]. Furthermore,

negativepsychopathologicalaspectsandvegetativesideeffectshavebeendescribedinVR

andhaveleadtothedescriptionofa“codeofconduct”regardVRresearch[24].Thuswe

9

investgatedparticipants’heart rate (stress level) andmotion sicknesswith thevalidated

motionsicknessscalebyKeshavarzetal.[25].

The presented technical data is a comparison of immersive and regular VR

simulation. This was investigated with the consectuvie performance of three tasks (peg

transfer,finedissection,andcholecystectomy)afteraninitialwarm-upphaseinregularVR

mode.Theselectedtasksrepresentdifferentaspectsoflaparoscopicsurgery,navigational

maneuver,finepreparationandproceduralaspects.Taskswerealwaysperformedinregular

modefirstandAVRsecondandintheabovementionedorder.Simulatormetricshavebeen

analysedusingthetotalz-scoreofeachtask.Additionally,metricshavebeengroupedinto

categoriestime,handlingeconomicsanderrorsaspreviouslydescribed[15].

Fig. 3: Screenshot of the artificial virtual reality (AVR) operating room. The virtual

environmentwasmodeled, based on a 3Dmodel kit from Vertigo Games, Rotterdam. A

virtualinstrumentallowsinteractionwiththeLapSimsimulator.Avirtualmonitor(center)

showsthelaparoscopysimulator'sgraphicoutput.

10

4Results

A general overview of the customVR setupwe developed is shown inFig. 2. All

software and hardware components and connections proved to be generally technically

feasible.Thesetupwasuserfriendly,easytoassemble,andhighlymobile.

ArtificialVirtualOperatingRoom

We firstdevelopedanartificial virtual reality (AVR)OR.OurAVRwasbasedona

commerciallyavailablemodular3Dassetkit,whichcontainedafullyequippedsurgicalOR

(VertigoGames,Rotterdam,Netherlands).Itfeaturedanimated3Dcharactersandseveral

modelsofsurgicaldevices,props,andfurniture.Thefeedbackfromtheclinicaldevelopment

teamledtorepositioningcertaincomponentsinthevirtualenvironment.Thepositionofthe

virtualdisplaywasadaptedtocorrespondtothesettinginourclinic.Thepositionsofthe

trocars, sterile equipment, and surrounding teammembers in the AVRwere changed to

ensurethepositionwascomfortablefortheuserandreflectedtheclinicalsetting(Fig.3).

Additionally,thesizeofthevirtualmonitorwasincreasedtoimprovevisibilityoffinedetails,

but at the same time, it remained realistic. A very small delay between the joystick

movementsandthecorrespondingmovementsintheVRsurgicalmonitorwasachievedon

the virtual display. Even after the AVR was optimized, clinical feedback from the

development team revealed that they remained aware of the fact that they were in an

artificialenvironmenttheentiretime,whichledtoalowdegreeofpresence.Thislimitation

stimulatedtheinitiativetoimprovethesurroundingsandcreateamorerealisticsimulation.

IntegrationofLaparoscopyInstruments

At first, the fourdevelopers remarked that the instruments in frontof themwere

missingduringtheAVRsimulation.TheSimballinputdevicesprovidedahapticsensationto

the user,whichwas similar to the sensation of holding real surgical instruments. These

instrumentsweretheonlymeansbywhichtheusercouldphysicallyinteractwiththevirtual

world.Therefore,itwasimportantthatvirtualrepresentationsoftheseinstrumentscould

beseenwhenwearingtheHMD.ThisvisualizationoftheinstrumentsinVRenabledtheuser

tolocateandgrabthedeviceswhilewearingtheHMD.Toachievethiseffect,theinputdata

fromtheSimballshadtobetranslatedintomovementsthatwerethenappliedtothevirtual

11

objects represented by the shapes of the simulated instruments. Because the virtual

environmentwasgeneratedintheVRsystem,andnottheLapSimsystem,theSimballsinput

datahadtobeinterceptedandsentfromonecomputertotheother.Anauxiliaryprogram,

theSimballDataLogger(Fig.2h),whichraninthebackgroundontheLapSimPC,recorded

thedatawithoutinterferingwiththeLapSimsoftware.Eeroutedthedatastreamtopass

throughalocalareanetwork,whichsentittothelaptopthatrantheVRsystem(Fig.3).The

movementsofthevirtualinstrumentswereveryaccurate,accordingtotheusers.Thedouble

foot-switchwasnotvisualizedwithinthevirtualenvironment.However,thisdidnotseem

tobeaproblemfortheclinicianssincethefootswitchisoftennotvisibleintherealoperating

roomaswellduetothesurgicalcovers.

Highly-ImmersiveVirtualOperatingRoom

AnalternativeVRenvironment,thehighlyimmersiveVR(IVR)OR,wascreatedwith

a360°camera(SamsungGear360,SamsungAG,Seoul,Korea).Sphericalvideosequences

wererecordedinsidearealsurgicalORattheUniversityHospitalMainz,Germany.Forthe

video,asurgicalstaffre-enactedthesituationofareallaparoscopicsurgery.Actualmembers

of the departments acted as the patient, scrub nurses, laparoscopy assistant, and

anesthesiologist (Fig.4).TheORwas first recordedwithout a scripteddialogue.Later, a

second scenariowas createdwith interactions between the fictional characters (e.g., the

scrubnurses).Thesecondscenarioincludedsounds,actions,andconversationsthatwere

typicalduringastandardlaparoscopicprocedure.

Thetestingclinicianswerehighlyexhilaratedinresponsetothepresencetheyfeltin

theOR,particularlyduringthesecondscenario.Thesensationofpresencewasevenmore

exhilaratingwhenperformingproceduralsimulationsettings,suchasacholecystectomy,on

the LapSim. Nonetheless, the display resolution was limited in the AVR and the IVR,

particularlyintaskslikefinedissection,wherebloodvesselsmustbedifferentiated.Despite

manychangesofsettings,thislimitationwasmentionedbyallparticipants.Thatfindingled

totheconclusionthattheresolutionwaslackingwiththeHMD.Down-sizingtheinputsignal

to960´540inUnitydidnotimpactthevisualqualityoftheimagesignificantly,becausethe

virtualmonitoronlycoveredpartoftheuser'sfieldofview.ThenativeresolutionoftheHTC

ViveHMDwasinsufficienttodisplaytheoriginalimageinfullHDresolutionatthatsize.

12

Fig.4:Screenshotofthe360°highlyimmersivevirtualreality(IVR)operatingroom(OR).A

sphericalvideowasgeneratedwithaSamsungGear360°cameraintheORattheUniversity

HospitalMainz,Germany.Thevirtualmonitorinthecentershowsthegraphicoutputfrom

thelaparoscopysimulator(LapSim).

TestingPhase

In a previously published pilot study, we found no significant difference in

performancebetweentheregularVRlaparoscopyandIVR.Theparticipatingsurgicalstaff

washighlyexhilaratedandindicatedahighlevelofpresence[26].Inadifferentapproach,

we previously investigated potential vegetative side effects. Nausea was present in this

previous investigation in 10% of participants (2/20). These two female surgeons had a

historyofmotionsickness.AlthoughheartrateswereelevatedduringIVRsimulations,the

elevationswerenotstatisticallysignificant[27].

ThecurrentanalysisoftechnicalperformancecomparedregularVRlaparoscopyand

AVRof16participants.Weobservednostatisticaldifferencesinthetotalz-scoresorinthe

categorizedz-scoresforhandlingeconomics,errors,andtime.Thesetechnicalresultsare

displayedinTable1.

13

Table1:Calculatedz-scoresfortasksperformedbyparticipants(n=16)

VRLSession AVRSession Taskparameters Median(IQR) Median(IQR) P*

Cholecystectomy Total 2.76(-1.18;5.40) 1.07(-2.18;6.14) 0.918Time 0.39(-0.60;0.74) 0.28(-0.86;0.85) 0.836Economics 2.47(-3.00;4.45) 0.93(-0.74;4.72) 0.836Errors 0.80(-1.48;2.33) 0.97(-1.77;2.03) 0.756 FineDissection Total 0.35(-1.58;3.10) 0.39(-3.53;3.35) 0.918Time 0.12(-0.71;0.57) -0.12(-0.59;0.86) 0.959Economics 1.22(-1.19;2.16) 0.12(-1.43;1.63) 1.000Errors -0.19(-1.57;1.64) -0.18(-1.23;1.49) 0.756 PegTransfer Total 0.59(-1.73;2.06) 1.39(-1.33;2.67) 0.756Time 0.19(-0.62;0.60) 0.28(-0.22;0.72) 0.796Economics 0.23(-0.24;0.79) 0.78(-1.29;1.24) 0.756Errors -0.61(-0.61;1.01) 0.27(-0.60;0.71) 0.679 IQR:interquartilerange;VRL:virtualrealitylaparoscopysimulation;AVR:artificialvirtualrealityoperatingroom*Wilcoxon-Signed-Rank-Test5Discussion

Previously,Bowmanetal.[28]discussedtheimportantaspectsofimmersion,andthe

sensation of presence in VR. These aspects require hardware components that provide

optimalrefreshrates,framerates,displaysizes,anddisplayresolutions.Inthecurrentstudy,

ourVRsetupincludedanHMDwithpotentiallythebesthardwarecomponentscommercially

available.Nevertheless, theresolutionintheHMDwaslimitedforveryfinepreparations,

anditmustbeoptimizedtodeterminethesweetspotwiththebestcombinationofasmooth

frame rate and video playback, an acceptable virtual monitor size, and good image

resolution.Accordingtothe“reality-virtualitycontinuum”[29],thecombinationoftheVR

laparoscopysimulationandavirtualORinthecurrentsetuprepresentedacombinationof

twovirtuallygeneratedworlds.TheVRlaparoscopysimulatorisusuallyperceivedascloser

14

toreality thanthenewlydevelopedsetup,becausethesimulation ispresentedona two-

dimensionalmonitorinanout-of-contextenvironment.Theonlyconnectionlefttorealityis

the haptic user interface (Simball joysticks) which makes the virtual display of the

instruments evenmore important.However, interaction inAVRand IVR is currentlynot

possible and should be improved by further technical advances. An overview of the

limitations in the setupwe developed is given inTable 2,with corresponding technical

solutionsandratingsofclinicalimportanceandtechnicalviability.Handtracking(e.g.,via

stereocameras)mayincreasespatialawarenessandproprioception,andthisfeaturemay

function as an interface for changing the simulator settings in VR. Furthermore, several

issuesremainwithHMDtechnologythataredisadvantagestoenhancingtheimmersion,and

intheseaspects,thetraditionalmonitoroutshinesthenewHMDs.Forexample,althoughthe

selectedHMDpossessedhighpixeldensity(above450ppi),atcloseviewingdistances,the

so-calledscreen-dooreffectwhichisdefinedasthevisibilityofinter-pixelspaceswasquite

noticeable [30]. In addition, from an ergonomic standpoint, the roughly half a kilogram

weight of theHMD can becomeuncomfortablewith prolonged use, and thisweight also

hindersimmersion.However,newadvancementsmaygiverisetohigher-resolution,lighter-

weight, and possibly wireless HMDs; thus, both the image quality and weight might be

reduced in future applications to increase user-friendliness. Furthermore, the cost and

complexitycouldbereducedbycombiningthetwocomputersetupsintoasinglecomputer

system. We demonstrated the technical feasibilities of both the AVR, produced with a

computer-generatedOR,andtheIVR,producedwiththe360°sphericalvideosequence.

Despitethementionedtechnicallimitations,thecurrentpilotstudy,whichcompared

theAVRtoregularVRlaparoscopy,showedthatparticipantstechnicalperformancewasnot

different toregularVR.Still,anon-randomizedstudydesign isa limitationregarding the

interpretationofthecurrentresults.CochranereviewscomparingregularboxtrainerstoVR

laparoscopyshowedequivalenceinthosetrainingmethods[14].Thecurrentresultsshow

no significant difference in performance compared to regular VR surgical simulation

techniques. This was consistent with our previous investigation, where we compared

regularVRlaparoscopyandIVR.Inthatstudy,althoughwefoundnodifferenceintechnical

performance, the questionnaires revealed that the users experienced a high degree of

presenceandexhilarationandaratherlowrateofmotionsicknesswiththeIVR[26,27].The

exhilarationexperiencedinVRlaparoscopycombinedwiththeHMDscenarioswasakeyaim

of the current approach, and it is likely to increase the attractiveness ofVR laparoscopy

15

simulation.Furtherstudieswithalargercohortmustbeperformedtoevaluatethegeneral

influenceofAVRandIVRonlaparoscopysimulations.

However,futureresearchshouldplacethegreatestemphasisoninteractionswithin

theVRenvironment,accordingto theuser’sperformanceontheLapSim.Here, itmaybe

possibletousetechnicaldatafromthesimulatororaspeechrecognitionfeaturetotrigger

differentenvironmentalscenarios,dependingonthesituation.Immersionmaybeimproved

byreplacingstaticvideopartswithhighresolution360°imagesorbyconstructingvirtual

environments with multiple 360° videos. Currently, these improvements might be

technicallydemanding;however,itwouldnotbedifficulttoimplementbinauralrecordings

ofORsoundsto increasepresenceinthegeneratedworld.Thedegreeof immersionmay

generallybehigher in thecomputer-generatedAVR than in the IVR,becauseAVRallows

movements about the virtual room (roomscale VR). However, the fact that the OR

surroundingsandtheactingindividualswerefamiliartoallparticipantsmayhaveincreased

thedegreeofpresenceintheIVR.Ontheotherhand,inIVR,movementinthevirtualroom

is currently impossible, due to the static video recording. Movement in IVR might be

achievedby fusingmultiple360° camera recordingsof avirtual environment to createa

photorealistictypeofAVR.Thelowdegreeofmotionsicknessachievedmightbeexplained

by the fact that all movements were controlled by the participants, and not by the VR

environment.TheabilitytosimulatecameranavigationwasnotusedinthisfirstVRsetup.

Apotential goalof futuredevelopmentsmightbe to integrateanassistant thatperforms

camera navigation; however, this integrationmight also affectmotion sickness. Another

promisingimprovementduetohighly-immersiveVRapplicationcouldbethesimulationof

stress training in the OR as part of the surgeon’s learning curve. Further research

additionallyneedstofocusonthesimulationofinteractivescenarios.IVRmaybeauseful

tooltosupportthisandinterfacesshouldbedevelopedforsurgicalsimulationsoftwarethat

triggeraspectsinavirtualsurrounding.

Inconclusion,wehavepresentedthetechnicalandclinicaldevelopmentofahighly

immersiveVRlaparoscopysimulationsetup.Thisnewgenerationofsimulationwillenable

clinicalstudiestoevaluatetheimpactofVRforsurgicaltraining.Furthertechnicaladvances

areneededtoimprovevisualizationandinteractivity.Clinicalanalysesshouldfocusonthe

influenceofAVRandIVRonlaparoscopytraining.

16

Table2:OverviewofcurrentlimitationsofthecustomimmersiveVirtualRealitySimulationandpossiblesolutionsforfutureresearch,includingratingsoftheirclinicalimportanceandtechnicalviability

Currentlimitation PossiblesolutionClinical

importanceTechnicalviability

Monoaudiorecordingof360°camera

Binauralrecordingofintraoperativesounds

3 3

Missingvisibilityofuser’shandmotion

Handtrackingwithdatagloves 1 2

Missinghaptic/tactilefeedbackfromobjects(patient,table,steriledrapes,otherpersons…)

Useofappropriatehapticfeedbackdevice

3 1

Flatspherewithmissingdepthoffield

Three-dimensional360°cameravideo

3 2

Static360°videoofsurroundings(noroom-scaleVRinIVR)

Multiple360°videostocreateaphotorealisticVRoperatingroomtoenableroom-scaleVR

2 2

NoreactionofsurroundingstoVRsimulatordata(e.g.,mistakes)

RecordingofsimulatordatathatcanbeusedtotriggerdifferentscenariosTextrecognitionofsimulatorcommands

1 3

InteractionwithVRsimulator(administrative)

Connectdataglovestosteersimulationsoftware

1 2

Cameranavigationnotincluded

Performvirtualsurgeriesasateamofsurgeons,andincludeacameranavigator,withtwoVRheadsets

2 1

Simpleoperatingroomscenario

Recorddifferentscenarios(e.g.,stresstraining)

1 3

Surroundingsfamiliartotheparticipants

Recorddifferentoperatingrooms 2 3

Rangeofclinicalimportanceratings:1=highimportance,3=lowimportanceRangeoftechnicalviabilityratings:1=difficult,3=feasible

17

Acknowledgments

TheauthorsthankS.Rohde,B.Golla,andM.Kosta(UniversityofMagdeburg)fortechnical

support. Additionally, we appreciate the support from and discussions with H. Hecht,

DepartmentofGeneralExperimentalPsychology,JohannesGutenbergUniversity,Mainz.

EthicalApproval

Forthistypeofstudy,formalconsentwasnotrequired.Thisstudydidnotincludepatients

oranimals.

Funding

Financial support for the laparoscopy simulatorwas provided by themedical education

project,“MAICUM”,fromtheMedicalCenteroftheJohannesGutenbergUniversityofMainz.

Fundingfortheadditionalimmersivevirtualrealityhardwarewasprovidedbyintramural

fundingfromtheMedicalCenteroftheJohannesGutenbergUniversityofMainz,Germany,

and educational intramural funding by the Otto-von-Guericke University Magdeburg,

Germany.

ConflictofInterest

TheauthorsTH,TW,MP,HL,WKandCHdeclarenoconflictsof interestor financial ties

relatedtothisstudy.

Informedconsent

Thisarticledoesnotcontainpatientdata.

18

References

1. AggarwalR,GrantcharovTP,EriksenJR,BlirupD,KristiansenVB,Funch-JensenP,

DarziA(2006)Anevidence-basedvirtualrealitytrainingprogramfornovice

laparoscopicsurgeons.AnnSurg244(2):310-314.

doi:10.1097/01.sla.0000218094.92650.44

2. vanDongenKW,vanderWalWA,RinkesIH,SchijvenMP,BroedersIA(2008)Virtual

realitytrainingforendoscopicsurgery:voluntaryorobligatory?SurgEndosc22

(3):664-667.doi:10.1007/s00464-007-9456-9

3. HuberT,KirschniakA,JohanninkJ(2017)SurveyofTraininginLaparoscopicSkillsin

Germany.ZentralblChir142(1):67-71.doi:10.1055/s-0042-116327

4. EarleD,BettiD,ScalaE(2017)Developmentofarapidresponseplanfor

intraoperativeemergencies:theCirculate,Scrub,andTechnicalAssistanceTeam.AmJ

Surg213(1):181-186.doi:10.1016/j.amjsurg.2016.01.047

5. ChiricoA,YadenDB,RivaG,GaggioliA(2016)ThePotentialofVirtualRealityforthe

InvestigationofAwe.FrontPsychol7:1766.doi:10.3389/fpsyg.2016.01766

6. SchmittYS,HoffmanHG,BloughDK,PattersonDR,JensenMP,SoltaniM,CarrougherGJ,

NakamuraD,ShararSR(2011)Arandomized,controlledtrialofimmersivevirtual

realityanalgesia,duringphysicaltherapyforpediatricburns.Burns37(1):61-68.

doi:10.1016/j.burns.2010.07.007

7. RizzoA,HartholtA,GrimaniM,LeedsA,LiewerM(2014)Virtualrealityexposure

therapyforcombat-relatedposttraumaticstressdisorder.Computer47(7):31-37

8. LaverKE,GeorgeS,ThomasS,DeutschJE,CrottyM(2015)Virtualrealityforstroke

rehabilitation.CochraneDatabaseSystRev(2):Cd008349.

doi:10.1002/14651858.CD008349.pub3

9. ParsonsTD(2015)VirtualRealityforEnhancedEcologicalValidityandExperimental

ControlintheClinical,AffectiveandSocialNeurosciences.Frontiersinhuman

neuroscience9:660.doi:10.3389/fnhum.2015.00660

10. RivaG(2003)Applicationsofvirtualenvironmentsinmedicine.MethodsInfMed42

(5):524-534

11. OlaskyJ,SankaranarayananG,SeymourNE,MageeJH,EnquobahrieA,LinMC,

19

AggarwalR,BruntLM,SchwaitzbergSD,CaoCG,DeS,JonesDB(2015)Identifying

OpportunitiesforVirtualRealitySimulationinSurgicalEducation:AReviewofthe

ProceedingsfromtheInnovation,Design,andEmergingAlliancesinSurgery(IDEAS)

Conference:VRSurgery.SurgInnov22(5):514-521.doi:10.1177/1553350615583559

12. PascholdM,SchroderM,KauffDW,GorbauchT,HerzerM,LangH,KneistW(2011)

Virtualrealitylaparoscopy:whichpotentialtraineestartswithahigherproficiency

level?IntJComputAssistRadiolSurg6(5):653-662.doi:10.1007/s11548-010-0542-4

13. LarsenCR,SoerensenJL,GrantcharovTP,DalsgaardT,SchouenborgL,OttosenC,

SchroederTV,OttesenBS(2009)Effectofvirtualrealitytrainingonlaparoscopic

surgery:randomisedcontrolledtrial.BMJ338:b1802.doi:10.1136/bmj.b1802

14. NagendranM,GurusamyKS,AggarwalR,LoizidouM,DavidsonBR(2013)Virtual

realitytrainingforsurgicaltraineesinlaparoscopicsurgery.CochraneDatabaseSyst

Rev(8):CD006575.doi:10.1002/14651858.CD006575.pub3

15. PascholdM,HuberT,KauffDW,BuchheimK,LangH,KneistW(2014)Preconditioning

inlaparoscopicsurgery--resultsofavirtualrealitypilotstudy.LangenbecksArchSurg

399(7):889-895.doi:10.1007/s00423-014-1224-4

16. KneistW,HuberT,PascholdM,LangH(2016)[3DVirtualRealityLaparoscopic

SimulationinSurgicalEducation-ResultsofaPilotStudy].ZentralblChir141(3):297-

301.doi:10.1055/s-0033-1350609

17. SorensenSM,SavranMM,KongeL,BjerrumF(2016)Three-dimensionalversustwo-

dimensionalvisioninlaparoscopy:asystematicreview.SurgEndosc30(1):11-23.

doi:10.1007/s00464-015-4189-7

18. TillouX,CollonS,Martin-FrancoisS,DoerflerA(2016)RoboticSurgerySimulator:

ElementstoBuildaTrainingProgram.JSurgEduc73(5):870-878.

doi:10.1016/j.jsurg.2016.04.008

19. SankaranarayananG,LiB,ManserK,JonesSB,JonesDB,SchwaitzbergS,CaoCG,DeS

(2016)Faceandconstructvalidationofanextgenerationvirtualreality(Gen2-VR)

surgicalsimulator.SurgEndosc30(3):979-985.doi:10.1007/s00464-015-4278-7

20. LaValleSM,YershovaA,KatsevM,AntonovMHeadtrackingfortheOculusRift.In:

RoboticsandAutomation(ICRA),2014IEEEInternationalConferenceon,2014.IEEE,

pp187-194

20

21. EvangelakosD,MaraMExtendedTimeWarplatencycompensationforvirtualreality.

In:Proceedingsofthe20thACMSIGGRAPHSymposiumonInteractive3DGraphicsand

Games,2016.ACM,pp193-194

22. CoburnJQ,FreemanI,SalmonJL(2017)AReviewoftheCapabilitiesofCurrentLow-

CostVirtualRealityTechnologyandItsPotentialtoEnhancetheDesignProcess.

JournalofComputingandInformationScienceinEngineering17(3):031013

23. NicholsS,HaldaneC,WilsonJR(2000)Measurementofpresenceanditsconsequences

invirtualenvironments.InternationalJournalofHuman-ComputerStudies52(3):471-

491.doi:http://dx.doi.org/10.1006/ijhc.1999.0343

24. MadaryM,MetzingerTK(2016)RealVirtuality:ACodeofEthicalConduct.

RecommendationsforGoodScientificPracticeandtheConsumersofVR-Technology.

FrontiersinRoboticsandAI3(3).doi:10.3389/frobt.2016.00003

25. KeshavarzB,HechtH(2011)ValidatinganEfficientMethodtoQuantifyMotion

Sickness.HumanFactors:TheJournaloftheHumanFactorsandErgonomicsSociety53

(4):415-426.doi:10.1177/0018720811403736

26. HuberT,PascholdM,HansenC,WunderlingT,LangH,KneistW(2017)New

dimensionsinsurgicaltraining:immersivevirtualrealitylaparoscopicsimulation

exhilaratessurgicalstaff.SurgEndosc.doi:10.1007/s00464-017-5500-6

27. HuberT,PascholdM,HansenC,LangH,KneistW(2017)["Iwilldolaparoscopy

somewhereelse":Total,highlyimmersivevirtualrealitywithoutsideeffects?].Der

Chirurg.doi:10.1007/s00104-017-0465-5

28. BowmanDA,McMahanRP(2007)VirtualReality:HowMuchImmersionIsEnough?

Computer40(7):36-43.doi:10.1109/MC.2007.257

29. MilgramP,TakemuraH,UtsumiA,KishinoFAugmentedreality:aclassofdisplayson

thereality-virtualitycontinuum.In,1995.pp282-292

30. ChoJ-m,KimY-d,JungSH,ShinH,KimTScreenDoorEffectMitigationandIts

QuantitativeEvaluationinVRDisplay.In:SIDSymposium2017.vol1.WileyOnline

Library,pp1154-1156