Multi-institutional development of a mastoidectomy ... · MULTI-INSTITUTIONAL DEVELOPMENT OF A...
Transcript of Multi-institutional development of a mastoidectomy ... · MULTI-INSTITUTIONAL DEVELOPMENT OF A...
MULTI-INSTITUTIONALDEVELOPMENTOFAMASTOIDECTOMYPERFORMANCEEVALUATIONINSTRUMENTThomasKerwin1,BradHittle1,DonStredney1,PaulDeBoeck2,GregoryWiet3
1InterfaceLab,OhioSupercomputerCenter,Columbus,Ohio,UnitedStates
2DepartmentofPsychology,OhioStateUniversity,Columbus,Ohio,UnitedStates
3DepartmentofOtolaryngology,OhioStateUniversity,Columbus,Ohio,UnitedStates
CORRESPONDINGAUTHOR:ThomasKerwin1
1224KinnearRd.,Columbus,Ohio,43212,UnitedStates
Emailaddress:[email protected]
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
MULTI-INSTITUTIONALDEVELOPMENTOFA1
MASTOIDECTOMYPERFORMANCEEVALUATION2
INSTRUMENT3
ABSTRACT4
OBJECTIVE5
Amethodforratingsurgicalperformanceofamastoidectomyprocedurethatisshowntoapply6universallyacrossteachinginstitutionshasnotyetbeendevised.Thisworkdescribesthe7developmentofaratinginstrumentcreatedfromamulti-institutionalconsortium.8
DESIGN9
UsingaparticipatorydesignandamodifiedDelphiapproach,amulti-institutionalgroupofexpert10otologistsconstructeda15elementtask-basedchecklistforevaluatingmastoidectomy11performance.Thisinstrumentwasfurtherrefinedintoa14elementchecklistfocusingonthe12conceptofsafetyafterusingittoratealargeandvariedpopulationofperformances.13
SETTING14
TwelveOtolaryngologicalsurgicaltrainingprogramsintheUnitedStates.15
PARTICIPANTS16
14surgeonsfrom12differentinstitutionstookpartintheconstructionoftheinstrument.17
RESULTS18
Byusing14expertsfrom12differentinstitutionsandaliteraturereview,individualmetricswere19identified,ratedastothelevelofimportanceandoperationallydefinedtocreatearatingscalefor20mastoidectomyperformance.Initialuseoftheratingscaleshowedmodestrateragreement.The21operationaldefinitionsofindividualmetricsweremodifiedtoemphasize“safe”asopposedto22“proper”technique.Asecondratinginstrumentwasdevelopedbasedonthisfeedback.23
CONCLUSIONS24
Usingaconsensusbuildingapproachwithmultipleroundsofcommunicationbetweenexpertsisa25feasiblewaytoconstructaratinginstrumentformastoidectomy.Expertopinionaloneusinga26Delphimethodprovidesfaceandcontentvalidityevidence,however,thisisnotsufficientto27developauniversallyacceptableratinginstrument.Acontinuedprocessofdevelopmentand28
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
experimentationtodemonstrateevidenceforreliabilityandvaliditymakinguseofalarge29populationofratersandperformancesisnecessarytoachieveuniversalacceptance.30
KEYWORDS31
mastoidectomy,assessment32
COMPETENCIES33
MedicalKnowledge,Practice-BasedLearningandImprovement34
INTRODUCTION35Skillassessmentisessentialtoalltypesoftraining,andotologicsurgeryisnoexception.Inaddition36toprovidingevidencethatabasiclevelofskillproficiencyhasbeenachieved,accuratefeedbackcan37acceleratelearning1.Surgicalresidencyprogramscurrentlyuseavarietyoftoolstoassesstrainees,38andnosingletoolhasemergedasthe"goldstandard".Ataminimum,agoodassessmenttoolmust39bereliable,feasible,fair,objective,andvalid2.Thetime-honoredassessmentcurrentlyusedbythe40AmericanBoardofOtolaryngology(ABOto)andtheAccreditationCouncilforGraduateMedical41Education(ACGME)isbaseduponboththeaccumulationof“adequate”casenumbersduring42trainingandalsotheattestationofthespecificresidencyprogramdirectorwheretheresident43trained.Notwithstanding,thereislittleevidenceofthereliabilityorvalidityofthecurrent44assessmentregimen.45
Auniversallyapplicablesetofmetricsthatcanbeagreeduponandusedforassessmentoftechnical46skillinperformingamastoidectomyhasnotbeendevelopedoradopted.Inordertodevelopsuch47anassessmenttool,caremustbegiventoformulateandvalidatethattooltakingintoaccount48differencesbetweentrainingprograms.Assessmenttoolsmustbedesignedbasedonwhatthe49measurementinstrumentwillbeusedforandwhatspecificinferenceswillbemadebasedonthe50results3.Thereisneedforaninstrumentforbothuserfeedbackintrainingandfordeterminingthe51levelofanindividual’sperformance(novice,intermediateorexpert)intermsoftechnical52performanceofamastoidectomywithfacialrecessapproach.53
Inthiswork,wedescribethecreationandevolutionofasetofmetricsspecificallyfordetermining54thelevelofanindividual'sperformanceinmastoidectomy.Weusedabroad-basedconsortiumof55surgeonsatdifferentinstitutionsinconsecutivefeedbackstepssothattheinstrumentcanbe56universallyappliedtoalltemporalbonedissectionperformancesregardlessofinstitutionor57background.58
PREVIOUSWORK59Ratinginstrumentsformastoidectomyhavebeendevelopedbyothergroups,buttheydonot60includesuchabroadbaseofexpertinput.Arecentreviewofthecurrentinstrumentsformeasuring61mastoidectomyperformancebySethiaetal.discusseseachoftheinstrumentsingreaterdetail4.62
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
AgroupatJohnsHopkinsdevelopedaninstrumentbasedontheworkofMartinetal.5for63mastoidectomyperformancecontainingbothaTask-BasedChecklist(TBC)andGlobalRatingScale64(GRS)6.Bothofthescalesusealistofevaluationitemswithratingsofonetofive.WorkbyLaeeqet65al.7andAwadetal.8showsomevalidityevidenceforthatinstrumentbutinonlyasmallnumberof66institutions.67
TheWellingScale(WS1)usesfinalproductanalysis(FPA)forevaluatingacomplete68mastoidectomywithfacialrecessperformedinthetemporalbonelab9,10.Itdefinesbinaryitems69thataresummedtoprovideanoverallscore.70
AsseeninthesurveyresultsfromButleretal.10,eventhoughasetofcommonevaluationitemsfor71mastoidectomycanbecreated,thereexistmanydifferencesbetweentheimportancegiventothose72itemsbyexpertsfromdifferentinstitutions.Additionalcaremustbegiventodevelopandevaluate73instrumentsthatcanbeusedbroadlyatallinstitutions.Inordertocreatesuchaninstrument,an74attemptwasmadebyWanetal.11touseamodifiedDelphimethodtofindconsensusonwhich75itemsshouldbeincorporatedintoaTBC.TheHopkinsscalewasalsodevelopedusingaDelphi76method,butincludedonlyJohnsHopkinsfacultymembersintheprocess.77
TheWanetal.studyreceivedresponsesfrom88membersoftheAmericanNeurotologySocietyor78AmericanOtologicalSocietyoncriteriaimportanttoasuccessfultemporalbonedissection.Based79onthoseresponses,alistofcriteriaorderedbyimportancewascreatedandusedinthisstudy.80
MATERIALSANDMETHODS81Inordertocreateaconsensus-based,cross-institutionalratinginstrumenttomeasuresurgical82performancewestartedwiththelistofassessmentitemsfromWanetal.11Theseitemswerethen83furtherrefinedusingaDelphimethoddescribedindetailbelowwithanexpertgroupconsistingof8414fellowship-trainedotologistsfrom12differentinstitutions(Table1).Inthisrefinement,the85individualitemsfromtheWanstudyweremoreexplicitlydefinedtoencourageauniform86interpretationfordeterminingsuccessorfailureforeachitem.Thislistwasthenreviewedbyall87individualsinthesamegroupofexpertsbymeansofanonlinesurvey.88
Inafirstround,membersoftheconsortiumwereaskedtorankthe5mostimportantand6least89importantmetricsonthelist.Resultsofthesurveyshowed24metricswithadditionalsuggestions90(Table2).91
Aface-to-facemeetingforactivediscussionregardingeachmetric,itsoverallimportanceandan92agreeduponoperationaldefinitionwasconvenedwiththemembersoftheexpertgroup.Inthis93meeting,eachmetricwaspresentedseparatelyalongwithanycommentsthatweremadewithin94thesurveycontext.AnexampleofametricresultanddiscussionispresentedinFigure1.95
Next,theexpertswereaskedtoassignanimportancemeasuretoeachmetric,asfollows:96
• Pass/Fail(P/F):Criticalmetricsthat,ifanyoneisviolated,thereisanautomaticfailure.97Violationsofthesemetricswillresultinseriousmorbiditytothepatient.98
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
• High:Dangerous,ifviolatedcouldpotentiallyresultinmorbiditytothepatient.99• Medium:Potentialcomplicationthatrequiresinterventionandcouldberectifiedor100
managedwithoutsignificantmorbiditytothepatient.101• Low:Potentialcomplicationwhichdoesnotrequireintervention–poortechnique.102
Then,inasecondround,expertswereaskedtoidentifywhichitemswereneededtobecompetent103inordertobeconsiderednovicelevel(readytooperateonpatientundersupervision),104intermediatelevel(readyforminimalsupervision–PGY4/5level),advancedlevel(practice105independentlyatfellowshiptrainedlevel).Usingthefollowingcriteria:106
• Novicelevel.(competencyoneachofthehighimportanceareasandnoFs).(readyfor107cadavericlab)108
• Intermediatelevel.(competencyonallofhighandmediumitemsandnoFs).(readyfor109SupervisedORexperience).110
• Advancedlevel.(expertonallmetricsandnoFs)(readyforindependentsurgery,doesnot111needsupervision).112
TheresultsoftheabovetworoundsarelistedinTable3asoriginalandfinalrelativeimportance.113TheitemslistedasP/F(Pass/Fail)includethoseitemsforwhichiftheywerenotachieved,the114globalperformanceautomaticallyresultedinafailingscoreregardlessofperformanceonanyother115metric.TheitemslistedasHighprioritywerethoseitemswithconditionstobefulfilledtobe116consideredasanoviceoperator,theitemslistedasMediumareitemswithconditionstobefulfilled117tobeconsideredasanintermediateandtheitemslistedaslowarenecessaryconditionsforan118advancedleveloperator.Byimplicationtheabsenceofmoreimportantviolationsisnecessaryas119wellforeachofthethreelevels.120
Atthispoint,underIRBapprovalfromTheOhioStateUniversityOfficeofResponsibleResearch,we121performedastudyusingourpreviouslydevelopedtemporalbonedissectionsimulator12,13across122the12institutions.Thisresultedinsixty-sixmastoidectomyperformancesforreview.Theycovered123awidedistributionofskilllevels:medicalstudents,PGY(Post-GraduateYear)2-5,fellowsand124attendingphysicians.Thissetcomprised36sessionscollectedfromfacultyand30collectedfrom125residentsandstudents.Eachoftwelveexpertreviewers,allconsideredexpertsinotologicsurgery,126wasassignedelevengradingtasks(individualmastoidectomyperformances).Theywereblindedto127theidentityofthesubjectperformingthedissectionanddidnotreviewtheirownperformances.128Thisresultedintwosetsofratingsusingtheinstrumentforeachvirtualmastoidectomyinthe129testingset.Afterexaminingthestatisticalmeasuresfromthistrial,amoderatelylowlevelof130agreementamongraterswasseen(overhalftheinterclasscorrelation12(ICC)valueswerebelow1310.4,whichisconsideredpooragreement).132
Asaresultoftherelativelyweakinter-rateragreement,weconcludedthatperhapsthismaybedue133topooragreementontheoperationaldefinitionofeachmetricandhowitshouldbescored.Asa134result,anadditionalface-to-faceDelphiprocesswasundertakentodiscussthepooragreement135scores.Itwastheconsensusofthegroupthattheoperationaldefinitionsofeachitemwereasource136ofcontinuedvariabilityinhowtheyshouldbeinterpreted.Thegrouprecommendedfurther137refinementbasedonthepremisethattheywouldbeusedtoidentify"safe"asopposedto"proper"138
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
surgicaltechnique.Itwasrecognizedthattherearevariousopinionsastowhatconstitutes139“proper”technique.Theconsensuswasthattherewouldbegreateragreementiftheoperational140definitionofindividualmetricscouldbejudgedonthebasisofits“safety”.Specifically,ifa141particularstyleoftechniquewasnotonethataparticularraterrecognizedas“proper”,itcouldstill142bejudgedonwhetherornotitwasconsideredhighriski.e.,notsafe.Basedonthisdiscussion,a143secondsetofassessmentitemswasdeveloped.Additionally,atthesuggestionoftheexpertgroup,144thetwoitemsintheoriginallistthatconcernedtheexternalauditorycanalwerecombinedinto145one.Theresultofthisseconddiscussiongroupwasthedevelopmentofasecondsetofmetrics146encompassingalistof15items.Theindividualitemsforbothmetricsetscanbeseenand147comparedinTable4.148
Anoverviewofthestepswetooktoconstructthemetricsandthereasonsbehindthemcanbeseen149inTable5.150
DISCUSSION151Aswithclinicalcare,itisimportantthatclearandrigorousevidenceexiststoobjectivelyappraise152theefficacyofoureducationalprograms.15Subjectivedeterminationsbyprogramdirectorsor153traineeself-reportingofnumberofproceduresmustevolveintomoreevidencebasedassessments.154Thisrequiresaconcertedefforttodevelopoutcomemeasuresthatareagreeduponanduniversally155translatable.Forassessmentstobevalid,theymustaccumulatevalidityevidenceinanumberof156areasincludingcontentevidence,responseprocess,internalstructure,relationswithother157variablesandconsequences.16Ourmetricsdemonstrate“contentevidence”basedonthenatureof158thedevelopmentprocessnotedabove.Thenextvalidationstepsincludethedemonstrationofa159sufficientlyhighintra-rateragreementandtherelationshipwithanexternalcriterionforthe160qualityofaperformance.161
WehavefollowedtheprocessoutlinedbyDauphineeandWood-Dauphineefordeveloping162evidenced-basedmedicaleducation.15Thisinvolvesdefiningtheparameterstobemeasured,163measuringthoseparameters,andbenchmarkingthoseparameterstoassesseducationaloutcomes.164Asnotedbyourwork,theefforttodefineoutcomemeasureswithanacceptablelevelofcontent165validityisinitselfoftenpainstaking,especiallyifthegoalincludesuniversalacceptance.Studies166conductedatoneinstitutionoftenarefraughtwithsubjectivebiasandlowsamplesizes.15This167makesdisseminationofrecommendationsandguidelinesforassessmentproblematic.168
Ourattemptatdevelopingaspecificsetofmetricsforaprocedureasspecializedasmastoidectomy169hasprovenextremelychallenging.Inmastoidsurgery,thereareanumberofassessmenttoolsin170existencetoday,noneofwhichprovidebroadenoughacceptanceanduniversality.4Itisthegoalof171thisresearchtocontinuetheprocessofpainstakinglyrefiningthemetricsestablishedandthe172ratingprocesssothattheycanshowthevalidityevidencenecessarytomakeassessmentsthat173correlatewithclinicalperformance.174
Identifying,definingandapplyingmetricssothattheycanbeuniversallyusefulandstillprovide175sufficientinformationtomakevaliddecisionsbasedontheiruseisdifficultevenattheearlystages.176
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
Forthenextstepswearenecessarilysubjecttomanysourcesofpossibleassessorerror17.These177includepossibledriftinassessorinterpretationofindividualmetrics,individualperformance178expectations,andlackoffamiliarityofbeinganassessorasopposedtoatrainer.Thesesourcesof179ratingerroraremultipliedwiththeexpandednumberofassessors.Thesesourceshowever,canbe180mitigatedinthefuturebymakingaconcertedefforttoprovidegoodoperationaldefinitionsofeach181metric,carefultrainingofassessors(perhapsagroupsessionwhereastandardizedperformanceis182ratedanddiscussedwithinthecontextofthegroup),andmonitoringoftheassessor’sperformance183assuggestedbyGallagheretal.17184
Inthefuture,wewilluseournewsetofmetricstoaccumulateadditionalvalidityevidence.185Emphasizingsafetyastheglobalconceptindefiningandadministeringtheitemsisonewaywecan186makeouroperationaldefinitionsmorewidelyapplicable.Wearecurrentlyinvestigatingdefining187ourmeasurementscalesintermsofthreeseparateaxes:boneremoval,toolcontrolandviolations188ofstructures(Table6).Thesecanfunctionasdistinctsubscales.Measurementscalesforskill189masterywillbebuiltforeachaxissuchthattheperformancesoftraineescanbeevaluatedinterms190ofdescriptiveandnormativemasterylevels.Thedescriptivelevelsarespecificpositionsonthe191measurementscaleswhilethenormativelevelsarelevelsthatmustbereachedtobeconsideredan192independentexpert(expertlevel)oranintermediateleveltrainee(intermediatelevel).The193approachtobeusedisatwo-foldextensionofitemresponsetheory(IRT).18-20IRTisafamilyof194statisticalmeasurementmodelsthathasbecomethestandardforthemeasurementofskillsinan195educationalandtrainingcontext.IRTscoresaremodel-baseddescriptivemasterylevels.196Additionally,wearedesigningamethodologytoeasily“traintheraters”sothatconsistencyin197interpretationandapplicationofthemetricsisplausible.198
CONCLUSION199Ourworkmovesclosertothegoalofdevelopingauniversallyacceptableandapplicablesetof200performancemetricsformastoidsurgery.Wehaveusedanextensiveparticipatoryprocessto201formulatealistofmetricsbasedonliteraturereview,multipleroundsofexpertfeedback,and202continuedrefinement.Basedonourmethodology,wefeelthatourresultsdemonstratesignificant203contentvalidity.Ourresultsdemonstrateconsiderableinputofdiverseexpertopinionbutstillneed204tobesupplementedwithothertypesofvalidityinamulti-institutecontext.205
ACKNOWLEDGEMENTS206ThisworkwassupportedbyTheNationalInstituteforDeafnessandotherCommunication207Disorders,NationalInstitutesofHealth,USA,R01DC011321.208
BIBLIOGRAPHY209210
1. CataniaAC.Learning.2nded.EnglewoodCliffs,N.J.:Prentice-Hall;1984.211
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
2. ShahJ,DarziA.Surgicalskillsassessment:anongoingdebate.BJUInt.2001;88(7):655-660.2123. MichelsonJD,ManningL.Competencyassessmentinsimulation-basedprocedural213
education.AmJSurg.2008;196(4):609-615.2144. SethiaR,KerwinTF,WietGJ.PerformanceAssessmentforMastoidectomy:StateoftheArt215
Review.OtolaryngolHeadNeckSurg.2016.2165. MartinJA,RegehrG,ReznickR,etal.Objectivestructuredassessmentoftechnicalskill217
(OSATS)forsurgicalresidents.BrJSurg.1997;84(2):273-278.2186. FrancisHW,MasoodH,ChaudhryKN,etal.Objectiveassessmentofmastoidectomyskillsin219
theoperatingroom.OtolNeurotol.2010;31(5):759-765.2207. LaeeqK,BhattiNI,CareyJP,etal.Pilottestingofanassessmenttoolforcompetencyin221
mastoidectomy.Laryngoscope.2009;119(12):2402-2410.2228. AwadZ,TornariC,AhmedS,TolleyNS.Constructvalidityofcadaverictemporalbonesfor223
trainingandassessmentinmastoidectomy.Laryngoscope.2015;125(10):2376-2381.2249. FernandezSA,WietGJ,ButlerNN,WellingB,JarjouraD.Reliabilityofsurgicalskillsscores225
inotolaryngologyresidents:analysisusinggeneralizabilitytheory.EvalHealthProf.2262008;31(4):419-436.227
10. ButlerNN,WietGJ.ReliabilityoftheWellingscale(WS1)forratingtemporalbone228dissectionperformance.Laryngoscope.2007;117(10):1803-1808.229
11. WanD,WietGJ,WellingDB,KerwinT,StredneyD.Creatingacross-institutionalgrading230scalefortemporalbonedissection.Laryngoscope.2010;120(7):1422-1427.231
12. WietGJ,BryanJ,DodsonE,etal.Virtualtemporalbonedissectionsimulation.Studiesin232healthtechnologyandinformatics.2000;70:378-384.233
13. WietGJ,StredneyD,KerwinT,etal.Virtualtemporalbonedissectionsystem:OSUvirtual234temporalbonesystem:developmentandtesting.Laryngoscope.2012;122Suppl1:S1-12.235
14. ShroutPE,FleissJL.Intraclasscorrelations:usesinassessingraterreliability.PsycholBull.2361979;86(2):420-428.237
15. DauphineeWD,Wood-DauphineeS.Theneedforevidenceinmedicaleducation:the238developmentofbestevidencemedicaleducationasanopportunitytoinform,guide,and239sustainmedicaleducationresearch.AcadMed.2004;79(10):925-930.240
16. CookDA,ZendejasB,HamstraSJ,HatalaR,BrydgesR.Whatcountsasvalidityevidence?241Examplesandprevalenceinasystematicreviewofsimulation-basedassessment.Advances242inhealthscienceseducation:theoryandpractice.2014;19(2):233-250.243
17. GallagherAG,O'SullivanGC.Fundamentalsofsurgicalsimulation:principlesandpractices.244London:Springer;2012.245
18. LindenWJvd.Handbookofitemresponsetheory:Models,statisticaltools,andapplications246BocaRaton:CRCPress;2016.247
19. DeBoekP,WilsonM.Explanatoryitemresponsemodels.Ageneralizedlinearandnonlinear248approach.NewYork:Springer;2004.249
20. DeChamplainAF.Aprimeronclassicaltesttheoryanditemresponsetheoryfor250assessmentsinmedicaleducation.Medicaleducation.2010;44(1):109-117.251
252
253
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
Table1:ParticipatingTrainingInstitutions254
BaylorUniversity
DukeUniversity
HenryFordHospitalSystem
UniversityofIowa
UniversityofMississippi
Montefiore/AlbertEinsteinCollegeofMedicine
StanfordUniversity
UniversityofCalifornia,Irvine
UniversityofCincinnati
UniversityofTexas,Southwestern
TheOhioStateUniversity
MedicalUniversityofSouthCarolina
255
256
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
257
Table2:Resultsfromsurveyonimportanceofindividualmetrics.13expertslisted5itemsashigh258importanceand6itemsaslowimportance.259
Metric Expertsselectingashigh
importance
Expertsselectingas
lowimportance
Maintainsvisibilityoftoolwhileremovingbone 6 1Selectappropriateburrtypeandsize 4 2Antrumentered 4 1Noviolationoffacialnervecanal 11 0Noviolationofsigmoidsinus 3 1Identifiestympanicsegmentoffacialnerve 0 2Doesnotdrillonossicle 5 1Doesnotuseexcessivedrillforcenearcriticalstructures 6 0
Identifieschordatympani 0 3Drillsinbestdirection(understandingofcuttingedge) 3 3
Canalwallup 1 3Identifiesfacialnerveatcochlearformprocess 0 4Appropriatedepthofcavity 0 3Drillswithbroadstrokes 1 3NoholesinEAC 2 2Completesaucerization 2 4Posteriorexternalauditorycanalwallthinned 2 2Facialrecesscompletelyexposed 2 1Identifiesfacialnerveatexternalgenu 1 2Lowfrequencyofdrill“jumps” 2 6Noholesinthetegmen 3 2Useofdiamondburrwithin2mmoffacialnerve 1 2Nocellsremainonsinoduralangle 0 10Sinoduralanglesharplydefined 0 7Otheradditionalmetric 1 0260
261
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
Table3:Originalandfinaldistributionsofmetricsbasedonlevelofimportanceandwhichmetric262expectedtobeachievedateachperformancelevel.263
Metric Importanceproposedtoexperts.
Final
Maintainsvisibilityofburrwhileremovingbone High High
Excessiveforcewillnotbeusednearcriticalstructures High High
Appropriatedepthofcavity Low LowNoholesintegmen Low LowSelectappropriateburr Medium MediumViolationofthesigmoidsinus Medium MediumIdentificationofchordatympaninerve High MediumDrillinbestdirection Medium MediumExternalauditorycanalwallwillremainup Medium Medium
Noholesinexternalauditorycanalwall Low Medium
Completesaucerization Medium MediumPosteriorexternalauditorycanalwallthinnedappropriately
Medium Medium
Violationofthefacialnerve P/F P/FViolationofthehorizontal(lateral)semi-circularcanal P/F P/F
Drillcontactwithossicles P/F P/FViolationofdura P/F264
265
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
Table4:Textofquestionsaskedduringmastoidectomyperformancereview.Question#10wasremoved266forthesecondinstrument,duetooverlapwithquestion#9.267
Number Instrument1 Instrument2
1 Maintainsvisibilityofburrwhileremovingbone
Maintainssafeviewoftheburrthroughouttheprocedure
2 Excessiveforcewillnotbeusednearcriticalstructures
Maintainssafeforcenearcriticalstructuresthroughouttheprocedure
3 Appropriatedepthofcavity Sufficientremovalofmastoidaircellsforpropervisualizationofdeepstructures
4 Noholesintegmen Maintainsintegrityoftegmen
5 Selectappropriateburr EfficientandSafeburrselection
6 Violationofthesigmoidsinus Maintainsintegrityofsigmoidsinus
7 Identificationofchordatympaninerve Identifieschordatympaninervesufficientlytoperformfacialrecessapproach
8 Drillinbestdirection Efficientandsafedirectionofdrilling(paralleltocriticalstructures)
9 Externalauditorycanalwallwillremainup
Sufficientthinningofposteriorexternalauditorycanalwalltovisualizefacialnerve
10 Noholesinexternalauditorycanalwall
11 Completesaucerization Sufficientsaucerizationforsafedrilling
12 Posteriorexternalauditorycanalwallthinnedappropriately
Avoidsoverthinningorholesinposteriorauditorycanalwall
13 Violationofthefacialnerve Maintainsintegrityoffacialnerve
14 Violationofthehorizontal(lateral)semi-circularcanal
Maintainsintegrityofhorizontalsemi-circularcanal
15 Drillcontactwithossicles Maintainsintegrityofossicles
16 Violationofdura Maintainsintegrityofdura
268
269
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
Table5:Stepstakentodeveloptheinstrument,inorder,withabriefreasonforeachone.270
Step ReasonStartwithlistofitemsfromWanetal.
Includeawidesampleofsurgicalexpertise
Surveytodeterminemostandleastimportantitems
Removeverylowpriorityitemsandestablishbroadlevelsofimportance
Meetingtopresentsurveyresultsanddefinemetrics
Reviseitemtextbasedonconsensusfromexperts
Classificationofmetricsfornovice,intermediate,expertachievementlevel.
Reflectimportancelevelsofitemsinthescoringoftheinstrument
Validationstudyusinginstrument TestinstrumentRevisionofinstrumentfocusingonsafety Attempttoincreaseinterraterreliability 271
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
Table6:MetricsandPerformanceAxisforAssessmentStrategy272
Metrics AxisSufficientremovalofmastoidaircellsforpropervisualizationofdeepstructures
BoneRemoval
Identifieschordatympaninervesufficientlytoperformfacialrecessapproach
BoneRemoval
Sufficientthinningofposteriorexternalauditorycanalwalltovisualizefacialnerve
BoneRemoval
Sufficientsaucerizationforsafedrilling BoneRemovalAvoidsoverthinningorholesinposteriorauditorycanalwall BoneRemovalMaintainssafeviewoftheburrthroughouttheprocedure ToolcontrolMaintainssafeforcenearcriticalstructuresthroughouttheprocedure
Toolcontrol
EfficientandSafeburrselection ToolcontrolEfficientandsafedirectionofdrilling(paralleltocriticalstructures)
Toolcontrol
Maintainsintegrityoftegmen ViolationMaintainsintegrityofsigmoidsinus ViolationMaintainsintegrityoffacialnerve ViolationMaintainsintegrityofhorizontalsemi-circularcanal ViolationMaintainsintegrityofossicles ViolationMaintainsintegrityofdura Violation 273
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017
Figure1:Exampleslideofanindividualmetriclevelofimportanceandoperationaldefinitiondiscussion274basedonsurveytogroupofexperts.275
276
PeerJ Preprints | https://doi.org/10.7287/peerj.preprints.2931v1 | CC BY 4.0 Open Access | rec: 18 Apr 2017, publ: 18 Apr 2017