1

RiskManagementinR&DPharmaceuticalProjects

TheHovioneCaseStudy

JoanaMariaLopesMarques

DepartmentofEngineeringandManagement,InstitutoSuperiorTécnico

AbstractTocatchupwiththeincreasingpressuretoadheretonewtechnologies,andtoachievedifferentiationfromcompetitors,pharmaceuticalcompaniesareexposedtoavastnumberofuncertaintieswithintheirResearch&Developmentprojects.HovioneFarmaciência isapharmaceuticalcompanythatdevelopsandproducesActivePharmaceutical Ingredients (APIs)and Drug Products (DPs) for branded pharmaceutical customers. Hovione is in pursuit of an efficient riskmanagementmethodology that can be applied in their projects. This paper proposes a risk management methodology that allowseffective,earlyanditerativeriskmanagementandperiodicriskreviews,whiledefiningthetoolsandtechniquestobeusedateachphaseoftheriskmanagementprocess,andtestsitintwoHovioneprojects.

Key-words:Risk,RiskManagement,Projects,R&Dprojects,PharmaceuticalProjects

1.Introduction

Due to its importance, risk management hasbecome a top-of-mind issue for senior executivesandboardsaround theworld, andnowheremorethan in pharmaceutical companies. Thepharmaceutical industryworks in a politically andeconomically turbulent environment. The riskspharmaceutical companies face, especially inclinical-trial design and execution, drug approval,product quality, and global commercial practices,areincreasingbothinfrequencyandmagnitudeofimpacts (Dhankharetal.,2018).Thispaperhasasmaingoal thedevelopmentofa riskmanagementmethodology to be applied in Hovione’s projects.The development of such tools and techniquesshouldallowthecompanytoimproveitsOTIF(OnTime In Full) performance and customersatisfaction.

2.ProblemDefinition

Hovione FarmaCiência is a Portuguesepharmaceutical company that dedicates itsactivities to helping pharmaceutical customers tobring new drugs to market. It has 60 years ofexperience in the development and compliantmanufacture of Active Pharmaceutical Ingredients(APIs)and15years inDrugProduct Intermediates(DPIs) using advanced technologies. The focus ofthispaper is theR&DServicesdivision.This is theareawhereHovioneneedsto improveprojectrisk

management. R&D Services handles externalclients. Theseexternal clients areusually brandedpharmaceutical companies who hire Hovione todeveloporproduceAPI,DPIorformulatedproductfor them, to support their drug developmentprocesses. Hovione follows a standard approachfor project management based on the ProjectManagement Body Of Knowledge (PMBOK), anindustry standard methodology from the ProjectManagement Institute. This approach follows asystematized methodology that includes thefollowingphaseswithinaprojectlifecycle:

1.Initiation2.Planning3.Execution4.CloseoutIn the past, Hovione has hired amanagement

and technology consulting firm to restructure theprocess of project management of the company.Amongstthetoolsandprocessesdeveloped,ariskmanagementmethodologywascreated.However,itwastoocomplextobeused inatimelymanneronshort-timeprojects,whoseduration is typicallybetween one to three months (from initiation tocloseout). So project managers started applyingonly a very basic methodology, which consists inidentifying and discussing, through brainstorming,thebiggestrisktotheprojectandwhatcouldbeitsimpactonthetimelines.Thepurposeofthispaperis to develop a methodology to substitute thecurrent one, described above, which has many

2

limitationswhenitcomestoaccuratelyidentifyingrisksandtreatingthem.

2.LiteratureReview

A project can be defined as an endeavor inwhichhuman,materialandfinancialresourcesareorganized in a novel way, to undertake a uniquescope of work, of a given specification, withinconstraints of cost and time, so as to achievebeneficial change defined by quantitative andqualitative objectives (Turner, 2009). A risk is anuncertaineventorconditionthat,ifitoccurs,hasapositive or negative effect on a project objective(PMI, 2017). The problem faced by projectmanagers is how to recognize which riskmanagement approach is appropriate for theparticularproject inhand.Therearea fewguidesto conduct project managers in decision makingunder risk and uncertainty. One of the mostrecognized of these guides is the ProjectManagement Institute’s (PMI®) A Guide to theProjectManagementBodyofKnowledge(PMBOK®Guide),which is an inclusive guide that describesthe sum of knowledge within the profession of

projectmanagement.AccordingtoPMI(2017),therisk management process comprises the sixfollowingstages:

1.RiskManagementPlanning;2.RiskIdentification;3.QualitativeRiskAnalysis;4.QuantitativeRiskAnalysis;5.RiskResponsePlanning6.RiskMonitoringandControl.Researchers have suggested different

techniques to manage risks in R&D andpharmaceutical projects, according to theircharacteristics. Some of the tools investigatedseem to be suitable to manage the risks ofHovione’sprojects.Authorshavefoundlimitationsin each other’s methodologies so that, after theanalysis of their researches, it is possible toconclude that the suitable framework toimplement needs to be flexible and adaptable toaccommodate theuncertaintyassociatedwith theresearch component of R&D projects. Themethodology described in the following chapters,takes into account the researches described inTable1.

Table1-MostrelevantstudiesforthedevelopmentoftheriskmanagementmethodologyforHovione

(Source:Author)

Reference DescriptionKwak&Dixon(2008)

BestpracticesusedinR&Dthatshouldhavegoodresultswhenappliedtopharmaceuticalprojects. Someof the identifiedpractices seem tobenecessarily addressedatHovione’sprojects,suchasassessingriskcontinuouslyandusingflexibletools.

Wageman(2004)

Suggestsasetof toolsandtechniques tobeapplied inR&Dprojects,andsomeof them,perexample thechecklistmethod, seemtobesimpleenoughtobeappliedatHovione’sshort-termprojects.

Lavallee(2010)

ProposedamethodologybasedonthePMBOK®Guidethat,combinedwiththerightsetofsimpletoolsandtechniques,andadaptedtoHovione’sfastpace,seemstobeappropriatetomanageDPD’srisks.

Marle&Gidel(2012)

Defined simplicity as a criterion to take into account in riskmanagement strategies. ForHovione,themethodologyhastobesimple,duetothehighworkloadsofemployees,shortdurationofprojectsandlackofhistoricaldataonriskstomanagethemmorecomplexly.

After studying the existing researches, it waspossible to conclude that a methodologycomprisingabaselinesetofprocesses,procedures,and templates that could be tailored to suit eachindividual project was a good starting point forHovione.Themethodologytoimplementmustnotbe intrusive. Which means that it must involverelatively simple processes and procedures suchthat it does not impose a significant overhead tothe execution of projects, as they are of short

duration. This risk management methodologyshouldbe,bynomeans,definiteorfinal.Rather,itis intended to provide to the projectmanagers, anew, simple, and straightforward approach forimplementing the fundamentalsofqualitative riskmanagement.Thismethodologyissupposedtobeastartingpointtoamorecompleteone,whichthecompany is not yet ready to implement, but itshouldbeinthefuture.

3

3.Themethodology

Themethodologydevelopedafteranalyzingthecompany and reviewing the existing literature onthe topic is a simplified version of the riskmanagement methodology espoused by theProject Management Institute in the PMBOK®

Guide. It combines the identification and analysisin one single stage, and it defines the timings ofeach phase by project meeting. Hovione followsthe PMBOK® Guide for managing projects, so itmadeonlysensetofollowitaswell inprojectriskmanagement. The preliminary methodologydevelopedisschematizedinFigure1.

3.1RiskBriefing

A pre-work/briefing phase is essential to thesuccessful implementation of the process. It iscritical that the team understands and identifiesthe need for risk management. The goals of thisstage is topreparetheteamtomanagerisks,andexplain to them how the process will elapse.Ideally,thisstagewouldendwithateamreadyandwilling to engage in the process. To achieve this,theprojectmanagermustassure that the team issoldonthebenefitsorriskmanagementandwhattheirinvestmentisgoingtobeintermsoftimeandeffort,andhowthisinvestmentwillpayoff.

It should be clear to the team that riskmanagement improves the capacity to forecastoutcomes and that when uncertainties areaddressed directly, the likelihood of the teamcompletingitsactivitiesonscheduleincreases.Thebriefing should happen before risk identification,whichshouldtakeplaceduringtheinternalkick-offmeeting. So, thebriefingmust happeneither in a

prior,separatemeeting,orontheinternalkick-offmeetingaswell.

3.2RiskIdentification

Beforethekick-offinternalmeeting,theprojectmanager should send an email to all teammemberswiththeriskchecklistasattachment,sothattheycantakealookatitandfillitbeforethemeeting. This risk checklist comprises 54 risksseparatedbycategoriesanditwasdevelopedwiththe help of Hovione’s projectmanagers. It leavesspace for team members to add new risks. Thechecklist is supposed to be a live document. (SeeAppendix3forthechecklist’sinitialversion).Afterthebriefing isdone, the teamcanproceed to riskidentification.Thisshouldbedoneduringthekick-offmeeting.Aseachriskischecked,itisconfirmedand can be immediately edited to ensure that ithas been captured correctly. Seeing the risks thathave already been recorded will reduce thenumber of duplicates and trigger new ideas aspeoplereadbackthroughthelist.

Selectonchecklist

Askclienttoidentifyrisks

RiskBriefing

RiskIdentificatio

n

RiskAnalysis

ResponsePlanning

Monitoring&Control

Identifycause

Assignriskowner

Qualifyprobability

Qualifyimpact

Developactionplaningroup

Registerstrategy

Definedeadline

Registerallriskandactionplanupdates

KoM

1stIM

IMs

RiskClose-out

Fillclosurecolumns

Presentresultstotheteam CoM

When?

Figure1-PreliminaryMethodologyProposed(Source:Author)

KoM:Kickoffmeeting;KoMw/Cust:KickoffMeetingwithCustomer;IM:InternalMeetingCoM:Close-outMeeting

4

Thesecondstageof the identificationphase isto involvetheclient.and it takesplaceduringtheKickoff meeting with the client. The projectmanagershouldaskwhataretheprojectrisksandadd them to the checklist to be analyzed by theteamintheanalysisphase.

3.3RiskAnalysis

Aftercompletingtheidentificationandcleanupof risks, it will be necessary to analyze each onewith regard to cause,probability that itwill occurand the impact that it will have on the overallprojectifitdoesoccur.

The assessment of probability and impact iscentraltodeterminingthehighestpriorityrisksforwhich action plans will be generated. Identifying

the cause of each risk makes developing action

planseasier,astheycanbeconstructedtodirectlyaddressriskcauses.

For each risk, the team should find its causeand register it. Then, they should assess theprobabilityof theeventoccurringandtherelativeimpact of its occurrence on the project. Bothprobabilities and impacts should be analyzedqualitatively, using the terms: Very Low, Low,Medium, High, and Very High. The approachdeveloped considers each risk and assigns it apriority ofhigh,medium or low. These categoriesreflecttherelativeimportanceofimplementinganaction plan for each risk. To help sorting similarlyratedrisksthroughalargelist,ariskmatrixwillbeusedtohelp(SeeFigure2).

Figure2-RiskMatrix(Source:Author)

It was decided that action plans would be

developedforbothhighandmediumpriorityrisks.Actionplans forhighpriority risks (red) shouldbeimplemented immediately. Yellow risks on theotherhand,shouldbemonitored,andunlesstheyaggravate into high priority risks, no action planshould be implemented. When determining thesubsetofrisksforwhichriskresponseplanningwillbe conducted, an evaluation of the potentialeffectivenessofmitigationactivitiesshouldalsobeconsidered.

The last step of this stage is to assign riskowners to risks,whowill developactionplans foreach high priority and medium priority risk. Riskownerassignmentsshouldbebasedonthelevelofknowledge and expertise needed for addressingthe risk. The person should understand theunderlyingconcernsandthetypesofactivitiesthat

could reduce the impact or probability ofoccurrence.

3.5 RiskResponsePlanning

Each risk ownerwill create an effective actionplan for their risk. The range of approaches thatcanbetakenisfairlybroad,butingeneral,willbeaimed at achieving one or more of the followingthreebasicobjectives:

- Actions that reduce the probability of a risk

occurrenceareidentified.

- Actions thatmitigate the impact if risk should

occurareidentified.

- Contingencyplans aredeveloped to execute if

theriskoccurs.

There are a number of options for respondingtorisk,whichwillbeprovidedtotheteam.

5

Risk owners prepare their draft plans andpresent them on the first weekly meeting. Theteam reviews the action plans to ensure they areacceptable. After action plans have beendevelopedforallhighandmediumpriorityrisks,adeadline for implementation in high priority risksshould be established and the risk owner is theresponsibleperson forassuring that theplan is infactexecuted.

3.6RiskMonitoringandControl

The team members should review eachassigned risk and check if the risk has changedregularly and let the rest of the team know of

updates on the weekly meetings. The risk mayhave occurred and became a certainty or thetriggerfortheriskoropportunitymayhavepassedand the risk is no longer worth consideration. Alowprobabilityriskthatmightnothavewarrantedan action plan may now have risen to a highenough probability to justify action planning. Inaddition, risk owners should also review theiractionplans todetermine if theyare still relevantand sufficient and they should adjust them asnecessarytoensuretheresponseisoptimized.

Oncetheprojectreachesitsclose-outphase,afull review of the risk templates should be doneand all data should be registered

4.CaseStudyResults

This methodology was tested through casestudy research, and refined based on theconclusions obtained from it. Conducting casestudy research allowed observing and analyzingthe reactions and behaviors of Hovione’semployees towards risk management and themethodology. It allowed adapting the steps, toolsand timings to Hovione’s people and work style,suitingbettertheirfastpaceandtheircurrentriskknowledge. The final version of the methodologywillbedescribedfurtherahead.

Project 1: The objective of Project 1 was todilute a feed solution in a lower concentration of

anAPIinordertoexpandtheprocessflexibilitytoobtain a high process yield and material withinspecifications and later sent it to the client to beformulated. The project execution phase wasplannedtolast26days.

The methodology described in the previoussection was applied. A total of twenty-one riskswere identified along the entire duration of theproject.18intheKOMand2.Sevenofthemwereconsideredhighpriority,tenweremediumpriorityand four were of low priority. These risks can beseeninthematrixinFigure3.

Figure3-RiskMatrixofidentifiedrisksinProject1

6

Figure4representsdechangesthatoccurredtoeachriskprioritylevel.Greenarrowsrepresentthechanges caused by the response plans and blackarrows the changes due to the naturaldevelopmentoftheproject.

The four red dots represent the risks thatimpacted.Theseriskswere:

1.ProcessRisk:Thisriskwasidentifiedbecausetheteamthoughtthatitwaspossiblethatthefinalpowderpropertieswereoutoftheexpectedrangeand also because necessary utilities could fail.These causes for the risk were not correctlyidentified, so when the action plan (monitoring),was put to action, the triggers of the real causes,whichwereveryunlikely,werenotdetected.

2. Equipment not fit for purpose: The pipeusedintheprocesshadneverbeenusedbefore,sotherewasaprobabilityoffailing.Theteamdecidedthat the action plan (mitigation) was to test thepipebeforeusingit.Nevertheless,thepipedidnotwork on the process but this issue was rapidlysolvedwithnoimpact

3.HumanResourcesover-allocated:Theactionplanforthisriskwastoaccept,becauseitwasnotpossibletoapplyanyotherstrategy.Therewasnoimpactontheprojectthough.

4. Delayed arrival of raw materials: Acontingency plan was defined, which was, to usedifferent raw materials

incasetheorderedonesdidnotarrived.Itwassuccessfulandeliminatedtheimpactoftherisk.

Aftertheendof theproject, itwaspossibletomakesomeconclusionsregardingtheefficiencyofthe

methodology and what alterations werenecessary. These alterations are summarized intable2.

4

32

1

Figure4-Project1RiskMatrixevolution

7

Table2-AlterationsmadetothemethodologyafterProject1

Project 2: The client was a pharmaceuticalcompany that focuses on investigating oral anti-inflammatorydrugs.Theprojectexecutionphaseisexpectedtolast13days.Duringtheentireproject

life cycle, eight risks were identified and actionplans were defined. Four of the risks wheremedium priority, two of high priority and one oflow priority. This can be seen in Figure 5.

Figure6representsdechangesthatoccurredtoeachriskprioritylevelandtherisksthatimpacted

InitialVersion ObservationsofProject1 AlterationstobemadeRiskIdentificationdoneintheKickoffmeeting

ThetimedestinedtoriskmanagementduringtheKickoffmeetingwasnotenoughforriskidentification

RiskIdentificationdoneinaseparateriskmeeting

Askclienttoidentifyprojectrisks

Clientcouldonlyidentifyprocessrisks Showclienttherisksidentifiedandmanageprocessrisksthatareathreattotheprojectsobjectives

Monitormediumpriorityrisks

Somemediumpriorityriskscouldbeeasilybesolved/preventedbyapplyingminimalresources

Applyactionplanstomediumpriorityrisks,iftheydon’trequiretoomanyresources

Riskownersdevelopactionplansindividually

Developactionplansingroupwasefficient Developactionplansingroup,onthefirstweeklymeeting

Figure5-RiskMatrixwithidentifiedrisksinProject2

v v

v

v

v

v

v

8

TherisksthatimpactedduringProject2arerepresentedbythereddotsandtheywerethefollowing:

1. Domino effect: In the equipment schedule,theproductionofProject2wasscheduledwithouttime buffer between other activities from otherprojects. Meaning that production delays in thepreceding projects could delay project 2. Theactionplanforthisriskwastomonitorit.Itstriggerwasdetected,andalthough itwasnotpossible toprevent the risk from occurring, this could havehadabiggerimpactif itwasnotdetectedintime.Themonitoringof this riskpreventedothers fromimpacting.

2 Issues with rawmaterial approval: The APIfor this project was not approved at first, so theteamconsideredthattherewasahighprobabilitythatthetestingfortheapprovalfailedagain,asthe

problem seem to be in the testing method.TheactionplanwastousetheAPIunderquarantineifthe testing failedagain.Thismeans that the teamoptedbyproducingtheproductwithoutapproval,expectingittobeapprovedsoon.Thiscontingencyplandidnotworkbecauseitwasneverpossibletoprovethattheproblemwasinthetestingprocess.

3. Equipment no fit for purpose: The pumprequired for theprojecthadneverbeenusedbefore,soitcouldfail.Theactionplanwastochoose an alternative pump in case the firston failed. The problemwas that none of thepumpsworked.

Again, after the endof the project, itwaspossible to observe that the methodologyneeded some alterations. These aresummarizedintable3.

1

23

Figure3-Project2RiskMartrixevolution

9

Table3-AlterationsmadetothemethodologyafterProject1

InitialVersion ObservationsofProject2 AlterationsRisk Identification andanalysis done in a RiskMeeting

There is time to conduct riskidentification and analysis insomeKick-offMeetings

Risk Identification done during Kickoffmeeting if there is enough time, if not, itshould be done, or continued, in a RiskMeeting

Send checklist to teammembersbye-mail

No one fills the checklist beforethemeeting

Print the checklist and bring it to themeetingtobefilled

Use Risk Response table toguideresponseplanning

Risk response table is of no useand team members havedifficulties developing actionplans

Ask 9 pre-defined questions to guideresponseplanning

5.SurveyResults

In the end of both projects, a questionnairewith questions regarding the efficiency of themethodologywassenttothecoreteammembersfrom both Project 1and Project 2. Eight teammembersansweredthesurvey,anditwaspossibleto construct the graph shown in Figure 32, thatdemonstratestheaverageanswertoeachquestionand the range of answers. The participants wereaskedtochoosealevelofagreementfrom1to5.1beingstronglydisagreeand5stronglyagree. Theanswersgivenby the teams’members show that,in general, they agree that the methodologyaccomplishes the objectives it was designed tocover. Being simple, adaptable, not too timeconsumingandstillbeeffective.

Some participants who gave low scores tosomeofthequestionswereaskedaboutwhytheydid it. For the affirmation “The methodology iseffective in solving risks”, team members fromProject 2 seemed to think that the action plansimplementeddidnotsolvetherisks.Althoughthismighthavehappened,and itmightstillhappen inthefuture,thiscannotbeconsideredaflawofthemethodology, but of the team members’perception of how to solve the risks. Throughdocumenting the risks and action plans, itwill bepossible inthefuture,tocheckthemeverytimearepetitiveriskappearsandtherewillbenoneedtotry new action plans, as the effective ones willalreadyberegisteredandproventobesuccessful.Asforaffirmation“Themethodologyiseffectiveinsolving risks”, participants from Project 1 thoughthat some process risks were not identified. Thishappenedbecausetheseriskswereveryunlikelytooccur,andagain,inthefuture,theywillalreadybe

considered as risks, because their probabilityincreased after this occurrence. Themethodologyis a work in progress and it will become moreefficientasrecordsarebeingsavedandmorerisksarebeingmanaged.

Thefactthatthenecessityofriskmanagementisrecognizedandthattheoverallexperienceoftheparticipants was rated positively should indicatethat the implementation of themethodology willbewellaccepted.

6.ConclusionsandFuturework

ThroughthecharacterizationofHovioneandaresearch on existing techniques formanaging riskin pharmaceutical and R&D projects, it waspossible. to suggest a risk managementmethodology to be implemented in Hovione’sprojects.Thismethodologytookintoconsiderationthree main characteristics of the projects. First,because of the fact that Hovione’s R&D projectsare of short duration, the methodology toimplementhadtobesimpleandshort.Thismeansthat, the analysis phase of the risk managementmethodology should be qualitative and notquantitative, and be done using the risk matrix.Theentiremethodologywasdesignedtobesimpleandfasttoapply.Second,astherisksarerecurrentfrom project to project, the solution proposed toidentify risks was the checklist method, whichtakes advantage of the fact that risks repeatthemselves and, at the same time, contributes tothe simplicity of the methodology. Third, theprojectteamswerealreadyoverloadedwithwork,so,performingriskmanagementcouldnotbeseenas a toomuch of extra work or a waste of time,thus the need for a risk management briefing atthe beginning of the project. This briefing should

10

diminish the overall time consumed by riskmanagement.Table3summarizesthesteps, tools

andtechniquesandinwhichmeetingeachstepoftheriskmanagementmethodologyshouldhappen.

Table3-FinalRiskManagementMethodology

Phase Steps ToolsandTechniques Meeting

RiskPlanning 1. RiskBriefing PowerPointpresentation KoMideally,butiftimeisnotenough,RiskMeeting

RiskIdentification

2 RiskIdentification3 CauseIdentification´4 AssignmentofRiskOwners

ChecklistBrainstormingExcel“Checklist”sheet

KoMideally,butiftimeisnotenough,RiskMeeting

RiskAnalysis 5 QualifyProbability6QualifyImpact

Excel“Analysis”sheetBrainstormingºRiskMatrixQualitative Scale with 5levels

KoMideally,butiftimeisnotenough,RiskMeeting

RiskResponsePlanning

7 Develop action plans ingroup

8 Registerriskstrategy9 Establishdeadlines

BrainstormingExcel “Response Plan”sheet8questionset

1stWeeklyMeeting

RiskMonitoringandControl

10RegisterRiskStatus11.Definenewprobabilities12.Definenewimpacts13. Definenewactionplan

BrainstormingExcel “ Response Plan”sheet

Weeklyinternalmeetings

7.FutureWork

This dissertation contributes to the study of riskmanagement in pharmaceutical projects, however, thefindingsandresultsshouldbeviewedtakingintoconsiderationthelimitationsoftheworkdone.Thisincludestakingintoaccountthatonlytwocasestudyprojectswereused,andbothwerefromthesamecompany.Also,the fact that the second pilot project started before the end of the first one and that the secondimplementationonlytookplacemid-project,mayhavebiasedtheresultsofthesecondimplementation,anditisnotpossibletodistinguishifsuchresultswouldstilloccuriftheimplementationwasdoneasplanned.Thishappened due to Hovione's requirement of having the dissertations results at a specific date and no otherproject besides Project 2would be finished in time. As such, future studies should take into account theseconsiderations.Itisalsoworthconsideringthatthecasestudyprojectswereshorttermprojectsandthatthemethodology implementedwas developed taking that into consideration.Most pharmaceutical projects arelong-term, so thework developed in this dissertationmay not be themost appropriate to be applied in alltypesofpharmaceuticalprojects.Theselimitationscanbeseenasanopportunityforfurtherresearchinthisarea,includingconductingcasestudyresearchonalargernumberofR&Dpharmaceuticalprojects,preferablyfromdifferentcompanies.Futureworkcanprovideafertilegroundonwhichtovalidatetheresultsinvariouscontexts,evaluatingdifferenttypesofpharmaceuticalprojects,fromdifferentpharmaceuticalcompanies.

References

Kwak,Y.,&Dixon,C.K.(2008).Riskmanagementframeworkforpharmaceuticalresearchanddevelopmentprojects.InternationalJournalofManagingProjectsinBusiness,1(4),552–565.

Lavallee,R.Courtland(2010)PharmaceuticalandBiomedicalprojectmanagementinachangingglobalenvironment(1sted.).Hoboken,NJ:JohnWiley&Sons

Marle,F.,&Gidel,T.(2012).AMulti-CriteriaDecisionMakingProcessforProjectRiskManagementMethodSelection.InternationalJournalofProjectOrganizationandManagement,2(2),189-

11

223.

ProjectManagementInstitute(2017).AGuidetotheProjectManagementBodyofKnowledge(PMBOK®guide)(6nded.).NewtounSquare,PA.

Turner,J.R.(2009).TheHandbookofProjectBasedManagement:LeadingStrategicChangeinOrganizations.Manager(3rded.).London,UK:McGraw-Hill.

Wageman,S.W.(2004).Riskmanagementonresearchanddevelopmentprojects.AACEInternationalTransactionspp.RI71–RI76.

12