Deliverable D1.2 Current and Emerging Commercial Optical ...
D1.2 Criteria to assess how patient engagement ... · D1.2 Criteria to assess how patient...
Transcript of D1.2 Criteria to assess how patient engagement ... · D1.2 Criteria to assess how patient...
D1.2 Criteria to assess how patient engagement expectations are met
777450 - PARADIGM
Patients Active in Research and Dialogues for an Improved Generation of Medicines
WP1 – Defining stakeholders’ preferences, needs and expectations Task 1.2 Co-Prioritize minimal expectations for patient engagement across key stakeholders
Task 1.3: Develop a set of criteria to assess patient engagement practices and processes against minimal expectations
Lead contributor Maria Jose Vicente Edo (IACS) [email protected]
Lucia Prieto Remón (IACS)
On Behalf of the IACS Team
Other contributors Chi Pakarinen (The Synergist) (WP1 co-leads)
Alexandra Moutet (UCB) (WP1 co-leads)
Ana Diaz (AE)
Begoña Nafria (SJD)
Elisa Ferrer (EURORDIS)
Neil Bertelsen (HTA)
2
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Experts Panel for the Delphi on Patient Engagement in Setting Research Priorities
NAME SURNAME INSTITUTIONLara Bloom EUPATI
Helen Rochford-Brenan AlzheimerEurope
Violeta Stoyanova MedicinesEvaluationBoard(Netherlands)
Matt Bradley GSK
Malca Resnick Pfizer
Emmanuelle Logette AssociationScléroseTubéreusedeBourneville(ASTB)
Blanca Novella ComunidaddeMadrid
Pedro AbizandaSoler AlbaceteUniversityHospital
Ana Calpe FundacionSanJoandeDeu
Encarnación González FundacionSanJoandeDeu
Daniel O'Connor Medicines&HealthcareproductsRegulatoryAgency(UK)
Heidrun HILDEBRAND Bayer
Paola Mosconi IstitutodiRicercheFarmacologicheMarioNegriIRCCS
Camilla KroghLauritzen LEOPharma
Kevin McCarthy European Commission-EuropeAid Development &Cooperation(DEVCO)
Tomáš Doležal ValueOutcomesCompany
Joseph Collins Sanofi
Daria Julkowska AgenceNationaleRecherche/E-Rare
Experts panel for the Delphi on Patient Engagement in the Designing of Clinical trials
NAME SURNAME INSTITUTION
Birgit Bauer EUPATI
Maria Piggin EUPATI
Natasha Ratcliffe Parkinson'sUK
Heidi Muller Johnson&Johnson
Gautam Gupta Pfizer
FrankMichael Amort EATG
3
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Tatiana Massarrah GregorioMarañónHospital
Stefanie Breitenstein Bayer
Victoria Thomas NICE
Rob Camp EURORDIS
Ana RuizCasado PuertadeHierroUniversityHospital
Nieves Tejeda FundacionSanJoandeDeu
Sofia Junquera FundacionSanJoandeDeu
Pamela Tenaerts ClinicalTrialsTransformationInitiative
Caridad Pontes CatSalut
Fabian Somers UCB
Jeannett Dimsits NovoNordisk
Juan Erviti ServicioNavarrodeSalud
Cynthia Grossman FasterCures
Sharareh Hosseinzadeh Novartis
Emilie Voltz Novartis
Annette Christoph Grunenthal
Debra Michaels DIA
Inge Schalkers Hartenraad
Traceann Rose Children'sTumorFoundation(CTF)
Experts panel for the Delphi on Patient Engagement in Early Dialogues with HTA and Regulators
NAME SURNAME INSTITUTION
Chris Roberts AlzheimerEurope
Jayne Goodrick AlzheimerEurope
Jenny Wilson GSK
Alicia Granados Sanofi
Adam Heatfield Pfizer
Marie-Pierre Gagnon UniversitéLaval
François Houÿez EURORDIS
Dimitrios Athanasiou EURORDIS/WorldDuchenneOrganization/UPPMD.
Encarnación CruzMartos ComunidaddeMadrid
Claire Champeix Eurocarers
Beatrix Friedeberg AmgenB.V.
Sandra Nestler-Parr ISPOR
4
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Deborah Morrison NICE
Nathalie Bere EMA
Luciana Ballini RegioneEmilia-Romagna
Michelle Mujoomdar CADTH
Isabelle Huys KatholiekeUniversiteitLeuven
Nigel Cook NovartisPharma&IMIPREFER
Due date 30/05/2019 Delivery date 20/05/2019 Deliverable type R Dissemination level PU
Description of Action Version Date V1.1 XXX
5
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Table of Contents
1. Publishable Summary.........................................................................................................................................8
2. Introduction............................................................................................................................................................10
3. Methodology..........................................................................................................................................................10
4. Results.....................................................................................................................................................................14
4.1. Representativeness of the three Delphi Expert Panels...........................................................14
4.2. Delphi on Patient Engagement in Setting Research Priorities............................................15
4.3. Delphi on Patient Engagement in Designing Clinical trials....................................................17
4.4. Delphi on Patient Engagement in Early Dialogues with HTA & Regulators................18
5. Discussion..............................................................................................................................................................20
6. Proposal for Utilising the Criteria..............................................................................................................22
Annex 1. List of sources...............................................................................................................................................24
Annex 2. Delphi questionnaires...............................................................................................................................30
Annex 3. Panel stakeholder profile........................................................................................................................42
Annex 4. Glossary for the Delphi in PE in Medicine R&D lifecycle.....................................................45
Annex 5. Results of the Delphi on PE in Setting Research Priorities................................................51
Annex 6. Results of the Delphi on PE in Designing Clinical Trials......................................................65
Annex 7. Results of the Delphi on PE in Early Dialogues........................................................................79
6
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Document History
Version Date Description V0.1 01/04/2019 First Draft V0.2 08/04/2019 Comments from experts panels V0.3 06/04/2019 Comments from first round reviewers V0.4 20/05/2019 Draft edited based on comments V1.0 30/05/2019 Final Version
7
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Partners of the PARADIGM Consortium are referred to herein according to the following codes: - EPF. EUROPEAN PATIENTS FORUM (Luxembourg) – Project Coordinator - EURORDIS. EUROPEAN ORGANISATION FOR RARE DISEASES ASSOCIATION (France) - EATG. EUROPEAN AIDS TREATMENT GROUP (Germany) - AE. ALZHEIMER EUROPE (Luxembourg) - AIFA. AGENZIA ITALIANA DEL FARMACO (Italy) - HTAi. HEALTH TECHNOLOGY ASSESSMENT INTERNATIONAL (Canada) - IACS. INSTITUTO ARAGONES DE CIENCIAS DE LA SALUD (Spain) - FSJD. FUNDACIO SANT JOAN DE DEU (Spain) - VU-ATHENA. STICHTING VU (The Netherlands) - UOXF-CASMI. THE CHANCELLOR, MASTERS AND SCHOLARS OF THE UNIVERSITY OF
OXFORD (United Kingdom) - EFGCP. EUROPEAN FORUM FOR GOOD CLINICAL PRACTICE (Belgium) - SYNERGIST. THE SYNERGIST (Belgium) - SYNAPSE. SYNAPSE RESEARCH MANAGEMENT PARTNERS SL (Spain) - EFPIA. EUROPEAN FEDERATION OF PHARMACEUTICAL INDUSTRIES AND
ASSOCIATIONS (Belgium) - Project Leader - MSD Corp. MERCK SHARP & DOHME CORP (United States) - UCB. UCB BIOPHARMA SPRL (Belgium) - ABPI. THE ASSOCIATION OF THE BRITISH PHARMACEUTICAL INDUSTRY (United Kingdom) - AMGEN. AMGEN LIMITED (United Kingdom) - BAYER. BAYER AKTIENGESELLSCHAFT (Germany) - GSK. GLAXOSMITHKLINE RESEARCH AND DEVELOPMENT (United Kingdom) - GRT. GRUENENTHAL GMBH (Germany) - JANSSEN. JANSSEN PHARMACEUTICA NV (Belgium) - LILLY. Eli Lilly and Company Limited (United Kingdom) - LUNDBECK. H. LUNDBECK AS (Denmark) - MERCK. MERCK KOMMANDITGESELLSCHAFT AUF AKTIEN (Germany) - NOVO NORDISK. NOVO NORDISK A/S (Denmark) - PFIZER. PFIZER LIMITED (United Kingdom) - ROCHE. F. HOFFMANN-LA ROCHE AG (Switzerland) - SERVIER. INSTITUT DE RECHERCHES INTERNATIONALES SERVIER (France) - VFA. VERBAND FORSCHENDER ARZNEIMITTELHERSTELLER EV (Germany) - SARD. SANOFI-AVENTIS RECHERCHE & DEVELOPPEMENT (France) - NOVARTIS. NOVARTIS PHARMA AG (Switzerland) - COVANCE. COVANCE LABORATORIES LTD (United Kingdom) - ALEXION. ALEXION SERVICES EUROPE (Belgium)
§ Consortium. The PARADIGM Consortium, comprising the above-mentioned legal entities § Consortium Agreement. Agreement concluded amongst PARADIGM participants for the implementation
of the Grant Agreement. Such an agreement shall not affect the parties’ obligations to the Community and/or to one another arising from the Grant Agreement.
8
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
1. Publishable Summary
ThisreportdescribestheworkthatwasdoneaspartofPARADIGM’s1workpackage1(WP1),which aimed to identify and define stakeholders’ preferences, needs and expectations inpatient engagement (PE) through an online survey and focus group consultations thatwereconductedin20182.
The objectives of thisworkwere two-fold: firstly to prioritise those needs and expectationsidentified,andsecondly,withthehelpoftopicexperts,turnthoseprioritisedneedsintoasetofcriteriaforagoodPEactivity.
TheDelphimethodology3wasused as themain tool to reach these objectives. Three expertpanelgroupswereformedtomatchthethreedecision-makingpointsParadigmprojectfocuseson[settingresearchpriorities(SRP),thedesignofclinicaltrials(DCT)andearlydialogueswithHealthTechnologyAssessment(HTA)bodiesandregulators(ED)]4.
Each expert group went through two online survey rounds, and a subsequent meeting inBrusselswhereexperts(whocompletedbothonlinerounds)discussedtoreachconsensusonthe final criteria toassessPEpractices.The format,procedureandmethodologyof all threemeetingswereidentical.
Each group produced a set of weighted criteria and categories for their dedicated decision-makingpoint.TheweightsallocatedrepresenttherelativeimportanceofthecriteriainthesettoassessPEpractices.
Keyoutcomes:
TheoutcomesofthethreeDelphigroupsshowedsomesimilarities.Someconvergencecanbeseenatthelevelofthemesasaimsandobjectives,andselectingtherighttargetpopulationtoengagewith scored as top criteria, and sustainabilitywhich scored the lowest in twoDelphigroups (SRP andDCT),while the thirdDelphi group (ED) droppedsustainability in its finalmeeting.
Some criteria were debated more than others due to differences in interpretations of theterminology.Forexample,conflictofinterest(COI)wasunderstoodas‘managing’,‘disclosing’or‘avoiding’bythedifferentstakeholders.Additionally,somebiasininterpretationcouldhavealsocomeduetotheEUandUSperspective.1 PARADIGM is an Innovative Medicines Initiative (IMI) - a public-private partnership co-funded by the European Commission and EFPIA (the European Federation of Pharmaceutical Industries (and Associations). For more information about IMI projects, please visit: https://www.imi.europa.eu/about-imi. For more information about PARADIGM, please visit www.imi-paradigm.eu 2 To learn more about WP1’s online survey and focus group consultations, please see the report: https://imi-paradigm.eu/wp-content/uploads/2019/03/D1.1-Survey-analysis-report.pdf 3 The Delphi methodology originates from the US and is meant to estimate the likelihood of future events. (RAND Corporation, 2019. Internet source, available here: https://www.rand.org/topics/delphi-method.html). In health research, it is widely used as a tool to obtain consensus on group opinion (Trevelyan, 2015. European Journal of Integrative Medicine. Available here: https://www.sciencedirect.com/science/article/abs/pii/S1876382015300160) 4 , “PARADIGM focus[es] on three decision-making points, during which integration of the patient perspective is critical (if not essential) for the medicine lifecycle” as requested by the IMI call (PARADIGM, 2018). More information here: https://www.imi.europa.eu/projects-results/project-factsheets/paradigm.,
9
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
TheseresultswillbeusedinotherPARADIGMworkpackages(i.e.WP2)inordertoco-createquestions for thegapanalysis tool. Inaddition, theauthorsproposeaway the resultsof theDelphi couldbeoperationalizedasa tool toassessPEpractices ineachof the three specificdecision-making points to determine whether they meet the expectations for a good PEpractice.
10
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
2. Introduction
PARADIGM, which stands for Patients Active in Research and Dialogues for an ImprovedGenerationofMedicines,isanEUprojectfundedbytheInnovativeMedicinesInitiative(IMI)andtheEuropeanFederationofPharmaceuticalIndustriesandAssociations(EFPIA).Itisa30-monthprojectthatinvolves34partners.ItsmissionistodevelopaPatientEngagement(PE)framework that allows structured, meaningful, sustainable and ethical patient engagementfocused on the three key decision-making points in the lifecycle of amedicine, i.e. researchpriority setting, design of clinical trials and early dialogue with regulators and HealthTechnology Assessment (HTA) bodies. The framework will include recommendations onprocesses,tools(withthecreationoftemplates)andmethodstomeasureanddemonstratetheaddedvalueofinnovativeandeffectiveapproachestopatientengagement.
This work is part of work package 1 (WP1: Defining stakeholders’ preferences, needs andexpectations) deliverables and is a continuation of the online survey and focus groupconsultations created in 20185 that intended tomap out the range and frequency of needs,expectations and desired outcomes for PE. To achieve a consensus for stakeholders’expectationsandneeds,aDelphimethodologywaschosentoaddressthelevelsofspecificityandin-depthinformationobtainedfromthesurveyandfocusgroups.TheDelphiexpertpanelsdetermined which priority and relative value (weights) each need and expectation for PEwouldgetineachofthethreedecision-makingpoints;howtheyrelatetoeachothertoform“goodpractice”;andwhicharemorelikelytopredictdesiredoutcomes.
3. Methodology
The Delphi methodology (Fig. 1) was used to firstly prioritize the stakeholders’ needs andexpectationsandthentoturnthemintoasetofcriteriatoassessPEpracticesandprocesses.
TheDelphiprocess isaconsensustechnique,widelyused inhealthcareresearch6 thatmeetsthe requirements of scientific methods7. This technique helps to identify and prioritise theexpectations and preferences regarding a specific issue. The results obtained from theinteraction of a structured group of experts aremore solid than those from individuals8. Inusingthistechnique,allparticipantsacceptoratleasthavenostrongdisagreementsregardingthe criteria and categories9 developed. Within PARADIGM, the Delphi modified technique
5Formoreinformation,see thereportfromthesurveyanalysishere:https://imi-paradigm.eu/wp-content/uploads/2019/03/D1.1-Survey-analysis-report.pdf6EubankBH,MohtadiNG,LafaveMR,WileyJP,BoisAJ,BoormanRS,etal.UsingthemodifiedDelphimethodtoestablishclinicalconsensusforthe diagnosis and treatment of patientswith rotator cuff pathology. BMCMedResMethodol [Internet]. 2016;16(1):1–15. Available from:http://dx.doi.org/10.1186/s12874-016-0165-87HutchingsA,RaineR,SandersonC,BlackN.Acomparisonofformalconsensusmethodsusedfordevelopingclinicalguidelines.JHealServResPolicy.2006;11(4):218–24.8 BlackN, MurphyM, LampingD,McKeeM, Sanderson C, Askham J, et al. Consensus developmentmethods: A review of best practice increatingclinicalguidelines.JHealServResPolicy.1999;4(4):236–48.9CriteriarefertotheattributesofPEpractices.Categoriesareelementsthatdescribethecharacteristicsandnuanceofeachcriterioninwhich
11
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
gatherstherelevantissuesinPEwithineachofthethreedecision-makingpointsinmedicinesdevelopment.
Figure1:OverviewofthePARADIGMDelphiprocess
This technique uses qualitative and quantitative data sequentially by combining two onlinesurveying rounds to elicit responses in a systematic manner, and a subsequent meeting inperson that allows expert panels to interact, discuss, debate and justify their viewpoints.Followingthetwoon-linesurvey,afacetofacemeetingwasheldtoachieveconsensusonasetofcriteriatoassessPEpracticesinmedicinesdevelopment.
Usingthis technique,expectationsandpreferences forPEcanbe identifiedandprioritisedinthe three specific points in time. The technique is based on the assumption that theconvergence of “good practice” elements and their relative value, steered by structuredinteractionamongagroupofindividualexpertsonPEprocesses,yieldsmoreaccurateresultsthan those from individuals; thus the consensus produced ismore likely to lend credibilitywithineachofthedecisionmakingpointinmedicinedevelopment.
Procedure
1-DevelopingDelphiQuestionnaires
First,theDelphiquestionnairesweredevelopedforthetwoonlinesurveyingrounds(Fig.1).Aseparatequestionnairewascreatedforeachofthethreedecision-makingpoints.Thesequestionnaireswerebuiltonthebasisoftheinformationobtainedfromvarioussources:
- ThepreviousworkcarriedoutwithinWP1:
• ResultsfromtheSurveyanalysisreportonstakeholder-specificpreferences,needsandexpectationscarriedoutwithinWP1(task1.1)
isincluded
12
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
• Three focus Groups by Alzheimer Europe, San Juan de Deu (paediatricgroup)andHTAbodies.
• Patient Focus Medicine Development (PFMD) PE Quality Guidance(https://patientfocusedmedicine.org/)
• Literature review fromNational,Europeanand Internationalpublic andprivateinstitutionsandorganisations,academics,regulatorybodies,HTAbodies,patient’sorganisationsetc.ThetypesofdocumentsconsultedandreviewedwerethoserelatedtoPEinthethreedecision-makingpoints10.
Followedthesearch,allgatheredinformationwasanalysed;thoseelementsdeemedrelevantfor the questionnaire were retrieved and organized into criteria with their categories, toanticipate the potential structure of the questionnaires for assessing PE practices11. ThePARADIGMconsortiumwasinvolvedthroughoutthedevelopmentofthequestionnaires.
2-Experts’recruitmentforthethreeDelphipanels
The recruitment process began (see Figure 1) at the same time as the questionnaireswerebeing constructed. The objective was select participants who share and bring a variety ofperspectivesandexpertiseandhavethepotentialtoproviderelevant,keyandrichinformationaboutPEforeachoftheDelphi.Toselecttheexperts,firsttheDelphiteam(WP1leadersandIACS) defined stakeholder and participant profiles for each panel. Themain requirement toparticipate was to have recognised expertise, experience and knowledge within their topicgroup12.Additionallytheselectioncriteriatookintoaccountthetypeofstakeholders,aswellascountryandgenderdiversity,toavoidbiasandsupportglobalapplicability.
For the recruitment process (Figure 2), the PARADIGMconsortiumwas invited to propose candidates fromwithin oroutside the consortium. Members of the PARADIGMInternational Liaison Group (PILG) were also invited toparticipate.Toensureadequaterepresentativenessofviewsaswell as to complement the level of expertise in the expertpanels,theDelphiteamrecruitedotherrelevantexpert(outsidethe Consortium) whose profile was still missing ensurerepresentativenessinthepanelgroups.
Figure2:Expertsrecruitmentandoutreachprocess
10Seeannex1forthelistofsources11Seeannex2forthequestionnaires
12SeeAnnex3forthepanelstakeholderprofile
13
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
3-Delphifirstandsecondonlineround(R1andR2)
Thefinishedquestionnaireswerecreatedanddistributedonanonlineplatformandaccesstoitwasgiventoexpertpanellistsindividually(seefigure1).Thisplatformprovidesparticipantsthesurveyinformationinreal-time(i.e.howmanyhavealreadyscoredeachcategoryandthecorrespondingmedianscore).Duringthesurveyperiods,thesystemalloweduserstostopandresumethetask(includingmodifyingthepreviousresponses)asmanytimesastheyneeded.Participants could also add comments and suggestions for the questions presented in theplatforminbothrounds.Thesecommentswereusedtopreparethefinalmeetingandsuggestto the experts possible merging and rephrasing of some of the criteria and categories. Aglossarywasincluded,tohelpexpertsunderstandtheconceptsusedinthequestionnaires13.
During the first and second rounds, experts had to rate each question based on theirknowledgeandexpertiseinPE.Concretelyduringthefirstround,expertswereaskedtoassesshowrelevantwaseachquestionwhenassessingPEpracticesusingaLikert-scale14[from1-3(not relevant at all) to 9 (highly relevant)]. The Mean, Median, Coefficient Variation (CV),Interquartile range (IQR), Quartile 1 (Q1) were calculated along the vote distribution.Agreementwasreachedandkeptforprioritysettinginthesecondroundforthosequestionswhen Q1≥7, IQR ≤ 1, CV <20%; andwhen themedian and the votes fittedwithin the samebracket (agreement converged around ‘relevant’) by the experts. Questions were droppedwhenQ1≤6, IQR≥2,CV<20%andtherewasagreementwhenthemedianandvotes fittedwithin the irrelevanceornot clearly relevant bracket.The restof thequestions thatdidnotreachanyagreementwerekeptforreassessmentforthesecondround
Forthesecondround,participantswereaskedtoindividuallyre-scorethecategories.Inthisround,questionsthatexpertshadagreedonasrelevantforassessingPEpracticesduringthefirst round were again presented for prioritisation (from 1=lowest priority to 9= highestpriority). The analysis focused on the Mean, Median, Coefficient deviation (CV), andInterquartilerange(IQR),Quartile1(Q1)aswellasthevotedistributiontodeterminewhetherexperts agreed on the level of priority (1-3 low priority; 4-6 moderate priority; 7-9 highpriority).Questionsthatreachednoagreementinthefirstroundwerepresentedagaininthesecondroundtotheexpertstore-rankusingtheproposedrelevancescalefromthefirstround(from1=’notrelevantatall’to9=‘highlyrelevant’).
4-Meetinginperson
After the twoonline survey rounds finished,only thoseexpertswhocompleted the first andsecondroundwereeligibleandinvitedtoparticipateinthethirdroundmeetinginBrusselstoreachconsensusaboutthefinalcriteriatoassessPEpractices.Therewasonespecificmeetingorganised foreachDelphipanelgroup. The format,procedureandmethodologyofall threemeetingswereidentical.
13SeeAnnex4fortheglossary14SeeAnnex4fortheglossary
14
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
A facilitator (from the IACS who was not part of the expert panel), helped conduct thediscussionsfollowingastructuredconsensusmethodology.Tworapporteurs(fromIACSwhowerenotpartof thepanelofexpertseither)providedsupport inrecordingvotingprocessesandmodificationsinphrasingandallocationaccordingtoexperts’consensus.Duringthegroupdiscussion,expertswereabletodiscussandelucidatetheirpointofview.
Theobjectivesofthesemeetingswereto
- identifysimilaritieswithinthedifferentcriteriaandcategories,
- resolveeventualredundancies,decidethefinalsetofcriteriaandcategories,
- allocatethecorrespondingweighttoeachcriteriaandcategory.
Inthefirstpartoftheprocess,questionsprioritisedduringthesecondroundwerepresentedto the experts; those questions that failed to reach agreement during the first and secondrounds were also presented to experts who were asked to decide if the questions wererelevanttoassesspracticesofPE,andwhethertokeepordropthem.Merging,rephrasingandreallocatingthecategoriesacrosseachcriterionwereallowedatthisstage.Oncefullconsensuswasreached,thepanelspenttheremainderofthemeetingvoting.
In thesecondpart, theexpertsvotedonthe finalhierarchyof theelements for thedecision-makingpointtoobtainrelativeweightsfortheconsolidatedcriteriaandcategories.
Expertsproceededtoprioritisethecategoriesbyweightingthembysharing100pointsamongthem. The final weight for each category was calculated by averaging total points by thenumberofvoters(dividingtotalpointsbynumberofexpertsandmultiplyingby100,sothesumofcategories’weightwithinacriterionwasalways100).
Theexpertswerethenaskedtoprioritiseandweightthecriteriainthesameway-bysharing100pointsamongallcriteria.Inbothcases,theprocesswasanonymousandprioritisationwasconductedusingadocumentthatincludedallfinalcategoriesandcriteriapreviouslyagreedonbytheexperts.
TheexpertsvotedonthefinalhierarchyofelementsofgoodPEpracticesforthecorrespondingdecision-makingpoint
4. Results4.1. RepresentativenessofthethreeDelphiExpertPanels
Thecompositionof theDelphi expertpanels isdescribed inFigure3.Themainrequirementwasthat theparticipantsholdrelevantknowledgeandexperience in the fieldofPEtheyareinvolvedintohavethepotentialtoproviderelevantinformationaboutpatientengagement.
15
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Partic ipants Initially Round 1
Round 2
Round 3 F2F Profiles in the F2F meeting
DE LPHI
S etting research priorities
24 20 16 10
Patient/Patient advocate/ Patient representative, Regulator,Pharma company, professional, Research Funder (public orprivate), Healthcare professionals, Academic, researcher,other
Design of clinical trials 31 29 26 18
Patient/Patient advocate/ Patient representative, Academic,researcher, Healthcare professionals , Pharma companyprofessional, HTA Body, Research Funder (public orprivate), other
Early Dialogues
with HTA and Regulators
26 20 18 12Regulator, Pharma company professional, Academic,researcher, Patient/Patient advocate/ Patient representative,HTA Body, other
TOTAL 81
Figure3:RepresentativenessoftheDelphiexpertsgroups
Nospecific informationwas collected regarding the specific reasons for thedropoutsduringtheprocess. However, somereportedtobeunable toattend themeeting inBrusselsdue toworkcommitments. Someothersinformedthatthereasonfortheirdroppingoutwasduetotheir institution rules and regulations; they were not allowed to attend any meeting withprivatecompaniespresenttoavoidpotentialconflictofinterest.
The reliability of the results could be compromised if many experts drop out of the studybeforeitiscompleted,puttingatrisksomestakeholders'representativeness.Accordingly,theDelphi team analysed the dropouts in each of the Delphi rounds to check for stakeholders’participation tomake sure that representativenesswas not compromised. It was concludedthat,despitethedropouts,responseratehasremainedhighandthedropoutsdidnotaffectanystakeholdergroups’representativeness.ThusthefinalresultsoftheDelphicanbeconsideredvalidwiththeexistingparticipation.
4.2. DelphionPatientEngagementinSettingResearchPriorities
ExpertpanelfortheDelphiinPEinSettingResearchPriorities
Thesocio-demographiccharacteristicsof therespondents in thisDelphi for thethreeroundsaredisplayedinFigure4.
Figure4:DemographicoftheDelphiexpertsinthegroupforsettingresearchpriorities
16
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Age (years)
Figure 4: Socio-demographic characteristics of the participants for the Delphi in SRP
Round 1
Gender
Women: 13 (65%) Men: 7 (35%)
Nº participants invited
24
Total responses
20 (84%) < 25: 1 (5%)35-44: 4 (20%)45-54: 10 (50 %)55-64: 4 (20 %) >65 : 1 (1 %)
Round 2 Women: 11 (68,7%)
Men: 5 (31,2%) 20 16 (80%) < 25: 1 (6,25%)35-44: 2 (12,5%)45-54: 9 (56,2 %)55-64: 3 (18,7 %) >65 : 1 (6,2 %)
Round 3F2F meeting
Women: 6 (60%) Men: 4 (40%)16 10 (62,5%) < 25: 1 (10%)
35-44: 2 (60 %)55-64: 1 (10 %)
DELPHISetting Research Priorities
Attendance
Country origen: Denmark, Spain, Germany, Ireland, Italy, Netherlands, USA, France, Czech Republic and United Kingdom
Round1(Online)–RelevanceAssessment
Thefirstroundlastedthreeweeks(November12th2018–December3rd2018).
Fromtheinitial56categories[49categoriesand7PEimpact(PEI)sub-categories]thatmadeupthelisttobeassessed,agreementwasreachedon20categories(17categories+3PEIsub-categories,(table2inAnnex5).Noconsensuswasreachedon32categories(28categoriesand4PEIsub-categories),(table3inAnnex5)and4weredropped,(table4inAnnex5).
Round2(Online)–RelevanceAssessmentandPrioritization
Thesecondroundlastedthreeweeks(January1st2019–January18th2019).
Regardingthe32categories(28categories+4PEIsub-categories)onwhichnoconsensuswasreachedinthefirstround,7(5categories+2PEIsub-categories)weredeemedrelevant(table5inAnnex5).
Asfortherestofthecategories,in14categories(13categories+1PEIsub-categories)stillnoconsensuswasreachedaboutthelevelofrelevance,sotheywereretainedfordiscussionattheface-to-facemeeting(table6inAnnex5)and11(10categories+1PEIsub-categories)weredropped(table7inAnnex5).
17
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Figure 5: Socio-demographic characteristics of the participants for the Delphi in DCT
Round 1
Gender
Women: 22 (75,8%) Men: 7 (24,1%)
Nº participants invited
31
Total responses
29 (93,5%)
< 25: 1 (3,4%)25-34: 2 (6,8%)35-44: 5 (17,2%)45-54: 18 (62,06%)55-64: 2 (6,8%) >65 : 1 (3,4%)
Round 2 Women: 20 (76,9%)
Men: 6 (23,07%) 29 26 (89,6%)
Round 3F2F meeting
Women: 6 (60%) Men: 4 (40%)26 18 (69,2%)
DELPHIDesignig Clinical Trials
Attendance
Country origen: Germany, Belgium, Spain, Austria, Denmark, Canada, Netherlands, USA, France and United Kingdom
< 25: 1 (3,8%)25-34: 2 (7,6%)35-44: 5 (19,2%)45-54: 17 (65,3 %)55-64: 1 (3,8%)
Age (years)
< 25: 1 (5,5%)25-34: 1 (5,5%)35-44: 4 (22,2%)45-54: 11 (61,11%)55-64: 1 (5,5%)
Fromthe20categories(17categories+3PEIsub-categories),agreedasrelevantinRound1,14were considered a high priority and 5were consideredmediumpriority and 1was lowpriority onRound2. Theywere kept for discussion at the face-to-facemeeting. The scoresobtainedforeachcategoryandthecorrespondingcriteriaaresummarizedintable8inAnnex5.
Round3(Meetinginperson)–DiscussionandFinalPrioritization
PEinSettingResearchPrioritiesExpertPanel
Theexpertpanel tookpart in round3meeting in twohalfdays inBrussels.The final setofcriteriawiththeirweightsrecommendedforassessingPEprocessinsettingresearchprioritiesaredetailedinTable9inAnnex5.
4.3. DelphionPatientEngagementinDesigningClinicaltrials
ExpertpanelfortheDelphiinPEinDesigningClinicalTrials
Thesocio-demographiccharacteristicsof therespondents in thisDelphi for thethreeroundsaredisplayedinFigure5
18
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Figure5:DemographicoftheDelphiexpertsinthegroupfordesigningclinicaltrials
Round1(Online)–RelevanceAssessment
Thefirstroundlastedthreeweeks(November12th2018–December3rd2018).
Fromtheinitial58categories[48categories+10PEimpact(PEI)sub-categories]thatmadeup the list to be assessed, agreement was reached on the relevance of 26 categories (21categories+5PEI sub- categories), (table10 inAnnex6).Noconsensuswas reachedon22categories(19categories+3PEIsub-categories),(table11inAnnex6)and10(8categories+2PEIsub-categories)weredropped,(table12inAnnex6).
Round2(Online)–RelevanceAssessmentandPrioritization
Thesecondroundlastedthreeweeks(January1st2019–January18th2019).
Forthissecondround,expertssuggestedincorporatingtwonewcategories(1categoryundertheCriterion7:Involvementandparticipationand1asPEIsub-category;bothareindicatedinthe tables) as they considered important and they were not initially included in thequestionnaireofthefirstround.
Regardingthe22categories(19categories+3PEIsub-categories)onwhichnoconsensuswasreached in the firstround,16 (13categories+3PEI sub- categories)weredeemedrelevant(table13inAnnex6).
Asfortherestofthecategories,in2categoriesand1ofthenewcategoriesstillnoconsensuswasreachedaboutthelevelofrelevance,sotheywereretainedfordiscussionattheface-to-facemeeting(table14inAnnex6)and4categoriesweredropped(table15inAnnex6).
Fromthe26categories(21categories+5PEIsub-categories),agreedasrelevantinRound1,12wereconsideredashighpriorityand11wereconsideredmediumpriorityand3waslowpriorityonRound2(table16 inAnnex6). Theywerekept fordiscussionat the face-to-facemeeting.
Round3(Meetinginperson)–DiscussionandFinalPrioritization
PEinDesigningClinicalTrialsExpertPanel
Theexpertpanel tookpart in round3meeting in twohalfdays inBrussels.The final setofcriteriawiththeirweightrecommendedforassessingPEprocessinindesigningclinicaltrialsaredetailedinTable17inAnnex6.
4.4. DelphionPatientEngagementinEarlyDialogueswithHTA&Regulators
ExpertpanelfortheDelphiinEarlyDialogueswithHTAandRegulators
19
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Figure 6: Socio-demographic characteristics of the participants for the Delphi in ED with HTA and Regulators
Round 1
Gender
Women: 14 (70%) Men: 6 (30%)
Nº participants invited
26
Total responses
20 (76,9%)25-34: 1 (5%)35-44: 4 (20%)45-54: 10 (50%)55-64: 5 (25%)
Round 2 Women: 13 (72,2%)
Men: 5 (27,7%) 20 18 (90%)
Round 3F2F meeting
Women: 9 (75%) Men: 3 (25%)18 12 (66,6%)
DELPHIEarly Dialogues with HTA and R
Attendance
Country origen: United Kingdom, Spain, Italy, Canada, France, Greece, Belgium and Sweden
Age (years)
25-34: 1 (5,5%)35-44: 3 (16,6%)45-54: 9 (50%)55-64: 5 (27,7%)
25-34: 1 (8,3%)35-44: 3 (25%)45-54: 4 (33,3%)55-64: 4 (33,3%)
Thesocio-demographiccharacteristicsoftherespondentsinthisDelphi,forthethreerounds,
aredisplayedinFigure6
Round1(Online)–RelevanceAssessment
Thefirstroundlastedthreeweeks(November12th2018–December3rd2018).
From the initial 62 categories [51 categories + 11 patient engagement impact (PEI) sub-categories]thatmadeupthelisttobeassessed,agreementwasreachedontherelevanceof30categories (23 categories + 7 PEI sub- categories) (table 18 inAnnex 7).No consensuswasreachedon20categories(18categoriesand2PEIsub-categories)(table19inAnnex7)and12weredropped(table20inAnnex7).
Figure6:DemographicoftheDelphiexpertsinthegroupforearlydialogueswithregulatorsandHTA
Round2(Online)–RelevanceAssessmentandPrioritization
Thesecondroundlastedthreeweeks(January1st2019–January18th2019).
20
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Regardingthe20categories(18categories+2PEIsub-categories)onwhichnoconsensuswasreachedinthefirstround,7(6categories+1PEIsub-categories)weredeemedrelevant(table21Annex7).
Asfortherestofthecategories,in7categories(6categoriesand1PEIsub-category)stillnoconsensuswasreachedaboutthelevelofrelevance,sotheywereretainedfordiscussionattheface-to-facemeeting(table22inAnnex7)and6categoriesweredropped(table23inAnnex7).
Fromthe30categories(23categories+7PEIsub-categories),agreedasrelevantinround1,9were considered a high priority and 16 were considered medium priority and 5 was lowpriorityinthisroundandwerekeptfordiscussioninroundthree(table24inAnnex7).
Round3(Meetinginperson)–DiscussionandFinalPrioritization
PEinEarlyDialogueswithHTA&RegulatorsExpertPanel
Theexpertpaneltookpartintheround3meetingintwohalfdaysinBrussels.ThefinalsetofcriteriawiththeirweightrecommendedforassessingPEprocessinearlydialogueswithHTAandRegulatorsaredetailedinTable25inAnnex7.
5. Discussion It isacknowledgedthat thedifferences in theoutcomes(Table1)of thethreeDelphigroupsresult from differences in interpretation of the questions by the expertswithin each group.Thus,all threeDelphioutputsarenotnecessarilydirectlycomparable(i.e.eachexpertpanelgroup and their questionnaires were different for each decision making point), but keysimilaritiesalsoexist.
Table1:SummaryoffinaloutputfromthethreeDelphipanelgroups
Some of the key findings discussed throughout the face-to-face meetings as follows. Someconvergence can be seen in the outputs of the three Delphi groups. For example, Aims and
21
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Objectivesofapracticerankedhigh,whichshowsthegeneralagreementbytheexperts thatthiscriterionisrelevantforagoodPEpractice.TheaimsandobjectivesshouldbeagreedonandunderstandablebyallstakeholdersinvolvedinthePEpractice.Atthisstageofthedesign,thePEpracticeshouldalsocontemplatetodefineandagreeontherolesandresponsibilitiesamong all participants and to include mechanism to ensure that that all participantsunderstandeachother’srolesandresponsibilities.
Foralldecision-makingpoints, therewasamajordiscussionabout thedifficulty inselectingtherightpatients(“…involvepatientsasindividualstoonoonlypatients’organisations”),withspecific characteristics and circumstances to engagewith, which reflect the diversity of thetargetpopulation.Establishingaclearmethodtoselectpatientswasalsodeemedrelevanttoensuretheengagementisnotmeaninglessandincomplete.
Establishingaregularfeedbacksystemtoinformpatientsaboutoutcomesorchangesthatmayhappen ineachphaseof the involvementwasdebatedduring themeetings.However, in thediscussion expertspointedout the difficulty sometimesof giving feedbackwithout breakingconfidentiality requirements, due to the nature of some information. Itwas alsohighlightedthat patients should be able to provide feedback on the information received related to theresults, documents, final output, etc. The experts agreed that feedback is essential after thecompletion of the process and that giving regular feedback closes the loop of patients’involvementintheimplementationofapractice.
Likewise, the criterion related to conflict of interest was extensively discussed in the threeseparate meetings. The communication and management of potential conflicts of interest,especiallythosereferredtopatientsandhealthcareprofessionalswaslargelydiscussed.Itwaspointedoutthat“…conflictsofinterestmustbeavoided,notjustmanagedordisclosed”inaPEpractice.However, as this is not always possible; and if the involvementof somehealthcareprofessionalsisavoidedbecauseoftheirconflictsofinterest,expertswithstrongknowledgeinthefieldwouldbeexcluded;therefore,avaluablecontributioncouldbemissed.Thediscussionalso revolved around payments and funding for participants involved in some PE practices.Therewas a diversity ofopinions and views.On one side, some experts considered, neitherpatients nor professionals should collect payments for their participation because of thepotentialconflictsofinterestthatthesepaymentscouldimply.Otherviewsconsideredthatanyactivityshouldberemuneratedtoanystakeholderinvolved(i.e.healthcareprofessionalsandpatients)becausetheyarecontributingwiththeirknowledge,expertise,timeandeffort.Therewasarangeofopinionsdependingonthetypeofstakeholder(privateorpublicsector)andbytheircountryoforigin.
Finally, another key criterion discussed during the meetings was related to sustainability.Although this criterionwas considered relevant by the experts, during the votingprocess itreached the lowest score in theDelphigroupsof SRP andDCT. In theDelphigroupof earlydialogueswithHTAandRegulatorssustainabilitywasincludedinthefirsttwoonlineroundsbut dropped in the finalmeeting. For someHTAbodies it is already very difficult to start aprocesslikePEso”longtermsustainabilityisnotyetconsideredintheearlydialogues”.Thus
22
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
sustainabilityshouldnotbeanelementtobeevaluatedinPEpracticesinthisdecision-makingpoint.ItwasalsorecognizedthatsustainabilityofPEforvulnerablepopulationsiskeyandthatthesepopulationsareexperiencingresource,processandculturalbarrierstosustainability.
Anothermajor element debatedwithin sustainabilitywas related to fostering allianceswithprivate and/or public institution to facilitate sustainability in the PE practices. There wereopposing points of view; on one side, some experts from public entities or patientsorganisationsseekpartnershipsinordertoensurelong-termandsustainablepractices,whichmight not be so necessary for the private sector (i.e. one explanation for this was thatsometimesthecontinuityofaprojectwasdecidedbytheinstitution’salreadysetplans).
Biases in the Delphi results may come from two known sources. The assumptions thatdefinitions of PE and patient and public involvement (PPI) relate to only English speakinginterpretationswere acknowledged as a caveat. Additionally,many respondents in the DCTDelphigroupwere fromNorthAmerica,hence somebias in interpretationof responsesandweightingsisinevitablefromanEUvs.USperspective.
6. Proposal for Utilising the Criteria
TranslatingDelphioutputsintoatooltoassessimplementedPEpractices
Inthissection,wesuggesthowtheresultsoftheDelphigroupscouldbetranslatedintoatoolthat can be used to assess PE practices to assess and determine whether they meet theexpectationsforagoodPEpracticeinthesespecificdecision-makingpoints.Withthistool,wepropose a way to operationalize the output of this work, namely the co-prioritised set ofminimumcriteriathatderivedfromstakeholder’sneedsandexpectationsforPE.Theresultsofthe assessment could help users to firstly identify successful practices and secondly toreplicatethemintotheirownsetting.
Itshouldbenoted, that theseproposedtoolsarenotwithinthescopeof thisworkandthushave not been further developed. However, aswe have been part of producing the criteria,thesetoolsareoursuggestionofhowthecriteriacouldbeoperationalized.
Howthecriteriahasbeenoperationalized
This proposed tool provides an approximate score for each category (0 to100)within eachcriteria and a final score (0 to100) for the practice. The scores of each categorywithin thecriteriahavebeenweightedaccordinglytotheresultsoftheprioritisationdonebytheDelphipanels.TheseweightsrepresenttheimportancegivenbytheexpertpanelwithregardstothecompletionofeachcriterioninthePEpracticedesignorimplementation.
Toconverttheweightsintoaglobalscore,theweightofeachcriterioniscalculatedbyaddingtheweightofitscategories.
Howtousethesetools
Thetoolusesaweighted5-itemLikert-scaletoassessPEpractices,however,itshouldbenoted
23
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
that it cannot fully counter the subjectivity of the userwith regards to the interpretationofhoweachcriteriaaddressedinthePEpractice.
When assessing a PEpractice, the user should choosewithin the likert-scale a score from0(minimum score) to 5 (maximum score) based on the degree of fulfilment of the practiceagainsteachofthecategorywithineachcriterionofthetool.Oncetheassessmentprocessisdone,thescoresareaddeduptogiveafinalscorethatcanbeusedtodemonstratehowwellapracticehasbeendesignedorimplementedaccordingtothecriteria.
To establish a threshold thatwill determine if a practice is good or not (based on the finalscore)couldbedecidedbytheuserpriortotheassessmentprocess.
ToaccesstheproposedtoolforPatientEngagementinSettingResearchPrioritiespleasefollowthislink:https://imi-paradigm.eu/wp-content/uploads/2019/06/Assessment_tool_PE-in-Setting-Research-Priorities.xlsx
ToaccesstheproposedtoolforPatientEngagementinDesigningClinicalTrialspleasefollowthislink:https://imi-paradigm.eu/wp-content/uploads/2019/06/Assessment_tool_PE-in-Design-Clinial-Trials.xlsx
To access the proposed tool forPatient Engagement in EarlyDialogueswithHTA bodies andRegulators Tool please follow this link: https://imi-paradigm.eu/wp-content/uploads/2019/06/Assessment_tool_PE-in-Eary-Dialogues-with-HTAR.xlsx
Annex 1. List of sources
AdelphiValues,Patient-CenteredOutcomes
Patientengagement:Thefutureforoutcomesresearchandclinicaltrials?
http://healthdocbox.com/Psychology_and_Psychiatry/68794253-Patient-engagement-the-future-for-outcomes-research-and-clinical-trials.html
AGREENextStepsConsortium
APPRAISALOFGUIDELINESFORRESEARCH&EVALUATION
https://www.agreetrust.org/wp-content/uploads/2013/10/AGREE-II-Users-Manual-and-23-item-Instrument_2009_UPDATE_2013.pdf
BPSU(BritishPaediatricSurveillanceUnit)
PPIGuidanceforresearcherssupportedbytheBPSU
https://www.rcpch.ac.uk/bpsu-patient-public-involvement
CADTH
PatientInputTemplateandIntegratingCompanionDiagnosticAssessmentIntoCADTH’sDrugReviewPrograms(2017)
https://www.cadth.ca/sites/default/files/cdr/cdr-pdf/CADTH-patient-input-template.docx
Health Technology Assessment of Drugs With Companion Diagnostics at CADTHhttps://cadth.ca/sites/default/files/cdr/cdr-pdf/CDx_Process.pdf
Template for Patient Group Input to CADTH Common Drug Review and pan-CanadianOncologyDrugReview
https://www.cadth.ca/pcodr
DurhamUniversityCentreforSocialJusticeandCommunity
ActionCommunity-basedparticipatoryresearchAguidetoethicalprinciplesandpractice
https://www.dur.ac.uk/resources/beacon/CBPREthicsGuidewebNovember20121.pdf
EMA
Stakeholder Engagement report 2017 Patients, consumers, healthcare professionals,
25
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
academicsandtheirorganisations.EMA2017
https://www.ema.europa.eu/documents/report/stakeholder-engagement-report-2017_en.pdf
Thepatient'svoiceintheevaluationofmedicinesHowpatientscancontributetoassessmentofbenefitandrisk.EMA.18October2013
https://www.ema.europa.eu/documents/report/report-workshop-patients-voice-evaluation-medicines_en.pdf
EMA-Incorporating patients' views during evaluation of benefit risk by the EMA ScientificCommittees.23October2014 https://www.ema.europa.eu/documents/other/incorporating-patients-views-during-evaluation-benefit-risk-european-medicines-agency-scientific_en.pdf
TheroleofpatientsasmembersoftheEMAHumanScientificCommittees
https://www.ema.europa.eu/documents/other/role-patients-members-european-medicines-agency-human-scientific-committees_en.pdf
EPF
PatientInvolvementinHealthTechnologyAssessmentinEurope(ResultsoftheEPFSurvey)
http://www.eu-patient.eu/globalassets/projects/hta/hta-epf-final-report2013.pdf
Fastercures
PatientPerspectiveValueFramework
http://www.fastercures.org/programs/patients-count/patient-perspective-value-framework/
FDA
Patient-Focused Drug Development Guidance Series for Enhancing the Incorporation of thePatient’sVoiceinMedicalProductDevelopmentandRegulatoryDecisionMaking.Guidance1:collecting comprehensive and representative input.https://www.fda.gov/Drugs/DevelopmentApprovalProcess/ucm610279.htm
PEAC Roundtable Scenarios & Questions. Patient Engagement Advisory CommitteeMeeting.October11and12,2017
https://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMaterials/PatientEngagementAdvisoryCommittee/UCM579154.pdf
PATIENT-FOCUSEDDRUGDEVELOPMENT PUBLICWORKSHOP ON GUIDANCE COLLECTING
26
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
COMPREHENSIVE AND REPRESENTATIVE INPUT DISCUSSION DOCUMENTWorkshop Date:December18,2017(FDA)
https://www.fda.gov/drugs/newsevents/ucm574725.htm
HQIP
Patientandpublicinvolvementstrategy2015-2016
https://www.hqip.org.uk/resource/hqip-patient-and-public-involvement-strategy-201516/#.W9hB7WhKiHs
HTAi
PatientGroupSubmissionTemplateforHTAofmedicines(2014)
https://htai.org/interest-groups/pcig/resources/for-hta-agencies-and-policy-makers/
PatientGroupSubmissionTemplateforHTAofnon-medicines
https://htai.org/interest-groups/pcig/resources/for-hta-agencies-and-policy-makers/
RESOURCES TO INVOLVE PATIENTS AND PATIENT GROUPS IN HTA: Patient GroupSubmissionTemplatesGenericpatientsubmissiontemplatesforanHTA,withusefulpromptsinEnglishandFrench(Medicine’sHTA)
https://htai.org/interest-groups/pcig/resources/for-patients-and-patient-groups/
INVOLVE
Publicinvolvementinclinicaltrials:Supplementtothebriefingnotesforresearchers
http://www.invo.org.uk/wp-content/uploads/2012/03/INVOLVEBriefingnotesforresearchersMarch2012.pdf
NIHR-INVOLVE
Beresford P. Beyond the usual suspects, towards inclusive user involvementhttps://www.invo.org.uk/beyond-the-usual-suspects-towards-inclusive-user-involvement/
IOM(InstituteofMedicine)
ClinicalPracticeGuidelinesWeCanTrust
https://www.ncbi.nlm.nih.gov/books/NBK209539/
27
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
ISPOR
PatientCenteredInterestGroup
https://www.ispor.org/member-groups/special-interest-groups/patient-centered
KCE
Desomer A et al. Use of patient-reported outcome and experience measures in patient inpatientcareandpolicy.Shortreport.KCEREPORT303Cs.2018
https://kce.fgov.be/sites/default/files/atoms/files/KCE_303C_Patient_reported_outcomes_Short_Report_0.pdf
MarylandCenterofExcellenceinRegulatoryScienceandInnovation
(M-CERSI) Assessing meaningful patient engagement in drug development: a definition,framework,andrubric.UniversityofMaryland
https://www.pharmacy.umaryland.edu/media/SOP/wwwpharmacyumarylandedu/centers/cersievents/pfdd/mcersi-pfdd-framework-rubric.pdf
MEDICALDEVICEINNOVATIONCONSORTIUM(MDIC)
Patientcenteredbenefit-riskprojectreport
http://mdic.org/wp-content/uploads/2015/05/MDIC_PCBR_Framework_Web.pdf
NationalHealthCouncil(NHC)
Dialogue/AdvancingMeaningfulPatientEngagementinResearch,Development,andReviewofDrugsSeptember22,2015
https://www.nationalhealthcouncil.org/sites/default/files/PatientEngagement-WhitePaper.pdf
NationalInstituteforHealthResearch
NationalStandardsforPatientInvolvement
http://www.invo.org.uk/wp-content/uploads/2018/06/Public_Involvement_Standards_v1.pdf
ParentProjectMuscularDystrophy(PPMD)
Key considerations for developing & integrating patient perspectives in drug development:examinationoftheDuchennecasestudy.
https://www.bio.org/sites/default/files/BIO_PPMD_Paper_2016.pdf
28
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Patient-CenteredOutcomesResearchInstitute(PCORI)
Engagementinhealthresearch
https://www.pcori.org/engagement
https://www.pcori.org/literature/engagement-literature
PatientFocusedMedicineDevelopment(PFMD)
PatientEngagementQualityGuidanceTool [PlanningaPEproject] [AssessinganongoingorcompletedPEproject]
http://patientfocusedmedicine.org/peqg/patient-engagement-quality-guidance-scenario-2.pdf
SpanishMinistryofHealth,ConsumptionandSocialWelfare
Patient Involvement in the Development of Clinical Guidelines. Methodological handbook.(ImplicacióndePacientesenelDesarrollodeGuíasdePrácticaClínica.ManualMetodológico)GrupodetrabajodeimplicacióndepacienteseneldesarrollodeGPC.ImplicacióndePacientesen el Desarrollo de Guías de Práctica Clínica: Manual Metodológico. Ministerio de Sanidad,ServiciosSocialeseIgualdad.InstitutoAragonésdeCienciasdelaSalud-IACS.GuíasdePrácticaClínicaenelSNS:IACSNº2010/01.
http://www.guiasalud.es/emanuales/pacientes/documentos/manual_pacientes.pdf
TheJamesLindAlliance
TheJamesLindAllianceGuidebook
http://www.jla.nihr.ac.uk/jla-guidebook/downloads/Version-8-JLA-Guidebook-for-download-from-website.pdf
Articles
Boutin et al. Culture and Process Change as a Priority for Patient Engagement inMedicinesDevelopment.TherInnovRegulSci.2017Jan;51(1):29–38
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5302122/
Dubois RW, Lauer M and Perfetto E. When is evidence sufficient for decision-making? Aframework for understanding the pace of evidence adoption. Journal of ComparativeEffectivenessResearch2(4),383–391(2013)
https://www.futuremedicine.com/doi/pdfplus/10.2217/cer.13.39
29
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Boudeset al.Whatdo stakeholdersexpect frompatientengagement:Are theseexpectationsbeingmet?HealthExpectations.2018;21:1035–1045
https://onlinelibrary.wiley.com/doi/full/10.1111/hex.12797
Sacristanetal.Patientinvolvementinclinicalresearch:why,when,andhow.Doverpress.27April2016Volume2016:10Pages631—640
https://www.dovepress.com/patient-involvement-in-clinical-research-why-when-and-how-peer-reviewed-fulltext-article-PPA#T1
SupleD.eta.Fromtokenismtomeaningfulengagement:bestpracticesinpatientinvolvementinanEUproject.ResearchInvolvementandEngagement.2015;1:5
https://researchinvolvement.biomedcentral.com/articles/10.1186/s40900-015-0004-9
30
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Annex 2. Delphi questionnaires
Questionnaire-DELPHIONPEINSETTINGRESEARCHPRIORITIESCriterion1:Aimandobjectives
R1. The aim and objectives of the practice on PE in research priority are agreed with allstakeholders(i.epatients,academics,researchers,pharmaceuticalcompaniesetc)involved.
R2. Theaimandobjectivesof thepracticeonPE inresearchpriorityarerelated topatients’needsandinterests.
R3.TheaimandobjectivesofthepracticeonPEareclearandunderstandableforallparticipants.
Criterion2:ContextualizationofthepracticeonPEinResearchPriority
R4. Thepracticeisbasedonanassessmentofthepatients’needsandpreferenceswhowillbeengagedinsettingresearchpriorityprocesses.
R5. There is a clear understanding of contextual factors that could affect the outcomes ofinvolving patients in setting research priority (e.g funders, policy makers, industry,researchpromoters…etc).
R6.Anyexistingpolicydirectives,legal(governancerequirements)and/orregulatoryframeworkabouthowtoengagepatientsinsettingresearchpriorityhavebeenconsidered
Criterion3:Targetparticipantsinvolvedinpatientengagement
R7.Theprofilesofthepatientsengagedinsettingresearchprioritiesreflecttherelevantdiversityof the population targeted. (e.g., gender, socioeconomic status, ethnicity, culturalbackground,geographicallocationetc)andincludesgendersensitiveinformation.
R8. Theprofilesof thepatientsengaged insettingresearchprioritiesareselected torepresentrelevant patients’ circumstances (e.g., former patients, patients who are at risk for adisease, but do not yet have the disease, severity of the disease, patients’ advocates,patients’ representatives, patient organizations, level of experience in setting researchpriority,patient’sscientificknowledge/background).
R9. The practice includes methods to identify individuals who have the necessary scientificbackgroundorexpertise,“lived”experiencetoparticipateinsettingresearchpriority(e.g.,patient’snetworks,healthcareprofessionals’networks,publicadvertiseetc.).
R10. Thepracticesupportspatients toengageand/orreachout tovulnerablegroupsto involvetheminsettingresearchpriorities,aswellastomentorotherpatients.
R11.Thepracticeincludesrelevantpointsofviewotherrelevantstakeholders(e.g.,carers,tutors,researchers,healthcareproviders,decision-makers,policymakers,funders).
R12. In the PE practice, the roles and responsibilities of all participants involved, includingpatients,areclearlydefinedandagreed.
R13.Thereisclearguidanceabouthowtoassignrolesamongallparticipantsinvolvedinsettingresearchpriority.
R14.Thepracticedisplayflexibilityandopen-mindednessinassigningtherolesforthepatients,according to the profile, interest and background of the patient (e.g. as sources of
31
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
information,asconsultantsorascollaboratingpartnersoftheprocess).
R15.Thepracticeincludesspecifictools(e.g.,rolesandresponsibilitycharts,proceduralguidancedocuments, protocols) andmechanisms (e.g.,meetings,workshops, training sessions), toensurethatallparticipantsunderstandtheirownandother’srolesandresponsibilities.
Criterion4:Codeofconduct
R16. The practice includes a code of conduct,which clearly states the rules andmechanism ofparticipationinsettingresearchpriorityforallstakeholdersinvolved.
R17.Thetermsandconditionsofconfidentialityagreementsareadequate,writteninaclearandaccessibleway,andadaptedtothetargetpopulationinvolved.
R18. Thepracticecontainsmechanisms(explainedtoallstakeholdersinaclearandaccessibleway) to manage potential conflicts of interest when patients engage with differentstakeholders(e.g.,policiesthatrequirefulldisclosure,transparency,andaccountability).
R19. The practice contains procedures (e.g., personalized meetings, ways of anonymousuncovering) to identify and address discriminatory, coercive, intimidating, and unethicalbehavioursresultingfromtheirparticipationintheresearchprioritiesprocess.
Criterion5:Resources
R20.Allparticipantsinvolvedinthesettingresearchprioritypracticeareinformed,inclearandaccessibleway,oftheresourcesavailabletosupporttheirtasksduringtheprocess.
R21. The practice includes an equitable financial compensation framework for all participants(e.g., reimbursement of expenses for travel, subsistence, time missed from work,child/elderlycareetc.).
R22.Fundingisallocatedtocoveralltheelementsofthepractice(e.g.,humanresources,materialresources),andisadequatefortheintendedwork.
R23.Theadequateinfrastructuresneededforpatients’andotherstakeholders’involvementsarein place (e.g., technological support, working space, information and communicationtechnologies) adapted to specific circumstances of participants (e.g possible physical,mental,cognitiveoranyimpairmentsetc.)
R24.Thepracticeincludesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateseffectivepatientengagementimplementationinaclearandaccessibleway.
Criterion6:Capacitybuilding
R25. The practice identifies the competencies required by all participants involved in settingresearchpriority(e.gknowledgeandskillsuchashowtoperformpatientengagementinsettingresearchpriority,etc).
R26. Thepracticeincludesspecifictrainingprogramsaboutsettingresearchpriorityprocessesand methods to all participants (e.g., written information, in-house training, online,personalisedassistance,andwebinars).
R27. Thetrainingmaterial isadapted,comprehensibleandaccessibletoallparticipantstakingintoconsiderationimpairments,literacylevels,culturalbackgroundandthecircumstancesofvulnerablepatientsinvolvedinsettingresearchpriority.
R28. The training program specifies time when training needs to be provided, duration and
32
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
frequencyofthesessions(e.g.,ondemand,once,twiceamonth/year).
R29.Thepracticeincludestrainingforpatientsandotherstakeholdersinvolvedindifferentareas(e.g legal and regulatory concepts and challenges in research or deliberative processes,concepts,terminologyetc).
R30. The practice incorporates training/coaching for stakeholders, other than patients (e.gresearchers,healthcareprofessional,fundersetc)regardingtheparticipationofpatientsinresearchprioritysetting.
Criterion7:involvementandparticipation
R31. The type of interaction among different stakeholders (e.g., cooperation, co-creation,information,advice…)isclearlydefined.
R32. Patients receive timely, clear, accessible and appropriate information about theirinvolvement (i.e what will be addressed, what is expected from them, what theirparticipationwillentail,howthepracticewillbeorganisedetc)
R33. The timing requirement of the engagement in setting research priority is appropriatelyplanned, taking into account patient’s needs, and based upon the requirements of theengagementprocess(e.gtimeoftheday,durationetc).
R34. Mechanisms to ensure the meaningful participation and contribution of all participantsinvolved in the PE activity are in place taking in consideration characteristics andcircumstancesofvulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc).
R35.Aregularfeedbacksystemtoinformpatientsabouttheoutcomes/changesisincludedforeachphaseoftheinvolvement.
R36. The information about the outcomes included for each phase of the involvement iscommunicatedinplainandrespectfullanguage,usingaccessible,clearandunderstandableformats, and taking into consideration potential impairments (e.g., impaired vision,blindness,deafnessandcognitiveproblems),lowliteracylevels,culturalbackgroundsandotherrelevantfactors(e.g.age,disease/condition,etc.).
R37.Thepracticeincludesasinglepointofcontactoranamedpersontowhompatientscanreachout to when needed, for information and/or support, throughout their involvement insettingresearchpriority.
R38. Mechanismsare inplace to guarantee allmembershave the same statuswithin theworkgroup,regardlessoftheirexperienceandknowledge.
Criterion8:EvaluationofthePEPracticeinSettingResearchPriority
R39. An evaluation framework is included (structure, process and results) and it enablesregularityevaluation(frequency,timing…).
R40. The evaluation framework is shared and agreed among the participants to fostercollaboration.
R41.Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice.
R42.TheevaluationoutcomesarelinkedtotheaimandobjectivesofthePEpracticeinsettingresearchpriority.
33
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
R43.Theevaluationoutcomesaresharedwithalltheparticipantsinvolvedinthepracticeusingappropriatechannelsandformatssuitedtotheparticipant’scharacteristicsandneeds.
R44.ThepracticeincludesproceduresbywhichtheconclusionsoftheevaluationofthepracticeonPEareused(e.g.,mechanismstointegratetheresultsoftheevaluationinthedesignofthenextpracticeorthecurrentonewithacontinuousimprovementprocess).
Criterion9:PatientEngagementImpact
R45. The practice identifies relevant outcomes and impacts regarding the involvement ofpatientsinsettingresearchpriority.
R46.Assesstherelevanceofthefollowingpossibleoutcomesandimpactsforpatientengagementinsettingresearchpriority:
46.1Researchtopicandprioritiesbecomemoreappropriate,basedonpatients’needs
46.2 The product profile (drug, technology) become more relevant and usable forpatients.
46.3Researchquestionsandoutcomes/endpointsbecomemorerelevantforpatients.
46.4 Increased probability of ending the study during the research process (reducedfailureoftheproject).
46.5 Researchproposalssubmittedtopublicfundershavemoreoptionstobefinanced(e.g.enhancedcredibility).
46. 6 Resource allocations when engaging patients within industry become moreappropriateandarebasedonpatients’needs.
46. 7 Funded studies will provide more useful information for patients and decision-makingprocesses
Criterion10:Sustainability
R47. The continuation of thepractice is ensured through ownership and it is embedded in theinstitutionororganisation.
R48.Humanandfinancialresourcesrequiredforthelong-termcontinuityofthepracticeonPEinsettingresearchpriorityareidentifiedandsecured.
R49.Thepracticeidentifiesandestablishesallianceswithotherprivateand/orpublicinstitutions,citizensandothers,toensureitscontinuity.
R50. Thereisadisseminationandcommunicationplan/strategiesoftheresultsinPEinsettingresearchprioritythroughculturallyappropriateandaccessiblemechanisms.
Questionnaire-DELPHIONPEINTHEDESIGNOFCLINICALTRIALSCriterion1:Aimandobjectives
D1.TheaimandobjectivesofthepracticeonPEinthedesigningclinicaltrialsareagreedwithall thestakeholders(i.epatients,academics, researchers,pharmaceuticalcompanies,etc.)involved.
D2.TheaimandobjectivesofthepracticeonPErespondtopatient’sneedsandinterests.
34
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
D3.TheaimandobjectivesofthepracticeonPEareclearandunderstandableforallparticipants.
Criterion2:ContextualizationofthepracticeonPEintheDesignofClinicalTrials
D4.Thepracticeisbasedonapre-assessmentofpatients’wantsandneedswhenparticipatinginthedesignofclinicaltrials.
D5. A clear understanding of contextual factors that would affect the outcomes of involvingpatients in the design of clinical trials is demonstrated (e.g. funding, health system andcoverage,characteristicsofthepopulation,socioeconomicenvironment).
D6.Anyexistingpolicydirectives,legal(governancerequirements)and/orregulatoryframework(e.gEMA)abouthowtoengagepatientsinclinicaltrials’designhavebeenfollowed.
Criterion3:Targetparticipantsinvolvedinpatientengagement
D7.Thepatientprofilesengagedinthedesignofclinicaltrialsreflecttherelevantdiversityofthepopulation targeted (e.g., gender, socioeconomic status, ethnicity, cultural background,geographicallocationetc)andincludesgendersensitiveinformation.
D8.Thetargetparticipant'sprofilewhoparticipateinthedesignoftheclinicaltrialsrepresentthetargetpopulationthatwillberecruitedtotheclinicaltrials(e.g.,formerpatients,peoplewhoareatriskforadisease,butdonotyethavethedisease,patients'advocates,patients'representatives,patientorganisations,levelofexperienceinthedesignofclinicaltrials).
D9.ThepracticeofPEincludesmethodstoidentifyindividualseligibletoengageinthedesignof clinical trials (e.g., patient's networks, health care professionals' networks, publicadvertiseetc).
D10.PracticesonPEinthedesignofclinicaltrialsconsidertheinclusionofotherrelevantpointsof view separated to that of the patients (e.g. carers, tutors, research and academia,healthcare professional, pharmaceutical/biotechnology/medical technology industry,policymaker,regulator,HTAagency,funder,payer…).
D11. The role and responsibilities of all participants involved in the design of clinical trials,includingpatients,areclearlydefinedandagreed.
D12. There is clearguidance abouthow toassign rolesamongallparticipants involved in thedesignofclinicaltrials.
D13.Thepracticedisplaysflexibilityandopen-mindednessinassigningtherolesforthepatients,according to theprofileandbackgroundof thepatient (e.g. assourcesof information, asconsultantsorascollaboratingpartnersoftheprocess).
D14.Thepracticeincludesspecifictools(e.g.,rolesandresponsibilitycharts,proceduralguidancedocuments, protocols) andmechanisms (e.g.,meetings,workshops, training sessions), toensurethatallparticipantsunderstandtheirownandother’srolesandresponsibilities.
Criterion4:CodeofConduct
D15. The practice includes a code of conduct,which clearly states the rules andmechanism ofparticipationindesignofclinicaltrialsforallstakeholdersinvolved.
D16. Effortsaremade toensure thatall individualsinvolvedknowtherulesofparticipation inclinicaltrialsdesign(e.g.,workshops,meetings).
D17.Thetermsandconditionsofconfidentialityagreementsareadequate,writteninaclearand
35
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
accessibleway,andadaptedtothetargetpopulationinvolved.
D18. Potential conflicts of interest are addressed andmanaged (e.g., policies that require fulldisclosure,transparency,andaccountability).
D19. The practices contains mechanisms (e.g., personalized meetings, ways of anonymousuncovering) to identify andaddress discriminatory, coercive, intimidation, and unethicalbehavioursresultingfromtheirparticipationduringthedesignoftheclinicaltrialsprocess.
D20.FundingresourcesforthepracticeonPEinthedesignofclinicaltrialsareclearlyexplainedto all stakeholders involvedprior to involvement,andany changes that occurduring thepracticearecommunicatedup-front.
Criterion5:Resources
D21. Allparticipants involved inthedesignofclinicaltrialspracticeareinformed, inclearandaccessibleway,oftheresourcesavailabletosupporttheirtasksduringtheprocess.
D22. An appropriate and equitable financial compensation framework is in place for allparticipants(e.g.professionals,patientsetc.)invitedtoparticipateinthePEpracticeinthedesignofclinicaltrials(e.g.,reimbursementofexpensesfortravel,subsistence,timemissedfromwork,child/elderlycare).
D23.Fundingisallocatedtocoverallelementsofthepractice(e.g.,humanresources,material),andisadequatefortheintendedwork.
D24.Theadequateinfrastructuresneededforpatients’andotherstakeholders’involvementinthedesign of clinical trials practice are in place (e.g., technological support, working space,information and communication technologies) and adapted to specific circumstances ofparticipants(e.gpossiblephysical,mental,cognitiveoranyimpairmentsetc.)
D25.Thepracticeincludesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateseffectivepatientengagementimplementationinaclearandaccessibleway.
Criterion6:Capacitybuilding
D26. The practice includes comprehensive and accessible training or induction material andprogramstorespondtotheneedsofthepeopleinvolvedinthedesignofclinicaltrials.
D27. The training program specifies timewhen training needs to be provided, duration andfrequencyofthesessions(e.g.ondemand,once,twiceamonth/year).
D28.Trainingforpatientsincludesareas,suchaslegalandregulatoryconceptsandchallengesinclinicaltrialsordeliberativeprocess,concepts(e.gbasicstatisticsetc),terminologyetc.
D29. PracticesonPE indesignofclinical trials incorporatetraining/coaching forstakeholders,otherthanpatients,regardinghowtoincorporatepatientsinthedesignofclinicaltrials.
Criterion7:Involvementandparticipation.
D30. The type of interaction among the different stakeholders (e.g., cooperation, co-creation,information,advice…)isclearlydefined.
D31. Consideration has been given to specific participants’ circumstances and characteristicslinkedtobutnotlimitedtopossiblephysicalormentalimpairments,culturalbackground,age and other relevant features (e.g., use of language, format of meetings, the venue,informationprovided).
36
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
D32. The timing requirementof the engagement in thedesignof clinical trials is appropriatelyplanned,takingintoconsiderationpatient’sneeds,andbasedupontherequirementsoftheengagementprocess(e.gtimeoftheday,durationetc).
D33. Mechanisms to ensure the meaningful participation and contribution of all participantsinvolved in the PE activity are in place taking in consideration characteristics andcircumstancesofvulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc).
D34. Aregularfeedbacksystemtoinformpatientsabouttheoutcomes/changesisincludedforeachphaseoftheinvolvement.
NEWCATEGORY:WhenengagingpatientsintheDesignofclinicaltrials,patient-orientedoutcomesareobtainedthroughvalidinstruments
D35. The information about the outcomes included for each phase of the involvement iscommunicated in plain and respectful language, using accessible, clear andunderstandable formats and taking into account potential impairments (e.g., impairedvision, blindness, and deafness and cognitive problems), low literacy levels, culturalbackgroundsandotherrelevantfactors(e.g.age,disease/condition,etc.).
D36. Asinglepointofcontact fororanamedpersontowhompatientscanreachout towhenneeded for information and/or support, throughout their involvement in the design ofclinicaltrials.
D37. Mechanismsare inplace to guarantee allmembershave the same statuswithin theworkgroup,regardlessoftheirexperienceandknowledge.
Criterion8:EvaluationofthePEPracticeinSettingResearchPriority
D38. An evaluation framework is included (structure, process and results) and it enablesregularityoftheevaluation(frequency,timing…)
D39. The evaluation framework is shared and agreed among the participants to fostercollaboration.
D40.Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice.
D41. The evaluation outcomes are linked to the aimand objectives of the PE practice in thedesignofclinicaltrials.
D42.Theevaluationoutcomesaresharedwithalltheparticipantsinvolvedinthepracticeusingappropriatechannelsandformatssuitedtotheparticipant’scharacteristicsandneeds.
D43.ThepracticeincludesproceduresbywhichtheconclusionsoftheevaluationofthepracticeonPEareused(e.g.,mechanismstointegratetheresultsoftheevaluationinthedesignofthenextpracticeorthecurrentonewithacontinuousimprovementprocess).
Criterion9:PatientEngagementImpact
37
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
D44.Thepracticeidentifiesrelevantoutcomesandimpactsregardingtheinvolvementofpatientsinthedesignofclinicaltrials.
D45.Assesstherelevanceofthefollowingpossibleoutcomesandimpactsforpatientengagementinthedesignofaclinicaltrial:
D45.1Improvedrecruitmentofstudyparticipants
D45.2Increasedretention(i.elowerdropout)ofstudyparticipants
D45.3Increaseddiversityofstudyparticipants
D45.4Betteradherencetotheresearchprotocol,lessamendments
D45.5Improvedstudyparticipants’trialexperience
D45.6Fasterstudycompletionasaresultofimprovedrecruitmentandadherencetotheprotocol
D45.7Decrease costs as a result of improved recruitment, retention and positivetrialexperience
D45.8Moreappropriate,inclusive,sensitiveandethicaltrialdesign
D45.9 More appropriate wording and timing of research instruments (e.g.questionnairesandinterventions)
D45.10Improvedpatientinformationaccessibility(e.g.laysummaries,informationsheets,consentform,recruitmentmaterial)
NEW CATEGORY: Identification clinical trial endpoints that better reflect functionaloutcomes/benefits for patients from using a given therapy compared totraditional/standardregulatoryendpoints
Criterion10:Sustainability
D46. The continuation of the practice is ensured through ownership and it is embedded in theinstitutionororganisation.
D47.Humanandfinancialresourcesrequiredforthelong-termcontinuityofthepracticeonPEinthedesignofclinicaltrialsareidentifiedandsecured.
D48. ThepracticeonPE in thedesignof clinical trials identifies andestablishesallianceswithotherprivateand/orpublicinstitutions,citizensandothers,toensureitscontinuity,aslongasconfidentialityisrespected.
D49. There is a dissemination and communication plan/strategies of the results in PE in thedesignofclinicaltrialsthroughculturallyappropriateandaccessiblemechanisms.
Questionnaire-DELPHIONPEINEARLYDIALOGUES
Criterion1:Aimandobjectives
38
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
E1. The aim and objectives of PE in Early Dialogues reflects patients’ needs, experiences andexpectationsintheearlydialoguesprocess.
E2. Theaimandobjectivesof thepracticeonPEinearlydialoguesareco-designedandagreedwithallthestakeholdersinvolved.
E3. The aim and objectives of PE in Early Dialogues are clear and understandable for allparticipants.
Criterion2:ContextualizationofthepracticeonPEinEarlyDialogues
E4. In the practice on PE in early dialogues, the process is based on the assessment of thepatient’spreferencesandneeds.
E5.Aclearunderstandingofcontextualfactorsthatinformtheprocessofpatientengagementisdemonstrated so that the quality of dialogue when involving patients in early dialogueswithHTAsandregulatorsisimproved.
E6. Anyexistingpolicy, legal (governancerequirements)andregulatory framework(e.gEMA)about how to engage patients in early dialogues with HTA and regulators have beenfollowed.
Criterion3:Targetparticipantsinvolvedinpatientengagement
E7. The profiles of patients engaged in earlydialogueswithHTA and regulators reflect therelevantdiversityofthepopulationtargeted.(e.g.,gender,socioeconomicstatus,ethnicity,culturalbackground,geographicallocationetc)andincludesgendersensitiveinformation.
E8. Patients engaged in early dialogues with HTA and regulators are selected to representrelevant patients’ circumstances (e.g., former patients, patients who are at risk for adisease, but do not yet have the disease, patients’ advocates, patients’ representatives,patient organizations, level of experience in early dialogues with HTA and regulators,patient’sscientificknowledge/background).
E9. ThepracticeofPEinearlydialoguesincludesmethodstoidentifypatientswhomatchtheclinical criteria needed for a particular dialogue (e.g., patient’s networks, health careprofessionals’networks,publicadvertiseetc.).
E10. Thepracticesupportspatientstoengageand/orreachouttovulnerablegroupstoinvolvetheminearlydialogueswithHTAandregulators,aswellastootherpatients.
E11.Relevantpointsofviewotherthanpatients(e.g.carers,parentsetc.)areconsidered.
E12. Inthepractice,theroleandresponsibilitiesofallparticipantsinvolved,includingpatients,areclearlydefinedandagreed.
E13. Thereisclearguidanceabouthowtoassignrolesamongallparticipantsinvolvedinearlydialogues.
E14.Guidance,trainingadequateisprovidedtoensurethatallparticipantsunderstandtheirownroleandresponsibilities,aswellastheresponsibilitiesandrolesofothers.
Criterion4:CodeofConduct
E15.Thepracticeincludesacodeofconduct,whichclearlystatestherulesofparticipationinearlydialogueswithHTAandregulatorsforallparticipantsinvolved.
39
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
E16. ThepracticeonPE inearlydialogues includesmechanisms,adequateandadapted to thetargetpopulationtoensurethatallindividualsinvolvedknowtherulesofparticipationinearlydialogueswithHTAandregulators(e.g.,workshops,meetings).
E17.Thetermsandconditionsofconfidentialityagreementsareadequate,writteninaclearandaccessibleway,andadaptedtothetargetpopulation.
E18. The practice on PE in early dialogues incorporates mechanisms (explained to allstakeholdersinaclearandaccessibleway)tomanagepotentialconflictsofinterestwhenpatients engage with different stakeholders (e.g., policies that require full disclosure,transparency,andaccountability).
E19. ThePEpractice containsprocedures (e.g., personalizedmeetings advocacy figure clearlyidentified) to identify and address potential discriminatory, coercive, intimidating, andunethicalbehavioursresultingfromtheparticipationintheearlydialogueprocess.
E20. Funding resources for the practice of PE in early dialogues are clearly explained to allstakeholdersinvolved,priortoinvolvement,andanychangesthatoccurduringthepracticeofPEinearlydialoguesarecommunicatedup-front.
Criterion5:Resources
E21. AllparticipantsinvolvedinearlydialogueswithHTAandregulatorsareinformedinclearandaccessibleway,oftheresourcesavailabletosupporttheirtasksduringtheprocess.
E22.Thepracticeincludesanequitablefinancialcompensationframeworkforallparticipants(e.g.professionals, patients etc.) invited to participate in the early dialogues with HTA andregulators(i.ereimbursementofexpensesfortravel,subsistence,timemissedfromwork,child/elderlycareetc.).
E23.Fundingisallocatedtocoveralltheelementsofthepractice(e.g.,humanresources,materialresources),andisadequatetotheintendedwork.
E24. Adequate infrastructuresneeded forPE inearlydialogues are inplace (e.g., technologicalsupport, working space, information and communication technologies) and adapted tospecific circumstances of participants (e.g. possible physical mental, cognitive or anyimpairmentsetc.).
E25.Thepracticeincludesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateeffectivepatientengagementimplementationinaclearandaccessibleway.
Criterion6:Capacitybuilding
E26. The practice on PE in early dialogues identifies the competencies required by allstakeholders involved in earlydialogueswithHTAbodies and regulators (e.g knowledgeandskillsuchashowtoperformpatientengagementinearlydialogues,etc.).
E27. The practice on PE in early dialogues includes training programs on HTA/Regulatoryprocesses/clinicaltrialdesignandmethodstopatients(e.g.,writteninformation,in-housetraining,online,personalizedassistance,andwebinars).
E28. The practice includes comprehensive and accessible training or induction material andprogramstorespondtotheneedsoftheparticipantsinvolvedinearlydialogueswithHTAandregulators.
40
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
E29. The trainingprogramspecifiestimewhentrainingneedstobeprovided,durationandfrequencyofthesessions(e.g.,ondemand,once,twiceamonth/year).
E30. ThepracticeonPE in earlydialogues incorporates training/coaching for all stakeholders,other thanpatients, regarding howto incorporatepatients in earlydialogueswithHTAandregulators.
Criterion7:Involvementandparticipation
E31. The mechanism of interaction among the stakeholder is adapted to their needs. (e.g.meetings,online,skype,etc.).
E32. Patients receive timely, clear, accessible and appropriate information about theirinvolvement (i.e what will be addressed, what is expected from them, what theirparticipationwillentail,howthepracticewillbeorganizedetc)
E33. Thepracticeconsidersspecificpatients’circumstancesandcharacteristicslinkedtobutnotlimited to possible physical or mental impairments, cultural background, age and otherrelevant features (e.g., use of language, format of meetings, the venue, informationprovided).
E34. ThetimingoftheengagementinearlydialogueswithHTAandregulatorsisappropriatelyplanned,basedupontherequirementsofthereviewprocess.
E35. The practice on PE in early dialogues sets mechanisms to ensure a proper deliberativeprocess(e.gmeetings,interviews,focusgroup,workshops…).
E36. A regular feedback system to informpatients about theoutcomes/changes is included ineachphaseoftheinvolvement.
E37. The information about the outcomes included for each phase of the involvement iscommunicatedinplainandrespectfullanguage,usingaccessible,clearandunderstandableformats, and taking into account potential impairments (e.g., impaired vision, blindness,anddeafness),lowliteracylevels,culturalbackgroundsandotherrelevantfactors(e.g.age,disease/condition,etc.).
E38.Thepracticeincludesasinglepointofcontactoranamedpersontowhompatientscanreachouttowhenneeded,forinformationand/orsupport,throughouttheirinvolvementinearlydialogues.
E39. Mechanismsare in place to guarantee allmembers have the same statuswithin theworkgroup,regardlessoftheirexperienceandknowledge.
E40.ThepracticeonPEinearlydialoguesfostersdiscussionandagreementwithpatientsaboutidentifyingandselectingpreferencesabouttheEDprocesswithHTAbodiesandregulators.
Criterion8:EvaluationofthePEPracticeinSettingResearchPriority
E41. An evaluation framework is included (structure, process and results) and it enablesregularityevaluation(frequency,timing…).
E42. The evaluation framework is shared and agreed among the participants to fostercollaboration.
E43.Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice.
E44. Theevaluationoutcomesare linked to theaimandobjectivesof thePEpractice inearly
41
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
dialogues.
E45.Theevaluationoutcomesaresharedwithalltheparticipantsinvolvedinthepracticeusingappropriatechannelsandformatssuitedtoparticipants’characteristicsandneeds.
E46.ThepracticeincludesproceduresbywhichtheconclusionsoftheevaluationofthepracticeonPEareused(e.g.mechanismstointegratetheresultsoftheevaluationinthedesignofthenextpracticeorinthecurrentonewithacontinuousimprovementprocess
Criterion9:PatientEngagementImpact
E47.ThepracticeidentifiesrelevantoutcomesandimpactsregardingtheinvolvementofpatientsinearlydialogueswithHTAandregulators.
E48.AssesstherelevanceofthefollowingpossibleoutcomesandimpactsforpatientengagementinearlydialogueswithHTAandRegulators:
48.1Betterunderstandingoftechnologies’impactinreallifecontextandthequalityoflifeaspects.
48.2Morepracticalandbetterdesigneddevelopmentplanformedicines.
48.3Higheraccuracyinmeasuringneedsandpreferencesofpatients.
48.4BetterqualityoftheevidenceassessedduringthelaterHTAassessment.
48.5Moreusefuladvicegiventoaidcompanydecisions.
48.6Higherrelevanceoftheadvicegiventothelocalcontextofapplication
48.7IncreasedpublicknowledgeandawarenessofHTAandregulatoryprocesses
48.8TransparencyofHTA/regulationprocesses.
48.9Acceptabilityofregulatorydecisions/HTAreportsforstakeholders
48.10 Inclusionofpatients’ relevantoutcomes andend-points in clinical trialsdata collectionandanalysis.
48.11Mutual understanding between all involved stakeholders including the patient communityabouttheregulatoryandHTAprocess.
Criterion10:Sustainability
E49. Thecontinuationofthepracticeisensuredthroughownershipandinstitutionalorpracticeorganizationanchoring.
E50.Humanandfinancialresourcesrequiredforthelong-termcontinuityofthepracticeonPEinearlydialoguesareidentifiedandsecured.
E51.ThepracticeonPEinearlydialoguesidentifiesandestablishesallianceswithotherprivateand/orpublic institutions,citizensandothers, toensure thecontinuityof thePE inearlydialoguesaslongasconfidentialityisrespected.
E52.Thereisadisseminationandcommunicationplan/strategiesoftheresultsoftheprocessofPEinearlydialoguesthroughculturallyappropriateandaccessiblemechanisms.
42
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Annex 3. Expert panel stakeholders’ profiles
Stakeholdercategories
Individuals working on orwithexperiencein
Rationaleoftheircontribution
Delphi 1PE inSettingResearchPriorities
ResearchFunders(publicand/orprivate)
*Responsibilityinmanagementand/ordesignofPublicfundingschemes for research(government agencies, publicorganizations, fundingprograms at universities, etc.).*Responsibilityinmanagementand/or design of Privatefunding schemes for research(research and developmentdepartments in companies,scientific societies, charitiesand other privateorganizations,etc.).
Viewsonhowpatientengagementfits into their decisions andprocesses to set priorities for theresearch tobe funded. Suitabilityof engagement practices in place.Howitshapesresults
Regulators Healthauthority atnationalorregional level (end user ofevidence to inform its decisionmaking)
Views on the impact of patientengagement on the researchagenda and how it affects theusability of research findings foradoption and reimbursementdecisions
HTABodies HealthTechnologyAssessment Viewsonhowpatientengagementin research priorities affects thesuitability of available evidencetoassessnewdrugs
Academics,researchers
Conducting and/ or managingresearch
Viewsonhowpatientengagementaffectstheirresearchagenda
HealthcareProfessionals
Healthcare providers(healthcaremanagers (primarycare and hospital),clinicians/health professionalsassociations).
Viewsonhowpatientengagementin research priorities affects theavailability and adoption of newdrugs
Patients/Patientorganizations
Participating (in the role ofpatient) in research prioritiessetting, research projects,researchdesign,etc.
Viewsonhowpatientengagementfits into funders' decisions andprocesses to set researchpriorities. Suitability of
43
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
engagementpracticesinplace
Delphi 2PE in theDesign ofClinicalTrials
Clinical trialspromoter(pharmacompanies orother)
Designing, preparing andconductingclinicaltrials
Viewsonhowpatientengagementfits into their decisions andprocessestoconductclinicaltrials
Academics,researchers
Designing, preparing andconductingclinicaltrials
Viewsonhowpatientengagementaffects their research agenda andthequalityoffindings
Healthcareprofessionals
Designing, preparing andconductingclinicaltrials
viewsonhowpatientengagementaffectsnewdrugsavailabilityandadoption
Regulators Clinicaltrialsregulation viewsonhowpatientengagementin design and analysis of clinicaltrialsaffectsthequality(includingethics-wise) and usability offindings
Patients/Patientorganizations
Participating (in the role ofpatient)indesigning,preparingandconductingclinicaltrials.
Viewsonhowpatientengagementfits into researchers and funders'decisions and processes to designclinical trials. Suitability ofengagement practices in place.How it shapes quality andusabilityoffindings
Delphi 3PE inEarlyDialogues
HTAbodies ParticipatinginEarlyDialogues Viewsonhowpatientengagementfits intoearlydialoguesdynamics.Suitability of engagementpractices in place. How it shapesresults
Pharmacompanies,Industry
ParticipatinginEarlyDialogues Viewsonhowpatientengagementfits intoearlydialoguesdynamics.Suitability of engagementpractices in place. How it shapesresults
Regulators Healthauthority atnationalorregional level (reimbursementdecision maker) participating
Viewsonhowpatientengagementfits intoearlydialoguesdynamics.Suitability of engagement
44
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
inearlydialogues practices in place. How it shapesresults
Patients/Patientorganizations
Participating (in the role ofpatient)inearlydialogues
Viewsonhowpatientengagementfits intoearlydialoguesdynamics.Suitability of engagementpractices in place. How it shapesresults
45
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Annex 4. Glossary for the Delphi in PE in Medicine R&D lifecycle
GLOSSARY
Accessible
Adherence
Carer
ClinicalEndpoint[ClinicalTrial]
ClinicalTrialDesign
Culturallyappropriate
CurrentPatient
EarlyDialogue
FormerPatient
HTAAssessment
HTABody
Inclusive
Industry
Likerscale
Outcome
PatientEngagementPractice
PatientOrganisations
PatientRepresentative/PatientAdvocate
Payer
Pointofcontact
PopulationatRisk
Practice
Regulator
RegulatoryFramework
ResearchPrioritiesSetting
Sensitive
Stakeholders
Tutor
46
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
VulnerablePopulations
Workgroup
AAccessibleEasilyusedoraccessedbypeoplewithspecialneeds.
AdherenceTakingmedications(orothertreatment)exactlyas instructedbyahealthcareprovider(theprescriber).
CCarerAnyone who cares, unpaid, for a friend or family member who, due to illness, disability, amentalhealthproblemoranaddictioncannotcopewithouthisorhersupport.
ClinicalEndpoint[ClinicalTrial]A characteristic or variable that reflects how a patient feels, functions, or survives. Clinicalendpointsaredistinctmeasurementsoranalysisofdiseasecharacteristicsreflectingtheeffectofatherapeuticinterventioninaclinicaltrialorstudy.
ClinicalTrialDesignProcedurebywhichthedetailsofaclinicaltrialaredefined.Theseaspectsincludethedesignoftheprotocol,thediscussionthepotentialburdenthatthepeoplewhowillparticipateinthetrial may experience or the discussion the outcomes of the trial, which could be moreimportanttopatients.
CulturallyappropriateConformingtoaculture'sacceptableexpressionsandstandardsofbehaviourandthoughts.
CurrentPatientA person that has been diagnosed of the condition studied in the practice at stake (SettingResearch Priorities, Designing a Clinical Trial, Early Dialogue) and is currently following atreatment.
E
47
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
EarlyDialogueEarly scientificdiscussions that can takeplacebetween industry,HTAbodiesandregulators(andinsomecaseswithpayers)todiscussdevelopmentalplansforamedicinalproductandtoensuretheseplansmeettherequirements.
FFormerPatientA person that has received the treatment of the condition studied in the practice at stake(SettingResearchPriorities,DesigningaClinicalTrial,EarlyDialogue),andisnotfollowinganytreatmentcurrently,exceptforfollow-upperiodicreviews.
HHTAAssessmentMultidisciplinary process that summarises information about the medical, social, economic,and ethical issues related to the use of a health technology in a systematic, transparent,unbiased,androbustmanner.Itsaimistoinformtheformulationofsafeandeffectivehealthpoliciesthatarepatientfocusedandseektoachievebestvalue.
HTABodyRegional and national organizations formed by multidisciplinary teams that providerecommendationsonmedicinesandotherhealthtechnologies(HTAassessment),thatcanbefinancedorreimbursedbythehealthcaresysteminaparticularMemberStateorregion.
IInclusiveNotexcludinganysectionofsocietyoranyparty involved.Deliberatelyavoidingusagesthatcouldbeseenasexcludingaparticularsocialgroup.
IndustryPrivate companiesworking in thedevelopmentand/or commercializationofmedicinesandnewhealthtechnologies.
LLikerscale
A Likert scale is a psychometric scale commonly involved in research that employs
48
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
questionnaires. It is a type of rating scale used to measure attitudes or opinions where,respondentsareaskedtorateitemsonalevelofagreement.
OOutcome
Outcomes refer to decisions made and things produced as a result of patient engagementpractices(includingchangedresearchprioritiesandtrialdesigns).Thisincludestheeffectsonthe development ofmedical products. Outcomes are influenced by context andmechanismsandhaveaninfluenceonimpact.
PPatientEngagementPracticeThe effective and active collaboration of patients, patient advocates, patient representativesand/or carers in a procedure of setting research priorities, designing clinical trials orparticipatinginearlydialogues.Byeffectiveandactivecollaborationsofpatients,wemeanthattheir views, preferences and values are gathered during the practice and are treated in thesamewayasthecontributionsmadebytheotherstakeholdersinvolved.
PatientOrganisationsNot-for profit organisationswhich are patient focused, andwhereby patients and/or carers(thelatterwhenpatientsareunabletorepresentthemselves)representamajorityofmembersingoverningbodies.
PatientRepresentative/PatientAdvocatePatients,carers,parentsortutorsand/orstafffrompatient’sorganizationthatactasaliaisonbetweenthepatientsandotherorganizations,representtheirinterestsandadvocatesfortheirrights.
PayerInsurerorotherorganizationthatpaysforhealthcareservices.
PointofcontactPersonorpersonsthatcanbeapproachedforinformationorassistance.
PopulationatRiskPopulationthatisexposedtodevelopingtheconditionstudiedinthepracticeatstake(SettingResearchPriorities,DesigningaClinicalTrial,EarlyDialogue).
PracticeThe customary, habitual or expected way, method or modality of performing an action. A
49
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
practice can be the designof the recruitmentprotocol of a clinical trialor the prioritizationexerciseofagivensetofresearchlines,amongothers.
RRegulatorHealthauthorityatnationalorregionallevel.
RegulatoryFrameworkThesetofrulesorordersissuedbyanexecutiveauthorityorregulatoryagencyofagovernmentandhavingtheforceoflaw.
ResearchPrioritiesSettingProcedurebywhichlinesorareasofresearchareidentifiedasapriorityinaresearch-fundingprogram.
SSensitiveCapableofmeasuringorrecordingverysmallchanges
StakeholdersAgencies,organisations,groupsorindividualsthathaverelevantknowledgeand/orexperienceinthepracticeatstake(SettingResearchPriorities,DesigningaClinicalTrial,EarlyDialogue).Theymakeuptheworkgroup.
TTutorThelegalguardianofapatient.
VVulnerablePopulationsThese terms are applied togroups of peoplewho, due to factors usually consideredoutsidetheircontrol;donothavethesameopportunitiesasother,more fortunategroups insociety.Examplesmightincludeunemployedpeople,refugeesandotherswhoaresociallyexcluded.
50
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
WWorkgroupGroupofstakeholderparticipatingintheclinicaltrialdesignpractice,researchprioritysettingpracticeorintheearlydialogue.
SourcesofInformation
1. MerriamWebsterOnlineDictionary2. Carerstrust3. EURORDISClinicalTrialGlossary4. GlossaryforHIVPrevention,StateofColorado20165. IMIPARADIGM6. EUPATIHTAGlossary7. EMA,StakeholdersandCommunicationsDivision8. OxfordOnlineDictionary9. OECDGlossaryofStatisticalTerms10. CollinsOnlineDictionary11. WHOGlossary12. https://www.statisticshowto.datasciencecentral.com/likert-scale-definition-and-
examples/13. https://en.wikipedia.org/wiki/Likert_scale
51
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Annex 5. Results of the Delphi on PE in Setting Research Priorities
ResultsofRound1(Online)–RelevanceassessmentTable2:RelevantcategoriesonPEinSettingResearchPriorities(Round1)
CriteriaID
CriteriaCategory
IDCategory
1 Aimandobjectives 1 TheaimandobjectivesofthepracticeonPEinresearchpriorityareagreed with all stakeholders (i.e. patients, academics, researchers,pharmaceuticalcompaniesetc.)involved.
2 TheaimandobjectivesofthepracticeonPEinresearchpriorityarerelatedtopatients’needsandinterests.
2 Contextualization ofthepracticeonPEinResearchPriority
4 The practice is based on anassessment of the patients’ needs andpreferences who will be engaged in research priority settingprocesses.
3 Target participantsinvolved in patientengagement
7 Theprofilesofthepatientsengagedinsettingtheresearchprioritiesreflecttherelevantdiversityofthepopulationtargeted.(e.g.,gender,socioeconomic status, ethnicity, cultural background, geographicallocationetc.)andincludesgendersensitiveinformation.
8 Patientsengagedprofileinsettingresearchprioritiesareselectedtorepresent relevant patients’ circumstances (e.g., former patients,patientswhoareatriskforadisease,butdonotyethavethedisease,patients’ advocates,patients’ representatives,patientorganizations,level of experience in research priority setting, patient’s scientificknowledge/background).
12 In thePE practice, the roles and responsibilities of all participantsinvolved,includingpatients,areclearlydefinedandagreed.
4 Codeofconduct 16 Thepracticeincludesacodeofconduct,whichclearlystatestherulesandmechanism of participation in setting research priority for allstakeholdersinvolved.
17 The terms and conditions of confidentiality agreements areadequate,written inaclearandaccessiblewayandadaptedto thetargetpopulationinvolved.
18 Thepracticecontainsmechanisms(explainedtoallstakeholdersinaclear and accessible way) to manage potential conflicts of interestwhenpatientsengagewithdifferentstakeholders(e.g.,policiesthatrequirefulldisclosure,transparency,andaccountability).
5 Resources 23 The adequate infrastructures needed for patients’ and otherstakeholders’ involvement is in place (e.g., technological support,working space, information and communication technologies)adapted to specific circumstances of participants (e.g. possiblephysical,mental,cognitiveoranyimpairmentsetc.)
24 Thepracticeincludesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateseffectivepatientengagementimplementationinaclearandaccessibleway.
52
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
6 Capacitybuilding 25 Thepracticeidentifiesthecompetenciesrequiredbyallparticipantsinvolvedinsettingresearchpriority(e.g.knowledgeandskillsuchashowtoperformpatientengagementinsettingresearchpriority,etc.).
7 Involvement andparticipation
32 Patientsreceivetimely,clear,accessibleandappropriateinformationabout their involvement (i.e. what will be addressed, what isexpected from them, what their participation will entail, how thepracticewillbeorganisedetc.)
36 Theinformationabouttheoutcomes includedforeachphaseof theinvolvement is communicated in plain and respectful language,usingaccessible,clearandunderstandableformats,andtakingintoconsideration potential impairments (e.g., impaired vision,blindness, deafness and cognitive problems), low literacy levels ,culturalbackgroundsandother relevant factors (e.g. age,disease/condition,etc.).
37 Thepracticeincludesasinglepointofcontactoranamedpersontowhom patients can reach out to when needed, for informationand/or support, throughout their involvement in setting researchpriority.
8 Evaluation 42 TheevaluationoutcomesarelinkedtotheaimandobjectivesofthePEpracticeinsettingresearchpriority.
9 Patient EngagementImpact
45 Thepracticeidentifiesrelevantoutcomesandimpactsregardingtheinvolvementofpatientsinsettingresearchpriority.
46.1 Research topic and priorities becomemore appropriate, based onpatients’needs
46.2 The product profile (drug, technology) becomemore relevant andusableforpatients.
46.3 Researchquestionsandoutcomes/endpointsbecomemorerelevantforpatients.
Table3:NoconsensuscategoriesonPEinSettingResearchPriorities(Round1)
CriteriaID
Criteria CategoryID
Category
1 Aimandobjectives 3 The aim and objectives of the practice on PE are clear andunderstandableforallparticipants.
2 Contextualizationofthepractice on PE inResearchPriority
5 Thereisaclearunderstandingofcontextualfactorsthatcouldaffecttheoutcomesof involvingpatients in research settingpriority (e.g.funders,policymakers,industry,researchpromoters…etc.).
6 Any existing policy directives, legal (governance requirements)and/or regulatory framework about how to engage patients insettingresearchpriorityhavebeenfollowed.
3 Target participantsinvolved in patientengagement
9 Thepractice includesmethodsto identify individualswhohavethenecessaryscientificbackgroundorexpertisetoparticipateinsettingresearchpriority(e.g.,patient’snetworks,healthcareprofessionals’networks,publicadvertiseetc.).
10 The practice supports patients by sharing responsibility ofengaging/reaching out to vulnerable groups to involve them in
53
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
settingresearchpriorities,aswellastomentorotherpatients.
13 There is clear guidance about how to assign roles among allparticipantsinvolvedinsettingresearchpriority.
14 Thepracticedisplayflexibilityandopen-mindednessinassigningtheroles for the patients, according to the profile, interest andbackground of the patient (e.g. as sources of information, asconsultantsorascollaboratingpartnersoftheprocess).
15 The practice includes specific tools (e.g., roles and responsibilitycharts,proceduralguidancedocuments,protocols)andmechanisms(e.g., meetings, workshops, training sessions), to ensure that allparticipants understand their own and other’s roles andresponsibilities.
4 Codeofconduct 19 Thepracticecontainsmechanisms(e.g.,personalizedmeetings,waysof anonymous uncovering) to identify and address discriminatory,coercive,intimidating,andunethicalbehavioursresultingfromtheirparticipationintheresearchprioritiesprocess.
5 Resources 20 Allparticipantsinvolvedinthesettingresearchprioritypracticeareinformed, in clearandaccessibleway,of the resourcesavailable tosupporttheirtasksduringtheprocess.
21 The practice includes an equitable financial compensationframeworkfor allparticipants(e.g.,reimbursementofexpenses fortravel,subsistence,timemissedfromwork,child/elderlycareetc.).
6 Capacitybuilding 26 The practice includes specific training programs about settingresearch priority process and methods to all participants (e.g.,written information, in-house training, online, personalisedassistance,andwebinars).
27 The trainingmaterial is adapted, comprehensible andaccessible toallparticipantstakingintoconsiderationimpairments,literacylevels,cultural background and the circumstances of vulnerable patientsinvolvedinsettingresearchpriority.
29 The practice includes training for patients and other stakeholdersinvolved in different areas (e.g. legal and regulatory concepts andchallenges in research or deliberative processes, concepts,terminologyetc.).
30 Thepractice incorporates training/coachingfor stakeholders,otherthanpatients(e.g.researchers,healthcareprofessional,fundersetc)regardingtheparticipationofpatientsinresearchprioritysetting.
7 Involvement andparticipation
31 The type of interaction among different stakeholders (e.g.,cooperation,co-creation,information,advice…)isclearlydefined.
33 The timing requirement of the engagement in setting researchpriorityisappropriatelyplanned,takingintoaccountpatient’sneeds,and based upon the requirements of the engagement process (e.g.timeoftheday,durationetc.).
34 Mechanismstoensurethemeaningfulparticipationandcontributionof allparticipants involved in thePEactivity are inplace taking inconsideration characteristics and circumstances of vulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc.).
35 Aregularfeedbacksystemto informpatientsabouttheoutcomes/changesisincludedforeachphaseoftheinvolvement.
54
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
38 Mechanisms are in place to guarantee allmembers have the samestatus within the work group, regardless of their experience andknowledge.
8 Evaluation 40 The evaluation framework is shared and agreed among theparticipantstofostercollaboration.
41 Expertise in evaluation process, methods, tools and monitoringsystemareinplacetocreateandpopulatetheevaluationframework.
43 The evaluation outcomes are shared with all the participantsinvolved in the practice using appropriate channels and formatssuitedtotheparticipant’scharacteristicsandneeds.
44 The practice includes procedures by which the conclusions of theevaluationareused(e.g.,mechanismstointegratetheresultsoftheevaluation in the design of the next practice or in the current onewithacontinuousimprovementprocess).
9 Patient EngagementImpact
46.4 Increased probability of success of the study during the researchprocess(reducedfailureoftheproject).
46.5 Researchproposals submitted topublic fundersaremore fundable(e.g.enhancedcredibility).
46.6 Resourceallocationwithinindustrybecomemoreappropriatebasedonpatients’needs.
46.7 Funded studies will provide more useful information for patientsanddecision-makingprocesses
10 Sustainability 47 The continuationof thepractice isensured throughownershipandinstitutionalororganisationalanchoring.
48 Humanandfinancialresourcesrequiredforthelong-termcontinuityof thepracticeonPEin setting researchpriorityare identifiedandsecured.
49 The practice identifies and establishes alliances with other privateand/or public institutions, citizens and others, to ensure itscontinuity.
50 There isadisseminationand communicationplan/strategiesof theresults in PE in setting research priority through culturallyappropriateandaccessiblemechanisms.
Table4:DroppedcategoriesonPEinSettingResearchPriorities(Round1)
CriteriaID
CriteriaCategory
IDCategory
3 Target participantsinvolved in patientengagement
11 The practice includes relevant points of view other relevantstakeholders(e.g.,carers,tutors,researchers,healthcareproviders,decision-makers,policymakers,funders).
5 Resources 22 Funding is allocated to coverall theelementsof thepractice (e.g.,human resources, material resources), and is adequate for theintendedwork.
6 Capacitybuilding 28 The training program specifies time when training needs to beprovided,durationandfrequencyof thesessions(e.g.,ondemand,
55
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
once,twiceamonth/year).
8 Evaluation 39 An evaluation framework is included (structure, process andresults)anditenablesregularityevaluation(frequency,timing…).
ResultsofRound2(Online)–PrioritizationTable5:RelevantcategoriesonPEinSettingResearchPriorities(Round2)
CriteriaID
Criteria CategoryID
Category
1 Aim andobjectives
3 TheaimandobjectivesofthepracticeonPEareclearandunderstandableforallparticipants.
3 Targetparticipantsinvolved inpatientengagement
15 The practice includes specific tools (e.g., roles and responsibility charts,proceduralguidancedocuments,protocols)andmechanisms(e.g.,meetings,workshops, training sessions), to ensure that all participants understandtheirownandother’srolesandresponsibilities.
7 Involvementandparticipation
33 The timing requirement of the engagement in setting research priority isappropriatelyplanned, taking intoaccountpatient’sneeds,andbaseduponthe requirementsof theengagementprocess(e.g. timeof theday,durationetc.).
35 Aregularfeedbacksystemtoinformpatientsabouttheoutcomes/changesisincludedforeachphaseoftheinvolvement.
8 Evaluation 41 Expertiseinevaluationprocess,methods,toolsandmonitoringsystemareinplacetocreateandpopulatetheevaluationframework.
9 PEImpact 46.6 Resource allocation within industry become more appropriate based onpatients’needs.
46.7 Funded studies will provide more useful information for patients anddecision-makingprocesses
Table6:NoconsensuscategoriesonPEinSettingResearchPriorities(Round2)
CriteriaID
Criteria CategoryID
Category
2 Contextualizationof the practice onPE in ResearchPriority
6 Any existing policy directives, legal (governance requirements) and/orregulatory frameworkabout how to engage patients in setting researchpriorityhavebeenfollowed.
3 Targetparticipantsinvolved inpatientengagement
13 There isclearguidanceabouthowtoassignrolesamongallparticipantsinvolvedinsettingresearchpriority.
56
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
5 Resources 20 All participants involved in the setting research priority practice areinformed, in clear and accessible way, of the resources available tosupporttheirtasksduringtheprocess.
21 Thepracticeincludesanequitablefinancialcompensationframeworkforall participants (e.g., reimbursement of expenses for travel, subsistence,timemissedfromwork,child/elderlycareetc.).
6 Capacitybuilding 27 The training material is adapted, comprehensible and accessible to allparticipants taking into consideration impairments, literacy levels,cultural background and the circumstances of vulnerable patientsinvolvedinsettingresearchpriority.
30 Thepractice incorporates training/coachingfor stakeholders,other thanpatients(e.g.researchers,healthcareprofessional,fundersetc.)regardingtheparticipationofpatientsinresearchprioritysetting.
7 Involvement andparticipation
34 MechanismstoensurethemeaningfulparticipationandcontributionofallparticipantsinvolvedinthePEactivityareinplacetakinginconsiderationcharacteristicsandcircumstancesofvulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc.).
8 Evaluation 40 Theevaluationframeworkissharedandagreedamongtheparticipantstofostercollaboration.
43 Theevaluationoutcomesaresharedwithall theparticipantsinvolvedinthe practice using appropriate channels and formats suited to theparticipant’scharacteristicsandneeds.
9 PEImpact 46.5 Research proposals submitted to public fundersaremore fundable (e.g.enhancedcredibility).
10 Sustainability 47 The continuation of the practice is ensured through ownership andinstitutionalororganisationalanchoring.
48 Human and financial resources required for the long-term continuity ofthepracticeonPEinsettingresearchpriorityareidentifiedandsecured.
49 Thepracticeidentifiesandestablishesallianceswithotherprivateand/orpublicinstitutions,citizensandothers,toensureitscontinuity.
50 Thereisadisseminationandcommunicationplan/strategiesoftheresultsin PE in setting research priority through culturally appropriate andaccessiblemechanisms.
Table7:DroppedcategoriesonPEinSettingResearchPriorities(Round2)
CriteriaID
Criteria CategoryID
Category
2 Contextualization ofthepracticeonPEinResearchPriority
5 There isaclearunderstandingofcontextual factorsthatcouldaffectthe outcomes of involving patients in research setting priority (e.g.funders,policymakers,industry,researchpromoters…etc.).
3 Target participantsinvolved in patient
9 The practice includesmethods to identify individualswho have thenecessaryscientificbackgroundorexpertisetoparticipate insetting
57
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
engagement research priority (e.g., patient’s networks, health care professionals’networks,publicadvertiseetc.).
10 The practice supports patients by sharing responsibility ofengaging/reachingouttovulnerablegroupstoinvolvetheminsettingresearchpriorities,aswellastomentorotherpatients.
14 Thepracticedisplayflexibilityandopen-mindednessinassigningtheroles for the patients, according to the profile, interest andbackground of the patient (e.g. as sources of information, asconsultantsorascollaboratingpartnersoftheprocess).
4 CodeofConduct 19 Thepracticecontainsmechanisms(e.g.,personalizedmeetings,waysof anonymous uncovering) to identify and address discriminatory,coercive, intimidating,andunethicalbehaviours resulting from theirparticipationintheresearchprioritiesprocess.
6 CapacityBuilding 26 The practice includes specific training programs about settingresearchpriorityprocessandmethodstoallparticipants(e.g.,writteninformation, in-house training, online, personalised assistance, andwebinars).
29 The practice includes training for patients and other stakeholdersinvolved in different areas (e.g. legal and regulatory concepts andchallenges in research or deliberative processes, concepts,terminologyetc.).
7 Involvement andparticipation
31 The type of interaction among different stakeholders (e.g.,cooperation,co-creation,information,advice…)isclearlydefined.
38 Mechanisms are in place to guarantee all members have the samestatus within the work group, regardless of their experience andknowledge.
8 Evaluation 44 The practice includes procedures by which the conclusions of theevaluationareused (e.g.,mechanisms to integrate the resultsof theevaluationinthedesignofthenextpracticeorinthecurrentonewithacontinuousimprovementprocess).
9 PEImpact 46.4 Increased probability of success of the study during the researchprocess(reducedfailureoftheproject).
Table8:RelevantcategoriesonPEinSettingResearchPrioritiesandtheirPriority(Round2)
CriteriaID
Criteria CategoryID
Category Priority
1 Aim andobjectives
1 The aim and objectives of the practice on PE in researchpriority are agreed with all stakeholders (i.e. patients,academics, researchers, pharmaceutical companies etc.)involved.
HP
2 The aim and objectives of the practice on PE in researchpriorityarerelatedtopatients’needsandinterests.
HP
58
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
3 Targetparticipantsinvolvedinpatientengagement
12 In the PE practice, the roles and responsibilities of allparticipantsinvolved,includingpatients,areclearlydefinedandagreed.
HP
4 CodeofConduct 16 Thepracticeincludesacodeofconduct,whichclearlystatestherulesandmechanismofparticipationinsettingresearchpriorityforallstakeholdersinvolved.
HP
17 Thetermsandconditionsofconfidentialityagreementsareadequate,writteninaclearandaccessiblewayandadaptedtothetargetpopulationinvolved.
HP
18 The practice contains mechanisms (explained to allstakeholders in a clear and accessible way) to managepotential conflicts of interest when patients engage withdifferent stakeholders (e.g., policies that require fulldisclosure,transparency,andaccountability).
HP
5 Resources 23 Theadequateinfrastructuresneededforpatients’andotherstakeholders’ involvement is in place (e.g., technologicalsupport, working space, information and communicationtechnologies) adapted to specific circumstances ofparticipants(e.gpossiblephysical,mental,cognitiveoranyimpairmentsetc.)
HP
7 Involvement andparticipation
32 Patients receive timely, clear, accessible and appropriateinformation about their involvement (i.e. what will beaddressed, what is expected from them, what theirparticipationwillentail,howthepracticewillbeorganisedetc.)
HP
36 The information about the outcomes included for eachphase of the involvement is communicated in plain andrespectful language, using accessible, clear andunderstandable formats, and taking into considerationpotential impairments (e.g., impaired vision, blindness,deafness and cognitive problems), low literacy levels ,cultural backgrounds and other relevant factors (e.g. age,disease/condition,etc.).
HP
8 Evaluation 42 The evaluation outcomes are linked to the aim andobjectivesofthePEpracticeinsettingresearchpriority.
HP
9 PEImpact 45 The practice identifies relevant outcomes and impactsregarding the involvement of patients in setting researchpriority.
HP
46.1 Research topic and priorities become more appropriate,basedonpatients’needs
HP
46.2 The product profile (drug, technology) become morerelevantandusableforpatients.
HP
46.3 Researchquestionsandoutcomes/endpointsbecomemorerelevantforpatients.
HP
59
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
2 Contextualizationof the practice onPE in ResearchPriority
4 The practice is based on an assessment of the patients’needs and preferences who will be engaged in researchprioritysettingprocesses.
MP
3 Targetparticipantsinvolvedinpatientengagement
7 Theprofilesofthepatientsengagedinsettingtheresearchpriorities reflect the relevant diversity of the populationtargeted. (e.g., gender, socioeconomic status, ethnicity,cultural background, geographical location etc.) andincludesgendersensitiveinformation.
MP
8 Patients engaged profile in setting research priorities areselectedtorepresentrelevantpatients’circumstances(e.g.,formerpatients,patientswhoareatrisk foradisease,butdo not yet have the disease, patients’ advocates, patients’representatives,patientorganizations,levelofexperienceinresearch priority setting, patient’s scientific knowledge/background).
MP
5 Resources 24 Thepracticeincludesguidance,andready-to-usetoolsandtemplates material that facilitates effective patientengagementimplementationinaclearandaccessibleway.
MP
7 Involvement andparticipation
37 Thepractice includesa singlepointof contactor anamedpersontowhompatientscanreachouttowhenneeded,forinformationand/or support, throughout their involvementinsettingresearchpriority.
MP
6 CapacityBuilding 25 The practice identifies the competencies required by allparticipants involved in setting research priority (e.g.knowledge and skill such as how to perform patientengagementinsettingresearchpriority,etc.).
LP
Round3(Meetinginperson)–DiscussionandFinalPrioritizationDiscussionandFinalPrioritizationprocess
Toeasediscussionsatthemeeting,thehighlyprioritiseditemsandnon-consensuscategoriesremainingfromthesecondroundwerefurtherelaboratedbythePARADIGMDelphiteam. Inordertoavoidredundancy,aproposalwasmadetomerge,rephraseorreallocatecategoriesandcriteria.Theywerepresentedtotheexpertpanelandthoroughlydiscussedatthemeeting.Comments provided by the experts in the first and second round were taken intoconsideration.
From the initial 10 thematic criteria, the experts agreed on 8 criteria that consisted of 29categories, and 6 subcategories. The experts rephrased categories in order to avoidredundancyandoverlapping.
Some parts of the discussions are captured below to provide more clarity on the keydiscussionsduringthemeeting.
Overall, experts hold that the disclaimers agreed at the meeting should be captured in thedocumentinaclearandaccessibleway.Besides,itwashighlightedthatitisnecessarytotake
60
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
into consideration any stakeholder and participant involved along the practice (e.g. patientrepresentatives, researchers, industrypartners, etc.). Furthermore, theyagreedon removingbracketsalongthedocumenttobereadwithease,andbecausetheywerealreadyincludedintheglossaryand/orfootnotes(Annex4).
Experts agreed on merging criterion 1, “Aim and objectives”, criterion 2,“Contextualization of the practice on patient engagement in research priority” andcriterion3,“Targetparticipantsinvolvedinpatientengagement”.Thenewcriterionwasnamed“Keyelementsofthepracticedesign”.Allthecategoriesincludedinthiscriterionwereconsideredbasicandrelevantforthepractice.
Expertspointedoutthatfeaturesapproachedincategories1,2and3overlapped,sotheywerecombinedinanewone.
Category 4was considered relevant but itwas rephrasedwith regard to every stakeholder:researchers,industryandpatients.
Categories7and8weremergedandexpertsconsidered that twonewdefinitionsshouldbeincluded in the glossary: “target population” and “patient’s circumstances”. Additionally,category15wasrephrased,makingiteasytoread,andwithclarificationsaboutsomewords:“specifictools”and“mechanisms”.
Criterion 4, “Code of conduct”, did not experience many changes. The three categoriesincludedinthiscriterionwererephrased.
The expert panel refused the idea of merged criteria 5, 6, and 7. All of them were barelychanged.
Regardingthecategorieswithinthecriterion5,“Resources”, somecategoriesweremergedand rephrased for the criterion to be clearer and more concise. The panel identifiedoverlappingofcategories20and21,whichweremergedtoavoidredundancy.Expertsmadeclearthat“itisnecessaryanequalframeworkandcompensationtosupportalltheparticipantsduringtheprocess”.Concerningcategories23and24,theywererephrased.
Criterion6,“Capacitybuilding”,wasbarelychanged.Expertsdiscussedcategory30andtheyconsideredthatthetrainingisreallyrelevant.Theyputintocontextthatwell-selectedpatientsmustbetrainedbeforetheirparticipation.
Again, categories within criterion 7, “Involvement and participation”, experiencedrephrasing andmerging to avoid repetition. Experts highlighted the need to clarify that thiscriterionreviewstheexpectationsoftheinvolvementinthepractice,notpatientexpectations.
Regarding formercriterion8, “Evaluationof thepractice insettingresearchpriorities”,categories 41 and 42 did not suffer any change. Categories 40 and 43 were rephrased. Inregard to category 43, the expert panel underlined that feedback is essential after thecompletionoftheprocess.
Expertsdiscussedaboutthelocationofcategory45,previouslypartofcriterion9,andfinallyreallocated in criterion 8. They considered that this category is an introductory step to the
61
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
following criterion “Patient engagement impact”. The experts added the term SMART to thecategory, to define relevance with specific and achievable outcomes. Each letter in SMARTreferstoadifferentcriterionforjudgingobjectives:Specific,Measurable,Assignable,RealisticandTimebound.
In relation to criterion 9, “Patient engagement impact” category 45 was removed from“Patient Engagement Impact” as mentioned before. The rest of the categories were barelychanged,withtheexceptionofsubcategories46.4and46.6.Theyfocusedthecategory46.4inenhancingpatients.Inregardtocategory46.4,thepanelconsideredthat“individualpatients,whoarenotmembersofanyorganisation,shouldbeinvolvedaswell”.
Respectingformersubcategory46.6,itwasrephrasedtoenforceitsmessage.
Finally,categories48and50fromcriterion10,“Sustainability”,wererephrasedtoclarifysometerms.Categories47and49weremerged;asitwasconsideredthatitscombinationgivesmorecontexts and enhances themessage. This criterionwas extensively discussed, because someexperts held that the public entities or patients organisations seek partnerships in order toparticipateinlong-termsustainablepractices,beingnotsoimportantfortheprivatesector.
Oncetheworkonnewspecificationwascompleted,theresultingcategoriesineachcriterionwereweighted. Later, expertsweighed the final 8 criteria by distributing 100 points amongthem.Thehighestweightwasassignedtocurrentcriterion1:“KeyelementsofPracticedesign”(19.5/100),whilethecriteriawiththelowestscorewere“Sustainability”(8/100),followedby“Resources”(10/100).
CurrentcriteriaandcategoryandtheircorrespondingweightsaredetailedinTable9.
Table9:Finalsetofcriteriaandcategories,andtheirweights,recommendedforevaluatingPEinsettingresearchpriorities
Newcriterianames
Criteriaweight
CategoriesCategoryweight
1.KeyelementsofPracticedesign
19.5
Theaimandobjectivesofthepracticeinsettingresearchprioritiesareagreedandunderstandablebyallstakeholders(e.g.patients,academics,researchers,pharmaceuticalcompanies,etc)involvedandrelatedtopatient'sneedsandinterest.
26.0
Thepracticeisbasedonanassessmentofthepatients’needsandpreferences. 22.5
Anyrelevantpolicydirectives,legal,ethics,governancerequirementand/orregulatoryframeworkabouthowtoengagepatientsinsettingresearchprioritieshavebeenconsidered.
11.5
62
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Thepatientsengagedinsettingresearchprioritiesreflectdiversityofthetargetpopulation*,patients'circumstances**andvulnerability.*Referstoculturalbackground,social-economicstatus,gender,age,ethnicity,educationallevel,disease,disabilityandvulnerability.**Formerpatients,patientswhoareatriskforadiseasebutdonotyethavethedisease,severityofthedisease,patient’sadvocate,carers,patient’srepresentatives,patientorganizations,levelofexperiences,patient'sscientificknowledge/background).
14.5
Therolesandresponsibilitiesareclearlydefined,agreedandassignedamongallparticipants 12.0
Thepracticeincludesspecifictools*andmechanisms**,toensurethatallparticipantsunderstandtheirownandother’srolesandresponsibilities.*(e.g.,rolesandresponsibilitycharts,proceduralguidancedocuments,protocols)**(e.g.,meetings,workshops,trainingsessions)
13.5
Totalmustbe100
2.Codeofconduct
11.5
Thepracticeincludesacodeofconduct,whichclearlystatestherulesandprocedures,includingethicalprinciples,ofparticipationinsettingprioritiesforallstakeholders.
44.0
Thetermsandconditionsofconfidentialityagreementsareinplace,presentedinaclearandaccessiblewaytothetargetpopulationinvolved.
30.0
Thepracticecontainsclearandaccessiblemechanismstomanagepotentialconflictsofinterest. 26.0
Totalmustbe100
3.Resources
10.00
Allparticipantsareinformedoftheavailableresources,includingequitablefinancialcompensationframeworktosupportthemduringtheprocess(e.g.,travelandsubsistenceexpenses,timemissedfromwork,child/elderlycareetc.).
27.5
Theinfrastructuresneededforparticipantsareinplace*andadaptedtospecificcircumstancesofparticipants***(e.g.,technologicalsupport,workingspace,communicationtechnologies)**(e.g.possiblephysical,mental,cognitiveoranyimpairment,etc)
41.0
Thepracticeincludesguidance,ready-to-usetoolsandtemplatesmaterialtofacilitateeffectivepatientengagementimplementation. 31.5
Totalmustbe100
4.Capacitybuilding
11.0
Thepracticeidentifiesthecompetenciesthatarerequiredtoperformpatientengagementinsettingresearchprioritiesbyallparticipants. 34.0
Thetrainingmaterialisadapted,comprehensibleandaccessibletoallparticipantstakingintoconsiderationimpairments,literacylevels,culturalbackgroundandthecircumstancesofvulnerablepatientsinvolvedinsettingresearchpriorities.
35.5
Thepracticeensuresthatallstakeholders(otherthanpatients)areadequatelytrainedfortheirrole. 30.5
Totalmustbe100
63
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
5.Involvementandparticipation
12.0
Patientsreceivetimely,clear,accessibleandappropriateinformationabouttheirinvolvementandwhatisexpectedofthem. 26.0
Thepatient'sparticipationwillbeproperlyplanned,takingintoaccounttimingrequirements,accessibility,andvulnerability. 18.5
Wheneverappropriate,andasagreedbyparticipants,regularfeedbackabouttheoutcomeiscommunicatedinaclearandadaptedway. 20.0
Allparticipantsaregiventheopportunitytogiveregularfeedbackabouttheprocess. 19.0
Thepracticeensuresthereisanamedkeycontactthatpatientscanreachoutthroughouttheprocess. 16.5
Totalmustbe100
6.EvaluationofthePEpracticeinsettingresearchpriority
12.0
Theevaluationframeworkisincludedandsharedamongtheparticipants. 15.5
Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice. 22.0
TheevaluationoutcomesarelinkedtotheaimandobjectivesofthePEpracticeinsettingresearchpriority. 24.5
Theevaluationoutcomesaresharedwithalltheparticipantsandfeedbackisenabledafterthecompletionoftheprocess. 14.5
ThepracticeidentifiesSMART*relevantoutcomesregardingtheinvolvementofpatientsinsettingresearchpriority.*SMART:SpecificMeasurableAssignableRealisticTimebound
23.5
Totalmustbe100
7.Patientengagementimpact
Impacts:
16.0
Researchtopicandprioritiesbecomemoreappropriate,basedonpatients’needs. 22.0
Theproductprofile(drug,technology)becomemorerelevantandusableforpatients. 19.0
Researchquestionsandoutcomes/endpointsbecomemorerelevantforpatients. 19.5
Credibilityofsubmittedresearchproposalsareenhancedwheninvolvingpatientsandpatients’organisations. 10.5
Resourceallocationswhenengagingpatientswithinindustrybecomemoreappropriateandarebasedonpatients’needs. 12.5
Completedstudieswillprovidemoreusefulinformationforpatientsanddecision-makingprocesswheninvolvingpatientsandpatientorganisations.
16.5
Totalmustbe100
8.Sustainability
8.00
Thepracticeisembeddedintheinstitutionororganisationand,whenrelevant,theallianceswithprivateand/orpublicinstitutionarefostered.
46.0
Humanandfinancialresourcesareallocatedforthelong-termcontinuityofthepracticeonPEinsettingresearchpriorities. 31.0
64
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
ThereisadisseminationandcommunicationplandemonstratingtheprocessandoutcomeofPE. 23.0
Totalmustbe100
Totalmustequal100
65
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Annex 6. Results of the Delphi on PE in Designing Clinical Trials
ResultsofRound1(Online)–RelevanceassessmentTable10:RelevantcategoriesonPEinDesigningClinicalTrials(Round1)
CriteriaID
Criteria CategoryID
Category
1 Aim andobjectives
3 TheaimandobjectivesofthepracticeonPEareclearandunderstandableforallparticipants
3 Targetparticipantsinvolved inpatientengagement
7 The patient profiles engaged in the design of clinical trials reflect therelevantdiversityof thepopulationtargeted(e.g.,gender,socioeconomicstatus, ethnicity, cultural background, geographical location etc.) andincludesgendersensitiveinformation.
11 The role and responsibilitiesof allparticipants involved in thedesignofclinicaltrials,includingpatients,areclearlydefinedandagreed.
4 CodeofConduct 15 Thepracticeincludesacodeofconduct,whichclearlystatestherulesandmechanismofparticipationindesignofclinical trials forallstakeholdersinvolved.
16 Effortsaremadetoensurethatallindividualsinvolvedknowtherulesofparticipationinclinicaltrialsdesign(e.g.,workshops,meetings).
17 The terms and conditions of confidentiality agreements are adequate,writteninaclearandaccessiblewayandadaptedtothetargetpopulationinvolved.
18 Potential conflicts of interest are addressed and managed (e.g., policiesthatrequirefulldisclosure,transparency,andaccountability).
19 Thepractices containsmechanisms (e.g., personalizedmeetings,waysofanonymousuncovering)to identifyandaddressdiscriminatory,coercive,intimidation, and unethical behaviours resulting from theirparticipationduringthedesignoftheclinicaltrialsprocess.
20 Funding resources for the practice on PE in design of clinical trials areclearly explained to all stakeholders involved prior to involvement, andanychangesthatoccurduringthepracticearecommunicatedup-front.
5 Resources 21 Allparticipants involvedindesignofclinical trialspracticeare informed,in clear and accessible way, of the resources available to support theirtasksduringtheprocess.
22 An appropriate and equitable financial compensation framework is inplace for all participants (e.g. professionals, patients etc.) invited toparticipate in the PE practice in design of clinical trials (e.g.,reimbursement of expenses for travel, subsistence, time missed fromwork,child/elderlycare).
6 Capacitybuilding 26 Comprehensive and accessible training or induction material andprograms to respond to the need of the people involved in design ofclinicaltrials.
7 Involvement andparticipation
30 Thetypeofinteractionamongthedifferentstakeholders(e.g.,cooperation,co-creation,information,advice…)isclearlydefined.
66
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
31 Considerationhasbeengiven to specificparticipants’ circumstancesandcharacteristics linked to but not limited to possible physical or mentalimpairments, cultural background, age and other relevant features (e.g.,useoflanguage,formatofmeetings,thevenue,informationprovided).
32 The timing requirement of the engagement in design of clinical trials isappropriately planned, taking into account patient’s needs, and basedupon the requirements of the engagement process (e.g. timeof the day,durationetc.).
35 The information about the outcomes included for each phase of theinvolvementiscommunicatedinplainandrespectfullanguageandusingaccessible, clear and understandable formats, taking into accountpotential impairments(e.g., impairedvision,blindness,anddeafnessandcognitive problems), low literacy levels, cultural backgrounds and otherrelevantfactors(e.g.age,disease/condition,etc.).
36 Asinglepointofcontactfororanamedpersontowhompatientscanreachout to when needed, for information and/or support, throughout theirinvolvementindesignofclinicaltrials.
8 Evaluation 41 The evaluation outcomes are linked to the aim and objectives of the PEpracticeindesignofclinicaltrials.
42 Theevaluationoutcomesaresharedwithall theparticipantsinvolvedinthe practice using appropriate channels and formats suited to theparticipant’scharacteristicsandneeds.
43 The practice includes procedures by which the conclusions of theevaluation are used (e.g., mechanisms to integrate the results of theevaluationinthedesignof thenextpracticeorinthecurrentonewithacontinuousimprovementprocess).
9 PEImpact 45.1 Improvedrecruitmentofstudyparticipants
45.2 Increasedretention(i.e.lowerdropout)ofstudyparticipants
45.4 Betteradherencetotheresearchprotocol,lessamendments
45.6 Faster study completion as a result of improved recruitment andadherencetotheprotocol
45.8 Moreappropriate,inclusive,sensitiveandethicaltrialdesign
10 Sustainability 46 The continuation of the practice is ensured through ownership andinstitutionalororganisationalanchoring.
Table11:NoconsensuscategoriesonPEinDesigningClinicalTrials(Round1)
CriteriaID
Criteria CategoryID
Category
1 Aimandobjectives 1 TheaimandobjectivesofthepracticeonPEindesignofclinicaltrialsare agreed with all the stakeholders (i.e. patients, academics,researchers,pharmaceuticalcompaniesetc.)involved.
2 The aim and objectives of the practice on PE respond to patient’sneedsandinterests.
2 Contextualization of thepractice on PE in the
4 The practice on PE in design of clinical trials is based on a pre-assessmentofpatients’wantsandneedswhenparticipatingindesign
67
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
DesignofClinicalTrials ofclinicaltrials.
5 A clear understanding of contextual factors that would affect theoutcomes of involving patients in design of clinical trials isdemonstrated (e.g. funding, health system and coverage,characteristicsofthepopulation,socioeconomicenvironment).
6 Any existing policy directives, legal (governance requirements)and/or regulatory framework (e.g. EMA) about how to engagepatientsinclinicaltrials’designhavebeenfollowed.
3 Target participantsinvolved in patientengagement
8 The targetparticipant’sprofilewhoparticipate in thedesignof theclinicaltrialrepresentthetargetpopulationthatwillberecruitedtotheclinical trials(e.g., formerpatients,peoplewhoareat risk for adisease,butdonotyethavethedisease,patients’advocates,patients’representatives,patientorganizations, levelof experience indesignofclinicaltrials).
9 The practice includes methods to identify individuals with theperspectives (or points of view) needed to engage in clinical trialsdesign(e.g.,patient’snetworks,healthcareprofessionals’networks,publicads).
14 The practice includes specific tools (e.g., roles and responsibilitycharts, procedures, procedural guidance documents, protocols) andmechanisms(e.g.,meetings,workshops, training sessions),areusedtoensurethatallparticipantsunderstandtheirownandother’srolesandresponsibilities.
5 Resources 23 Fundingisallocatedtocoverallelementsofthepractice(e.g.,humanresources,material),andisadequatefortheintendedwork.
24 The adequate infrastructures needed for patients’ and otherstakeholders’ involvement in design of clinical trials practice is inplace (e.g., technological support, working space, information andcommunicationtechnologies)andadaptedtospecificcircumstancesof participants (e.g. possible physical, mental, cognitive or anyimpairmentsetc.)
25 Thepracticeincludesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateseffectivepatientengagementimplementationinaclearandaccessibleway.
6 Capacitybuilding 29 Practices on PE in design of clinical trials incorporatetraining/coaching for stakeholders, other than patients, regardinghowtoincorporatepatientsinthedesignofclinicaltrials.
7 Involvement andparticipation
33 Mechanismstoensurethemeaningfulparticipationandcontributionof all participants involved in the PEactivity are in place taking inconsideration characteristics and circumstances of vulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc).
34 A regular feedback system to inform patients about theoutcomes/changesisincludedforeachphaseoftheinvolvement.
8 Evaluation 38 Anevaluationframeworkisincluded(structure,processandresults)anditenablesregularityoftheevaluation(frequency,timing…).
40 Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice.
9 PEImpact 44 Thepracticeidentifiesrelevantoutcomesandimpactsregardingtheinvolvementofpatientsinthedesignofclinicaltrials.
68
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
45.5 Improvedstudyparticipants’trialexperience.
45.9 Moreappropriatewordingand timingof research instruments (e.g.questionnairesandinterventions).
45.10 Improved patient information accessibility (e.g. lay summaries,informationsheets,consentform,recruitmentmaterial).
10 Sustainability 47 Humanandfinancialresourcesrequiredforthelong-termcontinuityof the practice on PE in design of clinical trials are identified andsecured.
48 The practice on PE in design of clinical trials identifies andestablishes alliances with other private and/or public institutions,citizensandothers,toensureitscontinuity,aslongasconfidentialityisrespected.
Table12:DroppedcategoriesonPEinDesigningClinicalTrials(Round1)
CriteriaID
Criteria CategoryID
Category
3 Targetparticipantsinvolvedinpatientengagement
10 PracticesonPEin in clinical trialsdesign consider the inclusionofotherrelevantpointsofviewseparatetothatofthepatients(e.g.carers, tutors,research and academia, healthcare professional,pharmaceutical/biotechnology/medical technology industry, policymaker,regulator,healthtechnologyassessment(HTA)agency,funder,payer…).
12 There is clearguidanceabouthow toassign roles amongallparticipantsinvolvedininthedesignofclinicaltrials.
13 Thepracticedisplaysflexibilityandopen-mindednessinassigningtherolesforthepatients,accordingtotheprofileandbackgroundofthepatient(e.g.assourcesofinformation,asconsultantsorascollaboratingpartnersoftheprocess).
6 Capacitybuilding 27 The trainingprogramspecifies timewhen trainingneeds tobeprovided,duration and frequency of the sessions (e.g. on demand, once, twice amonth/year).
28 Trainingforpatientsincludesareas,suchaslegalandregulatoryconceptsand challenges in research or deliberative process, concepts (e.g. basicstatisticsetc.),terminologyetc.
7 Involvement andparticipation
37 Mechanismsare in place toguaranteeallmembers have the same statuswithintheworkgroup,regardlessoftheirexperienceandknowledge.
8 Evaluation 39 Theevaluationframeworkissharedandagreedamongtheparticipantstofostercollaboration.
9 PEImpact 45.3 Increaseddiversityofstudyparticipants
45.7 Decreasecostsasaresultofimprovedrecruitment,retentionandpositivetrialexperience
10 Sustainability 49 Thereisadisseminationandcommunicationplan/strategiesoftheresultsin PE in design of clinical trials through culturally appropriate andaccessiblemechanisms.
69
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
ResultsofRound2(Online)–PrioritizationTable13:RelevantcategoriesonPEindesigningclinicaltrials(Round2)
(Newcategoriesinorange)
CriteriaID
Criteria CategoryID
Category
1 Aimandobjectives2 The aim and objectives of the practice on PE respond to patient’sneedsandinterests.
2 Contextualization ofthepracticeonPE intheDesignofClinicalTrials
6 Any existing policy directives, legal (governance requirements)and/or regulatory framework (e.g. EMA) about how to engagepatientsinclinicaltrials’designhavebeenfollowed.
3 Target participantsinvolved in patientengagement
9 The practice includes methods to identify individuals with theperspectives (or points of view) needed to engage in clinical trialsdesign (e.g., patient’snetworks,health careprofessionals’networks,publicads).
14 The practice includes specific tools (e.g., roles and responsibilitycharts, procedures, procedural guidance documents, protocols) andmechanisms(e.g.,meetings,workshops,trainingsessions),areusedtoensure that all participants understand their own and other’s rolesandresponsibilities.
5 Resources 23 Fundingisallocatedtocoverallelementsofthepractice(e.g.,humanresources,material),andisadequatefortheintendedwork.
24 The adequate infrastructures needed for patients’ and otherstakeholders’ involvement in design of clinical trials practice is inplace (e.g., technological support, working space, information andcommunication technologies) and adapted to specific circumstancesof participants (e.g. possible physical, mental, cognitive or anyimpairmentsetc.)
25 Thepracticeincludesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateseffectivepatientengagementimplementationinaclearandaccessibleway.
7 Involvement andParticipation
33 Mechanismstoensurethemeaningfulparticipationandcontributionof all participants involved in the PE activity are in place taking inconsideration characteristics and circumstances of vulnerablepopulation(e.g.,meetings,interviews,focusgroup,workshopsetc.).
34 A regular feedback system to inform patients about theoutcomes/changesisincludedforeachphaseoftheinvolvement.
38 Anevaluationframeworkisincluded(structure,processandresults)anditenablesregularityoftheevaluation(frequency,timing…)
8 Evaluation 40 Methods, tools and monitoring system are in place to evaluate PEpractice.
44 Thepractice identifiesrelevantoutcomesandimpactsregardingtheinvolvementofpatientsinthedesignofclinicaltrials.
70
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
9 PEImpact 45.5 Improvedstudyparticipants’trialexperience
45.9 More appropriatewording and timing of research instruments (e.g.questionnairesandinterventions)
45.10 Improved patient information accessibility (e.g. lay summaries,informationsheets,consentform,recruitmentmaterial)
Newcategory
Identification clinical trial endpoints that better reflect functionaloutcomes/benefitsforpatientsfromusingagiventherapycomparedtotraditional/standardregulatoryendpoints
10 Sustainability 47 Humanandfinancialresourcesrequiredforthelong-termcontinuityof the practice on PE in design of clinical trials are identified andsecured.
Table14:NoconsensuscategoriesonPEindesigningclinicaltrials(Round2)
(Newcategoriesinorange)
CriteriaID
Criteria CategoryID
Category
1 Aim andobjectives
1 TheaimandobjectivesofthepracticeonPEindesignofclinicaltrialsareagreed with all the stakeholders (i.e. patients, academics, researchers,pharmaceuticalcompaniesetc.)involved.
6 CapacityBuilding
29 PracticesonPE in design of clinical trials incorporate training/coachingfor stakeholders, other than patients, regarding how to incorporatepatientsinthedesignofclinicaltrials.
7 InvolvementandParticipation
Newcategory
When engaging patients in the design of clinical trials, patient-orientedoutcomesareobtainedthroughvalidinstruments
Table15:DroppedcategoriesonPEindesigningclinicaltrials(Round2)
CriteriaID
Criteria CategoryID
Category
2 Contextualization ofthe practice on PE inthe Design of ClinicalTrials
4 The practice on PE in design of clinical trials is based on a pre-assessment of patients’ wants and needs when participating indesignofclinicaltrials.
5 A clear understanding of contextual factors that would affect theoutcomes of involving patients in design of clinical trials isdemonstrated (e.g. funding, health system and coverage,characteristicsofthepopulation,socioeconomicenvironment).
71
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
3 Target participantsinvolved in patientengagement
8 Thetargetparticipant’sprofilewhoparticipateinthedesignoftheclinicaltrialrepresentthetargetpopulationthatwillberecruitedtotheclinicaltrials(e.g.,formerpatients,peoplewhoareatriskforadisease, but do not yet have the disease, patients’ advocates,patients’representatives,patientorganisations, levelofexperienceindesignofclinicaltrials).
10 Sustainability 48 The practice on PE in design of clinical trials identifies andestablishesallianceswith other privateand/orpublic institutions,citizens and others, to ensure its continuity, as long asconfidentialityisrespected.
Table16:RelevantcategoriesonPEindesigningclinicaltrialsandtheirPriority(Round2)
CriteriaID
Criteria CategoryID
Category Priority
1 Aim andobjectives
3 The aim and objectives of the practice on PE are clear andunderstandableforallparticipants
HP
4 CodeofConduct 16 Effortsaremadetoensurethatallindividualsinvolvedknowthe rules of participation in clinical trials design (e.g.,workshops,meetings).
HP
17 The terms and conditions of confidentiality agreements areadequate,writteninaclearandaccessiblewayandadaptedtothetargetpopulationinvolved.
HP
18 Potential conflicts of interest are addressed and managed(e.g., policies that require full disclosure, transparency, andaccountability).
HP
5 Resources 21 Allparticipantsinvolvedindesignofclinicaltrialspracticeareinformed, in clear and accessible way, of the resourcesavailabletosupporttheirtasksduringtheprocess.
HP
22 An appropriate and equitable financial compensationframework is inplace for allparticipants (e.g.professionals,patientsetc.)invitedtoparticipateinthePEpracticeindesignof clinical trials (e.g., reimbursement of expenses for travel,subsistence,timemissedfromwork,child/elderlycare).
HP
7 Involvement andparticipation
31 Consideration has been given to specific participants’circumstancesandcharacteristicslinkedtobutnotlimitedtopossible physical or mental impairments, culturalbackground, age and other relevant features (e.g., use oflanguage, format of meetings, the venue, informationprovided).
HP
35 Theinformationabouttheoutcomesincludedforeachphaseoftheinvolvementiscommunicatedinplainandrespectfullanguage and using accessible, clear and understandableformats, taking into account potential impairments (e.g.,impaired vision, blindness, and deafness and cognitiveproblems),lowliteracylevels,culturalbackgroundsandother
HP
72
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
relevantfactors(e.g.age,disease/condition,etc.).
8 Evaluation 41 TheevaluationoutcomesarelinkedtotheaimandobjectivesofthePEpracticeindesignofclinicaltrials.
HP
42 Theevaluationoutcomesaresharedwithalltheparticipantsinvolved in the practice using appropriate channels andformatssuitedtotheparticipant’scharacteristicsandneeds.
HP
9 PEImpact 45.8 Moreappropriate,inclusive,sensitiveandethicaltrialdesign HP
10 Sustainability 46 The continuation of the practice is ensured throughownershipandinstitutionalororganisationalanchoring.
HP
3 Targetparticipantsinvolved inpatientengagement
7 The patient profiles engaged in the design of clinical trialsreflect therelevantdiversityof thepopulationtargeted(e.g.,gender,socioeconomicstatus,ethnicity,culturalbackground,geographical location etc.) and includes gender sensitiveinformation.
MP
4 CodeofConduct 15 Thepractice includesacodeofconduct,whichclearlystatestherulesandmechanismofparticipationindesignofclinicaltrialsforallstakeholdersinvolved.
MP
20 FundingresourcesforthepracticeonPEindesignofclinicaltrialsareclearlyexplainedtoallstakeholdersinvolvedpriorto involvement, and any changes that occur during thepracticearecommunicatedup-front.
MP
6 CapacityBuilding
26 Thepractice includes comprehensiveandaccessible trainingorinductionmaterialandprogramstorespondtotheneedofthepeopleinvolvedindesignofclinicaltrials.
MP
7 Involvement andparticipation
30 Thetypeofinteractionamongthedifferentstakeholders(e.g.,cooperation, co-creation, information, advice…) is clearlydefined.
MP
36 A single point of contact for or a named person to whompatients can reach out to when needed, for informationand/or support, throughout their involvement in design ofclinicaltrials.
MP
8 Evaluation 43 Thepracticeincludesproceduresbywhichtheconclusionsofthe evaluation are used (e.g., mechanisms to integrate theresultsoftheevaluation inthedesignof thenextpracticeorinthecurrentonewithacontinuousimprovementprocess).
MP
9 PEImpact 45.1 Improvedrecruitmentofstudyparticipants MP
45.2 Increasedretention(i.e.lowerdropout)ofstudyparticipants MP
45.4 Betteradherencetotheresearchprotocol,lessamendments MP
45.6 Fasterstudycompletionasaresultof improvedrecruitmentandadherencetotheprotocol
MP
73
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
3 Targetparticipantsinvolved inpatientengagement
11 Theroleandresponsibilitiesofallparticipantsinvolvedinthedesignofclinicaltrials,includingpatients,areclearlydefinedandagreed.
LP
4 CodeofConduct 19 The practices contains mechanisms (e.g., personalizedmeetings, ways of anonymous uncovering) to identify andaddressdiscriminatory, coercive, intimidation, andunethicalbehaviours resulting from their participation during thedesignoftheclinicaltrialsprocess.
LP
7 Involvement andparticipation
32 The timing requirement of the engagement in design ofclinical trials is appropriately planned, taking into accountpatient’s needs, and based upon the requirements of theengagementprocess(e.g.timeoftheday,durationetc.).
LP
Round3(Meetinginperson)–DiscussionandFinalPrioritizationDiscussionandPrioritisation
Toeasediscussionsatthemeeting,thehighlyprioritiseditemsandnon-consensuscategoriesremainingfromthesecondroundwerefurtherelaboratedbythePARADIGMDelphiteam. Inordertoavoidredundancy,aproposalwasmadetomerge,rephraseorreallocatecategoriesandcriteria.Theywerepresentedtotheexpertpanelandthoroughlydiscussedatthemeeting.Comments provided by the experts in the first and second round were taken intoconsideration.
Fromtheinitial10criteriatheexpertsagreedon9criteriacomprisedof32categories,and11subcategories(Table17).
Theexpertsrephrasedcategoriesinordertoavoidredundancyandoverlapping.
Thefollowingparagraphsprovidemoreinformationaboutthekeydiscussionfollowedduringthemeetingofthecategories,whichcreatedmorecontroversyanddebate
Mainly, it was considered that the categories related to patients’ needs and types ofstakeholdersshouldbeclarifiedacrossthedocument.Furthermore,expertspreferredtheterm“patient representatives” to “patients”, as this term includes not only patients but carers,parents/tutors and/or staff from patient organisations too. It was replaced across thequestionnaire.Sometermshavebeenincludedasfootnotes.
Criterion1,“Aimandobjectives”experimentedseveralchanges.Ononehand,categories1and3weremerged,becauseofoverlapping,andcategory2wasrephrasedusingsynonyms.
On the other hand, category 32 was reallocated from “Involvement and participation” asexpertsconsideredthiscategoryshouldbeatthebeginningofthequestionnaire.
Criterion3,“TargetparticipantsinvolvedinPE”containedfourcategoriesattheendofthesecondround,allofthemwererephrasedatthemeeting.
74
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Criterion2,“ContextualisationofthepracticeonPEintheDesignofClinicalTrials”)and4, “Code of conduct” were merged into a new criteria named “Legal and ethicalconsideration”.Thesecriteriatriggeredadeepdebate,asthereweredifferentpointsofview.
Allthecategorieswererephrasedwiththeconsensusofthepanel.Besides,expertsreorderedcategories to facilitate the assessment. Language, aims of the categories, recipient of theassessment, etc.were exhaustively discussed.Themerges proposed by the PARADIGM teamwerenotaccepted,asexpertsconsideredthattheconceptscoveredacrossthecriterionwouldbereallydifficulttounderstand.
It isnoteworthy thatwithin this new criterion, category 16 was rephrased with the aim ofmakingclearthatthedescriptionoftheeffortsmadeintheprocessmustbecontemplated.
Likewise, category 18 generated an extensive debate about the management andcommunication of potential conflicts of interest, especially those referred to patients andhealthcareprofessionals.Someexpertsheldthatapatientorapatientrepresentativedoesnotnecessarilyrepresentalltherelevantpointsofview,whichcouldcauseaconflict.Moreover,itshould be clarified, in every practice, what a conflict of interest for a patient/patientrepresentativeis.
It was also pointed out that “…conflicts of interestmust be avoided, notmanaged” in a PEpractice. However, it was discussed that this is not always possible. For example, if theinvolvementofsomehealthcareprofessionalsisavoidedbecauseoftheirconflictsofinterest,experts with strong knowledge in the field would be excluded; therefore, a valuablecontributioncouldbemissed
Likewise, in the discussionof category 20, related to the funding resources for participants,discussionaboutpaymentsandfundingrecipientsweredebated.Therewasdiversityofpointsof views. On one side, some of the experts considered, neither patients nor professionalsshould collect payments for their participation because of the potential conflicts of interest.Otherviewsconsideredthatanyactivityshouldberemuneratedtoanystakeholder,healthcareprofessionals and patients involved because they are contributing with their knowledge,expertise,timeandeffort.
Criterion5,“Resources”,wasminimallychanged.Threeofthecategorieswererephrased.
After the analysis of the results in second-round, it was not clear that category 29 withincriterion6,“Capacitybuilding”,wasrelevant.Finally,itwasrephrased.
Categories 29 and 30 assessed the same activity but in different population, so they weremaintainedseparated.
Criterion7,“Involvementandparticipation”,sufferedseveralchanges.Thepanelofexpertsacceptedthesuggestionofmergingthecategories21and33.Likewise,theyacceptedtomergecategories 34 and 36 into amore comprehensive category, considering that to give regularfeedbackclosestheloopofpatientrepresentatives’involvementintheimplementationofthepractice.
75
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Asmentionedbefore,category32wasremovedfromthiscriterionandreallocatedincriterion1.
The new category proposed by an expert in the previous Delphi rounds (Criterion 7:Involvementandparticipation:WhenengagingpatientsintheDesignofclinicaltrials,patient-oriented outcomes are obtained through valid instruments, when presented to the panel ofexperts,itwasdiscardedbecausetheyconsideredthat“validinstruments”weretoodifficulttohaveinthiscontextsotheydiscardedthecategory.
Criterion 8, “Evaluation of the PE practice in the design of clinical trials” was kept asoriginallypresented.
Thepaneldetectedoverlappingamongcategories38and40,allidentifyingdifferentfeaturesforwhichthePEpracticeshouldbeevaluated(methods,toolsandmonitoringsystems).
Categories41and43wererephrased.Inthecaseofcategory42,expertsrephrasedto“closethe feedback loop”. The aim was to “make sure the information is shared and patientrepresentativesreceiveit”.
Category44wasdroppedsinceitwasconsideredtobeconfusing.However,itwasthoughttobetterfitasanewsubcategorywithinthecategory“Impact”.
Criterion9, “Patientengagement impact”, triggeredadeepdebateabouthow tomeasuresomecategories,how tomake the rephrasingethical and inclusive, etc. Finally, categories inthiscriterionwereorganisedassubcategorieswithincategory45:“Impacts”.
Category44wasrephrasedandreallocatedasanewsubcategoryandanewsubcategorywasaddedbythepanel.
Criterion10,“Sustainability”,hastwocategories.Category46wasrephrased;itwassaidthatit’snotalwaysnecessarytoensurethecontinuationofthepracticebecauseitissystematicallyembedded.
Finally,category47waskeptasoriginallypresentedtothepanel.
Oncethisworkwascompleted,theresultingcategoriesineachcriterionwereweightedaswellasthefinal8criteria,bydistributing100pointsamongthem.
The highest weight was assigned to current criterion 1: “Patient engagement impact” and“Aims and objectives” (both of themwith a weight of 14/100), while the criteria with thelowestscorewere“Sustainability”(8/100),followedby“Resources”(10/100)and“EvaluationofthePEpracticeinsettingclinicaltrials”(10/100).
CurrentcriteriaandcategoryandtheircorrespondingweightsaredetailedinTable17.Table17:Finalsetofcriteriaandcategories,andtheirweights,recommendedforassessingPEpracticesinthedesignofclinicaltrials(followingtheassessmentpracticeprocess).
Newcriterianames
Criteriaweight
CategoriesCategoryweight
76
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
1. Aims andobjectives 14
ThereisgeneralagreementontheaimsandobjectivesofthepracticeonPEandtheseareunderstandablebyallrelevantstakeholdersinvolvedinthedesignofCT.
40.6
Theaimsandobjectivesof thepracticeonPE focusonpatient’sneedsandinterests. 34.7
The timeline for the engagement in the design of clinical trials isappropriately planned and allows for incorporation throughout theprocess.
24.7
Totalmustequal100
2. Targetparticipantsinvolved
12
The patient representatives* engaged in the design of clinical trialsreflect the relevant diversity** of the target population.* Patients, carers, parents or tutors and/or staff from patient’sorganization that act as a liaison between the patients and otherorganisations, represent their interests and advocates for theirrights.** Including but not limited to cultural background, socio-economicstatus, gender, age, ethnicity, educational level, disease, disability,vulnerability,etc.
32.9
ThepracticeofPEincludesacleardescriptionoftheprocessandcriteriafollowedtoidentifypatientrepresentativestoparticipateinthedesignofclinicaltrials.
19.0
The role and responsibilities of all target participants involved in thedesign of clinical trials, including patient representatives, are clearlydefinedandagreed.
23.7
The practice includes a clear description of the process followed* tomake sure thatallparticipantsunderstand theirownandother's rolesand responsibilities.*Includingdocuments,charts,guidance,training,meetings,workshops.
24.4
Totalmustequal100
3. Legal andethicalconsideration
11
The practice includes a code of conduct, which clearly states theprinciplesofparticipationindesignofclinicaltrialsforallstakeholdersinvolved.
18.9
The practice contains procedures to identify and address unethicalbehaviours, towards all stakeholders, during their participation in thedesignoftheclinicaltrialsprocess.
10.9
There is a clear description of efforts made to ensure that allstakeholders involved know the principles of participation in clinicaltrialsdesign(e.g.,workshops,meetings).
11.0
The terms and conditions of all legal agreements are written andcommunicated ina clear andaccessiblewayandadapted to the targetpopulationinvolved.
14.2
Potential conflicts of interest are addressed and managed (up toavoidance). For this, policies that require full disclosure, transparencyandaccountabilityaredeveloped.
15.7
Funding resources for thepracticeonPEin thedesignof clinical trialsareclearlydocumentedandexplainedtoallstakeholdersinvolvedpriorto involvement, and any changes that occur during the practice arecommunicatedup-front.
14.0
77
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
All relevantpolicydirectives, legal and/or regulatory frameworkhavebeenfollowedwhenengagingpatientsinthedesignofclinicaltrials. 15.3
Totalmustequal100
4.Resources 10
All participants involved in the design of clinical trials practice areinformed, in clear and accessible way, of the resources available tosupporttheirtasksduringtheprocess.
19.3
Aclear,transparentandequitablefinancialcompensationframeworkisinplaceforpatientrepresentativeswhoparticipateinthePEpracticeinthedesignof clinical trials (e.g., reimbursementof expenses for travel,timemissedfromwork,subsistence,child/elderlycare,stipends).
24.1
Fundingisallocatedtocovergovernance,administrationandoperationsof the practice in PE on design of clinical trials, is adequate for theintendedwork.
18.6
Infrastructure is in place to support the practice in PE on design ofclinical trials and is adapted to specific circumstances of patientrepresentatives (e.g. technological support,working space, informationandcommunicationtechnologies).
21.4
Theresourcesincludeguidelines,andready-to-usetoolsandtemplates,materialthatfacilitateseffectivepatientengagement implementationinaclearandaccessibleway.
16.6
Totalmustequal100
5. Capacitybuilding 10
Comprehensive and accessible trainings or induction materials andprograms are available to respond to the needs of the patientrepresentativesinvolvedinthedesignofclinicaltrials.
54.8
PracticesonPEindesignofclinicaltrialsincorporatetraining/coachingfor stakeholders, other than patient representatives, regarding how toincorporatepatientsinthedesignofclinicaltrials.
45.2
Totalmustequal100
6. Involvementandparticipation
11
The type of interaction among the different stakeholders is defined ateachstage(e.g.co-creation,advice,consultation...). 18.4
Therearemechanisms(e.g.language,formatofmeetings,thevenue,timeof the day, etc.) in place to ensure participation of patientrepresentatives, taking into consideration participant’s characteristics,andcircumstancesofvulnerablepopulation.
29.4
Patient representatives receive timelyand regular feedbackabout theoutcomes/changes for each phase of the involvement, in a clear andunderstandableformatandadaptedtopatients'circumstances.
31.7
Aclearpointofcontactisavailabletowhompatientrepresentativescanreachout towhenneeded for informationand/or support, throughouttheirinvolvementinthedesignofclinicaltrials.
20.6
Totalmustequal100
7.Evaluationofthe PE practicein settingclinicaltrials
10
Therearemethods,toolsandmonitoringsystemsinplacetoevaluatethePEpractice. 24.2
Theevaluation criteria are linked to theaimsandobjectivesof thePEpractice. 23.3
78
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Theevaluationoutcomesaresharedwithallthestakeholdersinvolvedinthe practice using appropriate channels and formats suited to patientperspectives'circumstancesandneeds.
25.6
The evaluation outcomes are used to improve future PE practices fordesignofclinicaltrials. 26.9
Totalmustequal100
8. Patientengagementimpact
14
Impacts:
Improvedrecruitment. 7.2
Improvedretention(i.e.fewerdropouts)ofstudyparticipants. 6.8
Betteradherencetotheresearchprotocol. 7.1
Feweramendmentstotheresearchprotocol. 7.3
Improvedtrialexperienceforstudyparticipant. 11.1
Moretimelystudycompletion. 7.3More inclusive, sensitive and ethical trial design, which areappropriate for specific needs and circumstances of the targetpopulation.
12.4
Thewordingandtimingofresearch instruments(e.g.questionnairesand interventions) are appropriate for specific needs andcircumstancesofthetargetpopulation.
9.8
Patient information and education materials (e.g. lay summaries,information and education sheets, consent form, recruitmentmaterial)areappropriatetothespecificneedsandcircumstancesofthetargetpopulation.
10.9
Additional study-specific relevant measure are included tocomplementthedescribedones. 5.6
Identificationofmeaningfulendpointsforpatients. 14.5 Totalmustequal100
9.Sustainability 8
Thecontinuationofthepracticeisensuredthroughownershipanditissystematically embedded in the institution or organisation for allstakeholders.
47.8
Humanandfinancialresourcesrequiredforthe long-termcontinuityofthe practice on PE in the design of clinical trials are identified andsecured.
52.2
Totalmustequal100
Totalmustequal100
79
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Annex 7. Results of the Delphi on PE in Early Dialogues
ResultsofRound1(Online)–Relevanceassessment
Table18:RelevantcategoriesonPEinEarlyDialogueswithHTA&Regulators(Round1)
CriteriaID
Criteria CategoryID
Category
1 Aimandobjectives 3 The aim and objectives of the practice on PE in early dialogues areclearandunderstandableforallparticipants.
2 Contextualization ofthe practice on PE inEarlyDialogues
6 Any existing policy, legal (governance requirements) and regulatoryframework(e.g.EMA)abouthowtoengagepatientsinearlydialogueswithHTAandregulatorshavebeenfollowed.
4 CodeofConduct 17 Thetermsandconditionsofconfidentialityagreementsareadequate,written in a clear and accessible way and adapted to the targetpopulation.
18 The practice on PE in early dialogues incorporates mechanisms(explainedtoallstakeholdersinaclearandaccessibleway)tomanagepotential conflicts of interest when patients engage with differentstakeholders (e.g., policies that require full disclosure, transparency,andaccountability).
19 Thepracticecontainsprocedures(e.g.,personalizedmeetings,waysofanonymous uncovering) to identify and address discriminatory,coercive, intimidating, and unethical behaviours resulting from theparticipationduringtheearlydialogueprocess.
5 Resources 21 Allparticipants involved inearlydialogueswithHTAand regulatorsareinformedinclearandaccessibleway,oftheresourcesavailabletosupporttheirtasksduringtheprocess.
22 Thepracticeincludesanequitablefinancialcompensationframeworkfor all participants (e.g. professionals, patients etc.) invited toparticipate in the early dialogues with HTA and regulators (i.ereimbursementofexpensesfortravel,subsistence, timemissedfromwork,child/elderlycareetc.).
23 Funding is allocated to cover all the elements of the practice (e.g.,humanresources,materialresources),andisadequatetotheintendedwork.
24 AdequateinfrastructuresneededforPEinearlydialoguesareinplace(e.g., technological support, working space, information andcommunicationtechnologies)andadaptedtospecificcircumstancesofparticipants (e.g. possible physical mental, cognitive or anyimpairmentsetc.).
25 Thepractice includesguidance,andready-to-usetoolsandtemplatesmaterial that facilitateseffectivepatientengagement implementationinaclearandaccessibleway.
6 Capacitybuilding 26 The practice on PE in early dialogues identifies the competenciesrequired by all stakeholders involved in early dialogues with HTAbodies and regulators (e.g. knowledge and skill such as how to
80
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
performpatientengagementinearlydialogues,etc.).
28 The practice includes comprehensive and accessible training orinduction material and programs to respond to the need of theparticipantsinvolvedinearlydialogueswithHTAandregulators.
7 Involvement andparticipation
31 Themechanism of interaction among the stakeholder is adapted totheirneeds.(e.g.meetings,online,skype..etc.).
32 Patientsreceivetimely,clear,accessibleandappropriateinformationabouttheirinvolvement(i.e.whatwillbeaddressed,whatisexpectedfromthem,whattheirparticipationwillentail,howthepracticewillbeorganizedetc.)
33 The practice considers specific patients’ circumstances andcharacteristicslinkedtobutnotlimitedtopossiblephysicalormentalimpairments, cultural background, age and other relevant features(e.g., use of language, format of meetings, the venue, informationprovided).
34 The timing of the engagement in early dialogues with HTA andregulators is appropriatelyplanned,basedupon therequirementsofthereviewprocess.
35 The practice on PE in early dialogues sets mechanisms to ensure aproper deliberative process (e.g. meetings, interviews, focus group,workshops…).
36 A regular feedback system to inform patients about theoutcomes/changesisincludedineachphaseoftheinvolvement.
37 The information about the outcomes included for each phase of theinvolvementiscommunicatedinplainandrespectfullanguage,usingaccessible,clearandunderstandableformats,andtakingintoaccountpotentialimpairments(e.g.,impairedvision,blindness,anddeafness),low literacy levels , cultural backgrounds and other relevant factors(e.g.age,disease/condition,etc.).
38 Thepractice includesa singlepointofcontactoranamedperson towhompatientscanreachouttowhenneeded,forinformationand/orsupport,throughouttheirinvolvementinearlydialogues.
8 Evaluation 41 Anevaluationframeworkis included(structure,processandresults)anditenablesregularityevaluation(frequency,timing…).
44 TheevaluationoutcomesarelinkedtotheaimandobjectivesofthePEpracticeinearlydialogues.
45 Theevaluationoutcomesaresharedwithalltheparticipantsinvolvedin the practice using appropriate channels and formats suited toparticipants’characteristicsandneeds.
9 PEImpact 48.1 Betterunderstandingof technologies’ impact in real life context andthequalityoflifeaspects
48.2 Morepracticalandbetterdesigneddevelopmentplanformedicines.
48.3 Higheraccuracyinmeasuringneedsandpreferencesofpatients.
48.4 Better quality of the evidence assessed during the later HTAassessment.
48.5 Moreusefuladvicegiventoaidcompanydecisions.
81
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
48.6 Higherrelevanceoftheadvicegiventothelocalcontextofapplication
48.10 Inclusion of patients’ relevant outcomes and end-points in clinicaltrialsdatacollectionandanalysis.
Table19:NoconsensuscategoriesonPEinEarlyDialogueswithHTA&Regulators(Round1)
CriteriaID
Criteria CategoryID
Category
1 Aimandobjectives 1 The aim and objectives of PE in Early Dialogues reflects patients’needs,experiencesandexpectationsintheearlydialoguesprocess.
2 Contextualization ofthe practice on PE inEarlyDialogues
4 In thepracticeonPE inearlydialogues, theprocess isbasedon theassessmentofthepatient’spreferencesandneeds.
5 Aclearunderstandingofcontextualfactorsthatinformtheprocessofpatient engagement is demonstrated so that the quality of dialoguewheninvolvingpatientsinearlydialogueswithHTAsandregulatorsisimproved.
3 Target participantsinvolved in patientengagement
7 The profiles of patients engaged in early dialogues with HTA andregulators reflect the relevant diversity of the population targeted.(e.g., gender, socioeconomic status, ethnicity, cultural background,geographicallocationetc.)andincludesgendersensitiveinformation
9 The practice of PE in early dialogues includes methods to identifypatients who match the clinical criteria needed for a particulardialogue(e.g.,patient’snetworks,healthcareprofessionals’networks,publicadvertiseetc.).
10 The practice supports patients to engage and/or reach out tovulnerable groups to involve them in early dialogueswithHTA andregulators,aswellastootherpatients.
11 Relevantpointsofviewotherthanpatients(e.g.carers,parentsetc.)areconsidered.
12 In the practice, the role and responsibilities of all participantsinvolved,includingpatientsareclearlydefinedandagreed.
14 Guidance,trainingadequateisprovidedtoensurethatallparticipantsunderstand their own role and responsibilities, as well as theresponsibilitiesandrolesofothers.
4 CodeofConduct 15 Thepracticeincludesacodeofconduct,whichclearlystatestherulesof participation in early dialogues with HTA and regulators for allparticipantsinvolved.
20 FundingresourcesforthepracticeonPEinearlydialoguesareclearlyexplainedtoallstakeholders involved,priorto involvement,andanychanges that occur during the practice of PE in early dialogues arecommunicatedup-front.
6 Capacitybuilding 27 ThepracticeonPE inearlydialogues includes trainingprogramsonHTA/Regulatory processes/clinical trial design and methods topatients (e.g., written information, in-house training, online,personalizedassistance,andwebinars).
82
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
30 ThepracticeonPEinearlydialogues incorporatestraining/coachingfor all stakeholders, other than patients, regarding how toincorporatepatientsinearlydialogueswithHTAandregulators.
8 Evaluation 42 The evaluation framework is shared and agreed among theparticipantstofostercollaboration.
43 Methods,toolsandmonitoringsystemareinplacetoevaluatethePEpractice.
46 The practice includes procedures by which the conclusions of theevaluation are used (e.g. mechanisms to integrate the results of theevaluationinthedesignofthenextpracticeorinthecurrentonewithacontinuousimprovementprocess).
9 PEImpact 47 The practice identifies relevant outcomesand impacts regarding theinvolvementofpatientsinearlydialogueswithHTAandregulators.
48.8 TransparencyofHTA/regulationprocesses.
48.11 MutualunderstandingbetweenallinvolvedstakeholdersincludingthepatientcommunityabouttheregulatoryandHTAprocess.
10 Sustainability 50 Humanandfinancialresourcesrequiredforthe long-termcontinuityofthepracticeonPEinearlydialoguesareidentifiedandsecured.
Table20:DroppedcategoriesonPEinEarlyDialogueswithHTA&Regulators(Round1)
CriteriaID
Criteria CategoryID
Category
1 Aim andobjectives
2 The aim and objectives of the practice on PE in early dialogueswere co-designedandagreedwithallthestakeholdersinvolved.
3 Targetparticipantsinvolved inpatientengagement
8 PatientsengagedinearlydialogueswithHTAandregulatorsareselectedtorepresent relevant patients’ circumstances (e.g., former patients, patientswho are at risk for a disease, but do not yet have the disease, patients’advocates, patients’ representatives, patient organizations, level ofexperience inearlydialogueswithHTA and regulators,patient’s scientificknowledge/background).
13 There is clear guidance about how to assign roles among all participantsinvolvedinearlydialogues.
4 Codeofconduct 16 ThepracticeonPE inearlydialogues includesmechanisms, adequateandadapted to the target population to ensure that all individuals involvedknowtherulesofparticipationinearlydialogueswithHTAandregulators(e.g.,workshops,meetings).
6 Capacitybuilding 29 The training program specifies time when training needs to be provided,duration and frequency of the sessions (e.g., on demand, once, twice amonth/year).
7 Involvement andparticipation
39 Mechanisms are in place to guarantee all members have the same statuswithintheworkgroup,regardlessoftheirexperienceandknowledge.
40 ThepracticeonPEinearlydialoguesfostersdiscussionandagreementwithpatients about identifying and selecting preferencesabout the ED processwithHTAbodiesandregulators.
9 PEImpact 48.7 Increased public knowledge and awareness of HTA and regulatory
83
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
processes.
48.9 Acceptabilityofregulatorydecisions/HTAreportsforstakeholders.
10 Sustainability 49 The continuation of the practice is ensured through ownership andinstitutionalorpracticeorganizationanchoring.
51 The practice on PE in early dialogues identifies and establishes allianceswithotherprivateand/orpublicinstitutions,citizensandothers,toensurethecontinuityofthePEinEDaslongasconfidentialityisrespected.
52 There isadisseminationandcommunicationplan/strategiesof theresultsof theprocessofPE inearlydialogues throughculturally appropriateandaccessiblemechanisms.
ResultsofRound2(Online)–PrioritizationTable21:RelevantcategoriesonPEinEarlyDialogueswithHTA&Regulators(Round2)
CriteriaID
Criteria CategoryID
Category
3 Targetparticipantsinvolved inpatientengagement
11 Relevantpointsofviewotherthanpatients(e.g.carers,parentsetc.)areconsidered.
12 In the practice, the role and responsibilities of all participants involved,includingpatientsareclearlydefinedandagreed.
14 Guidance, training adequate is provided to ensure that all participantsunderstand their own role and responsibilities, as well as theresponsibilitiesandrolesofothers.
4 Code ofConduct
15 Thepracticeincludesacodeofconduct,whichclearlystatestherulesofparticipation in early dialogues with HTA and regulators for allparticipantsinvolved.
20 Funding resources for the practice on PE in early dialogues are clearlyexplained to all stakeholders involved prior to involvement, and anychanges that occur during the practice of PE in early dialogues arecommunicatedup-front.
9 PEImpact 47 The practice identifies relevant outcomes and impacts regarding theinvolvementofpatientsinearlydialogueswithHTAandregulators.
48.8 TransparencyofHTA/regulationprocesses.
Table22:NoconsensuscategoriesonPEinEarlyDialogueswithHTA&Regulators(Round2)
CriteriaID
Criteria CategoryID
Category
84
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
1 Aim andobjectives
1 The aim and objectives of the practice respond to patient’s needs whenparticipatingintheearlydialoguesprocess.
3 Targetparticipantsinvolved inpatientengagement
9 ThepracticeonPEinearlydialogues includesmethods to identify individualseligible toparticipate inearlydialogueswithHTAand regulators (e.g., agencydatabases, those involved in other parts of HTA or regulatory processes,patients’networks,healthcareprofessionals’networks,publicadvertiseetc.).
10 Thepracticesupportspatientstoengageand/orreachouttovulnerablegroupstoinvolvetheminearlydialogueswithHTAandregulators,aswellastootherpatients.
8 Evaluation 43 Methods,toolsandmonitoringsystemareinplacetoevaluatePEpractice
46 Thepracticeincludesproceduresbywhichtheconclusionsoftheevaluationareused(e.g.mechanismstointegratetheresultsoftheevaluationinthedesignofthenextpracticeorinthecurrentonewithacontinuousimprovementprocess).
9 PEImpact 48.11 Mutualunderstandingbetweenall involved stakeholders including thepatientcommunityabouttheregulatoryandHTAprocess.
10 Sustainability 50 Human and financial resources required for the long-term continuity of thepracticeonPEinearlydialoguesareidentifiedandsecured.
Table23:DroppedcategoriesonPEinEarlyDialogueswithHTA&Regulators(Round2)
CriteriaID
Criteria CategoryID
Category
2 Contextualizationof the practice inPE in EarlyDialogues
4 In the practice on PE in early dialogues, the process is based on theassessmentofpatients’preferencesandneeds.
5 A clear understanding of contextual factors that inform the process ofpatientengagementisdemonstratedsothatthequalityofdialoguewheninvolving patients in early dialogues with HTAs and regulators isimproved.
3 Targetparticipantsinvolvedinpatientengagement
7 The profiles of patients engaged in early dialogues with HTA andregulatorsreflecttherelevantdiversityofthepopulationtargeted.(e.g.,gender, socioeconomic status, ethnicity, cultural background,geographicallocationetc.)andincludesgendersensitiveinformation
6 Capacitybuilding 27 The practice on PE in early dialogues includes training programs onHTA/Regulatoryprocesses/clinicaltrialdesignandmethodstopatients(e.g., written information, in-house training, online, personalizedassistance,andwebinars).
30 ThepracticeonPEinearlydialoguesincorporatestraining/coachingforall stakeholders, other than patients, regarding how to incorporatepatientsinearlydialogueswithHTAandregulators.
9 Evaluation 42 Theevaluationframeworkissharedandagreedamongtheparticipantstofostercollaboration.
85
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
Table24:RelevantcategoriesonPEinEarlyDialogueswithHTA&RegulatorsandtheirPriority(Round2)
CriteriaID
Criteria CategoryID
Category Priority
1 Aimandobjectives3 The aim and objectives of the practice on PE in earlydialogues are clear and understandable for allparticipants.
HP
4 Codeofconduct 17 Thetermsandconditionsofconfidentialityagreementsareadequate,writteninaclearandaccessiblewayandadaptedtothetargetpopulation.
HP
18 The practice on PE in early dialogues incorporatesmechanisms (explained to all stakeholders in a clearand accessible way) to manage potential conflicts ofinterest when patients engage with differentstakeholders (e.g., policies that require fulldisclosure,transparency,andaccountability).
HP
7 Involvement andparticipation
36 Aregularfeedbacksystemtoinformpatientsabouttheoutcomes/changes is included in each phase of theinvolvement.
HP
37 Theinformationabouttheoutcomesincludedforeachphaseoftheinvolvementiscommunicatedinplainandrespectful language, using accessible, clear andunderstandable formats, and taking into accountpotentialimpairments(e.g.,impairedvision,blindness,and deafness), low literacy levels , culturalbackgrounds and other relevant factors (e.g. age,disease/condition,etc.).
HP
9 PEImpact 48.1 Betterunderstandingoftechnologies’impactinreallifecontextandthequalityoflifeaspects
HP
48.2 More practical and better designed development planformedicines
HP
48.3 Higheraccuracyinmeasuringneedsandpreferencesofpatients
HP
48.10 Inclusionofpatients’relevantoutcomesandend-pointsinclinicaltrialsdatacollectionandanalysis.
HP
2 ContextualizationofthepracticeinPEinEarlyDialogues
6 Any existing policy, legal (governance requirements)and regulatory framework (e.g EMA) about how toengage patients in early dialogues with HTA andregulatorshavebeenfollowed.
MP
4 Codeofconduct 19 The practice contains procedures (e.g., personalizedmeetings,ways of anonymous uncovering) to identifyandaddressdiscriminatory,coercive,intimidating,andunethical behaviours resulting from the participation
MP
86
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
duringtheearlydialogueprocess.
5 Resources 21 All participants involved in early dialogues with HTAand regulators are informed in clear and accessibleway, of the resources available to support their tasksduringtheprocess.
MP
22 The practice includes an equitable financialcompensation framework for all participants (e.g.professionals,patientsetc.)invitedtoparticipateintheearly dialogues with HTA and regulators (i.ereimbursement of expenses for travel, subsistence,timemissedfromwork,child/elderlycareetc.).
MP
23 Funding is allocated to cover all the elements of thepractice (e.g., human resources, material resources),andisadequatetotheintendedwork.
MP
24 Adequate infrastructures needed for PE in earlydialogues are in place (e.g., technological support,working space, information and communicationtechnologies)andadaptedtospecificcircumstancesofparticipants(e.g.possiblephysicalmental,cognitiveoranyimpairmentsetc.).
MP
25 Thepractice includesguidance,andready-to-usetoolsandtemplatesmaterialthatfacilitateseffectivepatientengagement implementation in a clear and accessibleway.
MP
6 Capacitybuilding 26 The practice on PE in early dialogues identifies thecompetenciesrequiredbyall stakeholders involved inearly dialogues with HTA bodies and regulators (e.gknowledge and skill such as how to perform patientengagementinearlydialogues,etc).
MP
7 Involvement andparticipation
31 Themechanismofinteractionamongthestakeholderisadaptedtotheirneeds.(e.g.meetings, online,skype..etc).
MP
32 Patients receive timely, clear, accessible andappropriate information about their involvement (i.ewhat will be addressed, what is expected from them,what their participation will entail, how the practicewillbeorganizedetc)
MP
33 Thepracticeconsidersspecificpatients’circumstancesandcharacteristicslinkedtobutnotlimitedtopossiblephysical or mental impairments, cultural background,age and other relevant features (e.g., use of language,formatofmeetings,thevenue,informationprovided).
MP
34 The timingof theengagement in earlydialogueswithHTA and regulators is appropriately planned, basedupontherequirementsofthereviewprocess.
MP
87
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
38 The practice includes a single point of contact or anamedpersontowhompatientscanreachouttowhenneeded, for information and/or support, throughouttheirinvolvementinearlydialogues.
MP
9 PEImpact 48.4 BetterqualityoftheevidenceassessedduringthelaterHTAassessment.
MP
48.5 Moreusefuladvicegiventoaidcompanydecisions MP
48.6 Higherrelevanceoftheadvicegiventothelocalcontextofapplication.
MP
6 Capacitybuilding 28 The practice includes comprehensive and accessibletrainingorinductionmaterialandprogramstorespondto the need of the participants involved in earlydialogueswithHTAandregulators.
LP
7 Involvement andparticipation
35 ThepracticeonPEinearlydialoguessetsmechanismstoensureaproperdeliberativeprocess(e.g.meetings,interviews,focusgroup,workshops…).
LP
8 Evaluation 41 An evaluation framework is included (structure,process and results) and it enables regularityevaluation(frequency,timing…).
LP
44 The evaluation outcomes are linked to the aim andobjectivesofthePEpracticeinearlydialogues.
LP
45 The evaluation outcomes are shared with all theparticipantsinvolvedinthepracticeusingappropriatechannels and formats suited to participants’characteristicsandneeds.
LP
Round3(Meetinginperson)–DiscussionandFinalPrioritization
DiscussionandFinalPrioritisation
Toeasediscussionsatthemeeting,thehighlyprioritiseditemsandnon-consensuscategoriesremainingfromthesecondroundwerefurtherelaboratedbythePARADIGMDelphiteam. Inordertoavoidredundancy,aproposalwasmadetomerge,rephraseorreallocatecategoriesandcriteria.Theywerepresentedtotheexpertpanelandthoroughlydiscussedatthemeeting.Comments provided by the experts in the first and second round were taken intoconsideration.
Fromtheinitial10thematiccriteria,theexpertsagreedon8criteriamadeupof29categoriesand7subcategories(seetable25).
Foreachcriterionandcategory,consensuswasreachedonwhethertoretain,rephrase,mergewithanothercategory,ordiscardit.
Thefollowingparagraphsprovidemoreinformationaboutthekeydiscussionfollowedduring
88
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
themeetingofthecategorieswhichcreatedmorecontroversyanddebate
In general, it was agreed by the expert panel to use the term "PE practice" throughout thediscussion.Furthermore,theterm"patient"wasalsoreplacedwith"patientrepresentative",inordertoreflectthatthisisarolethatrepresentsthewholetargetpopulation.Expertsagreedonremovingbracketsalongthedocumenttobereadwithease.
Criterion1,“Aimandobjectives”consistedoftwocategories.Category1wasbarelymodified.
Criterion 2, “Contextualization of the practice on PE in Early Dialogues” was dropped,movingCategory6toCriterion4.
Criterion 3, “Target participants involved in patient engagement” consisted of fivecategories,twoofwhichweremerged.Thiscriteriongenerateddiscussiononconceptsrelatedto representativeness and diversity of the participants involved in ED. They flagged thedifficultytofindapatientwiththerightprofileorspecificexperienceforaparticularED,andwith the capacity to participate. They also acknowledged that the problem of targetingindividual patients led to end up contacting the same patients due to the difficulty itencounterstofindtherightpatient.
Category 9 was slightly modified and experts decided to add the concept of “vulnerablepopulation” to the glossary (see Annex 4). Category 10was rephrased to reflect that, whenassessing the practice, evaluators should take into account whether the practice tries tocapture diversity (regardless ofwhether it ultimately succeeds). Likewise, Category 11wasmodified toshow therelevanceof representingotherpointsof view,different from thoseoftheirspecificpatients.Finally,Category12and14weremerged;theexpertpanelagreedthatallparticipants inaPEpracticeshouldbeclearabout theirrole.Toachievethis, informationandguidanceshouldbeprovided.
Criterion 4, “Code of conduct” contained four categories. All of themwere rephrased andCategory6was included,movedfromCriterion2“Contextualizationof thepracticeonPE inEarlyDialogues”.Categories15,17and19wererewrittentosimplifyandavoidoverlappingconceptsandCategory18was rephrased toreflect everykindof conflictsof interest.For itspart, Category 6 was modified by including concepts about the regulatory frameworkdeterminedbybodiessuchastheEMA(European,national,etc.).
Criterion5,“Resources” included6categories,oneofthemwasdiscardedandtherestwererephrased.Firstly,Category20wasrephrasedtoincludethatfinancialarrangementrepresentstheframeworkestablishedbythedifferentpartiesintheED.Ontheotherhand,Category21was dropped due to its contentswere also consideredwithin Category20. Category 22wasrewritteninordertoclarifythedefinitionoffinancialframework.Thus,explicitmentionwasmadetotheideathatPEpracticesaresubjecttoacontext/rules.Besides,Categories23and24were rephrased due tooverlaps and to clarify concepts. ThenCategory 25was rewritten toindicatethattoolsusedshouldbebasedonexistingmethodology.
Criterion 6, “Capacity building” was renamed "Capability building". The two existing
89
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
categories in this criterion were maintained, one of themwas rephrased. Category 26 wasmodifiedinordertodifferentiatepatients’experiencefromtheknowledgerequirements.
Criterion 7, “Involvement and participation” initially consisted of six categories, one ofwhichwasdiscardedandtwoweremerged,theremainingcategorieswererephrased.HenceCategory31wasmergedwithCategory33andCategory32wasdiscardedsincetheroleofthepatient representative was already described. Category 34 was adapted to explain that theprocessplanshouldtakeintoaccountPEpreparationtime,withaviewtoinvolvethepatientfrom the beginning and Category 35was rewritten to determine that an equal deliberativeprocessisrequired.Likewise,Category36wasrephrased.Expertsplacedemphasisonhavingan iterative processwhen including patients in ED. It was considered important to providefeedbackabout their contributionduring theEDprocess. However, in thediscussion itwaspointedoutthedifficultyofgivingfeedbackwithoutbreakingconfidentiality,duetothenatureofthisinformation.Itwasalsohighlightedthatpatientsshouldbeabletoprovidefeedbackontheinformationreceivedrelatedtoresults,documents,finaloutput,etc.Foritspart,Category37 was amended to highlighted the importance of considering possible "disabilities andimpairments"ratherthanlimitationsduetoculturalbackgroundorlowliteracy.Category38was adapted to take into account that contact information should be updated, because ofchangesthatmayoccurwithintheinstitutions.
Criterion 8, “Evaluation of the PE practice in early dialogues with HTA bodies andregulators”was composed of five categories.An additional categorywas included from the"Patient Engagement Impact" criterion, three categories were merged and the rest wererephrased.Category47wasaddedasananalogouscategorytothatexistingwithinthe"PatientEngagement Impact" domain; in this case the word "impact" was removed to avoidoverlapping.Likewise,Categories41,43and44weremergedbasedontheproposalmadebythe PARADIGM Delphi Team. Furthermore, Categories 45 and 46 were rephrased to avoidredundancies.
Criterion9, “PatientEngagement Impact” consistedof twomain categories,Categories47and 48, which experts decided to merge and rephrased. The panel discussed about thepracticaldifficultiesforobservationandmeasurementthemwhenthePEpracticeisassessed.ThereforesubcategorieswereinterpretedjustasalistofexamplesofpotentialimpactsofaPEpracticeonED,accordinglythepanelagreednotassigningweightstoeachsubcategory.
Individually,thesubcategorieswerediscussedbythepanel:Subcategories48.1and48.2wererephrased.BesidesSubcategory48.3andSubcategory48.4wererephrased;andSubcategory48.5 was discarded because overlaps with Category 48. Subcategory 48.8 was adapted todescribetheconceptoftransparencyandSubcategory48.10remainsunchanged,butitsorderwasmodifiedtoimprovethecoherenceofthelistofexamplesofpotentialimpacts.Eventually,Subcategory48.11isdiscardedbecauseofitsoverlapwithSubcategory48.8.
Criterion10,“Sustainability”itwasproposedtomergewithCriterion"Resources",giventhatissuesrelatedtosustainabilityhadbeencovered.Finally,itwasestablishedbythepanelthatthiscriterionanditscontentscouldbeincludedsomewhereelseinthePARADIGMproject,but
90
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
itshouldnotbeanelementtobeevaluatedineachPEpractice.
Oncetheworkonspecificationwascompleted,theresultingcategoriesineachcriterionwereweighted.Later, expertsweighed the final8 criteriabydistributing100pointsamong them.ThehighestweightwasassignedtocurrentCriterion1,“Aimandobjectives”(17.9/100),whilethecriteriawiththelowestscorewere“Evaluation”(8/100),followedby“PatientEngagementImpact”(10/100).
Currentcriteriaandcategoryandtheircorrespondingweightsaredetailedintable25.Table25:Finalsetofcriteriaandcategories,andtheirweights,recommendedforassessingPEpracticesinearlydialogues
CriteriaCriteriaweight
N Categories Weight
1.Aimandobjectives 17.9
1 The practice aims to meet patients' expectations whenparticipatingintheearlydialogueprocess. 46.5
2 TheaimandobjectivesofPEinEarlyDialoguesareclearandunderstandableforallstakeholdersinvolved. 53.5
Totalmustbe100
2.Targetparticipantsinvolvedinpatient
engagement
15.3
3
The practice of PE in early dialogues includes rigorousmethodstoidentifypatientsorpatientrepresentatives*fromthe relevant target population needed for a particulardialogue.*Patients, vulnerable population, carers, parents or tutorsand/orstaff frompatient’sorganization thatactasa liaisonbetween the patients and other organizations, representtheirinterestsandadvocatesfortheirrights.
30.5
4 The process captures the diversity of the target populationandtheirrangeofperspectives. 25
5 Relevantpointsofviewotherthanthosepatients(e.g.carers,parentsetc.)arealsoconsidered. 16.5
6Guidanceandadequatetrainingareprovidedtoensurethattheroleandresponsibilitiesofallstakeholdersinvolvedareclearlydefinedandunderstoodbyall.
28
Totalmustbe100
3.Codeofconduct
11.3
7
Any applicable policies, legislative and regulatoryframeworks (e.g. EMA) about when and how to engagepatients in early dialogues with HTA and regulators havebeenfollowed
26.5
8Thepracticeincludesacodeofconduct,whichclearlystatesthe rules of participation in early dialogues for allstakeholdersinvolved.
25
9 The terms and conditions of all policies and confidentiality 16.5
91
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
agreementsareadequate,clearandaccessible.
10 The PE practice incorporates mechanisms to disclose andmanageallpotentialconflictsofinterestofallstakeholders. 18
11
ThePEpracticecontainsprocedurestoidentifyandaddresspotential discriminatory, coercive, intimidating, andunethical behaviours before, during and after theparticipation.
14
Totalmustbe100
4.Resources 10.9
12 Thepractice uses relevant PEmethodologies, guidance andtools. 20.2
13
The practice includes a fair financial framework* forparticipatingpatientrepresentatives.
*Mechanismforpayment/reimbursement
19.7
14 Sufficientfundingisinplacetoensurethatalltheelementsofthepracticearecoveredinitsentirely. 19.7
15
Financial arrangements for the practice of PE in earlydialoguesareclearlyexplained toall stakeholders involved,priortoinvolvement,andanychangesthatoccurduringthepracticearecommunicatedup-front.
17.2
16
An adequate infrastructure is in place (e.g., technologicalsupport, working space, information and communicationtechnologies) and adapted to specific requirements ofparticipants (e.g. possible physicalmental, cognitive or anyimpairmentsetc.).
23.2
Totalmustbe100
5.Capacitybuilding 11.1
17The practice on PE describes the relevant competencies,expertise and experience required by all stakeholders toeffectivelyengageinthisprocess.
42.9
18
ThepracticeincludescomprehensiveandaccessibletrainingorinductionmaterialandprogramstorespondtotheneedsoftheparticipantsinvolvedinearlydialogueswithHTAandregulators.
57.1
Totalmustbe100
6.Involvement
andparticipation
14.7
19Theprocessrespectstheneedforappropriateplanningandpreparation time, to allow patient representatives toeffectivelyengagefromthebeginning.
19
20
Themechanismofinteractionwithpatients’representativesis adapted to their needs to ensure effective PE. Morespecifically, itconsidersspecificpatients’circumstancesandcharacteristics linked tobutnot limited topossiblephysicalormental impairments, cultural background, age and otherrelevant features (e.g., recordings, virtual communication,
28.5
92
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
use of language, format of meetings, the venue, andinformationprovided).
21The practice on PE in early dialogues sets mechanisms toensure a fair deliberative process* that allows equalopportunityforparticipants'contribution.
16.2
22
A regular feedback system is in place to inform patientrepresentativesabouttheoutcomes/changes,asappropriate,includingtheoptionforpatients’representativestocommentonthefinaloutput.
10.8
23
Theinformationabouttheoutcomesincludedforeachphaseof the involvement is communicated in clear and plainlanguage, using accessible formats, and taking into accountpotentialdisabilitiesandimpairments,asappropriate.
10
24
Thepracticeincludesanup-to-datesinglepointofcontactora named person to whom patients can reach out to whenneeded, for information and/or support, throughout theirinvolvementinearlydialogues.
15.5
Totalmustbe100
7.Evaluation 8.8
25
Thepracticeincludesanevaluationframeworklinkedtotheaimandobjectives, includingmethods,toolsandmonitoringsystem,whichenables systematicevaluation atappropriatephasesoftheprocess.
29.8
26The practice identifies relevant outcomes regarding theinvolvement of patients in early dialogues with HTA andregulators.
31.8
27 Theevaluationoutcomesaresharedwithalltheparticipantsinvolved. 16.6
28Thepracticeincludesproceduresbywhichtheconclusionsofthe evaluation are used to support a continuousimprovementprocess.
21.8
Totalmustbe100
8.PatientEngagementImpact
10 29
Thepracticeidentifiespotentialimpactfromthepractice,forexample:
● Better understanding of the impact of healthtechnologiesand treatments in real life contextand thequalityoflifeaspects.
● Morepracticalandbetterdesigneddevelopmentplanforhealthtechnologiesandtreatments.
● Betterreflectionoftheneedsandpreferencesofpatientsindevelopmentplansfornewtechnologies.
● Inclusionofpatients’ relevantoutcomesandend-points
93
Copyright 2019 PARADIGM Consortium - 777450 – PARADIGM D X.Y
inclinicaltrialsdatacollectionandanalysis.
● Better quality of theevidenceassessedduring the laterregulatoryandHTAassessment.
● Higherrelevanceoftheadvicegiventothelocalcontextofapplication.
● Transparencyof regulatoryandHTAprocesses leads tobetter understanding and trust of the scientificprocedures.