Development and Testing of a Clinical …...Final score: sum of domain score of all relevant domains...
1
Development and Testing of a Clinical Documentation Tool to Assess Individual Case Safety Reports in an International Setting Introduction Methods In order to raise a signal, there must be a certain causal relationship between the drug and ADR reported in the Individual Case Safety Reports (ICSR). If relevant clinical information is present, it increases the possibility to do a good causality assessment. If relevant clinical information is lacking, causality is more difficult or even impossible to assess which hampers signal detection. The various reporter types in spontaneous reporting systems are quite diverse (ranging from members of the general public to medical specialists) and the modes of reporting are increasing (from paper form, to web-based forms, to apps and Electronic Health Records). With an increasing number of reporter types and means of reporting, it is important to monitor the quality of the reported information to make sure that the information received in the ICSR contains enough information in order to be useful for signal detection. Therefore, it is desirable to express the quality of clinical information in a quantitative manner to see if there is any variation in quality between the reporter types or reporting mediums. The aim of this study is to develop and test a Clinical Documentation tool for case-by-case assessment of the clinical documentation of ICSRs in an international pharmacovigilance setting. Results Discussion Conclusion Acknowledgments Funding ClinDoc was developed in several phases (see Table 1). The tool was developed based on several discussion rounds by the authors. In each round, ICSRs with different grades of clinical documentation were selected both from the Lareb database as well as the EudraVigilance database to check the applicability of the tool in an international setting. After each discussion round ClinDoc was adapted based on the assessors’ comments. Face validity of the first version of the ClinDoc tool was performed by two pharmacovigilance assessors from pharmacovigilance organizations of four different countries. To test the performance, the tool was tested on 14 ICSRs (Table 1, stage 3) from the Lareb database whose clinical documentation was rated based on expert judgement of the authors. The same assessors, who had performed the face validity, assessed these ICSRs using ClinDoc without seeing the final clinical documentation outcome. ClinDoc contains four domains with several subdomains which are considered to be relevant in order to assess the documentation of clinical information of ICSRs: ADR, drug, patient characteristics and chronology, See Table 2. ADRs can be very diverse, and the information required for a good clinical assessment varies depending on the type of ADR. Therefore, the tool is not created as a static model, but as a flexible model which takes this diversity into account. First, the assessor indicates which subdomains are relevant for assessing the clinical documentationof the specific ICSR and then if the information is present or not. The score given to each domain is the proportion of information present in relation to the information deemed relevant. To translate the score to a clinical documentation measure, the following cut offs are being used: poor (≤ 45%), moderate (from 46- 74%) and well (≥ 75%). After the first round of validity and reliability testing there was a low agreement for validity as well as reliability. After a second round of validity and reliability testing (after adaptation of the tool) we found an overall ‘substantial’ validity and ‘moderate’ agreement for reliability. The mean level of agreement across all pairs of assessors was substantial. The advantage of ClinDoc compared to existing tools that look at the completeness of ICSRs is that it takes the content and the relevance of the information into account, not only the presence. Furthermore, it has been developed and tested in an international setting. A limitation is that a case-by-case assessment is necessary in order to use the clinical documentation tool, and that it is dependent on the knowledge and experience of the assessor to deem a subdomain relevant or not. For countries already working with case-by-case review adding the tool to their assessment is a small time investment. For countries not working with case-by-case assessment, using the tool on a routine basis may be time consuming. However, the tool can still be used to compare for example the clinical quality between different reporting groups, or different means of reporting, by analysing a specific sample only. ClinDoc is the first tool to give insight in the documentation of relevant clinical information during the case-by-case analysis of ICSRs. The tool is not created as a static model, but as a flexible model which takes the diversity of all types of ADRs and drugs into account. ClinDoc can internationally be used in order to measure the clinical documentation of ICSRs and to compare it between reports coming from different reporting methods and different types of reporters in pharmacovigilance. The authors would like to thank the assessors who helped us testing the ClinDoc tool: Morana Pavičić and Željana Margan Koletić of HALMED; Agency for Medicinal Products and Medical Devices of Croatia, Ola Caster and Rebecca Chandler of the Uppsala Monitoring Centre, Julia Cowen and Natalie Bandoo of the UK Medicines & Healthcare products Regulatory Agency and Anne-Marie van Gorp and Annet van Boekel of the Netherlands PharmacovigilanceCentre Lareb. Furthermore, we would like to thank dr. Niklas Norén of the Uppsala Monitoring Centre for his creative input during the development of the tool. The development of the Clinical Documentation tool was part of the WEB-RADR project (www.web-radr.eu) which is a public private partnership coordinated by the Medicines and Healthcare products Regulatory Agency. The WEB-RADR project has received support from the Innovative Medicine Initiative Joint Undertaking (www.imi.europa.eu) under Grant Agreement n° 115632, resources of which are composed of financial contribution from the European Union's Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution. QR code (download poster as pdf ) 1 ADVERSE DRUG REACTION (ADR) Relevant? yes, no Present? yes, no a Proper descripon of the ADR 1* b Specificaon reacon 'localizaon' and 'characterizaon' To strengthen the diagnosis (subdomain c or d or e applicable): c Treatment; or d Visual material (photo, video); or e Lab values, test 2 CHRONOLOGY Relevant? yes, no Present? yes, no a Latency 1* b Descripon of the course of the ADR c Acon taken on drug 1* d Outcome of the ADR 1* 3 SUSPECTED DRUG Relevant? yes, no Present? yes, no a Brand name in case of drug substuon? b Different forms or route of administraon for suspected drug? c Dose-relaonship with ADR? d Batch number of relevance? 4 PATIENT CHARACTERISTICS Relevant? yes, no Present? yes, no a Risk factors/medical history/comorbidity/indicaon b Concomitant medicaon 1* c Age/gender/length/weight d Paent’s life style or other risk factors CALCULATION OF SCORE Domain score: number of present subdomains / number of relevant subdomains * 100% Final score: sum of domain score of all relevant domains / number of relevant domains * 100% Cut off values: poor (≤45%), moderate (from 46- 74%) and well (≥75%) STAGE 1 Formulation of (sub)domains for clinical documentation ClinDoc was drafted Discussion rounds with 5 pharmacovigilance experts STAGE 2 Face validity testing Adaption of ClinDoc according to comments. ClinDoc seems feasible First check of ClinDoc and its user instruction by two pharmacovigilance assessors from pharmacovigilance organizations of four different countries STAGE 3 Validity and reliability testing Cohen’s kappa was calculated be- tween: -Assessors of each agency individually versus expert panel (validity) -two assessors in one organization and mean level of agreement across all pairs of assessors (reliability) Kappa’s were acceptable for imple- mentation Testing 14 ICSRs with ClinDoc by two pharmacovigilance assessors from pharmacovigilance organizations of four different countries STAGE DEVELOPMENT METHOD CONCLUSION Table 1 ClinDoc Development Phases Table 2 ClinDoc Subdomains Leàn Rolfes, Ingrid Oosterhuis, Corine Ekhart, Annemarie Muller-Hansma and Linda Härmark, Netherlands Pharmacovigilance Centre Lareb, ‘s-Hertogenbosch, the Netherlands
Transcript of Development and Testing of a Clinical …...Final score: sum of domain score of all relevant domains...
Development and Testing of a Clinical Documentation Tool to Assess Individual Case Safety Reports in an International Setting
Introduction
Methods
In order to raise a signal, there must be a certain causal relationship between the drug and ADR reported in the Individual Case Safety Reports (ICSR). If relevant clinical information is present, it increases the possibility to do a good causality assessment. If relevant clinical information is lacking, causality is more di�cult or even impossible to assess which hampers signal detection.
The various reporter types in spontaneous reporting systems are quite diverse (ranging from members of the general public to medical specialists) and the modes of reporting are increasing (from paper form, to web-based forms, to apps and Electronic Health Records). With an increasing number of reporter types and means of reporting, it is important to monitor the quality of the reported information to make sure that the information received in the ICSR contains enough information in order to be useful for signal detection. Therefore, it is desirable to express the quality of clinical information in a quantitative manner to see if there is any variation in quality between the reporter types or reporting mediums.
The aim of this study is to develop and test a Clinical Documentation tool for case-by-case assessment of the clinical documentation of ICSRs in an international pharmacovigilance setting.
Results
Discussion
Conclusion
Acknowledgments
Funding
ClinDoc was developed in several phases (see Table 1). The tool was developed based on several discussionrounds by the authors. In each round, ICSRs with di�erent grades of clinical documentation were selected bothfrom the Lareb database as well as the EudraVigilance database to check the applicability of the tool in aninternational setting. After each discussion round ClinDoc was adapted based on the assessors’ comments.
Face validity of the �rst version of the ClinDoc tool was performed by two pharmacovigilance assessors from pharmacovigilance organizations of four di�erent countries.
To test the performance, the tool was tested on 14 ICSRs (Table 1, stage 3) from the Lareb database whose clinical documentation was rated based on expert judgement of the authors. The same assessors, who had performed the face validity, assessed these ICSRs using ClinDoc without seeing the �nal clinical documentation outcome.
ClinDoc contains four domains with several subdomains which are considered to be relevant in order to assess the documentation of clinical information of ICSRs: ADR, drug, patient characteristics and chronology, See Table 2. ADRs can be very diverse, and the information required for a good clinical assessment variesdepending on the type of ADR. Therefore, the tool is not created as a static model, but as a �exible model which takes this diversity into account. First, the assessor indicates which subdomains are relevant for assessing the clinical documentationof the speci�c ICSR and then if the information is present or not. The score given to each domain is the proportion of information present in relation to the information deemed relevant. To translate thescore to a clinical documentation measure, the following cut o�s are being used: poor (≤ 45%), moderate (from 46- 74%) and well (≥ 75%).
After the �rst round of validity and reliability testing there was a low agreement for validity as well as reliability. After a second round of validity and reliability testing (after adaptation of the tool) we found an overall ‘substantial’ validity and ‘moderate’ agreement for reliability. The mean level of agreement across all pairs of assessors was substantial.
The advantage of ClinDoc compared to existing tools that look at the completeness of ICSRs is that it takes thecontent and the relevance of the information into account, not only the presence. Furthermore, it has beendeveloped and tested in an international setting. A limitation is that a case-by-case assessment is necessary in order to use the clinical documentation tool, and thatit is dependent on the knowledge and experience of the assessor to deem a subdomain relevant or not. For countries already working with case-by-case review adding the tool to their assessment is a small time investment. For countries not working with case-by-case assessment, using the tool on a routine basis may be time consuming. However, the tool can still be used to compare for example the clinical quality between di�erentreporting groups, or di�erent means of reporting, by analysing a speci�c sample only.
ClinDoc is the �rst tool to give insight in the documentation of relevant clinical information during thecase-by-case analysis of ICSRs. The tool is not created as a static model, but as a �exible model which takes the diversity of all types of ADRs and drugs into account. ClinDoc can internationally be used in order to measurethe clinical documentation of ICSRs and to compare it between reports coming from di�erent reporting methods and di�erent types of reporters in pharmacovigilance.
The authors would like to thank the assessors who helped us testing the ClinDoc tool: Morana Pavičić and Željana Margan Koletić of HALMED; Agency for Medicinal Products and Medical Devices of Croatia, Ola Caster and Rebecca Chandler of the Uppsala Monitoring Centre, Julia Cowen and Natalie Bandoo of the UK Medicines & Healthcare products Regulatory Agency and Anne-Marie van Gorp and Annet van Boekel of the Netherlands PharmacovigilanceCentre Lareb. Furthermore, we would like to thank dr. Niklas Norén of the Uppsala Monitoring Centre for his creative input during the development of the tool.
The development of the Clinical Documentation tool was part of the WEB-RADR project (www.web-radr.eu) which is a public private partnership coordinated by the Medicines and Healthcare products Regulatory Agency. The WEB-RADR project has received support from the Innovative Medicine Initiative Joint Undertaking (www.imi.europa.eu) under Grant Agreement n° 115632, resources of which are composed of �nancial contribution from the European Union's Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.
QR code (download poster as pdf )
1 ADVERSE DRUG REACTION (ADR) Relevant? yes, no
Present? yes, no
a Proper description of the ADR 1* b Specification reaction 'localization' and 'characterization'
To strengthen the diagnosis (subdomain c or d or e applicable):
c Treatment; or d Visual material (photo, video); or e Lab values, test 2 CHRONOLOGY Relevant?
yes, no Present? yes, no
a Latency 1* b Description of the course of the ADR c Action taken on drug 1* d Outcome of the ADR 1* 3 SUSPECTED DRUG Relevant?
yes, no Present? yes, no
a Brand name in case of drug substitution? b Different forms or route of administration for suspected drug? c Dose-relationship with ADR? d Batch number of relevance? 4 PATIENT CHARACTERISTICS Relevant?
yes, no Present? yes, no
a Risk factors/medical history/comorbidity/indication b Concomitant medication 1* c Age/gender/length/weight d Patient’s life style or other risk factors
CALCULATION OF SCORE Domain score: number of present subdomains / number of relevant subdomains * 100% Final score: sum of domain score of all relevant domains / number of relevant domains * 100% Cut off values: poor (≤45%), moderate (from 46- 74%) and well (≥75%)
STAGE
1Formulation of (sub)domains for clinical documentation
ClinDoc was drafted Discussion rounds with 5 pharmacovigilance experts
STAGE
2Face validity testing Adaption of ClinDoc according to
comments.ClinDoc seems feasible
First check of ClinDoc and its user instruction by two pharmacovigilance assessors from pharmacovigilance organizations of four di�erent countries
STAGE
3Validity and reliability testing
Cohen’s kappa was calculated be-tween:-Assessors of each agency individually versus expert panel (validity)-two assessors in one organization and mean level of agreement across all pairs of assessors (reliability)Kappa’s were acceptable for imple-mentation
Testing 14 ICSRs with ClinDoc by two pharmacovigilance assessors from pharmacovigilance organizations of four di�erent countries
STAGE DEVELOPMENT METHOD CONCLUSION
Table 1 ClinDoc Development Phases
Table 2ClinDoc Subdomains
Leàn Rolfes, Ingrid Oosterhuis, Corine Ekhart, Annemarie Muller-Hansma and Linda Härmark, Netherlands Pharmacovigilance Centre Lareb, ‘s-Hertogenbosch, the Netherlands
