New Data Disasters and How to Prevent Them - WordPress.com · 2017. 1. 26. · Data Disasters and...
Transcript of New Data Disasters and How to Prevent Them - WordPress.com · 2017. 1. 26. · Data Disasters and...
Data Disasters and How to Prevent Them
Colleen W. Marano, PhD
Director Clinical Immunology
Janssen R &D, LLC
&
Anne Martin Robinson, Pharm.D.
Group Scientific Director, Immunology
Global Pharmaceutical Research and Development
AbbVie, Inc.
Disclosure slide
• Colleen Marano is an employee of Janssen R & D, LLC
• Anne Robinson is an employee of AbbVie, Inc.
2
What are data disasters...
• Any issue that potentially interferes with study interpretation or validity, for example:
– missing data,
– confounded data (i.e. endpoints compromised by other procedures or biased by surveys administered out of sequence),
– out of window data,
– inaccessible or lost data,
– data you can't use
Why do data issues arise?
• Protocols are unclear/complicated/internally conflicting
• Study systems are confusing or do not communicate with each other or do not require essential entries
• Humans make errors (or intentionally provide misinformation)
• Lab samples get lost/hemolyzed
• Transit agencies go on strike
• Technology fails to work correctly or batteries fail
• Schedules/vacations/holidays interfere with pre-specified visit plans
• Sick people need to be treated outside of protocol-defined parameters
The type of questions you are asking drives the data to be collected....
• Is the study hypothesis generating (exploratory) or hypothesis testing?
• What data is necessary to support the objectives of the study?
– What is essential?
– What is nice-to-have?
• How much, at what times, & for what duration should the data be collected?
Biostatistics Biomarkers Clinical Therapeutic Area Clinical Operations Clinical Pharmacology Data Management Health Economics Medical Writers Patient-reported Outcomes Pharmacovigilance Programming Quality Management and Compliance Regulatory
A Study is a Cross-functional Team Effort
• Protocol • ICF • Case report forms; data
cleaning & monitoring • Patient diaries • Site/vendor selection • Quality Plan • IND/regulatory
submissions • Statistical Analysis Plan • Data Presentation Plan • TLFs • Study reports
Coordinated with vendors, CROs, Ethics Committees , and Health Authorities
Protocol Considerations (I) • Does the data collected support the study objectives?
• Has the protocol & its requirements been reviewed by individuals outside of your study team?
– Obtain input of key investigative sites, including study coordinators, or patient representatives
– Are there any safety considerations requiring unique sample or data collection?
– What are the relevant concomitant medications?
– What medications should be discontinued prior to study drug?
• Consider eligibility requirements and procedures to minimize protocol deviations & missing data
• Are they in line with patient’s perception of burdens of invasive procedures (i.e. endoscopy [number and interval]
• Are screening procedures consistent with local practices/preferences (e.g. TB testing)
Protocol Considerations (II) • Will there be a blinded assessor?
– For example, an endoscopist blinded to treatment & patient-reported data
– Is the study flow (e.g. visit procedures/sequences) logistically feasible?
• How will missing or out of window data be handled?
• Define end of study completion and impact of patient drop out on study validity?
– Will patients who discontinue study prematurely be replaced?
Informed Consent Considerations • Confirm that ICF explains all protocol-specified
procedures and study risks in patient appropriate language/format.
• Does your ICF language allow banking and future analysis of biological specimens? – Is the collection of genetic samples permitted at each
site/country?
• Does your ICF allow follow-up after patients prematurely end study participation? – For example, collection of colectomy or safety follow-
up through planned final study visit.
Data Collection & Integrity • Collect the right data at the right time
• Consider how will data be collected: paper/electronic case report forms or ePRO
– Data review & query process
– Data cleaning & monitoring: frequency
– Handling of missing data
– Treatment failure
• Avoid double data collection: consider data consistency & reconciliation issues
– Concomitant medications affecting treatment response: targeted med review pages linked to concomitant medication CRF
– Pre-programmed real-time edit checks or backend edit checks
• Multiple data sources
– IWRS/CRF/Central Lab or Imaging Vendor/ePRO
A Few Examples
– Disease indices that are incorrectly calculated
– Missing data
• A missing subscore of a multiple component index
• Missing entries for data that relies on multiple days for calculation
• Failure to follow protocol visit specifications
Simple Endoscopic Score (SES-CD) Ileum Right colon Transverse
colon Left colon Rectum
Presence/size of ulcers 0: none 1: < 0.5 cm 2: 0.5-2 cm 3: > 2 cm
Extent of ulcerated surface 0: 0% 1: < 30% 2: 10-30% 3: > 40%
Extent of affected surface 0: 0% 1: < 50% 2: 50-75% 3: > 75%
Narrowings 0: none 1: single, can be passed 2: multiple, can be passed 3: cannot be passed
Daperno, et al. Gastrointest Endosc 2004;60:505-12.
Incorrectly Calculated Disease Activity Index: SES-CD Hypothetical Example
Ileum Right colon Transverse colon
Left colon Rectum
Presence/size of ulcers 0: none 1: < 0.5 cm 2: 0.5-2 cm 3: > 2 cm
2 1 0 0 0
Extent of ulcerated surface 0: 0% 1: < 30% 2: 10-30% 3: > 40%
1 0 0 0 0
Extent of affected surface 0: 0% 1: < 50% 2: 50-75% 3: > 75%
0 1 0 0 0
Narrowings 0: none 1: single, can be passed 2: multiple, can be passed 3: cannot be passed
0 3 0 0 0
Scores present for ileum although right colon has score for “impassable stenosis”
Incorrectly Calculated Disease Activity Index: SES-CD Hypothetical Example
Ileum Right colon Transverse colon
Left colon Rectum
Presence/size of ulcers 0: none 1: < 0.5 cm 2: 0.5-2 cm 3: > 2 cm
2 1 0 0 0
Extent of ulcerated surface 0: 0% 1: < 30% 2: 10-30% 3: > 40%
1 0 0 0 0
Extent of affected surface 0: 0% 1: < 50% 2: 50-75% 3: > 75%
1 1 0 0 0
Narrowings 0: none 1: single, can be passed 2: multiple, can be passed 3: cannot be passed
0 3 0 0 0
Scores ileum although right colon has score for “impassable stenosis”
Ulcerated surface subscore is 0 although size of ulcer score indicates aphthi are
present
Pediatric Crohn’s Disease Activity Index (PCDAI) Variable Score
Abdominal pain 0, 5, 10
Stool frequency 0, 5, 10
General well-being 0, 5, 10
Hematocrit 0, 2.5, 5
Erythrocyte sedimentation rate 0, 2.5, 5
Albumin 0, 5, 10
Weight 0, 5, 10
Height 0, 5, 10
Abdomen 0, 5, 10
Perirectal disease 0, 5, 10
Extra-intestinal manifestations 0, 5, 10
Total 0-100
Hyams, et al. J Pediatr Gastroenterol Nutr 1991;12:439-47.
Missing PCDAI Subscore Hypothetical Example (Lost Lab Sample)
Variable Score
Abdominal pain 0
Stool frequency 5
General well-being 5
Hematocrit n/a
Erythrocyte sedimentation rate 0
Albumin 0
Weight 0
Height 0
Abdomen 0
Perirectal disease 0
Extra-intestinal manifestations 0
Total 10?
Is the total really 10?
Is the patient in remission?
Is this PCDAI “missing?”
How will you decide?
Missing SES-CD Segment Hypothetical Example (Ileum Not Intubated)
Ileum Right colon
Transverse colon
Left colon
Rectum
Presence/size of ulcers 0: none 1: aphthous < 0.5 cm 2: 0.5-2 cm 3: > 2 cm
n/a 1 0 0 0
Extent of ulcerated surface 0: 0% 1: < 30% 2: 10-30% 3: > 40%
n/a 1 0 0 0
Extent of affected surface 0: 0% 1: < 50% 2: 50-75% 3: > 75%
n/a 1 0 0 0
Narrowings 0: none 1: single, can be passed 2: multiple, can be passed 3: cannot be passed
n/a 0 0 0 0
Is the total really 3?
Does it matter why the ileum wasn’t
intubated?
Is this SES-CD “missing?”
How will you decide?
Missing Data: PROs Based on Multiple Days of Patient Entries
• Example: CDAI is based upon seven days of patient reported data for stool frequency, abdominal pain, general well-being
• Things to consider • Which seven days? (i.e., from what period can the 7 days be
selected?)
• Do the seven days need to be consecutive?
Day S M T W Th F S
Liquid/Soft Stool frequency
3 2 2 1 1 2 1
Missing CDAI Entries Hypothetical Example
• What if this happens?
• Are four days of entries sufficient?
Day S M T W Th F S
Stool frequency
3 2 2 1
Missing CDAI Entries Hypothetical Example
• What if this happens?
• Or this?
Day S M T W Th F S
Stool frequency
3 2 2 1
Day S M T W Th F S
Stool frequency
3 12 1 1 1 2 1
Missing CDAI Entries Example
• What if this happens?
• Or this?
Day S M T W Th F S
Stool frequency
3 2 2 1
Day S M T W Th F S
Stool frequency
3 12 1 1 1 2 1
Endoscopy prep
Endoscopy
Should days around endoscopy preparation and procedure be
excluded (and if so, should they be replaced?)
Missing Data Based on Visit Scheduling: Hypothetical Example
• Subject is randomized, but last minute travel requires patient to leave country at Week 5 for 30 days; subject is willing to undergo endoscopy at Week 5
• Is this acceptable?
• What if the endoscopy is at Week 11, and the subject received “rescue” treatment at Week 10?
0 Week 8
Endoscopy endpoint
Treatment Period (concomitant medications are kept stable) Off Treatment
12
Safety follow up
Missing Data Based on Visit Scheduling: Hypothetical Example
• Subject is randomized, but last minute travel requires patient to leave country at Week 5 for 30 days; subject is willing to undergo endoscopy at Week 5
• Is this acceptable?
• What if the endoscopy is at Week 11, and the subject received “rescue” treatment at Week 10?
• Be sure to pre-specify study visit windows, so data are consistently assigned to the appropriate visit
• Consider clinical relevance of visit windows and potential issue of missing data
0 Week 8
Endoscopy endpoint
Treatment Period (concomitant medications are kept stable) Off Treatment
12
Safety follow up
Closing thoughts
• If something can go wrong, it will
• Anticipate issues and pre-specify how to handle them
• It is not OK to change the rules after the fact without specifying what was done ad-hoc
• Resist the lure of adding too many variables, procedures, and data collection instruments to a study
• Involve your statistician early and often!
• Consider logic checks (e.g., CDEIS should not total > 44) or systems that force data entry
• Have an external study coordinator review your protocol
• Monitor data periodically in in order to identify issues while there is still time to address them (before data are unblinded)