Risk-Based Monitoring Quantitative Metrics · Risk-Based Monitoring Quantitative Metrics Toolkit 04...

Risk-Based Monitoring

Quantitative Metrics

Toolkit

04 November 2016

Copyright ©2015 TransCelerate BioPharma Inc., All rights reserved. 2

Contents

Purpose of this Document

Objectives

Process

Core Metrics

Recommendations for Historical Control Calculations

Recommendations for Historical Control Periods

Guidance for Potential Surrogates

Establishing Target Ranges

Expected Observations and Potential Alternative Observations

Final Considerations

RBM Metrics Report 1Q2016 (Bi-annual)

Trial Inventory (RBM Uptake, trials planned or in progress)

How to read the metrics (Stacked charts)

Actual metrics report from 1Q2016

3

4

5

6-7

8

9

10

11-12

13-14

15

16-26

17

18

19-26


Purpose of this Document

As the TransCelerate RBM initiative developed the methodology and framework for voluntary RBM implementation, there was a recognized need to quantitatively evaluate progress and impact.

The team developed 8 core metrics which could be used to evaluate the impact of the TransCelerate RBM methodology on clinical development programs across three broad categories; Quality, Efficiency and Cycle Time

A process was developed to support sponsors in determining historical controls, setting target values, measuring the metrics, and assign a dashboard rating. All TransCelerate RBM metrics have been reported anonymously to a neutral third party for aggregation.

Overtime, it was recognized that companies needed the flexibility of defining the metrics differently due to internal systems, procedures, metrics, etc. Guidance was provided to companies in an effort to move toward some consistency in measurements for some metrics in accordance with regulatory agency requests. Companies may determine the extent to which they follow the recommendations.

Additional guidance is also provided to assist with historical control calculations and periods, potential surrogates and setting target ranges.

All information, recommendations or guidance contained herein is voluntary for sponsors to utilize.


Quantitative Metric Collection Objectives

Determine the impact and effectiveness of the proposed RBM

methodology on managing quality and risks associated with the

conduct of clinical trials.

Determine if the RBM methodology works from the standpoint of

operational impact on an organization, clinical sites and investigators.

Keep in Mind:

Benefits realized must be accompanied by either an improvement or

maintenance of current criteria in data quality and subject safety.


Determine Historical Control

Set Target Values

Measure Metrics

Assign Dashboard

Rating

RBM Quantitative Metric Analysis Process

Recommendation

Generate historical

control at one of four

levels:

1.) IP level

2.) therapy area level

3.) cumulative

4.) split study

This will remain constant

through assessment

Recommendation

Determine target values

on three levels:

1.) Improvement at X%

2.) Worsening at Y%

3.) Negligible change

(About the same)

between X% and Y%

Measure the metrics

that are feasible to be

quantified

Compare quarterly

metric to target values

and assign dashboard

rating of better, about

the same or worse


Core RBM Quantitative Metrics (1 of 2)

Indicator Metric Optional Guidance: For consistency amongst

sponsors

Quality Number and classification of

major/critical audit findings

per site audit

• Better + 25%

• Worse -25%

• About the Same > -25% & < +25%

Quality Number of unreported,

confirmed SAEs as discovered

through any method

• Better + 10%

• Worse -10%

• About the Same > -10% & < +10%

Quality Significant Protocol Deviation

rate per treated subject (total

# of deviations/ total # of

subjects for the protocol)

Recommend normalize to per treated subject or patient.

Do not include bio or stat programmed reports. Include only

Significant PDs identified by central or site monitoring or data

cleaning activities.

• Better + 10%

• Worse -10%

• About the Same > -10% & < +10%

Efficiency Average Monitoring (all types)

cost per site

• Better + 10%

• Worse -10%

• About the Same > -10% & < +10%

Efficiency Average interval between on-

site monitoring visits per site

Recommend measuring using Start Date to Start Date

• Better + 10%

• Worse -10%

• About the Same > -10% & < +10%


Core RBM Quantitative Metrics (2 of 2)

Indicator Metric Optional Guidance: For consistency amongst

sponsors

Cycle Time Median number of days from

visit to eCRF data entry

Primary recommendation is to use first data entered.

• Better + 10%

• Worse -10%

• About the Same > -10% & < +10%

Cycle Time Median number of days from

query open to close

Recommend focus on site activity, use the response by site for

consistency, exclude any auto generated queries

• Better + 10%

• Worse -10%

• About the Same > -10% & < +10%

Cycle Time Median days from

significant/major issue open to

close

Recommend focus on significant or major findings if able to.

Companies should define what they feel are significant.

• Better + 10%

• Worse -10%

• About the Same > -10% & < +10%


Recommendations for Historical Control Calculation

Metric Determination – historical controls and metrics

Average number of major/critical

audit findings per site audit

• Calculate the number of findings and divide by the number of site audits

per quarter

• Potential sources to include: QA audit reports

Percentage of unreported, confirmed

SAEs as compared to total SAEs as

discovered through any method

• Calculate the number of findings and divide by the total number of SAEs

• Potential sources to include: Monitoring reports

Significant Protocol Deviation rate per

treated subject (total # of deviations/

total # of subjects for the protocol)

• Calculate the number of findings and divide by the total number treated

subjects.

• The definition of “Significant” to be defined by each sponsor

• Potential sources to include: EDC platform

Average Monitoring (all types) cost

per site

• Compile all costs associated with monitoring the trial and divide by the

number of sites

• Potential sources to include: CTMS, Finance

Average interval between on-site

monitoring visits per site

• Determine the interval between on-site monitoring visits for all sites and

divide by the number of sites.

• If a site has not had a second visit to perform analysis in the quarter, omit

that site from analysis

• Potential sources to include: CTMS

Median number of days from patient


• Calculate median


Median number of days from query

open to close



Median number of days from

significant/major issue open to close


• The terms “issue”, “open” and “close” to be defined by each sponsor

• Potential sources to include: Issue Tracking System


Recommendations for Historical Control Periods

Metric Historical Control Period and Rationale

Average number of major/critical

audit findings per site audit

Due to lower frequencies, consider at least a 2 year sample

Percentage of unreported,

confirmed SAEs as compared to

total SAEs as discovered through

any method

Consider at least a 1 year sample

Significant Protocol Deviation rate

per treated subject (total # of

deviations/ total # of subjects for

the protocol)


Average Monitoring (all types)

cost per site

Due to fluctuations in costs and time value of money, consider at most a

1 year sample





patient visit to eCRF data entry



query open to close



significant/major issue open to

close



Guidance for Potential Surrogates

Metric Potential Surrogates

Average number of major/critical audit

findings per site audit

Percentage of unreported, confirmed

SAEs as compared to total SAEs as

discovered through any method

If a field does not exist on monitoring report, consider using data

entry of SAEs vs. on-site monitoring visit date.

E.g. Calculate the number of SAEs with start date that were

data entered >/= 2 days after an on-site monitoring visit and

divide by the total number of SAEs

Significant Protocol Deviation rate per

treated subject (total # of deviations/

total # of subjects for the protocol)

Consider defining “Significant” as those protocol deviations

impacting primary or secondary endpoints.

Average Monitoring (all types) cost per

site

If direct costs cannot be obtained, consider collaborating with

finance to estimate

Also, consider determining average cost of visit and utilize decreased

visit frequency to estimate



Determine the interval between on-site monitoring visits for all sites

and divide by the number of sites. If a site has not had a second visit

to perform analysis in the quarter, omit that site from analysis

Median number of days from patient


Median number of days from query

open to close


significant/major issue open to close


Establishment of Target Ranges (1 of 2)

Using historical controls as a guide, set ranges that will indicate “better”, “about

the same” or “worse" since last quarter

If unable to calculate a historical control for any reason, the quantitative target

ranges will have to be created using best judgment qualitatively

Example: Median Number of Days from Query Open to Close – historical control =

10 days

In this example, the metric, when compared to the target ranges, will inform the dashboard

ranking

• </= to 8 days would be better (e.g. could be top 10% or -2 standard deviations)

• 8-12 days would be about the same (e.g. could be mid – 80% or +/- 1 standard deviation)

• >/= 12 days would be worse (e.g. could be bottom 10% or +2 standard deviations)

Historical Control = 10 days

8 days 12 days

Better About the Same Worse


Example: Average number of major/critical audit findings per site audit –

historical control = 2 findings

In this example, the metric, when compared to the target ranges, will inform the dashboard

ranking

• </= 1 finding would be better (e.g. could be top 10% or -2 standard deviations)

• 1-2 findings would be about the same (e.g. could be mid – 80% or +/- 1 standard deviation)

• >/= 2 findings would be worse (e.g. could be bottom 10% or +2 standard deviations)

Note the range is tighter for this particular metric

• If no audits have been performed, the metric would be 0 and per this target range the dashboard ranking would be better

Establishment of Target Ranges (2 of 2)

Better About the Same

Historical Control = 2 findings

1 finding 2 findings

Worse


These Metric Narratives illustrate the expectations

of RBM’s impact as well as alternative observations

Metric Expected Observations Potential Alternative

Observations

Average number of

major/critical audit

findings per site audit

Given RBM, it would be expected

that the average number of

major/critical findings per site audit

will decrease

Early during implementation, findings may

rise due to it being a new procedure not

necessarily a focus on critical data and

processes with expectation that they

would decrease over time

Percentage of unreported,

confirmed SAEs as

compared to total SAEs as

discovered through any

method


that number of unreported,

confirmed SAEs will decrease


rise due to shift in focus from SDV to SDR

with expectation that they would

decrease over time

Significant Protocol

Deviation rate per treated

subject (total # of

deviations/ total # of

subjects for the protocol)


that number of Significant Protocol

Deviations will decrease


rise due to shift in focus from SDV to SDR

with expectation that they would

decrease over time

New process implementation for Protocol

Deviation review may reflect unexpected

increases


These Metric Narratives illustrate the expectations

of RBM’s impact as well as alternative observations

Metric Expected Observations Potential Alternative

Observations

Average Monitoring (all

types) cost per site


that average monitoring costs will

decrease

Costs may remain flat until second quarter

of analysis or later

Average interval between

on-site monitoring visits per

site


that interval between on-site

monitoring visits will increase


monitoring visits may remain flat until

second quarter of analysis or later

Median number of days

from patient visit to eCRF

data entry

Given RBM, there are no

expectations for the median

number of days from patient visit to

eCRF data entry, however, a

decrease would be beneficial

This metric measures an unintended

consequence of RBM, namely, the site’s

delay in performing a crucial function that

empowers central monitoring due to the

potential decrease in on-site visits


from query open to close

Given RBM, there are no

expectations for the median

number of days from query open to

close, however, a decrease would

be beneficial

This metric measures an unintended

consequence of RBM, namely, the site’s

delay in performing a crucial function due

to the potential decrease in on-site visits


from significant/major issue

open to close


that the median number of days

from issue open to close will

decrease


rise if issues management process is new to

the organization with expectation that they

would decrease over time


Final Considerations

Once the historical controls for your metrics are determined, they

should remain static for the duration of analysis

As RBM becomes more pervasive in a company (“business as

usual”), controls will switch from “non-RBM trials vs RBM trials” to your normal baseline for control, (e.g., previous calendar year)


Progression of RBM Uptake* at Member Companies

Blinded Inventory of RBM Trials (planned and ongoing)

0

20

40

60

80

100

120

140

160

180

2Q13 2Q14 2Q15 3Q15 1Q16

# Companies # Therapeutic Areas # Trials

Voluntary adoption information is reported to a blinded third party for aggregation


RBM Metrics Context for Collection and Analysis

• Collection process and limitations across 81 trials eligible to report data across 7 Companies for periods ending Q42015, Q12016 (out of inventory of 162 trials planned or in progress)

• Analysis

– Trial data with similar level of maturity (“RBM + x months”) is aggregated into

stacked charts. In example below, the first quarter of metric data was grouped

together into stacked charts(red circles), the second quarter of metric data

was grouped together for analysis (green circles) and so on, regardless of

actual calendar quarter. Metric #1 Example

RBM + 3 Months RBM + 6 Months RBM + 9 Months RBM + 12 Months

RBM + 15 Months

RBM + 3 Months RBM + 6 Months RBM + 9 Months RBM + 12 Months

RBM + 3 Months RBM + 6 Months

RBM + 3 Months

Trial 1

Trial 2

Trial 3

Trial 4

Note that not all metrics

are reported for all trials

across all time periods


How to read the metrics (stacked charts)

• What do the X and Y axis’ represent?

– The Y-axis (stacks and the numbers inside the stacks) represent the total

number of trials reported by member companies (via voluntary reporting)

that have utilized RBM methods during this report period.

– The X-axis shows how long the trials have been running using RBM methods

(e.g., “RBM + 3 months” = trial using RBM methodology for 3 months)

• What do the colors mean?

– Dark Blue indicates this metric for this reporting period for these trials is Better

compared to the control

– Light Blue indicates this metric for this reporting period for these trials is About

The Same compared to the control

– Yellow indicates this metric for this reporting period for these trials is Worse

compared to the control

Refer to slides 6-7 for optional guidance defining “Better”, “About

the Same” and “Worse”


Quality: Audit findings per audited siteAverage number of major/critical audit findings per audited site

Responses compiled, blinded and aggregated by third party before dissemination to member companies. Not all metrics are reported for all trials across all time periods.


Quality: SAE reportingPercentage unreported, confirmed SAEs as compared to total SAEs



Quality: Significant Protocol DeviationsSignificant Protocol Deviation rate per treated subject



Efficiency: Overall Monitoring CostAverage Monitoring (all types) cost per site



Efficiency: On-site visit intervalAverage interval between on-site monitoring visits per site



Cycle Time: eCRF EntryMedian number of days from patient visit to eCRF data entry



Cycle Time: Query Open to CloseMedian number of days from query open to close



Cycle Time: Issue Open to CloseMedian number of days from issue open to close


Risk-Based Monitoring Quantitative Metrics · Risk-Based Monitoring Quantitative Metrics Toolkit 04...

Documents

Transcript of Risk-Based Monitoring Quantitative Metrics · Risk-Based Monitoring Quantitative Metrics Toolkit 04...