Clinical Assessment – Part II William Sacks, PhD, MD

12/10/02 Sacks - Clinical Assessment 1

Clinical Assessment – Part IIWilliam Sacks, PhD, MD

COMPUTERIZED THERMAL IMAGING, INC.

Breast Cancer System BCS 2100

P010035

Radiological Devices Advisory PanelDecember 10, 2002


Safety of the BCS has two aspects:• Adverse events• Accuracy of BCS output


There were only 4 minor adverse events, two of which were thought not to be device related, during a clinical trial involving 2407 subjects.


From the point of view of the BCS output:

• Safety is more closely related to sensitivity (cancers), or FNR, but also FPR• Effectiveness is more closely related to specificity (benign masses)


POINTS TO BE COVERED

• What BCS 2100 (BCS) is and is not intended to do• How BCS does it• What the clinical trial demonstrated• Labeling issues


CLINICAL SUBMISSIONS• PMA • Amendment 4• Amendment 5• Amendment 7


After reviewing the PMA, the FDA sent a letter to CTI listing a number of deficiencies. CTI’s response was Amendment 4.


In Amendment 4, for their conclusions concerning BCS effectiveness (p. 11), CTI retrospectively selected from the PMA data one of two analytical indices (Se/Sp, not ROC) and two of three lesion types (masses, and arch. dist., not microcalcifications).


In Amendment 4 the revised labeling further deleted architectural distortion, and referred to masses alone.


The FDA sent another deficiency letter to CTI. The response was Amendment 5.


Amendment 5 was offered as a test of BCS in additional subjects, because Amendment 4 had contained retrospective selections. CTI called the additional dataset PPMA (Post-PMA).


Amendment 5 confined its analysis of the PPMA data to the newly chosen analytical index (Se/Sp) in the newly chosen subgroup (masses).


In addition to presenting data on a new set of subjects (PPMA), Amendment 5 also contained analysis of the combined datasets from Amendment 4 and the PPMA.


Because of the retrospective selections in Amendment 4, FDA asked CTI to justify combining that data with the PPMA. CTI’s response was Amendment 7.


In Amendment 7 CTI applied a Bonferroni correction in an attempt to compensate for• retrospective selection of data in Amendment 4 and • the smallness of the additional (PPMA) sample in Amendment 5.


Two overriding issues:• Adequacy of data• Interpretation of data, i.e., do the data demonstrate S&E of BCS?


Adequacy of data

Can data from Amendment 4 contribute to judgment of S&E, when it consists of retrospective selection of mammographic masses, because the Se was too low in microcalcifications and in the tested population as a whole?


Adequacy of data (cont.)

Is a Bonferroni correction applicable in this context?


Adequacy of data (cont.)

Are the data fromthe PPMA alone (inAmendment 5) adequate tojudge S&E?


Interpretation of the dataIt is noteworthy for the following discussion that no formal hypotheses were explicitly put forward for testing, either in the PMA or in the subsequent amendments.


To qualify as a testable hypothesis there must be a quantitative criterion, whereby either• a point estimate may imply rejection, or• a confidence interval may entail exclusion.


There were two implicit hypotheses:

• that the ROC area for BCS and mammography combined would exceed that of mammography alone (with statistical significance) [Protocol, Section 6.0 Statistical Analysis]

• that the point estimate for sensitivity would be at least 99.3% (in at least 75% of simulations with the data)

[PMA, Section 5.8.7.2, p. 489].


The protocol otherwise contained only non-quantitative statements of what CTI hoped to achieve in the clinical trial. For example,

The objective of the study is to determine if the (BCS 2100), when used in conjunction with clinical examination and/or diagnostic mammography, increases the ability of physicians to differentiate benign from malignant, or suspicious, breast abnormalities. [Protocol, Section 3.1 Study Objective]


In the original PMA submission, the comparison of ROC areas failed to achieve statistical significance, except as an artifact of too few points in the mammography-alone curve, so it was not pursued in any of the amendments.


In addition, the sensitivities failed to achieve a level of 99.3% with 75% confidence, in any of the datasets.


PPMA (n=15)

(Se 93.3%, Sp 25.4%)

Mammography alone

PMA (n=187)(Se 97.1%,Sp 14.3%)

Am. 4 (n=90)(Se 100%,Sp 18.0%)

Combined(n=105)

(Se 99.0%, Sp 19.2%)

72.1%

99.7%

94.1%

98.8%

95.6%96.7%

Reference Se 99.3%

SPECIFICITY

SENS

ITIV

ITY

100%

0%

Chance Line


Potential safety and effectiveness in U.S. population:


Percent of U.S. biopsies potentially obviated by the BCS if used on all eligible women: • 1.3 million U.S. women biopsied each year,• of which ~45% (585,000) have mammographic masses, • of which ~80% (468,000) are benign, • of which 15% to 20% (70,000 to 94,000) would be BCS(-) and saved a biopsy.• Thus ~5% to 7% of 1.3 million U.S. biopsies would be obviated,• and that’s if BCS were used on all 585,000 women with mammographic masses on their way to biopsy.• In addition to saving biopsies on these benign masses, approximately 1% to 6% of the malignant masses, and 0.5% to 3% of all breast cancers, might be delayed in diagnosis.

(


POINTS TO BE COVERED

• What BCS 2100 (BCS) is and is not intended to do• How BCS does it• What the clinical trial demonstrated• Labeling issues


Labeling issues

• Size of mass• Depth of mass


Size of mass

Effect of small lesion size on device sensitivity difficult to evaluate, since only 2/105 cancers < 5mm.


Size of mass (cont.)

Number of masses Malignant Benign<0.5 cm 2 12

0.5 cm - 1.0 cm

50 188

>1.0 cm 53 185

(Amendment 4, p.8, Table A4.4,

Amendment 5, p.23, Table A5.14)


Depth of mass

With the chosen IOS threshold (20.59) there was no definite effect of lesion depth on BCS result, but...


Depth of mass (cont.)

...the effect of lesion depth was difficult to evaluate, since depth is not easily gauged on mammography.


Conclusions


In summary• Only 4 out of 2407 subjects had an adverse event, all minor.• There were no explicit quantitative hypotheses.• There were two implicit quantitative hypotheses.• Neither hypothesis was fulfilled.• Most data were selected retrospectively.• Bonferroni correction is not applicable in this context.• Using the trial results, if the BCS were in general use in the U.S., it would obviate 5% to 7% of the 1.3 million biopsies per year. • Approximately 1% to 6% of these obviated biopsies would turn out to be malignant, and their diagnoses would thus be delayed.

Clinical Assessment – Part II William Sacks, PhD, MD

Documents

Transcript of Clinical Assessment – Part II William Sacks, PhD, MD