Issues of Simultaneous Tests for Non-Inferiority and Superiority

Tie-Hua Ng*, Ph. D.U.S. Food and Drug Administration

Ng@cber.fda.gov

Presented atMCP 2002

August 5-7, 2002Bethesda, Maryland

_______

* The views expressed in this presentation are not necessarily of the U.S. Food and Drug Administration.

2

Simultaneous Tests for Non-Inferiority and Superiority

• Multiplicity adjustment is not necessary– Intersection-union principle (IU)

• Dunnett and Gent (1996)

– Closed testing procedure (CTP)• Morikawa and Yoshida (1995)

• Indisputable

3

A Big Question

Is Multiplicity Adjustment Necessary?

4

Is Is MultiplicityMultiplicityAdjustment Adjustment Necessary?Necessary?

5

Outline• Assumptions and Notations

• Switching between Superiority and Non-Inferiority

• Is Simultaneous Testing Acceptable?

• Use of Confidence Interval in Hypothesis Testing --- Pitfall

• Problems of Simultaneous Testing

• Conclusion

6

Assumptions/Notations• Normality and larger is better• T: Test/Experimental treatment (t)

• S: Standard therapy/Active control (s): Non-Inferiority Margin (> 0)• For a given d (real number), define

– Null: H0(d): T S - d – Alternative: H1(d): T > S - d

• Non-Inferiority: d = • Superiority: d = 0

7

Non-Inferiority (d = )

H0(): T S - against H1(): T > S -

H0() H1()

•°

TBoundaryWorse

Better

Mean Response

S

8

Superiority (d = 0)

H0 (0): T S against H1 (0): T > S

H0(0)H1(0)

•°

TBoundary

Worse

Better

Mean Response

S

9

Switching between Superiority and Non-Inferiority

CPMP (Committee for Proprietary Medicinal Products), European Agency for the Evaluation of Medicinal Products

Points to Consider on Switching Between Superiority and Non-Inferiority, 2000.

http://www.emea.eu.int/htms/human/ewp/ewpptc.htm

10

Switching between Superiority and Non-Inferiority (2)

• Non-Inferiority Trial– If H0() is rejected, proceed to test H0(0)

– No multiplicity issue, closed testing procedure

• Superiority Trial– Fail to reject H0(0), proceed to test H0()

– No multiplicity issue

– Post hoc specification of

11

Switching between Superiority and Non-Inferiority (3)

• Non-inferiority Trial– Intention-to-treat (ITT)– Per protocol (PP)

• Superiority Trial– Primary: Intention-to-treat (ITT)– Supportive: Per protocol (PP)

• Assume ITT = PP

12

Simultaneous Testing

One-sided 100(1 - )% lower Confidence Interval for T - S

Test is worse

Test is betterMean Difference (T – S)

0-

Superiority

Non-inferiority

Neither

13

Simultaneous Testing (2)

• Multiplicity adjustment is not necessary– Dunnett and Gent (1996)

• Intersection-Union (IU):

Superiority: Both H0() and H0(0) are rejected

– Morikawa and Yoshida (1995)• Closed Testing Procedure (CTP):

Test H0(0) when H0()H0(0) is rejected

14

Simultaneous Testing (3)

• Discussion Forum (October 1998)– London

– PSI (Statisticians in Pharmaceutical Industry)

• Is Simultaneous Testing of Equivalence [Non-Inferiority] and Superiority Acceptable?– Superiority trial:

• Fail to reject H0 (0)

• No equivalence/non-inferiority claim

– Ok: Morikawa and Yoshida (1995)

• Ref: Phillips et al (2000), DIJ

15

Is Simultaneous

TestingAcceptable?

16

Use of Confidence Interval in Hypothesis Testing

H0(d): T S - d (at significance level )

One-sided 100(1-)% lower CI for T-S

Reject H0(d) if and only if the CI excludes -d

Test is worse

Test is better

Mean Difference (T – S)

-d

Reject H0(d)

Do not reject H0(d)

17

Use of Confidence Interval in Hypothesis Testing (2)

• If CI = (L, ), then H0(d) will be rejected for all -d < L.

• A Tricky Question– Suppose CI = (-1.999, ), L = -1.999

• H0(2): T S - 2 is rejected (d=2) since -d < L• Can we conclude that T > S - 2?

• Yes, if H0(2) is prespecified.• No, otherwise.

18

Use of Confidence Interval in Hypothesis Testing (3)

Post hoc specification of

H0(d) is a

No No

19

Simultaneous Testing: Problems

H0(d1) and H0(d2), for d1 > d2

One-sided (1 - )100% lower CI for T - S

Test is worse

Test is betterMean Difference (T – S)

-d2-d1

Reject H0(d2)

Reject H0(d1)

Neither

20

Simultaneous Testing: Problems (2)

H0(d1), H0(d2) and H0(d3), for d1 > d2 > d3

One-sided (1 - )100% lower CI for T - S

Test is worse

Test is betterMean Difference (T – S)

-d3-d1

Reject H0(d3)Reject H0(d2)

None

-d2

Reject H0(d1)

21

Simultaneous Testing: Problems (3)

H0(d1), H0(d2),…, H0(dk), for d1 > d2 > … > dk

One-sided (1 - )100% lower CI for T - S

Test is worse

Test is betterMean Difference (T – S)

-dk-d1

Reject H0(dk)

Reject H0(d2)

None

-d2

Reject H0(d1)

…

…

.

.

.

-d3

22

Simultaneous Testing: Problems (4)

• Choose k large enough Pr[-d1 < Lower limit < -dk] close to 1

Max |dk - dk-1| < a given small number

Simultaneous testing of H0(di), i = 1,…, k

Post hoc specification of H0(d)

23

Simultaneous Testing: Problems (5)

Number of Nested hypotheses

Exploratory (many H0(d))

Confirmatory (one H0(d))

1 2 3 4 …………. k …………

Simultaneous H0() and H0(0)

24

Simultaneous Testing: Problems (6)

• What is wrong with IU and CTP?

• Nothing

• Pr[Rejecting at least one true null] • What kind of problems?

25

Simultaneous Testing: Problems (7)

• Post hoc specification of H0(d)

Let -d0 = 100(1 - )% lower limit -

Reject H0(d0), since -d0 < lower limit

Repeat the same trial independently

Pr[Rejecting H0(d0)] = 0.5 +

26

Simultaneous Testing: Problems (8)

• Simultaneous testing of many H0(d)

– Repeat the same trial independently

– Low probability of confirming the finding

• 1st trial: Reject H0(dj) but not H0(dj+1)

• 2nd trial: Pr[Rejecting H0(dj)] is low (e.g., 0.5+)

27

Simultaneous Testing: Problems (9)

• Simultaneous testing of H0() and H0(0)?

• Confirm the finding = 2 Known variance Let T - S Significance level = 0.025 80% power for H0() (at = 0)

28

Simultaneous Testing: Problems (10)

f() = Pr[Rejecting H0() | ]

f0() = Pr[Rejecting H0(0) | ]

29

Simultaneous Testing: Problems (11)

• Test one null hypothesis H0()

• Suppose that H0() is rejected

• Repeat the same trial independently

• Pr[Rejecting H0() again] = f()

30

Simultaneous Testing: Problems (12)

• Test H0() and H0(0) simultaneously

• Suppose that H0() or H0(0) is rejected

• Repeat the same trial independently• Pr[Rejecting the same null hypothesis again] =

[1 - w()] · f() + w() · f0()

= f() - f0() [1 – f0()/f()],

where w() = f0()/f()

31

Simultaneous Testing: Problems (13)

[1 - w()] · f() + w() · f0()

where w() = f0()/f()

Simultaneous tests in the 2nd trial

32

Simultaneous Testing: Problems (14)

• Ratio: 1 – [f0()/f()] [1 – f0()/f()]

• Ratio may be as low as 0.75

33

Conclusion

• Many H0(d): Problematic

• Not type I error rate

• H0() and H0(0): Acceptable?

• If “zero tolerance policy”: No

• If 25% reduction cannot be tolerated: No

• If 25% reduction can be tolerated: Yes

34

Is Simultaneous Testing of

H0() and H0(0) Acceptable?

35

You be the judge

36

References• Dunnett and Gent (1976), Statistics in Medicine, 15,

1729-1738.

• Committee for Proprietary Medicinal Products (CPMP; 2002). Points to Consider on Switching Between Superiority and Non-Inferiority. http://www.emea.eu.int/htms/human/ewp/ewpptc.htm

• Morikawa T, Yoshida M. (1995), Journal of Biopharmaceutical Statistics, 5:297-306.

• Phillips et al., (2000), Drug Information Journal, 34:337-348. 

37