Judith D. Singer & John B. Willett Harvard Graduate School of Education Discrete-time survival...

Judith D. Singer & John B. WillettHarvard Graduate School of

Education

Discrete-time survival analysis ALDA, Chapters 10, 11, and 12

Times change, and we change with themAnonymous, quoted in Holinshed’s Chronicles, 1578

What we will cover….What we will cover….

Making sure we’re all on the same page: A quick review of basic descriptive statistics for discrete-time event data (Ch 10) using the “Age at first intercourse” study

Specifying a discrete-time hazard model (§11.1 & 11.2)—both heuristic and formal representations

Model fitting, interpretation and comparison (§11.3-11.6)—very similar to logistic regression

Alternative specifications of the baseline in the discrete-time hazard model (§12.1)—more parsimonious representations of TIME

Including time-varying predictors (§12.3)—use of a person-period data set makes them easy to include (although interpretations require care)

Evaluating and relaxing the proportionality assumption (§12.5)—not all predictors have time constant effects

Making sure we’re all on the same page: A quick review of basic descriptive statistics for discrete-time event data (Ch 10) using the “Age at first intercourse” study

Specifying a discrete-time hazard model (§11.1 & 11.2)—both heuristic and formal representations

Model fitting, interpretation and comparison (§11.3-11.6)—very similar to logistic regression

Alternative specifications of the baseline in the discrete-time hazard model (§12.1)—more parsimonious representations of TIME

Including time-varying predictors (§12.3)—use of a person-period data set makes them easy to include (although interpretations require care)

Evaluating and relaxing the proportionality assumption (§12.5)—not all predictors have time constant effects

© Singer & Willett, page 2

The life table: Describing the distribution of event occurrence over timeThe life table: Describing the distribution of event occurrence over time

(ALDA, Section 10.1, pp 326-329)

n at riskjn censoredj

n eventsj

Recall the grade of 1st intercourse study: 180 middle school boys were tracked from 7th through 12th grades. By the end of data collection (at the end of 12 th grade),

n=126 (70.0%) had had sex; n=54 (30%) were censored (still virgins)

Recall the grade of 1st intercourse study: 180 middle school boys were tracked from 7th through 12th grades. By the end of data collection (at the end of 12 th grade),

n=126 (70.0%) had had sex; n=54 (30%) were censored (still virgins)


The discrete-time hazard function: Assessing the conditional risk of event occurrenceThe discrete-time hazard function: Assessing the conditional risk of event occurrence

(ALDA, Section 10.2.1, pp 330-339)

6 7 8 9 10 11 12

Grade

0.00

0.10

0.20

0.30

h(t)Discrete-time hazard

Conditional probability that individual i experiences the target event in time period j (T i = j) given that s/he didn’t experience it in any earlier time period (T i j)

h(tij)=Pr{Ti= j|Ti j}

Easy to estimate because each value of hazard is based on that interval’s risk set.

As a probability, discrete time hazard is bounded by 0 and 1. This is an issue for modeling we’ll need to address

Discrete-time hazardConditional probability that individual i experiences the target event in time period j (T i = j) given that s/he didn’t experience it in any earlier time period (T i j)

h(tij)=Pr{Ti= j|Ti j}

Easy to estimate because each value of hazard is based on that interval’s risk set.

As a probability, discrete time hazard is bounded by 0 and 1. This is an issue for modeling we’ll need to address

j

jj risk at n

events nth )(

j

jj risk at n

events nth )(

0833.0180

15)(ˆ 7 th

3250.080

26)(ˆ 12 th


The survivor function (and median lifetime): Cumulating risk over time The survivor function (and median lifetime): Cumulating risk over time


6 7 8 9 10 11 12

Grade

0.00

0.25

0.50

0.75

1.00

S(t)

Discrete-time survival probabilityProbability that individual i will “survive” beyond time period j (Ti > j) (i.e., will not experience the event until after time period j).

S(tij)=Pr{Ti > j}By definition, at the beginning of time, S(t0)=1.0

Strategy for estimation: Since h(tij) tells us about the probability of event occurrence, 1-h(tij) tells us about the probability of non-occurrence (i.e., about survival)

Discrete-time survival probabilityProbability that individual i will “survive” beyond time period j (Ti > j) (i.e., will not experience the event until after time period j).

S(tij)=Pr{Ti > j}By definition, at the beginning of time, S(t0)=1.0

Strategy for estimation: Since h(tij) tells us about the probability of event occurrence, 1-h(tij) tells us about the probability of non-occurrence (i.e., about survival)

)](ˆ1)[(ˆ)(ˆ1 jjj thtStS

)](ˆ1)[(ˆ)(ˆ1 jjj thtStS

ML = 10.6

Estimated median lifetime

9167.0]0833.01[0.1)(ˆ7 tS

7444.0]1519.01[8778.0)(ˆ9 tS


Towards a discrete time hazard model: Inspecting sample plots of within-group hazard functions: In raw and transformed scales

Towards a discrete time hazard model: Inspecting sample plots of within-group hazard functions: In raw and transformed scales

(ALDA, Section 11.1.1, pp 358-361)

0

0.25

0.5

6 7 8 9 10 11 12

Grade

Haz

ard

PT=1

PT=0

Questions to ask when examining sample hazard functions:

What is the shape of each hazard function?

Does the relative level of hazard differ across groups?

Suggests the appropriateness of the dual partition introduced earlier, but how do we deal with the bounded nature of hazard?

Questions to ask when examining sample hazard functions:

What is the shape of each hazard function?

Does the relative level of hazard differ across groups?

Suggests the appropriateness of the dual partition introduced earlier, but how do we deal with the bounded nature of hazard?

0

0.2

0.4

0.6

0.8

1

6 7 8 9 10 11 12

Grade

Odd

s

PT=1

PT=0

Transform into odds

Solves the upper bound problem but not the lower bound

Transform into odds

Solves the upper bound problem but not the lower bound

hazard

hazardodds

1

-4

-3

-2

-1

0

6 7 8 9 10 11 12

Grade

logi

t(h

azar

d)

PT=1

PT=0

Transform into logits

Usually regularizes distances between functions—stretches distances between small values and compresses distances between large values

Not bounded at all (although you need to get used to negative #’s)

Transform into logits

Usually regularizes distances between functions—stretches distances between small values and compresses distances between large values

Not bounded at all (although you need to get used to negative #’s)

hazard

hazardoddslogit

1log)log(

+

++

+

+

+


What population model might have generated these sample data? Sample hazard estimates, alternative hypothesized models, parameterizing the DT hazard model

What population model might have generated these sample data? Sample hazard estimates, alternative hypothesized models, parameterizing the DT hazard model

(ALDA, Section 11.1.1, pp 366-369)

Flat population logit hazard, shifted when PT switches from

0 to 1

Linear population logit hazard, shifted when PT switches from 0

to 1

General population logit hazard, shifted when PT switches from 0 to 1

When PT=1, you shift this entire baseline vertically by 1

PT1

1

How do we fit this model to data?How do we fit this model to data?

][)(hlogit 121277 DDDt jjij

Baseline logit hazard function (when PT=0)7

9

10

11

12

(D7=1) (D8=1) (D11=1) (D12=1)...


The person-period data set: The key to fitting the discrete-time hazard modelThe person-period data set: The key to fitting the discrete-time hazard model

(ALDA, Section 10.5.1, pp 351-354)

Person level data setid T censor pt

126 12 0 1

193 9 0 1

407 12 1 0

ijjij PTDDDthlogit 1121277 ][)(

100010019193

100001008193

100000107193

0100000012407

0010000011407

0001000010407

000010009407

000001008407

000000107407

All parameter estimates, standard errors, t- and z-statistics, goodness-of-fit statistics, and tests will be correct for the discrete-

time hazard model

All parameter estimates, standard errors, t- and z-statistics, goodness-of-fit statistics, and tests will be correct for the discrete-

time hazard model

Person-Period data setptD12D11D10D9D8D7eventTIMEid

1100000112126

1010000011126

1001000010126

100010009126

100001008126

100000107126


Model A: A “baseline discrete-time hazard model” with no substantive predictorsModel A: A “baseline discrete-time hazard model” with no substantive predictors

(ALDA, Section 11.4.1, pp 386-388)

Because there are no predictors in Model A, this baseline is for the

entire sample• If estimates are approx equal, baseline is

flat• If estimates decline, hazard declines• If estimates increase (as they do here),

hazard increases

Because there are no predictors in Model A, this baseline is for the

entire sample• If estimates are approx equal, baseline is

flat• If estimates decline, hazard declines• If estimates increase (as they do here),

hazard increases

12128877j ...)h(tlogit :A Model DDD


Models B & C: Uncontrolled effects of substantive predictorsModels B & C: Uncontrolled effects of substantive predictors

(ALDA, Section 11.4.2 & 11.4.3, pp 388-390)

Continuous predictorsAntilogging still yields a

estimated odds-ratio associated with a 1-unit difference in the

predictor:

56.14428.0ˆee PAS

The estimated odds of first intercourse are 1.56 times (just

over 50% higher) for boys whose parents score one unit higher on this antisocial behavior index.

PASDDD

PTDDD

212128877j

112128877j

...)h(tlogit :C Model

...)h(tlogit :B Model

4.28736.0ˆee PT

The estimated odds of first intercourse for boys who have

experienced a parenting transition are 2.4 times higher than the odds for boys who did

not experience such a transition.

Dichotomous predictorsAs in regular logistic regression,

antilogging a yields the estimated odds-ratio associated with a 1-unit difference in the

predictor:

^


Comparing nested models using deviance statistics (and non-nested models information criteria)

Comparing nested models using deviance statistics (and non-nested models information criteria)


TIME dummies

Deviance smaller value, better fit, 2

dist., compare nested models

AIC, BIC smaller value, better fit,

compare non- nested models

Model B vs. Model A provides an uncontrolled test of H0: PT=0Deviance=17.30(1), p<.001

Model C vs. Model A provides an uncontrolled test of H0: PAS=0Deviance=14.79(1), p<.001

Model D vs. Models B&C provide controlled tests

[Both rejected as well]


Displaying fitted hazard and survivor functionsSubstitute in prototypical predictor values and compute fitted values

Displaying fitted hazard and survivor functionsSubstitute in prototypical predictor values and compute fitted values

(ALDA, Section 11.5.1, pp 392-394)

PTDDD

PTDDD

8736.01791.1...7001.39943.2)(thlogit

...)h(tlogit

1287j

112128877j

PTDDD

PTDDD

8736.01791.1...7001.39943.2)(thlogit

...)h(tlogit

1287j

112128877j

Model BModel B

In logit hazard scale, a constant vertical separation of

0.8736

In hazard scale, a non-constant vertical separation

(no simple interpretation because this a proportional

odds model, not a proportional hazards model!)

Effect of PT cumulates into a large difference in estimated

median lifetimes(9.9 vs. 11.8 2 years)


Pros and cons of the dummy specification for the “main effect of TIME”Pros and cons of the dummy specification for the “main effect of TIME”


][][)( 1111 kkjjj XXDDthlogit

The dummy specification for TIME is:• Completely general, placing no constraints on

the shape of the baseline (logit) hazard function;• Easily interpretable—each associated

parameter represents logit hazard in time period j for the baseline group

• Consistent with life-table estimates

PRO

The dummy specification for TIME is also:

• Nothing more than an analytic decision, not a requirement of the discrete-time hazard model

• Completely lacking in parsimony. If J is large, it requires the inclusion of many unknown parameters;

• A problem when it yields fitted functions that fluctuate erratically across time periods because of nothing more than sampling variation

CON

Three reasons for considering an alternative specification

Your study involves many discrete time periods (because data collection is long or time is less coarsely discretized)

Hazard is expected to be near 0 in some time periods (causing convergence problems)

Some time periods have small risk sets (because either the initial sample is small or hazard and censoring dramatically diminish the risk set over time)

Three reasons for considering an alternative specification

Your study involves many discrete time periods (because data collection is long or time is less coarsely discretized)

Hazard is expected to be near 0 in some time periods (causing convergence problems)

Some time periods have small risk sets (because either the initial sample is small or hazard and censoring dramatically diminish the risk set over time)

The variable PERIOD in the person-period data set can be treated as continuous TIME


0ONE 1(TIME-c)2Linear1 Centering constant helps interpretation

0ONE 1(TIME-c) 2(TIME-c)2 3(TIME-c)34cubic3

0ONE 1(TIME-c) 2(TIME-c)23quadratic2Common choices

0ONE 1(TIME-c) 2(TIME-c)2 3(TIME-c)34(TIME-c)4

53 stationary points

4

0ONE 1(TIME-c) 2(TIME-c)2 3(TIME-c)34(TIME-c)45(TIME-c)5

64 stationary points

5

Rarely adopted but gives a sense of whether you should stick with completely general specification

0ONE1Constant0 Always the worst fit (highest deviance)

Comparing the general specification to an ordered set of polynomialsNot necessarily “the best,” but a systematic set of informative choices

Comparing the general specification to an ordered set of polynomialsNot necessarily “the best,” but a systematic set of informative choices

(ALDA, Section 12.1.1, pp 409-412)

1D1 + … + JDJ JGeneraln/a

Model:

logit h(tij)=

n parameters

Behavior of logit hazard

Order of polynomial

Always the best fit (lowest deviance)

Strategy for model comparison

Because each lower order model is nested within each higher order model, Deviance statistics can be directly

compared to help make analytic decisions


Examining alternative polynomial specification for TIME:Deviance statistics and fitted logit hazard functions

Examining alternative polynomial specification for TIME:Deviance statistics and fitted logit hazard functions

(ALDA, Section 12.1.1, pp 412-419)

The quadratic looks reasonably good, but can we test whether it’s “good enough”?

The quadratic looks reasonably good, but can we test whether it’s “good enough”?

0 1 2 3 4 5 6 7 8 9

Years after hire

0.0

-1.0

-2.0

-3.0

-4.0

-5.0

-6.0

Fitted logit(hazard)

GeneralConstant

Linear

Quadratic

Cubic

Sample: 260 faculty members (who had received a National Academy of Education Post-Doc)Each was tracked for up to 9 years after taking his/her first academic jobBy the end of data collection, n=166 (63.8%) had received tenure; the other 36.2% were censored (because they might eventually receive tenure somewhere).

Sample: 260 faculty members (who had received a National Academy of Education Post-Doc)Each was tracked for up to 9 years after taking his/her first academic jobBy the end of data collection, n=166 (63.8%) had received tenure; the other 36.2% were censored (because they might eventually receive tenure somewhere).

Gamse and Conger (1997) Abt Associates Comparisons always worth making

Is the added polynomial term necessary?

Is this polynomial as good as the general spec?

* Constant is terrible* Linear is better, but not as good as general* Quadratic is better still, and nearly as good as general

* Cubic on up seem thoroughly unnecessary


Including time-varying predictors: Age of onset of 1st depressive episodeIncluding time-varying predictors: Age of onset of 1st depressive episode

Sample: 1,393 adults ages 17 to 57 387 (27.8%) reported a first depression onset between ages 4 and 39

Specification of baseline hazard functionMany person-periods (36,997) and very few actual events (387)Annual data between ages 4 and 39 requires 36 TIME dummies—hardly parsimonious A cubic function of TIME fits nearly as well (2=34.51, 32 df, p>.25) as a completely general specification and measurably better (2=5.83, 1 df, p<.05) than a quadratic

Time-varying predictor: First parental divorce n=145 (10.4%) experienced a first parental divorce while still at risk of first depression onsetPD is time-varying, indicating whether the parents of individual i divorced during, or before, time period j.

PDij=0 in periods before the divorce

PDij=1 in periods coincident with or subsequent to the divorce

Sample: 1,393 adults ages 17 to 57 387 (27.8%) reported a first depression onset between ages 4 and 39

Specification of baseline hazard functionMany person-periods (36,997) and very few actual events (387)Annual data between ages 4 and 39 requires 36 TIME dummies—hardly parsimonious A cubic function of TIME fits nearly as well (2=34.51, 32 df, p>.25) as a completely general specification and measurably better (2=5.83, 1 df, p<.05) than a quadratic

Time-varying predictor: First parental divorce n=145 (10.4%) experienced a first parental divorce while still at risk of first depression onsetPD is time-varying, indicating whether the parents of individual i divorced during, or before, time period j.

PDij=0 in periods before the divorce

PDij=1 in periods coincident with or subsequent to the divorce

id age female pd event

40 4 1 0 0

40 5 1 0 0

40 6 1 0 0

40 7 1 0 0

40 8 1 0 0

40 9 1 1 0

40 10 1 1 0

40 11 1 1 0

40 … … … …

40 22 1 1 0

40 23 1 1 1

Data source: Blair Wheaton and colleagues (1997) Stress & adversity across the life course

(ALDA, Section 12.3, p 428)

33

221 )18()18()18()( AGEAGEAGEONEthlogit ijoij

ID 40: Reported first depression onset at 23; first parental divorce at age 9


Including a time-varying predictor in the discrete-time hazard modelIncluding a time-varying predictor in the discrete-time hazard model

(ALDA, Section 12.3.1, p 428-434)

ijijoij PDAGEAGEAGEONEthlogit 13

32

21 )18()18()18()(

What does 1 tell us ?Contrasts the population logit hazard for people who have experienced a parental divorce with those who have not,

But because PDij is time-varying, membership in the parental divorce group changes over time so we’re not always comparing the same peopleThe predictor effectively compares different groups of people at different times!But, we’re still assuming that the effect of the time-varying predictor is constant over time.

What does 1 tell us ?Contrasts the population logit hazard for people who have experienced a parental divorce with those who have not,

But because PDij is time-varying, membership in the parental divorce group changes over time so we’re not always comparing the same peopleThe predictor effectively compares different groups of people at different times!But, we’re still assuming that the effect of the time-varying predictor is constant over time.

Sample logit(proportions) of people experiencing first depression onset at each age, by PD status at that age

Hypothesized population model (note constant effect of PD)

Implicit particular realization of population model (for those whose parents divorce when they’re age 20)


Interpreting a fitted DT hazard model that includes a TV predictorInterpreting a fitted DT hazard model that includes a TV predictor

(ALDA, Section 12.3.2, pp 434-440)

FEMALEPD

AGEAGEAGEONEthlogit

ij

ijij

5455.04151.

)18(0002.0)18(0074.0)18(0596.05866.4)(ˆ 32

e0.4151=1.51 Controlling for gender, at every age from 4 to 39, the estimated odds of first depression onset are about 50% higher for

individuals who experienced a concurrent, or previous, parental divorce

e0.5455=1.73 Controlling for parental divorce, the estimated odds of first depression onset are

73% higher for women

What about a woman whose parents divorced when she

was 20?


The proportionality assumption:Is a predictor’s effect constant over time or might it vary?

The proportionality assumption:Is a predictor’s effect constant over time or might it vary?

(ALDA, Section 12.5.1, pp 451-456)

0 1 2 3 4 5 6 7 8

Time period

-5.00

-4.00

-3.00

-2.00

-1.00

Logit hazard

Predictor’s effect is constant over time

0 1 2 3 4 5 6 7 8

Time period

-5.00

-4.00

-3.00

-2.00

-1.00

Logit hazard

Predictor’s effect

increases over time

0 1 2 3 4 5 6 7 8

Time period

-5.00

-4.00

-3.00

-2.00

-1.00

Logit hazard

Predictor’s effect decreases over time

0 1 2 3 4 5 6 7 8

Time period

-5.00

-4.00

-3.00

-2.00

-1.00

Logit hazard

Predictor’s effect is particularly pronounced in

certain time periods


Discrete-time hazard models that do not invoke the proportionality assumptionDiscrete-time hazard models that do not invoke the proportionality assumption

(ALDA, Section 12.5.1, pp 454-456)

][][)(ˆ 111111 JJJJj DXDXDDthlogit

A completely general representation:

The predictor has a unique effect in each period

on... soand ,Xthlogit :2 period time In

Xthlogit :1 period time In

j

j

122

111

)(ˆ

)(ˆ

)(][)(ˆ 121111 cTIMEXXDDthlogit JJj A more parsimonious representation:

The predictor’s effect changes linearly with time 1 assesses the effect of

X1 in time period c

2 describes how this effect linearly increases (if positive) or decreases

(if negative)

LATEXXDDthlogit JJj 121111 ][)(ˆ Another parsimonious representation:

The predictor’s effect differs across epochs 2 assesses the additional effect of X1

during those time periods declared to be “later” in time


The proportionality assumption: Uncovering violations and simple solutionsThe proportionality assumption: Uncovering violations and simple solutions

(ALDA, Section 12.4, pp 443)

Data source: Graham (1997) dissertationSample: 3,790 high school students who participated in the Longitudinal Survey of American Youth (LSAY)Research design:

Tracked from 10th grade through 3rd semester of college—a total of 5 periodsOnly n=132 (3.5%) took a math class for all of the 5 periods!

RQs:When are students most at risk of dropping out of math?What’s the effect of gender?Does the gender differential vary over time?

Data source: Graham (1997) dissertationSample: 3,790 high school students who participated in the Longitudinal Survey of American Youth (LSAY)Research design:

Tracked from 10th grade through 3rd semester of college—a total of 5 periodsOnly n=132 (3.5%) took a math class for all of the 5 periods!

RQs:When are students most at risk of dropping out of math?What’s the effect of gender?Does the gender differential vary over time? HS 11

HS 12 C 1 C 2 C 3

Term

0.0

-1.0

-2.0

Sample logit(hazard)

Risk of dropping out zig-zags over time—peaks at 12th and 2nd semester of college

Magnitude of the gender differential varies over time—smallest in 11th grade and increases over time

Suggests that the proportionality assumption is being violated


Checking the proportionality assumption: Is the effect of FEMALE constant over time?Checking the proportionality assumption: Is the effect of FEMALE constant over time?

(ALDA, Section 12.5.2, pp 456-460)

All models include a completely general specification for TIME using 5 time dummies: HS11, HS12, COLL1, COLL2, and COLL3

Model C: Interaction between FEMALE and time

HS 11

HS 12 C 1 C 2 C 3

Term

0.0

-1.0

-2.0

Fitted logit(hazard)

8.04 (4) ns

6.50 (1) p=0.0108


Judith D. Singer & John B. Willett Harvard Graduate School of Education Discrete-time survival...

Documents

Transcript of Judith D. Singer & John B. Willett Harvard Graduate School of Education Discrete-time survival...