Receiver Operating Characteristic (ROC) curve analysis. 19.12

An assessment of the self-reported version of the Swedish Strengths and Difficulties Questionnaire among children and adolescents 12-16

years old

Kenisha S. Russell Jonsson

Irina Vartanova

SDQ studies • Study type 1: Examination of the psychometric properties (alpha

coefficients) Internal consistency

Retest reliability ??

• Study type 2: Factor stucture (factor analysis & SEM)Controversy of the five versus three structure versus bifactor

• Study type 3: Validity (ROC Analysis, mean comparison) specificity & sensitivity

Convergent validity ??

Data • Community Sample

Survey of children and young peoples mental health (Grodan) conducted in 2009, collected data from students in grade 6 and 9 (roughly between age 11-17). In total there were 172,000 respondents.

• Service Contact SampleDuring 1 mars – 30 september 2014 data a collected from 2 648 children and

young people from 27 municipalities in Sweden who visited a healthcarecenter.

Psychometric properties (1)Internal consistency reliability (Cronbachs Alpha) of the total difficulty scores and subscores

SDQ scale Community Service Contact

Widenfeltet al. (2003)

Goodman (2001)

Koskelainen et al. (2000)

Total difficulties 0.63 0.56 0.70 0.80 0.71

Emotional symptoms

0.69 0.67 0.63 0.66 0.69

Conduct problems 0.55 0.55 0.47 0.60 0.57

Hyperactivity-inattention

0.66 0.71 0.66 0.67 0.66

Peer Problems 0.54 0.59 0.39 0.41 0.63

Prosocial 0.68 0.67 0.60 0.66 0.69

Factor structure (1) Community sampleSpecified 5 factor analyses. Oblimin rotation (community sample)

Prosocial Emotion Hyper Conduct Peer

Somatic -0.004 0.582 0.019 0.105 -0.040

Worries 0.044 0.733 -0.014 -0.041 0.040

Unhappy 0.044 0.730 -0.081 0.130 0.154

Clingy -0.059 0.548 0.116 -0.181 0.094

Afraid 0.050 0.493 0.063 -0.100 0.103

Tantrum -0.070 0.408 0.107 0.335 -0.031

Robeys -0.406 0.070 0.032 0.187 -0.205

Fights -0.068 0.011 0.057 0.788 -0.006

Lies -0.022 0.024 0.167 0.489 0.180

Steals -0.062 0.034 0.038 0.544 0.079

Restles -0.006 -0.003 0.757 0.024 -0.044

Fidgety 0.034 -0.047 0.877 0.008 0.037

Distrac -0.137 0.359 0.362 0.076 -0.047

Reflect -0.456 0.137 0.094 0.150 -0.157

Rattends -0.451 0.315 0.212 0.026 -0.110

Loner 0.027 0.198 -0.003 -0.026 0.530

Friend -0.228 0.103 -0.072 0.079 0.638

Popular -0.383 0.117 0.046 -0.152 0.475

Bullied 0.112 0.203 0.032 0.280 0.558

Oldbest 0.250 -0.030 0.069 0.029 0.470

Consid 0.553 0.129 -0.001 -0.244 -0.121

Shares 0.461 0.046 0.031 0.088 -0.165

Caring 0.615 0.258 -0.002 -0.067 -0.217

Kind 0.498 -0.001 0.048 -0.258 -0.002

Helpout 0.779 -0.036 -0.0002 0.003 0.029

Factor structure (2) Service contact sampleSpecified 5 factor analyses. Oblimin rotation (community sample)

Prosocial Emotion Hyper Conduct Peer

Somatic 0.104 0.506 0.058 0.099 0.010

Worries 0.026 0.759 -0.069 -0.108 -0.002

Unhappy 0.108 0.680 -0.017 0.060 0.142

Clingy -0.110 0.553 0.039 -0.131 0.077

Afraid 0.023 0.459 -0.100 0.008 0.034

Tantrum -0.036 0.366 0.094 0.504 0.015

Robeys -0.254 -0.005 0.111 0.413 -0.091

Fights -0.038 -0.026 0.017 0.739 0.024

Lies 0.052 -0.181 0.111 0.558 0.301

Steals -0.191 0.046 0.089 0.366 0.030

Restles -0.013 -0.031 0.914 -0.032 0.045

Fidgety 0.006 -0.012 0.885 0.0002 -0.028

Distrac -0.046 0.396 0.384 0.203 -0.064

Reflect -0.124 -0.015 0.147 0.494 -0.078

Rattends -0.137 0.337 0.294 0.231 -0.036

Loner -0.191 0.247 -0.108 -0.151 0.435

Friend -0.089 0.061 -0.034 -0.089 0.751

Popular -0.198 0.073 0.044 0.095 0.587

Bullied 0.140 -0.022 0.098 0.129 0.661

Oldbest 0.148 0.080 -0.046 0.124 0.426

Consid 0.491 0.052 0.094 -0.479 0.020

Shares 0.561 0.034 -0.013 0.046 -0.144

Caring 0.727 0.104 0.007 -0.023 -0.134

Kind 0.435 -0.025 0.057 -0.275 -0.017

Helpout 0.765 -0.038 -0.090 -0.001 0.066

Validity (1) Descriptive stats

Validity (2) Descriptive stats

Validity (3) ROC Analysis Receiver operating curves

In a ROC curve the true positive rate (Sensitivity) is plotted as a function of the false positive rate (100-Specificity) for different cut-off points of a parameter.

The area under the ROC curve (AUC)

a measure of how well a parameter can distinguish between two diagnostic groups (community/service contact)a method for reducing the entire ROC curve to a single

quantitative index of diagnostic accuracy

Validity (4) Caseness

True positive: caseswith condition

classified as positive

False positive: caseswithout condition

classified as positive False negative: cases with conditionclassified as negative

True negative: caseswithout condition

classified as negative

Validity (5) Sensitivity–Specificity Report

Emotional Problems: Detailed report of sensitivity and specificity

Cutpoint Sensitivity Specificity

Correctly

Classified LR+ LR-

( >= 0 ) 100.00% 0.00% 0.53% 1.0000

( >= 1 ) 96.75% 16.52% 16.94% 1.1590 0.1965

( >= 2 ) 91.64% 35.90% 36.19% 1.4295 0.2330

( >= 3 ) 81.65% 54.18% 54.33% 1.7821 0.3387

( >= 4 ) 71.79% 68.67% 68.68% 2.2909 0.4109

( >= 5 ) 55.93% 79.78% 79.65% 2.7661 0.5524

( >= 6 ) 41.95% 87.91% 87.67% 3.4693 0.6604

( >= 7 ) 27.97% 93.21% 92.86% 4.1158 0.7729

( >= 8 ) 17.10% 96.42% 96.00% 4.7774 0.8597

( >= 9 ) 7.37% 98.42% 97.94% 4.6694 0.9412

( >= 10 ) 3.00% 99.38% 98.87% 4.8338 0.9761

( > 10 ) 0.00% 100.00% 99.47% 1.0000

Cutpoint:indicate the rating used to classifysubjects with/without a condition

Probability of correctlyclassifying those with a condition

Probability of correctlyclassifying those without a condition

The ratio of theprobability of anegative test amongtruly positive subjectsto the probability of anegative test amongtruly negative subjects

The ratio of theprobability of a positivetest among truly positivesubjects to theprobability of a positivetest among truly negativesubjects

Validity (6) AUC

OBS AUC SD LLCI ULCI

Total difficulties 151803 0.7076 0.0089 0.69017 0.72508

Emotional 151803 0.7541 0.0083 0.73778 0.77049

Conduct 151803 0.5761 0.0097 0.55715 0.59510

Hyperactivity-inattention 151803 0.5768 0.0104 0.55634 0.59728

Peer 151803 0.6104 0.0102 0.59051 0.63038

Prosocial 151803 0.5595 0.0098 0.54025 0.57879

Validity (7) ROC-Emotion

0.0

00

.25

0.5

00

.75

1.0

0

Se

nsitiv

ity

0.00 0.25 0.50 0.75 1.001 - Specificity

Area under ROC curve = 0.7541

service contact versus community sample

Emotional Problems

Validity (8) ROC-Conduct

0.0

00

.25

0.5

00

.75

1.0

0

Se

nsitiv

ity

0.00 0.25 0.50 0.75 1.001 - Specificity



Conduct Diffiulties

Validity (9) ROC -Hyper

0.0

00

.25

0.5

00

.75

1.0

0

Se

nsitiv

ity

0.00 0.25 0.50 0.75 1.001 - Specificity



Hyperactivity-inattention

Validity (10) ROC- Peer

0.0

00

.25

0.5

00

.75

1.0

0

Se

nsitiv

ity

0.00 0.25 0.50 0.75 1.001 - Specificity



Peer Problems

Validity (11) ROC-Prosocial

0.0

00

.25

0.5

00

.75

1.0

0

Se

nsitiv

ity

0.00 0.25 0.50 0.75 1.001 - Specificity



Prosocial Behaviour

Validity (12) ROC-Total difficulties

0.0

00

.25

0.5

00

.75

1.0

0

Se

nsitiv

ity

0.00 0.25 0.50 0.75 1.001 - Specificity



Total Difficulties

Dilemma• AUC low for some of the subscores -> community sample is too high

or the service contact sample is too low or vice versa.

How to solve this???

• Compare results with other countries (specifically nordic sample)

• Further analyses, restricting/more emphasis on service samplereason for the visit

number of visit

who contacted the service center (parent/ child/teacher/other adult)

reason for contact

FACTOR STRUCTURES- THE SWEDISH CONTRIBUTION

Explorative (EFA) vs Confirmative (CFA) Factor Analysis• In EFA, the factor structure is inferred from the obtained correlation

matrix.

• In CFA,the obtained correlation matrix is compared with a specified theoretical model.

• The result of comparison is goodness of fit of the specified model. Thus, we can compare different factor structures for better understanding of the analyzed questionnaire.

EFA vs CFACorrelation matrix

Factor structure

Correlation matrix

Theoretical model

compared

Model fit

Bifactor models – the latest suggestion of model fit improvement

Kobor et al., 2013 Casi et al., 2015

Alternative models fitModel chisq df RMSEA CFI TLI

Original 5-factor model 198,004 265 0.068 0.896 0.882

5-factor model with acquiescence style 125,592 259 0.051 0.942 0.932

Alternative 3-factor model 269,164 272 0.080 0.853 0.838

3-factor model with acquiescence style 232,670 268 0.073 0.880 0.865

Bifactor model (Kobor et al., 2013) 96,664 240 0.044 0.961 0.952

Bifactor model (Casi et al., 2015) 164,093 252 0.063 0.914 0.898

Different model fit measures

Best fit model

Model currentlytesting

Reference

Caci, H., Morin, A. J., & Tran, A. (2015). Investigation of a bifactor model of the Strengths and Difficulties Questionnaire. European child & adolescent psychiatry, 24, pp 1291-1301.

Choi,B.C.K. 1998. Slopes of a receiver operating characteritic curve and the likelihood ratio for a diagnostic test.American Journal of Epidemiology 148:1127-1132.

Di Riso, D., Salcuni, S., Chessa, D., Raudino, A., Lis, A., & Altoè, G. (2010). The Strengths and Difficulties Questionnaire (SDQ). Early evidence of its reliability and validity in a community sample of Italian children. Personality and Individual Differences, 49(6), 570-575.

Essau, C. A., Olaya, B., Anastassiou‐Hadjicharalambous, X., Pauli, G., Gilvarry, C., Bray, D., ... & Ollendick, T. H. (2012). Psychometric properties ofthe Strength and Difficulties Questionnaire from five European countries. International journal of methods in psychiatric research, 21(3), 232-245.

Goodman, R. (1997). The Strengths and Difficulties Questionnaire: a research note. Journal of child psychology and psychiatry, 38(5), 581-586.

Goodman, R., Meltzer, H., & Bailey, V. (1998). The Strengths and Difficulties Questionnaire: A pilot study on the validity of the self-report version. European child & adolescent psychiatry, 7(3), 125-130.

Goodman, A., Lamping, D. L., & Ploubidis, G. B. (2010). When to use broader internalising and externalising subscales instead of the hypothesised five subscales on the Strengths and Difficulties Questionnaire (SDQ): data from British parents, teachers and children. Journal of abnormal child psychology,38(8), 1179-1191.

Hanley,J.A and B.J. McNeil.1982.The meaning and the use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:9-36.

Kóbor, A., Takács, Á., & Urbán, R. (2013). The bifactor model of the Strengths and Difficulties Questionnaire. European Journal of PsychologicalAssessment, 29, pp. 299-307.

Receiver Operating Characteristic (ROC) curve analysis. 19.12

Documents

Transcript of Receiver Operating Characteristic (ROC) curve analysis. 19.12