description: tags: REL 2007034

8/14/2019 description: tags: REL 2007034

1/175

I S S U E S & A N S W E R S

U . S . D e p a r t m e n t o f E d u c a t i o n

Measuringresilienceand youthdevelopment:the psychometricproperties of

the HealthyKids Survey

R E L 2 0 0 7 N o . 0 3 4

At WestEd


2/175

Measuring resilience and youthdevelopment: the psychometric

properties of the Healthy Kids SurveySeptember 2007

Prepared by

Thomas L. HansonRegional Educational Laboratory West

Jin-Ok KimRegional Educational Laboratory West

I S S U E S &ANSWERS R E L 2 0 0 7 N o . 0 3 4

U . S . D e p a r t m e n t o f E d u c a t i o n

At WestEd


3/175

Issues & Answers is an ongoing series o reports rom short-term Fast Response Projects conducted by the regional educa-tional laboratories on current education issues o importance at local, state, and regional levels. Fast Response Project topicchange to re ect new issues, as identi ed through lab outreach and requests or assistance rom policymakers and educa-tors at state and local levels and rom communities, businesses, parents, amilies, and youth. All Issues & Answers reportsmeet Institute o Education Sciences standards or scienti cally valid research.

September 2007

Tis report was prepared or the Institute o Education Sciences (IES) under Contract ED-06-CO-0014 by Regional Edu-cational Laboratory West administered by WestEd. Te content o the publication does not necessarily re ect the views orpolicies o IES or the U.S. Department o Education nor does mention o trade names, commercial products, or organiza-tions imply endorsement by the U.S. Government.

Tis report is in the public domain. While permission to reprint this publication is not necessary, it should be cited as:

Hanson, . L., & Kim, J. O. (2007). Measuring resilience and youth development: the psychometric properties o the HealthyKids Survey.(Issues & Answers Report, REL 2007No. 034). Washington, DC: U.S. Department o Education, Institute o Education Sciences, National Center or Education Evaluation and Regional Assistance, Regional Educational LaboratoryWest. Retrieved romhttp://ies.ed.gov/ncee/edlabs

Tis report is available on the regional educational laboratory web site athttp://ies.ed.gov/ncee/edlabs.

WA

OR

ID

MT

NV

CAUT

AZ

WY

ND

SD

NE

KSCO

NM

TX

OK

CO

AR

LA

MS ALGA

SC

NC

VAWV

KY

TN

PA

NY

FL

AK

MN

WI

IA

IL IN

MI

OH

VT

NH

ME

MO

At WestEd
http://ies.ed.gov/ncee/edlabshttp://ies.ed.gov/ncee/edlabshttp://ies.ed.gov/ncee/edlabshttp://ies.ed.gov/ncee/edlabs


4/175

Summary

This report summarizes fndings rom astudy o the psychometric properties o the resilience and youth developmentmodule, a key component o the HealthyKids Survey. The study aims to improveresilience assessment and research sothat educators can shape the school envi-ronment to promote academic resilience.

Te Healthy Kids Survey (HKS) is a compre-hensive student sel -report tool or monitoringthe school environment and student healthrisks. Tis report ocuses on one module o thesurvey, the resilience and youth development

module (RYDM), which assesses environmen-tal and internal assets associated with posi-tive youth development and school success.Environmental assets re er to meaning ul andpro-social bonding to community, school,amily, and peers. Internal assets are personalresilience traits, such as sel -e cacy andproblem-solving skills

A part o the resilience and youth developmentmodule is administered to 600,000 studentsin Cali ornia every year. School districtsand schools, which receive both single-yearprevalence data and trend data gathered bythe module, use the data to evaluate their localprograms and guide decisionmaking. TeHealthy Kids Survey and the resilience andyouth development module were designed as an

epidemiological surveillance tool to track ag-gregate levels o health risk and resilience. Temodule increasingly is being used in evaluationwork to assess student-level changes over time.

However, widespread use o the module,particularly or evaluation, may be premature.Te psychometric properties o speci c scalesassessed by the elementary school modulehave yet to be established. Te secondaryschool module has not been validated since2000, when the instrument was rst tested inthe eld. Te instrument has since undergoneseveral modi cations, however, and must be re-

validated. Moreover, measurement equivalenceacross di erent grades, males and emales, andracial and ethnic groups has never been exam-ined. Given Cali ornias diversity, demonstrat-ing the cultural appropriateness o the moduleor di erent racial and ethnic groups is critical.

Using HKS data processed or school districtsby WestEds Health and Human DevelopmentProgram, Regional Educational Laboratory

West analyzed the modules psychometricproperties. Tis report describes the results o this analysis, provides recommendations onthe proper use o the instrument, and suggestsmodi cations to the module.

For the secondary school module, the resultsare consistent with the instruments current

M s s th

m t: th s h m tt s th H th K s S


5/175

Summary

use as an epidemiological tool and with itsconceptual oundation. It provides compre-hensive and balanced coverage o eight envi-ronmental resilience assets and our internal

resilience assets; its subscales exhibit goodinternal consistency and are associated withstudent risk actors in expected ways. And i certain items are dropped, the module alsodemonstrates measurement equivalence acrossracial/ethnic groups, males and emales, andgrades. Te secondary school RYDM scales ex-hibit low test-retest reliability, however, whichsuggests that the module is not well suited orexamining student-level changes over time.Te instrument was not designed to examineindividual di erences across students andshould not be used this way. Moreover, twoo the six internal assets that the secondary

school module was designed to measurecooperation and goals/aspirationscould notbe assessed validly. Several measures wouldbene t i additional items were included in

derived scales to increase domain coverage.

Te elementary school module was designedto assess seven environmental resilience assetsand three internal resilience assets, but it canreliably assess only two environmental as-sets and one internal asset. Most o the scalesmeasured by the elementary school instru-ment have poor psychometric properties. Teelementary school instrument should thus bemodi ed considerably to make it suitable orresearch.

September 2007


6/175

Table o conTenTS

Why this study? 1

Developing a risk and resilience assessment tool 2Te Healthy Kids Surveyassessing risk and protective actors 2

Te resilience and youth development moduleassessing the other side o risk 4Evaluating the psychometric properties o the resilience and youth development module 8

Results o the analysis o the secondary school module 10Results o the analysis o the elementary school module 11

Recommendations 12Secondary school environmental resilience assets 12Secondary school internal resilience assets 12Elementary school environmental and internal assets 13

Appendix A Analytic strategy 15

Appendix B Results 21Appendix C Results and model selection details 44

Appendix D Other assessments o resilience and related actors 53

Appendix E Detailed tables 55

Notes 164

Re erences 165

Boxes

1 Speci cations o the Healthy Kids Survey 32 Data and analytic strategies 9

Figures

1 Conceptual model or the resilience and youth development module 5

A1 Hypothetical example o MIMIC approach or testing or measurement equivalence 18

C1 Secondary environmental resilience asset scree plot, total analytic samples 44

C2 Elementary school environmental resilience asset scree plot, total analytic samples 46

C3 Secondary school internal resilience asset scree plot, total analytic samples 50

C4 Elementary school internal resilience asset scree plot, total analytic samples 52

Tables

1 Items on the secondary school resilience and youth development module by construct, 2006/07 6

2 Elementary school resilience and youth development module items by construct, 2006/07 8

3 Recommended measures o environmental resilience assets among secondary school students 13

4 Recommended measures o internal resilience assets among secondary school students 14


7/175

5 Recommended measures o environmental and internal resilience assets among elementary schoolstudents 14

A1 Missing data patterns or secondary and elementary samples rom the resilience and youth developmentmodule 16

B1 Secondary school environmental resilience asset exploratory actor analysis results, main sample, 8- actorsolution 22

B2 Secondary school environmental resilience asset exploratory actor analysis results, validation sample,8- actor solution 23

B3 Final secondary school environmental assets model, main sample 24

B4 Correlations among secondary school environmental resilience assets, nal con rmatory actor analysismodel 25

B5 Elementary school environmental resilience asset exploratory actor analysis results, main sample, 4- actorsolution 25

B6 Elementary school environmental resilience asset exploratory actor analysis results, validation sample,4- actor solution 26

B7 Final elementary school environmental resilience assets model, main sample 27

B8 Secondary school internal resilience asset exploratory actor analysis results, main sample, 4- actormodel 28

B9 Secondary school internal resilience asset exploratory actor analysis results, validation sample, 4- actormodel 29

B10 Final secondary school internal resilience assets model, main sample 30

B11 Elementary school internal resilience asset exploratory actor analysis results, main sample, 2- actormodel 30

B12 Elementary school internal resilience asset exploratory actor analysis results, validation sample, 2- actormodel 31

B13 Final elementary school internal resilience asset model, main sample 31

B14 Secondary school internal consistency reliability coe cients by demographic subgroup 32

B15 Elementary school internal consistency reliability coe cients by gender 32

B16 est-retest reliability o secondary school environmental resilience asset constructs and items 33

B17 est-retest reliability o secondary school internal resilience asset constructs and items 34

B18 est-retest reliability o elementary school resilience asset constructs and items 35

B19 Secondary school subscale means by demographic subgroup 36

B20 Elementary school subscale means by gender 37

B21 Correlations between secondary school environmental resilience assets and criterion variables 38

B22 Correlations between secondary school internal resilience assets and criterion variables 39


8/175

B23 Correlations between elementary school resilience assets and criterion variables 40

B24 Current and recommended measures o environmental resilience assets among secondary schoolstudents 41

B25 Current and recommended measures o internal resilience assets among secondary school students 42

B26 Current and recommended measures o environmental resilience assets among elementary schoolstudents 43

B27 Current and recommended measurement o internal resilience assets among elementary schoolstudents 43

C1 Secondary school environmental resilience assets, total analytic sample, goodness-o - t in ormation orexploratory actor analysis models 44

C2 Secondary school environmental resilience asset, total analytic sample, goodness-o - t in ormation orcon rmatory actor analysis models 46

C3 Measurement intercept di erences or environmental resilience assets, secondary school sample 47

C4 Elementary school environmental resilience assets, total analytic sample, goodness-o - t in ormation orexploratory actor analysis models 48

C5 Elementary school environmental resilience asset, total analytic sample, goodness-o - t in ormation orcon rmatory actor analysis models 49

C6 Gender measurement intercept di erences or environmental resilience assets, elementary school sample 49

C7 Secondary school internal resilience assets, total analytic sample, goodness-o - t in ormation or exploratoryactor analysis models 50

C8 Secondary school internal assets, total analytic sample, goodness-o - t in ormation or con rmatory actor

analysis models 51C9 Measurement intercept di erences or internal resilience assets, secondary school sample 51

C10 Elementary school internal resilience assets, total analytic sample, goodness-o - t in ormation orcon rmatory actor analysis models 52


9/175

Why thiS Study? 1

Th s ts mm z s

s m st ths h m t

t s th s

thm t

m , km t

th H thK s S . Th

st ms tm sss ssm t s h s th t

t s sh th s h

m tt m t

ms .

WHy THiS STudy?

As improvements to curriculum and instructionraise academic standards, researchers are look-ing more and more at what actors account or

the varied in uence o these improvements. Mosthave ocused on risk actors or academic ailure,such as poverty or racial and cultural minoritystatus. But researchers are beginning to look at theother side o riskresilienceand have identi-ed several traits common to resilient youth thatenable the youth to overcome barriers to academicsuccess. Tere is little research, however, on howto measure these traits within the general studentpopulation and how to determine the role o theschool environment in promoting these traits.

Te Healthy Kids Survey (HKS) is one o the ewlarge-scale surveys to assess both risk and resil-ience. Te surveys resilience and youth develop-ment module (RYDM) is based on the premisethat youth who experience high levels o environ-mental assets in three areashigh expectationsrom adults, caring relationships with adults, andopportunities or meaning ul participationwill develop the resilience traits, the connectionto school, and motivation to learn that lead topositive academic, social, and health outcomes(Constantine, Benard, & Diaz, 1999).

Te resilience and youth development modulewhich has both elementary and secondary schoolversionswas designed as an epidemiologicalsurveillance tool to track aggregate levels o pro-tective actors. In Cali ornia an average o about600,000 students take the Healthy Kids Surveyand a part o the resilience and youth developmentmodule every year. School districts and schoolsuse the resulting prevalence and trend data toguide programmatic decisionmaking. With suchwidespread administration, school districts andindependent evaluators are increasingly using thesurvey data to evaluate local programs by examin-ing student-level changes over time. Capitalizingon the mandated administration o a standardinstrument or local evaluation has the bene to reducing the survey burden or students and


10/175

2 meaSuring reSilience and youth development: the pSychometric propertieS o the healthy KidS Survey

provides comparable outcome data across di erentprogram evaluations.

Widespread use o the module or research andlocal evaluation may be premature, however.

Te psychometric properties o speci c scalesassessed by the elementary school module haveyet to be established. And the secondary schoolmodule has not been validated since 2000, whenthe instrument was rst tested in the eld. Teinstrument has since been modi ed several times,making validation o the current secondary schoolresilience and youth development module neces-sary. In addition, measurement equivalence acrossracial and ethnic groups, males and emales, anddi erent grades has never been systematically

examined. Te stakes are thus high to ensure thatall parts o the module are valid and reliable.

o guide urther improvements o this importantassessment tool, Regional Educational Labora-tory West conducted psychometric analyses o theproperties o the resilience and youth developmentmodule, using a large set o recent survey data.1 Tis report describes the results o these analyses,makes recommendations on the proper use o themodule, and suggests modi cations to improvethe instrument.

For the secondary school module, the resultsare consistent with the instruments current useas an epidemiological tool and with its concep-tual oundation. It provides comprehensive andbalanced coverage o eight environmental resil-ience assets and our internal resilience assets;2 its subscales exhibit good internal consistencyand are associated with student risk actors inexpected ways. And i certain items are dropped,

the module also demonstratesmeasurement equivalence acrossracial/ethnic groups, males andemales, and grades. Te second-ary school RYDM scales exhibitlow test-retest reliability, however,which suggests that the moduleis not well suited or examiningstudent-level changes over time.

Te instrument was not designed to examineindividual di erences across students and shouldnot be used this way. Moreover, two o the sixinternal assets that the secondary school modulewas designed to measurecooperation and goals/

aspirationscould not be assessed validly. Severalmeasures would bene t i additional items wereincluded in derived scales to increase domaincoverage.

Te elementary school module was designed toassess seven environmental resilience assets andthree internal resilience assets, but it can reli-ably assess only two environmental assets andone internal asset. Most o the scales measuredby the elementary school instrument have poor

psychometric properties. Te elementary schoolinstrument should thus be modi ed considerablyto make it suitable or research.

developing a riSK and reSilienceaSSeSSMenT Tool

Te Healthy Kids Survey is a comprehensive healthrisk and resilience data collection system thatrelies on student sel -reporting. Te surveys coremodule tracks health risks and problem behaviorsthat are signi cant barriers to learning amongstudents. Te resilience and youth developmentmodule assesses individual and environmentalassets associated with positive youth developmentand school success. Tis section provides a brie background on how the survey and the resilienceand youth development module were developedand are now used in Cali ornia.

The Healthy Kids Surveyassessingrisk and protective factors

Te Healthy Kids Survey is the largest e ort in thenation to require school districts to assess studentresilience and risk behaviors (box 1). Te Cali or-nia Department o Education requires all schooldistricts with ederal itle IV unding or with stateobacco Use Prevention and Education grants toadminister the survey every two yearsthe case

Th s thm t m

s ss s m t ss ts

s t w th s tth m t s h s ss


11/175

developing a riSK and reSilience aSSeSSment tool 3

or 85 percent o Cali ornia school districts. Inmandating the survey, the Cali ornia Departmento Education aims to promote accountability anddata-driven decisionmaking and to improve healthand prevention programs in schools.

Te survey was developed in 1997 by WestEdsHealth and Human Development Program in col-laboration with Duerr Evaluation Resources andan advisory committee o researchers, teachers,prevention and health program practitioners,and public agency representatives. Te Cali orniaDepartment o Education unded the develop-ment o the survey in response to ederal require-ments that schools implement the Principles o E ectivenessto collect and use data to assessstudent needs, justi y program unding, guideprogram development, and monitor progress inachieving program goals. Te immediate impetusor mandating the biennial administration o the

survey, however, was meeting the requirements o the No Child Lef Behind Act ( itle IVSa e andDrug-Free Schools and Communities Act).

Te Healthy Kids Survey consists o a general coremodule, the resilience and youth developmentmodule, and our optional modules on speci crisk behaviors. It can be customized to meet localneeds:

Te required core module assesses demo-

graphic in ormation and health risks relat-ing to school violence, harassment, physicalhealth, mental health, school-related behavior(such as truancy), and alcohol, tobacco, andother drug use.

Te resilience and youth development moduleassesses environmental actors (environmentalassets) and individual traits (internal assets)

Box 1

Specifcations o the Healthy Kids Survey

MandateMandated (since all 2003) by theCali ornia Department o Educa-tion or compliance with No ChildLef Behind and state obacco UsePrevention and Education ( UPE)grants

Survey typeComprehensive health risk andresilience surveyStudent sel -reportAnonymous, voluntary,con dentialModular secondary schoolinstrument; single elementaryschool version

Grade levelsGrades 5, 7, 9, 11, and students incontinuation schools

SamplingRepresentative district sample;school-level surveys optional

Required modules(secondary school)

Core (required)A.Resilience and youth develop-B.ment (school and communityasset scales required)

Optional modules(secondary school)

Resilience and youth develop-B.ment (home, peer, and internalasset scales)Sa ety (violence and suicide)C.and alcohol and other druguseobaccoD.Physical healthE.Sexual behavior (pregnancy andF.HIV/AIDS risk)Custom module ( or addingG.questions)

SourcesItems based on the Cali ornia StudentSurvey, Youth Risk Behavior Survey,and Cali ornia Student obacco Useand Evaluation Survey

RequirementsBiennial administrationModule A and school & commu-nity asset scales in module BModule D by state UPE granteesWritten parental consent; passiveconsent optional since all 2004Representative district samples

AdministrationBy school, ollowing detailedinstructionsProcessing and reporting byWestEds Health & Human De-velopment Program

ProductLocal reports and aggregated statedatabase


12/175


associated with academic per ormance, posi-tive youth development, and protection romrisky behaviors. Te Cali ornia Department o Education mandates that the sections on schooland community assets be administered to all

students who take the Healthy Kids Survey.

Four optional, topical modules (and onecustomizable module) collect urther detailon subjects covered by the core module, suchas violence and alcohol and other drug use(module C); tobacco use and tobacco educa-tion (module D); physical activity and diet(module E); and sexual behavior, pregnancy,and HIV risk (module F).

A custom module that allows schools to incor-porate their own items.

Te survey was designed as a district surveillancetool to provide prevalence estimates representativeo students in the school districts that administer thesurvey rather than o students in the state as a whole.It was not designed to evaluate student-level changesover time or individual di erences across students.Te Cali ornia Department o Education requiresthat districts administer the survey to 900 randomlyselected students rom each targeted grade (5, 7, 9,and 11). In districts with ewer than 900 students pergrade (the case or 85 percent o Cali ornia districts),all students in the targeted grades are surveyed. I a district has more than 10 schools per grade, atleast 50 percent o schools are randomly sampled.(Los Angeles Uni ed School District has di erentrequirements because o its size.)

WestEds Health and HumanDevelopment Program providesschool districts administeringthe survey with technical as-sistance and with a report on thedistrict-level data collected in eachmodule.

Although several adolescentbehavior surveys, such as theYouth Risk Behavior Surveillance

System, assess student risk actors and problembehavior, the Healthy Kids Surveys assessment o student supports, strengths, and competencies setsit apart. While some surveys incorporate protec-tive actors, the resilience and youth development

module is one o the ew assessments that speci -cally addresses this dimension and does so with astrong theoretical oundation.

The resilience and youth development moduleassessing the other side of risk

Secondary school module. In early 1998 the HKSAdvisory Committee asked WestEd to develop asurvey module to assess middle and high schoolstudent strengths, competencies, and positive so-

cial and health attitudes, eeling that the HKS coremodule did not give practitioners enough in orma-tion about the actors behind positive developmentand school success (Constantine et al., 1999).

WestEd ormed a Resilience Assessment ExpertPanel to develop and validate a new survey moduleon youth resilience. Te assessment needed to bebrie enough to be widely administered along withthe HKS core module; have a strong theoreticaloundation; demonstrate reliability, validity, andcultural and developmental appropriateness whenadministered in Cali ornia school settings; andprovide a comprehensive, research-based assess-ment o environmental actors (environmentalassets) and resilience traits (internal assets).Environmental assets re er to meaning ul and pro-social bonding to community, school, amily, andpeers. Internal assets are personal resilience traits,such as sel -e cacy and problem-solving skills(Benard, 1991, 1995, 2004).

Failing to nd a survey that met its theoreti-cal and psychometric criteria, the panel builton research on resilience and healthy humandevelopment systemsparticularly the work o Benard (1991, 1995, 2004)to develop a theoreti-cal ramework that describes resilience actorsand their interrelationships ( gure 1). Te result-ing module or secondary school students wasdesigned to measure 11 environmental assets,

t sh t m t ts th t s h m tt , th ts h t

th t m w k h t s s st s tht t sh s


13/175


asking students their perception o adult highexpectations, their perceptions o caring rela-tionships with adults, and their opportunitiesor meaning ul participation in school, home,and community environments. Te module alsoassesses caring relationships and high expecta-tions in the peer domain. Tese external sup-ports promote positive outcomes, discouragingrisky behavior and stimulating academic success(Benard, 2004; Constantine et a l., 1999; Hawkins,Catalano, & Miller, 1992; Masten & Coatsworth,1998; Resnick et al., 2000; Rutter, 1987; Werner &Smith, 1982, 1992).

Internal resilience assetsthe personal strengthso a resilient childinclude social competence,problem solving, autonomy, and sense o pur-pose, which can each be broken down urther(Benard, 1991, 2004). Socia l competence, or ex-ample, entails social communication skills, em-pathy and caring, and the ability to elicit positiveresponses rom others (responsiveness) (Benard,2004; Masten, 2001). Problem solving involvesplanning, exibility, and resource ulness;

autonomy entai ls sel -e cacy, sel -awareness,and mind ulness; and sense o purpose in-cludes goal direction, achievement motivation,optimism, and hope (Benard, 2004). Internalresilience assets develop both naturally and inresponse to environmental resilience assets. Teresilience and youth development module wasdesigned to measure six internal assets: empathy,problem solving, sel -e cacy, sel -awareness,cooperation and communication, and goals andaspirations.

A pool o 128 potential items was piloted in onemiddle and one high school in all 1998. Re-searchers, classroom teachers, and other schoolpractitioners helped select and modi y itemsrom the pool and revise the ormat and instruc-tions. Te rst eld test o the resilience andyouth development module, with 92 resilienceitems, was administered to 1,000 high schoolstudents in three school districts in winter 1999.Cognitive processing interviews with studentswere also conducted to nd out students inter-pretation o the items. Based on analysis o the

Improved

health,

social, and

academic

outcomes

Internal resilience assets

Environmental resilience assets Youth needs

School

Home

Community

Peers

igure 1c t m th s th m t m


14/175


taBle 1it ms th s s h s th m t m st t, 2006/07

c s i d sEnvironmental resilience assets

S h ss ts

c s s sSchlCare r6

r8r10

a s , s s w . . . s b .s w i .

s s w i s s .

h s sSchlHigh r7

r9r11

a s , s s w . . .s w i j b.

w s w s b s .b s i w b s ss.

m sSchlPart r12

r13r14

a s . . .i s s.i s k ss s s.i s k .

H m ss tsc s s

HomeCare r49r51r53

i , s s . . .w s s s w k.w ks w b b s.w s s w i s s .

h s HomeHigh r48

r50r52

i , s s . . .w s w s.w b s i w b s ss.w w s w s b s .

m HomePart r54

r55r56

a . . .i s s w s s.i s k .i k s s w .

c mm t ss tsc s s

ComCare r15r17r20

o s s , s . . .w s b .w s w i s b s .w i s .

h s ComHigh r16

r18r19

o s s , s . . .w s w i j b.w b s i w b s ss.w w s w s b s .

m ComPart r21

r22r23

o s s , i s s. . .i bs, s s s, / , s.i s , , , s s bb .i .

p ss ts

c s s w sPeerCare r42

r43r44

i b w . . .w s b .w ks w b b s.w s w i .

p -s sPeerHigh r45

r46r47

m s. . . b . w s . w s .


15/175


cognitive interview data, requency distributions,and estimated Cronbachs alpha coe cients, thenumber o resilience items was reduced rom92 to 51 (table 1). In 2001 the resilience instru-ment was modi ed again, based on the results o grade-, gender-, and race/ethnic-speci c explor-atory actor analyses o data collected duringthe 1999/2000 academic year. Te constructedresilience scales based on the 1999/2000 eld testdata orm the basis o the current RYDM reportsprovided to school districts, even though themodule has since been modi ed urther.

Since 2003 all districts administering the HealthyKids Survey must also administer the school andcommunity asset parts o the module.3 Tirty- vepercent o districts choose to administer the ullresilience and youth development module, re ect-ing widespread interest in assessing resilience.WestEd provides districts with the data or each

scale and a report on the meaning and use o thedataand on how schools can create supportivelearning environments that promote school con-nectedness and achievement. WestEd also pro-vides state-level data to researchers and evaluatorswho apply or it.4

Elementary school module.Pools o resilienceitems were not independently developed or theelementary school module. Tey were selectedrom the secondary school module afer ocusgroups with elementary school students. Initially,the elementary school module used the sameconstructs as the secondary school module, butwith two items per construct instead o three.Analysis o the 1999 eld test data and cognitiveprocessing interviews with students suggesteditem deletions and changes in item wordings andresponse options. Te nal version has 21 items(table 2).

c s i d sInternal resilience assets

c Coop r31

r36

r37

h w s s s b s ?i w k w s w s s .i j w k w s s .

i s s w s w .S -

Sel Ef r29r30r32

h w s s s b s ?i w k b s.i s s i . t s i w .

eEmpathy r33

r34r38

h w s s s b s ?i b w s s s .i s w .i s w k.

p b -sProbSolv r35

r27

r28

h w s s s b s ?W i i f s k w .i k w w w b .

i w k b s b k w b .S - w ss

Sel Aware r39r40r41

h w s s s b s ? t s s .i s s s.i s w i w i .

g s s sGoals r24

r25r26

h w s s s b s ?i s s .i s .i s s s .

Note: Possible responses include (1) not at all true, (2) a little true, (3) pretty much true, (4) very much true.


16/175


evaluaTing THe pSycHoMeTricproperTieS o THe reSilience andyouTH developMenT Module

o better understand and improve the psychomet-ric properties o the resilience and youth develop-ment module, this report analyzes local HKS dataprocessed between 1998 and spring 2005, askingthe ollowing questions:

How should school districts and local evalu-ators best use the module? Should the instru-ment be used exclusively to assess prevalenceo environmental and internal assets orshould it also be used to assess student-levelchanges across time?

What are the psychometric properties o speci c scales assessed by the secondary andelementary school resilience and youth devel-opment modules (including the dimensional-ity o scales, scale reliability, and constructvalidity)?

Does the module exhibit measurementequivalence across racial and ethnic groups?In other words, is it culturally appropriateor di erent racial and ethnic groups? Does itexhibit measurement equivalence or malesand emales? Across di erent grades?

What modi cations should be made to im-prove the module?

taBle 2e m t s h s th m t m t ms st t, 2006/07

c s i d sEnvironmental resilience assets

S h ss ts

c s s sSchlCare

1013

d s w - s s b ?d s w - s s s w s s ?

h s sSchlHigh

1114

d s w - s s w j b?d s w - s s b j b?

m sSchlPart

915

d k ss s s s s ?d s b s ?

H m ss ts

c s s HomeCare

5255

d s s w - b s w k?d s s w - s w s s ?

h s HomeHigh

5354

d s s w - b j b?d s s w - w b s ?

m HomePart

5657

d ?d k s s s ?

p ss ts

h s w sPeerHigh

5051

d b s s b ?d b s s ?


eEmpathy

3738

d s w ?d b w s s s ?

p b -sProbSolv

3940

d k w w w b ?d w k b s b k w b ?

g s s s

Goals

41

4216

d b s ?

d s s ?d s s s ?

Note: Possible responses include (1) no, never, (2) yes, some of the time, (3) yes, most of the time, (4) yes, all of the time.


17/175

evaluating the pSychometric propertieS o the reSilience and youth development module 9

Box 2

Data and analytic strategies

Te authors used the ollowing dataand analytic strategies to analyzethe psychometric properties o thesecondary and elementary schoolresilience and youth developmentmodules.

Datawo mutually exclusive analyticsamplesa main sample and avalidation samplewere drawnrom an aggregate data le thatincluded all HKS data processedbetween the spring 2003 and thespring 2005 administrations o theHealthy Kids Survey. For the second-ary school analysis, separate sampleswere drawn or each grade (7, 9, and11), gender, and ethnicity (ChineseAmerican, A rican American, Mexi-can American, and white EuropeanAmerican)with 500 respondentsrandomly sampled per cell (12,000total). Equal numbers were used oreach gender and ethnic group so thatmodels that do not adjust or genderand/or ethnicity would not be a -ected by gender/ethnic di erences inthe sample.

For the elementary school analysis,random samples o 1,000 males and1,000 emales (2,000 total) weredrawn rom the aggregated HKS datale. Tus, or the elementary schoolresilience and youth developmentmodule, only gender di erences inmeasurement structure were exam-ined. Respondents with missing dataon more than hal the resilience itemswere excluded rom the analysis. Forestimating models with missing data,

maximum likelihood estimation withmissing at random (MAR) assump-tions were used, which assumes thatvalues are missing at random con-ditional on the other observed itemsin the data (Little & Rubin, 2002;Muthn & Muthn, 2006).

Statewide data was supplementedwith two sets o HKS data originallycollected or local evaluation. Datacollected in 2006 rom a large urbanschool district in Southern Cali orniawere used to describe the temporalstability o the derived scales (test-retest reliability). Te elementaryschool Healthy Kids Survey and thesecondary school core module and re-silience and youth development mod-ule were administered two times intwo weeks to 132 fh-grade studentsand 90 ninth-grade students. Datacollected in 2004/05 rom students ina large county in Southern Cali orniawere used to examine the relation-ship between the RYDM constructsand standardized test scores.

Exploratory and confrmatory actor analysesAnalyses were conducted to test em-pirically whether the actor structureo the resilience instrument is con-sistent with current usage and withits underlying conceptual model. Foreach sample and subsample (grade,gender, ethnicity), the measurementstructure o the resilience instrumentwas established by tting a series o exploratory and con rmatory actoranalysis models. Exploratory actoranalysis (EFA) models were estimatedto determine roughly the number o actors underlying the data and themeasurement structure o the latent

actors. A combination o criteria wasused to determine the number o ac-tors to retain in the EFAs, includingt indices, scree plots, the number o eigenvalues greater than 1, concep-tual clarity, and simplicity. Modelswith the ewest possible actors andmodels with no cross-loadings wereavored over more complex models.

Te results o the exploratory actoranalysis models were then used as astarting point or a series o nestedcon rmatory actor analysis (CFA)models. Measures o model t, cor-relations among the latent constructs( actors), and actor-loading patternswere used to make decisions aboutmodels. Tis process was replicatedor each grade, gender, and ethnicgroup, and or the main sample andthe validation sample.

o derive estimates or the EFA andCFA models, Muthn and Muthns(2006) Mplusstatistical modelingprogram was used. Because all theitems used to measure resilienceassets are ordinal, Muthns (1984)approach to exploratory and con r-matory actor analysis with ordinalindicators was used.

Confrmatory actor analysismodels with covariatesMeasurement equivalence across de-mographic subgroups was examinedby estimating con rmatory actoranalysis models with covariates.MIMIC modelingmultiple indica-tor, multiple cause structural equa-tion modelswas used to test ordi erential item unctioning acrossschool grade, gender, and ethnic-ity. An applied strategy was used to

(continued)


18/175


Tis report nds that both the secondary schooland elementary school modules are used pri-marily to report aggregate data on prevalenceand district-level changes across time. Althoughseveral modi cations should be made, the RYDMscales are generally consistent with current use o the instruments and with the conceptual ounda-tion o the module. (See box 2 and appendixes Aand B or a discussion o the analytic strategy andthe results o the analysis.)

Results of the analysis of the secondary school module

Te secondary school module is a short instru-ment (51 items) suitable or widespread adminis-tration. It provides comprehensive and balanced

coverage o both environmental (eight dimen-sions) and internal ( our dimensions) resilienceassets.5 Its subscales exhibit good internal consis-tency and are associated with student risk actorsin expected ways. I certain items are dropped,the module also demonstrates measurementequivalence across racial/ethnic groups, males andemales, and grades.

Te secondary school instrument is appropriateas an epidemiological tool, but is not well suitedor evaluating student-level changes over time orindividual di erences across students. Te instru-ment exhibits low test-retest reliability, suggestingthat the RYDM constructs are temporally speci c.Estimates o student-level changes across time are

ascertain whether group di erencesin measurement intercepts have

implications or evaluation research.Recommendations or item changesare made only when the measure-ment intercepts are substantively di -erent across groups ( 0.20 standarddeviations) in both the main sampleand the validation sample.

Additional reliability and validity analysesInternal consistency estimates o

reliability o the derived scales werecalculated using Cronbachs alpha oreach grade, gender, and ethnic groupin both the main sample and thevalidation sample. Nunnalys (1978)criterion o 0.70 was used as the cuto or determining acceptable internalconsistency reliability or the second-ary school survey. Because o the no-toriously low internal consistency evi-dent in surveys o elementary school

students, this criterion was relaxedslightly to 0.60 or the elementary

school module. o examine test-retestreliability, RYDM survey data col-

lected rom a small sample o fh andninth graders who took the resilienceand youth development module twicein two weeks was used.

Di erences in resilience scale scoresacross the demographic subgroupswere also examined. o make demo-graphic di erences in the resiliencescales more interpretable, e ect sizeswere calculated to represent the mag-

nitude o such di erences (Cohen,1988). With two groups (male/ emale),the di erence in scale means betweeneach group was divided by the pooledstandard deviation (Cohensd ). Tusthe standardized di erence representsthe di erence between each group instandard deviation units. With morethan two groups (race/ethnicity),the standardized di erences wererepresented by multiplying Cohens

by 2which is roughly equivalent tothe standardized di erence calculated

or two groups when the numbero observations in each cell is equal

(Cohen, 1988).

Construct validity was assessed byexamining the relationship o thederived resilience scales to other theo-retically related constructsinclud-ing substance use, school violence,school-related behavior, and stan-dardized test scores. o examine theserelationships using a common metric,correlations between resilience con-

structs and criterion variables romcon rmatory actor analysis modelswere estimated using the main andvalidation samples. Latent constructsrepresent continuous variables, whilethe criterion variables are eitherdichotomous or ordinal. Tus, polyse-rial correlations are presented, whichrepresent the correlation between acontinuous variable and a dichoto-mous or ordinal variable that re ects

an underlying continuous variable(Bedrick & Breslin, 1996).

Box 2 (continued)

Data and analytic strategies


19/175

evaluating the pSychometric propertieS o the reSilience and youth development module 11

likely to be imprecise because o the instability o the resilience measures. Even with low student-level stability, however, the module is valuable ortracking school and district prevalence estimateso resilience assets. Student-level errors in mea-

surement likely cancel each other out when thedata are aggregated at the school, district, andstate levels.

Te secondary school module contains eight in-ternally consistent and valid measures o environ-mental resilience assets:

Tree measures representing supportive rela-tionships in the school, community, and homeenvironments. Tese supportive relationships

include both caring relationships with andhigh expectations messages rom adults. Onlythe measure or supportive relationships inthe home environment, however, demon-strates su cient test-retest reliability or usein research.

Tree measures o meaning ul participationor involvement in relevant, engaging, andinteresting activities with opportunities orresponsibility and contribution in school, inthe community, and at home.

wo measures o environmental assets in thecontext o peerscaring relationships andhigh expectations (a liation with pro-socialpeers).

Tat the scales or caring relationships and highexpectations in the school environment turn outto measure the same actor is consistent withknowledge that has emerged since the resilienceand youth development module was developed inthe late 1990s. In ocus groups conducted by HKSsta , when students were asked what they considerto be actions that re ect that a teacher cares aboutyou, they most ofen mentioned that the adultis a good listener, sets high standards, expectsresponsibility rom the student, praises successes,and encourages the student through setbacks.Akey (2006) ound that supportive teachers and

clear, high expectationsor behavior are key todeveloping both stu-dent engagement andperceived competence.

eachers whom studentssee as supportive andwho set clear expecta-tions or behavior createan atmosphere wherestudents eel in controland con dent about their ability to succeed inschool. Akeys ndings suggest that supportiveteacher relationships, high academic expectations,and high-quality pedagogy combine to enhancestudent engagement and academic competence,

which lead to higher achievement, consistent withthe RYDM conceptual ramework. Te school andhome supportive relationships measures, however,exhibit better psychometric qualities than manyother instruments designed to measure similarconstructs.

Scores on our o the internal asset scalessel -e cacy, empathy, problem solving, and sel - awarenessare internally consistent and adequateor general research purposes. But the RYDMitems designed to measure cooperation and goals/aspirations do not, however, provide valid assess-ments o these constructs.

Although the consistency o the associations o environmental and internal resilience assets toother related constructssuch as substance use,school violence, school-related behavior, and stan-dardized test scoressuggests that the measuresdemonstrate construct validity, the associationsare weak. Tus the constructs exhibit only moder-ate construct validity.

Results of the analysis of the elementary school module

Te elementary school resilience and youth devel-opment module uses 21 items to assess seven en-vironmental assets and three internal assets. Reli-ably assessing so many actors with so ew itemsis di cult, however, especially with a student

Th ss h st m ts t s

m t ,t s t w s t

t st t-h s t m

f sss st ts


20/175


sel -report instrument. Notsurprisingly, the module reliablyassesses only two environmentalasset measures and one internalasset measure, leaving consider-

able room or improvement.

Te elementary school modulemeasures meaning ul participa-

tion, pro-social peers, and supportive relationshipsin the school and home environments, but only theschool supportive relationships and home support-ive relationships scales exhibit su ciently high in-ternal consistency or urther use. Only one reliableinternal resilience asset measure was detected orelementary school studentsempathy. Te second

actor detected, goals/aspirations, was not reliableenough to be recommended or urther use. Tethird actor, problem solving, was not identi ed.

recoMMendaTionS

Tis report recommends that neither the second-ary school nor the elementary school resilienceand youth development module be used to evaluatestudent-level changes over time or individual di -erences across students. Estimates o student-levelchanges across time are likely to be imprecise be-cause o the instability o the resilience measures.Other, longer, companion instruments should bedeveloped to assess student-level changes. Teresilience and youth development module is stilluse ul as an epidemiological surveillance tool orreporting aggregate district-level data, however.

Te ollowing sections provide recommendationsto drop or revise speci c items in the module.ables 3, 4, and 5 present the recommended mea-sures. (See appendix tables B24, B25, B26, and B27or a side-by-side comparison o the current andrecommended measures.)

Secondary school environmental resilience assets

Recommendation 1Combine the caring relation-ships and high expectations items.o maximize

construct validity and reduce redundancy acrossscales, the caring relationships and highexpectations items should be combined to ormone scale representing supportive relationships.Caring relationships and high expectations are

indistinguishable as currently measured by themodule. Te new supportive relationships scaleshould continue to be assessed separately orschool, community, and home environments.

Recommendation 2Drop Item R23 (I help other people). Tis item should not be used to indicatecommunity meaning ul participation because theitem unctions di erently, and thus has a di er-ent meaning, or emales and Mexican Americanyouth. A new item that taps involvement in activi-

ties in the community should be developed.

Recommendation 3Drop Item R54 (I do unthings or go un places with my parents or other adults).Te item is not developmentally appro-priate or older adolescents because 11th gradersreport substantially lower participation in suchactivities or a given level o home meaning ulparticipation. Tis item distorts developmentaltrends on the home meaning ul participation scaleand should be dropped. A di erent item should bedeveloped to replace it.

Recommendation 4Drop item R45 (My riends get into a lot o trouble).Because it is a biased in-dicator o pro-social peers or emales and ChineseAmerican students, an alternative item should bedeveloped to measure this construct.

Secondary school internal resilience assets

Recommendation 5Drop the cooperation/com-munication construct.wo o the items used tomeasure cooperation/communication measuremore than one construct: Items R36 (I enjoyworking together with other students my age)and R37 (I stand up or mysel without put-ting others down). Item R31 (I can work withsomeone who has di erent opinions than mine)should be moved to the sel -e cacy scale. Temeasurement models suggest that this item

Th m t s hm ss ss s

tw m ts t m s s t ss t m s ,

sm m m t


21/175

recommendationS 13

measures sel -e cacy better than it does coopera-tion and communication.

Recommendation 6Drop the goals and aspira-tions construct.wo o the three items used tomeasure this constructR24 (Goals and plansor the uture) and R26 (I plan to go to college orsome other school afer high school) unctiondi erently across racial/ethnic groups.

Recommendation 7Drop item R27 (I knowwhere to go or help with a problem).As an

indicator o problem solving, this item should bedropped because it unctions di erently or malesand emales. An alternative item should be devel-oped to assess problem solving.

Elementary school environmental and internal assets

Recommendation 8Develop more elementaryresilience items.Te elementary school resilienceand youth development module tries to assess toomany actors with too ew items. Because havingan elementary school resilience assessment that

taBle 3r mm m s s m t s ss ts m s s h st ts

c s i

S s

a w s b .a w s w i .

a w s s w i s . . .a w s w i j b.a w w s w s b s .a w b s i w b s ss.

S i s s.i s k ss s s.i s k .

c s

a w s b .a w s w i s b . . .a w i s .a w s w i j b.

a w b s i w b s ss.a w w s w s b s .

c i bs, s s s, / , . . .i k ss s s , , . . .

h s

a w s s s w k.a w ks w b b s.a w s s w i s . . .a w s w s.a w b s i w b s ss.a w w s w s b s .

h i s k .

i k s s w .

p s sa w s b .a w ks w b b s.a w s w i .

p -s sm s w s .m s w s .


22/175


is aligned with the secondary school module isimportant, additional resilience items should bedeveloped or the elementary school survey. Eacho the elementary school RYDM scales demon-strates inadequate domain coverage and marginalinternal consistency, at least one additional itemshould be developed or each o the school sup-portive relationships, home supportive relation-ships, and empathy subscales. wo additionalitems should be developed or the meaning ulparticipation at school and at home scales i it isretained in the survey.

Recommendation 9Combine the caring rela-tionships and high expectations items. As withthe secondary school module, the caring relation-ships and high expectations items should becombined to orm one scale representing support-ive relationships in both the school environmentand the home environment.

Recommendation 10Drop meaning ul participa-tion. Te meaning ul participation scale shouldeither be dropped or redeveloped because o low

internal consistency. Moreover, item R15 (Doyou do things to be help ul at school?) shouldnot be used to indicate meaning ul participationbecause the item unctions di erently or malesand emales.

Recommendation 11Drop pro-social peers. Tepro-social peers scale should be dropped becauseone o the two items used to measure it unctionsdi erently or males and emales. Perhaps itemsrom other instruments that assess this constructshould be used instead.

Recommendation 12Drop goals and aspirations. Te goals and aspirations scale should be droppedor modi ed because o its low internal consistency.

Recommendation 13Develop a sel -efcacymeasure.Items should be developed to assesssel -e cacy because this important construct iscurrently not assessed.

taBle 4r mm m s s t sss ts m s s h st ts

c s i

S -

i w k w s w s

s .i w k b s.

i s s i .

t s i w .

e

i b w s s s .

i s w .

i s w k.

p b s

W i i f s k w .

i w k b s b k w b .

S - w ss

t s s .

i s s s.

i s w i w i .

taBle 5r mm m s s m t

t s ss ts mm t s h st ts

c s i

Environmental resilience assets

S s

d s . . . s b ?

t s . . . s w . . . s s ?

t s . . . w j b?

t s . . . b j b?

h s

p . . . b s w k?

p . . . s w s s ?

p . . . b j b?

p . . . w b s ?


e

d s w ?

d b w s s s ?


23/175

Appendix A 15

Appendix AAnAlytic strAtegy

o describe the psychometric properties o thesecondary and elementary school resilience and

youth development modules, this report examines

Te dimensionality o scales by using explor-atory and con rmatory actor analysis models.

Measurement equivalence across demo-graphic subgroups by estimating con rma-tory actor analysis models with covariates(such as multiple indicator, multiple causestructural equation models).

Scale reliability by estimating internal consis-tency and test-retest reliability coe cients.

Construct validity by examining the relation-ship o scales to other theoretically relatedconstructs and mean di erences across demo-graphic subgroups.

Data

Statewide data rom the local administration o theHealthy Kids Survey. Te data or the analyses inthis report are rom local administration o theHealthy Kids Survey (HKS) in elementary, middle,and high schools. Tese data were drawn rom adatabase o all local HKS data processed between1998 and spring 2005 by WestEds Health andHuman Development Program (approximately2.1 million observations). Analyzing such a largesample size would, however, make almost everyparameter estimate statistically signi cant, wouldin ate chi-square values o model t, and wouldmake assessing substantive signi cance moredi cult. Tus, two mutually exclusive analyticsamples were used in the analyses: a main sampleand a validation sample. Te samples were drawnrom the aggregate data le that included all HKSdata processed between the spring 2003 and thespring 2005 administrations o the Healthy KidsSurvey. For the secondary school analysis, separatesamples were drawn or each grade (7, 9, and 11),

gender, and ethnicity (Chinese American, A ricanAmerican, Mexican American, and white Euro-pean American)with 500 respondents randomlysampled per cell (12,000 total). Equal numberswere used or each gender and ethnic group so

that models that do not adjust or gender and/orethnicity would not be a ected by gender/ethnicdi erences in the sample.

Te elementary school Healthy Kids Survey is ad-ministered only to fh graders and does not askstudents about their ethnic/racial group. Randomsamples o 1,000 males and 1,000 emales (2,000total) were drawn rom the aggregated HKS datale. Tus, or the elementary school resilience andyouth development module, only gender di er-

ences in measurement structure were examined.Respondents with missing data on more thanhal the resilience items were excluded rom theanalysis. For estimating models with missing data,maximum likelihood estimation with missing atrandom (MAR) assumptions were used, whichassume that values are missing at random con-ditional on the other observed items in the data(Little & Rubin, 2002; Muthn & Muthn, 2006).(See section on missing data patterns.)

Te same procedures were used to draw thevalidation samples or both the secondary schooland elementary school samplesexcept thatrespondents included in the main sample were ex-cluded rom the validation sample. Te data wereweighted by grade, race/ethnicity, and gender torepresent the characteristics o HKS respondentssurveyed rom spring 2003 to spring 2005.

Local evaluation HKS data. Statewide data wassupplemented with two sets o HKS data originallycollected or local evaluation. Data collected in2006 rom a large urban school district in South-ern Cali ornia were used to describe the temporalstability o the derived scales (test-retest reliability).Te elementary school Healthy Kids Survey and thesecondary school core module and resilience andyouth development module were administered twotimes in two weeks to 132 fh-grade students and90 ninth-grade students. Data collected in 2004/05


24/175

16 MeAsuring resilience And youth developMent: the psychoMetric properties o the heAlthy Kids survey

rom students in a large county in Southern Cali-ornia were used to examine the relationship be-tween the RYDM constructs and standardized testscores. Standardized test score and school/com-munity asset data were available or 2,898 students,

while test score and home and internal asset datawere available or 651 students.6 English LanguageArts and Mathematics Cali ornia Standards estscale scores were used as criterion variables.

Missing data patterns.Approximately 0.5 percento respondents in the elementary and secondarymodules were excluded rom the sampling poolbecause o missing data on more than hal theresilience items (table A1). In the secondary schoolsamples, approximately 65 percent o respondents

provided answers to all the survey items in theresilience and youth development module; an ad-ditional 18 percent had missing values on one ortwo items; 8 percent had missing values on 3 to 10items; and 8 percent had missing values on 11 ormore items. Respondents with missing values on11 or more items had lower scores on about one-quarter o the secondary RYDM itemsscoringapproximately 912 percent o a standard devia-tion lower on these items. Tese results held orboth the main and validation samples. Di erencesin item means were diminished signi cantly afercontrolling or one or two o the remaining items,

suggesting that the missing at random assumptionis reasonable.

Approximately 81 percent o elementary studentsprovided valid answers to all the RYDM items

and 15 percent answered all but one or two items.Respondents with missing values on two or moreitems had lower scores on seven o the elementaryRYDM items (averaging 0.24 standard deviations).Tese di erences were no longer apparent afercontrolling or any two o the remaining items,again suggesting that maximum likelihood esti-mation with missing at random assumptions willyield unbiased parameter estimates.

Exploratory and con rmatory actor analyses

Analyses were conducted to test empiricallywhether the actor structure o the resilience in-strument is consistent with current usage and withits underlying conceptual model. For each sampleand subsample (grade, gender, ethnicity), the mea-surement structure o the resilience instrumentwas established by tting a series o exploratoryand con rmatory actor analysis models. Explor-atory actor analysis (EFA) models were estimatedto determine roughly the number o actors under-lying the data and the measurement structure o the latent actors. A combination o actors was

tAble A1M a a a a a m a am mh a h v m m

n m mm

s a e m a

Ma am va a am Ma am va a am

n m p

n m p

n m p

n m p

0 7,819 65.2 7,865 65.5 1,627 81.4 1,622 81.1

1 1,634 13.6 1,615 13.5 266 13.3 249 12.5

2 585 4.9 545 4.5 55 2.8 59 3.0

35 497 4.1 539 4.5 33 1.7 45 2.3

610 445 3.7 437 3.6 15 0.8 14 0.7

11 m 1,020 8.5 999 8.3 4 0.2 11 0.6

t a 12,000 100 12,000 100 2,000 100 2,000 100

Note: Analytic samples randomly drawn rom students surveyed between spring 2003 and spring 2005. Secondary school resilience and youth developmentmodule has 51 survey items. The elementary school module has 21 survey items.


25/175

Appendix A 17

used to determine the number o actors to retainin the EFAs, including t indices, scree plots, thenumber o eigenvalues greater than 1, conceptualclarity, and simplicity. Models with the ewestpossible actors and models with no cross-loadings

were avored over more complex models.

Te results o the exploratory actor analysis mod-els were then used as a starting point or a series o nested con rmatory actor analysis (CFA) models.Measures o model t, correlations among thelatent constructs ( actors), and actor-loading pat-terns were used to make decisions about models.Tis process was replicated or each grade, gender,and ethnic group, and or the main sample and thevalidation sample.

o derive estimates or the EFA and CFA models,Muthn and Muthns (2006) Mplusstatisticalmodeling program was used. Because all the itemsused to measure resilience assets are ordinal,Muthns (1984) approach to exploratory andcon rmatory actor analysis with ordinal indica-tors was used.

In the general actor analysis model, the relation-ship between the indicators ( y* ) and the under-lying constructs () can be represented by:

(A1) y* = + +

where is a vector o measurement intercepts, isa matrix o measurement slopes ( actor loadings),and is a matrix o residuals, assumed to be inde-pendent o and with zero expectation. Te modelimplies the ollowing covariance matrix o y* :

(A2) = +

where is the covariance matrix o and is thecovariance matrix o (see Long, 1983).

In general, the indicators y* are assumed to benormally distributed, latent continuous variables.A persons observed score on item ydepends onher/his position on y* . I the observed item is con-tinuous, y* is directly observed ( y= y* ). However,

i the observed item is dichotomous or ordinal, theobserved categorical variable ( y) is linked to thelatent continuous variable ( y* ) in a nonlinear waythrough a model o thresholds (see Muthn, 1984).Te relationships between an observed ordinal or

dichotomous item ywithc categories to y* can beexpressed as:

(A3) y= c, i c < y* c+1

or c = 0, 1, 2, . . . , c1. Te s represent thresholdparameters. Muthns (1987) approach modelsthe relationships among these more undamentallatent y* variables. With ordinal items, polychoriccorrelations represent the correlations o theunderlying continuous y* variables.

Te measurement model is estimated by mini-mizing the weighted least squares (WLS) ttingunction

(A4) WLS= ( s ) W 1 (s )

wheres is a matrix o sample statistics (probitthresholds and polychoric correlations), is amatrix o the population counterparts tos impliedby equation [A2], andW is the covariance matrixor the vector or sample statistics.7

Con rmatory actor analysis models with covariates

MIMIC modelingmultiple indicator, multiplecause structural equation modelswas usedto test or di erential item unctioning acrossschool grade, gender, and ethnicity. A simplegraphical example o this approach is presented ingure A1. Panel A shows a classic MIMIC modelthat assumes there are no emale/male di erencesin measurement intercepts. Te three arrowsconnecting school meaning ul participation toitems R12, R13, and R14 are actor loadings andrepresent the strength o the relationships betweenthe underlying constructs and the items used tomeasure them. Te arrows pointing rom right tolef toward the items (R12, R13, R14) are residualsand represent true measurement error and item-speci c variation. Finally, the arrow pointing rom


26/175


emale to school meaning ul participation indi-cates that the means o the underlying constructare allowed to be di erent or males and emales.Te actor loadings are not allowed to be di er-ent or males and emales, and there is no directe ect o emale on the individual items. Te modelassumes that the items unction identically ormales and emales in measuring school meaning-ul participation.8

Te measurement model in panel B allows oremale/male nonequivalence in the measurementintercept or item R14. Tat is, it allows or a directe ect o emale on R14 that is not dependent onthe underlying construct. Tis is indicated by thearrow going directly rom emale to R14. A sig-ni cant emale/male di erence in measurement

intercept indicates that the item unctions di -erently or emales and males in measuring theunderlying construct. For example, i the measure-ment intercept or R14 is 25 percent o a standarddeviation ( emale R14) lower or emales than

males, then or a given level o school meaning ulparticipation, emales score 25 percent o a standarddeviation lower on R14. In this example, a givenscore on item R14 does not mean the same thing ormales and emalesat least not with re erence tothe school meaning ul participation construct.

An applied strategy was used to ascertain whethergroup di erences in measurement intercepts haveimplications or evaluation research. Recommen-dations or item changes are made only when the

measurement intercepts are substantively di erentacross groups ( 0.20 standard deviations) in boththe main sample and the validation sample.

Fit indices

A mean- and variance-adjusted 2 test o model tis obtained by multiplying the minimum unc-tion by twice the total sample size and dividing bya scaling correction actor ( or more details, seeMuthn, 1984, 1987; Muthn & Muthn, 2006).Afer adjusting or the scaling correction ac-tor (see Satorra, 2000; Satorra & Bentler, 1999;Muthn & Muthn, 2006), the di erence in 2 testsor two nested models ollows a 2 distributionand can be used to test whether a model resultsin a statistically signi cant improvement in t.However, 2 di erence tests are sensitive to samplesize and can be in uenced by substantively mean-ingless parameter di erences in large samples.For this reason, the analysis also relied on severalother indices o model t.

For EFA models, root mean square residual(RMSR) and root mean square error o approxi-mation (RMSEA) values were used to assess modelt (Hu & Bentler, 1999). RMSR is the square rooto the mean o the squared residuals and indexesthe di erence between the sample variance/covari-ance matrix and the variance/covariance matrixpredicted by the model. Hu and Bentler (1999)

R12

R13

R14

Schoolmeaningful

participation

Panel A MIMIC modelingno measurement invariance

Female

R12

R13

R14

Schoolmeaningful

participation

Panel B MIMIC modelinghypothetical gender measurement intercept invariance (differential item

functioning for R14)

Female

igure A1H h a am MiMic a a h

m a m q va

Note: MIMIC re ers to multiple indicators multiple causes structuralequation models.


27/175

Appendix A 19

suggest that RMSR values less than 0.05 indicategood t. Te RMSEA is also based on di erencesbetween the observed and predicted variance/covariance matrices, but penalizes or model com-plexity. RMSEA is computed by:

(A5) RMSEA = 2

(n*df) 1// ( )nwhere 2 is the model chi-square value,n is thetotal sample size, andd is the degrees o ree-dom. RMSEA penalizes or model complexityby dividing 2 by (n*d ). Hu and Bentler (1999)recommend RMSEA values o 0.06 or less asthe cut-o or good model t. Based on Hu andBentlers recommendations, more emphasis is

placed on RMSEA than on RMSR in EFA modelselection.

In addition to RMSEA, several additional tindices were used to assess CFA models, includ-ing Bentlers comparative t index (CFI), theucker-Lewis index ( LI), and Muthn andMuthns (2006) weighted root mean squareresidual (WRMR). As implemented in Mplus, boththe CFI and LI compare estimated CFA modelsto baseline models with uncorrelated variables(independence model). CFI and LI are calculatedas ollows:

(A6) CFI =1max 2Hodf Ho, 0

max 2Hodf Ho, 2Bdf B, 0

(A7) TLI =

2Bdf B

2Hodf Ho

2Bdf B 1

where 2 and d Ho denote the chi-squared valueand degrees o reedom o the estimated modeland 2 and d B denote the same or the baselinemodel. Both CFI and LI are not appreciablyin uenced by sample size. By convention, CFI andLI values greater than 0.95 indicate good t (Hu& Bentler, 1999).

Yu and Muthn (2001) recently developed WRMRto identi y good- tting models with categoricaloutcomes. It is de ned as ollows:

(A8) WRMR =e

(s r r)v r

e

r

wheresr is an element in the sample variance/covariance (or probit threshold/polychoric cor-relation) matrix, r is the element in the variance/covariance matrix predicted by the model, r is an estimate o the variance o sr , and e is thenumber o elements in the variance/covariancematrix. According to Muthn, WRMR is suitableor models where sample statistics have widelyvarying variances, when sample statistics are on

di erent scales, and in models with categoricaloutcomes. Yu and Muthn (2001) suggest WRMRvalues less than or equal to 1.00 or good modelswith categorical outcomes. Because WRMR hasbeen tested or models with categorical outcomes,greater weight is placed on this index in CFAmodel selection.

Modi cation indices and 2 di erence testingwere also used to compare nested con rmatoryactor analyses models, particularly or testingmeasurement intercept invariance.

Additional reliability and validity analyses

Internal consistency estimates o reliability o thederived scales were calculated using Cronbachsalpha or each grade, gender, and ethnic group inboth the main sample and the validation sample.Nunnalys (1978) criterion o 0.70 was used as thecuto or determining acceptable internal consis-tency reliability or the secondary school survey.Because o the notoriously low internal consis-tency evident in surveys o elementary schoolstudents, this criterion was relaxed slightly to 0.60or the elementary school resilience and youthdevelopment module. o examine test-retest reli-ability, RYDM survey data collected rom a smallsample o fh and ninth graders who took theresilience and youth development module twice intwo weeks was used.


28/175


Di erences in resilience scale scores across thedemographic subgroups were also examined.o make demographic di erences in the resil-ience scales more interpretable, e ect sizes werecalculated to represent the magnitude o such

di erences (Cohen, 1988). With two groups (male/emale), the di erence in scale means betweeneach group was divided by the pooled standarddeviation (Cohensd ). Tus, the standardizeddi erence represents the di erence between eachgroup in standard deviation units. With morethan two groups (race/ethnicity), the standard-ized di erences were represented by multiplyingCohens by 2which is roughly equivalent to thestandardized di erence calculated or two groupswhen the number o observations in each cell is

equal (Cohen, 1988). Cohens was calculated by

(A9) f =

(mi m)2

kk

i=1

wheremi represents the mean or each subgroupi,m represents the population mean,k the numbero subgroups, and the pooled standard deviation.

Construct validity was assessed by examining

the relationship o the derived resilience scales toother theoretically related constructsincludingsubstance use, school violence, school-relatedbehavior, and standardized test scores. o exam-ine these relationships using a common metric,correlations between resilience constructs andcriterion variables rom con rmatory actor analy-sis models were estimated using the main andvalidation samples. Latent constructs representcontinuous variables, while the criterion variablesare either dichotomous or ordinal. Tus, poly-

serial correlations are presented, which representthe correlation between a continuous variable anda dichotomous or ordinal variable that re ectsan underlying continuous variable (Bedrick &Breslin, 1996).


29/175

Appendix b 21

Appendix Bresults

Tis appendix presents the results o the analysesconducted to evaluate the psychometric properties

o the resilience and youth development module.

Secondary school environmental resilience assets

Exploratory actor analysis results.EFA models wereestimated or each subpopulation and or the mainand validation samples to determine the numbero actors underlying the items. Te EFA modelssuggested that the environmental resilience assetsitems measure eight actors.9 Te actor pattern andloadings or the main sample and cross-validation

sample are displayed in tables B1 and B2, respec-tively. Te 8- actor EFA solutions show conceptu-ally clear actor-loading patterns that are mostlyconsistent with the underlying theory guiding thedevelopment o the instrument. Te pattern o ac-tor loadings across all the demographic subgroupsis consistent with those displayed in tables B1 andB2.10Distinct actors are apparent or support andmeaning ul participation in the school, community,and home environments, as well as caring and pro-social relationships in the peer environment.

However, the actor pattern evident in the 8- actorsolution is inconsistent with how the instrumentcurrently is being used in Cali ornia because theresults suggest that caring relationships and highexpectations at school, in the home, and in thecommunity arenot distinct actors.

Confrmatory actor analysis results.A CFA modelequivalent to the 8- actor EFA models in tablesB1 and B2 was estimatedexcept that all but thehighest magnitude loadings rom the EFA modelwere constrained to be zero.11 Tat is, each itemwas orced to load on only one actor. As with theEFA models, the results were consistent acrosseach sample. Te CFA models indicated that itemR45 (My riends get into a lot o trouble) has arelatively small actor loadingsuggesting that anassociation with peers who get into a lot o troubleis a less sensitive indicator o pro-social peers

than the other two items assessing this construct.Moreover, there was a relatively high correlationbetween home support and home meaning ulparticipation (0.78 and 0.79), which suggests thatthese two constructs may not be distinct.

Te CFA models were re-estimated to include covari-ates to detect di erences in measurement interceptsacross demographic subgroups. Several measure-ment intercepts di ered by demographic subgroup:

Te results or R23 (I help other people)suggest that or a given level o communitymeaning ul participation, emale and Mexi-can American youth report between one- fhand one-third o a standard deviation higher

or helping other people. Te item thus has adi erent meaning or these two populations.

For R54 (I do un things or go un placeswith my parents), 11th graders reportsubstantially lower levels o participation inun activities with parents or a given level o home meaning ul participation than do sev-enth and ninth graders (0.29 to 0.33 standarddeviations). Tis represents a developmentaldi erence in the appropriateness o this item.

Female and Chinese American youth reportlower requencies on R45 (My riends get intoa lot o trouble) or a given level o pro-socialpeersre ecting the di erent meaning at-tached to this item by these populations.

Each o these measurement intercept di erences issubstantively signi cant. Tat is, these particularitems assess the underlying constructs di erentlyor demographic subgroups and thus should not beused as indicators. Dropping these items, however,leaves three subscales with only two items, which isar rom ideal. able B3 presents revised CFA mod-els afer dropping the items with non-invariantmeasurement intercepts. able B4 reports latentactor correlations.12 Note that the correlationsbetween home support and home meaning ul par-ticipation remain relatively high (0.73), indicating ahigh degree o overlap between these two actors.


30/175


tAble b1s a h v m a a a aa a , ma am , 8- a

o a a

i m i m 1 2 3 4 5 6 7 8

r6 s ca s a w a a a m . 0.75 0.08 0.02 0.02 0.07 0.03 0.06 0.01r8 s ca s a w w im . 0.79 0.02 0.01 0.03 0.03 0.04 0.04 0.06

r10 s ca s a w mw i a m . . . 0.86 0.02 0.01 0.01 0.02 0.04 0.02 0.00

r7 s h s a w m w i a j . 0.82 0.02 0.00 0.01 0.02 0.01 0.02 0.02

r9 s h s a w a wa wa m m . 0.92 0.05 0.02 0.03 0.05 0.06 0.03 0.02

r11 s h s a w a i w a . 0.83 0.01 0.05 0.00 0.03 0.01 0.05 0.04

r12 s pa s i a . 0.08 0.57 0.01 0.19 0.08 0.06 0.01 0.01

r13 s pa s i k a a 0.02 0.91 0.02 0.09 0.01 0.02 0.00 0.00

r14 s pa s i a mak a . 0.04 0.79 0.04 0.01 0.02 0.05 0.00 0.04

r15 c mca c mm a w a a a m . 0.04 0.05 0.95 0.03 0.04 0.04 0.02 0.00

r17 c mca c mm a w w i am a . . . 0.02 0.03 0.90 0.05 0.01 0.07 0.05 0.04

r20 c mca c mm a w m i . 0.02 0.04 0.82 0.02 0.03 0.08 0.00 0.00

r16 c mh c mm a w m w i a j . 0.01 0.01 0.90 0.02 0.03 0.04 0.01 0.01

r18 c mh c mm a w a i w a . 0.02 0.05 0.90 0.02 0.10 0.05 0.02 0.03

r19 c mh c mm a w a wa wa m m . 0.04 0.01 0.95 0.00 0.05 0.08 0.03 0.04

r21 c mpa i am a , am , /m , . . . 0.03 0.06 0.02 0.82 0.01 0.03 0.04 0.03

r22 c mpa i am ak m , a , a . . . 0.02 0.07 0.03 0.97 0.00 0.01 0.01 0.06

r23 c mpa i . 0.05 0.10 0.09 0.46 0.09 0.19 0.08 0.07

r49 h m ca h m a w m w k . 0.02 0.07 0.03 0.00 0.86 0.01 0.01 0.02

r51 h m ca h m a w a k w m a m m . 0.03 0.08 0.02 0.12 0.77 0.27 0.01 0.10r53 h m ca h m a w m

w i a m . . . 0.02 0.01 0.03 0.12 0.76 0.32 0.03 0.06

r48 h m h h m a w m w . 0.01 0.02 0.01 0.13 0.76 0.18 0.06 0.09

r50 h m h h m a w a i w a . 0.04 0.02 0.07 0.03 0.83 0.02 0.00 0.03

r52 h m h h m a w a wa wa m m . 0.03 0.08 0.05 0.09 0.89 0.08 0.01 0.06

r54 h m pa i a wm a . . . 0.01 0.08 0.01 0.04 0.30 0.63 0.02 0.04

r55 h m pa i a m a mak a . 0.02 0.11 0.00 0.08 0.09 0.68 0.00 0.08

r56 h m pa i mak w m am . 0.03 0.02 0.02 0.01 0.23 0.70 0.03 0.01

r42 p ca A w a a a m . 0.02 0.04 0.04 0.05 0.04 0.04 0.83 0.06

r43 p ca A w a k w m a m m . 0.02 0.03 0.01 0.03 0.01 0.01 0.96 0.00

r44 p ca A w m w im a a a m . 0.00 0.00 0.00 0.03 0.02 0.02 0.92 0.02

r45 p h M a . 0.05 0.05 0.03 0.03 0.01 0.01 0.02 0.45

r46 p h M w a . 0.02 0.02 0.02 0.07 0.03 0.05 0.04 0.92

r47 p h M w . 0.02 0.03 0.01 0.01 0.05 0.05 0.02 0.68

Note: Analytic samples consist o 12,000 7th-, 9th-, and 11th-grade respondents sampled rom surveys administered between spring 2003 and spring 2005.Weighted data. Loadings with largest absolute values bolded.


31/175

Appendix b 23

tAble b2s a h v m a a a aa a , va a am , 8- a

o a a

i m i m 1 2 3 4 5 6 7 8

r6 s ca s a w a a a m . 0.76 0.03 0.03 0.01 0.10 0.06 0.05 0.03r8 s ca s a w w im . 0.78 0.03 0.02 0.03 0.04 0.04 0.03 0.04

r10 s ca s a w mw i a m . . . 0.85 0.02 0.02 0.03 0.04 0.04 0.01 0.01

r7 s h s a w m w i a j . 0.82 0.02 0.01 0.01 0.01 0.01 0.01 0.02

r9 s h s a w a wa wa m m . 0.90 0.07 0.01 0.03 0.05 0.05 0.04 0.01

r11 s h s a w a i w a . 0.84 0.07 0.01 0.03 0.06 0.05 0.01 0.02

r12 s pa s i a . 0.11 0.59 0.01 0.18 0.06 0.07 0.04 0.03

r13 s pa s i k a a 0.03 0.88 0.03 0.09 0.01 0.01 0.00 0.00

r14 s pa s i a mak a . 0.02 0.80 0.04 0.00 0.02 0.04 0.01 0.03

r15 c mca c mm a w a a a m . 0.02 0.06 0.95 0.03 0.08 0.01 0.03 0.02

r17 c mca c mm a w w i am a . . . 0.01 0.01 0.89 0.06 0.02 0.08 0.03 0.03

r20 c mca c mm a w m i . 0.00 0.02 0.83 0.01 0.04 0.06 0.02 0.03

r16 c mh c mm a w m w i a j . 0.03 0.01 0.89 0.02 0.01 0.07 0.00 0.01

r18 c mh c mm a w a i w a . 0.02 0.08 0.89 0.02 0.11 0.07 0.01 0.00

r19 c mh c mm a w a wa wa m m . 0.04 0.02 0.95 0.01 0.07 0.12 0.03 0.01

r21 c mpa i am a , am , /m , . . . 0.03 0.06 0.02 0.83 0.02 0.01 0.02 0.03

r22 c mpa i am ak m , a , a . . . 0.00 0.07 0.01 0.97 0.02 0.03 0.02 0.05

r23 c mpa i . 0.04 0.13 0.08 0.47 0.08 0.16 0.09 0.05

r49 h m ca h m a w m w k . 0.03 0.05 0.03 0.03 0.85 0.03 0.01 0.00

r51 h m ca h m a w a k w m a m m . 0.05 0.08 0.04 0.12 0.74 0.30 0.02 0.09r53 h m ca h m a w m

w i a m . . . 0.02 0.03 0.06 0.12 0.73 0.32 0.02 0.07

r48 h m h h m a w m w . 0.01 0.03 0.00 0.11 0.75 0.20 0.07 0.12

r50 h m h h m a w a i w a . 0.06 0.02 0.07 0.03 0.81 0.04 0.01 0.03

r52 h m h h m a w a wa wa m m . 0.08 0.08 0.05 0.08 0.85 0.05 0.03 0.03

r54 h m pa i a wm a . . . 0.04 0.07 0.01 0.05 0.23 0.67 0.06 0.06

r55 h m pa i a m a mak a . 0.03 0.15 0.03 0.10 0.06 0.68 0.04 0.05

r56 h m pa i mak w m am . 0.02 0.02 0.00 0.01 0.16 0.77 0.02 0.02

r42 p ca A w a a a m . 0.04 0.05 0.04 0.04 0.03 0.02 0.83 0.05

r43 p ca A w a k w m a m m . 0.02 0.03 0.02 0.03 0.00 0.02 0.96 0.01

r44 p ca A w m w im a a a m . 0.01 0.01 0.01 0.03 0.03 0.01 0.91 0.03

r45 p h M a . 0.07 0.07 0.04 0.01 0.02 0.03 0.09 0.42

r46 p h M w a . 0.02 0.02 0.01 0.04 0.01 0.02 0.07 0.85

r47 p h M w . 0.01 0.04 0.04 0.05 0.02 0.05 0.01 0.77

Note: Analytic samples consist o 12,000 7th-, 9th-, and 11th-grade respondents sampled rom surveys administered between spring 2003 and spring 2005.Weighted data. Loadings with largest absolute values bolded.


32/175


33/175

Appendix b 25

Elementary school environmental resilience assets

Exploratory actor analysis results.An identi-cal strategy was used to analyze the elementaryschool RYDM environmental resilience items.EFA models suggested that a 4- actor model bestrepresents the environmental resilience items,

with distinct actors or school support (car-ing relationships and high expectations), homesupport, meaning ul participation (in the schooland home domains), and pro-social peers (tablesB5 and B6). Tese results were ound or both themain sample and the validation sample and orboth boys and girls.

tAble b4c a am a h v m a a , a ma a a a m

a

Ma am (1) (2) (3) (4) (5) (6) (7) (8)

(1) s 1.00(2) s m a a a 0.59 1.00

(3) c mm 0.54 0.42 1.00

(4) c mm m a a a 0.42 0.58 0.46 1.00

(5) h m 0.47 0.37 0.59 0.44 1.00

(6) h m m a a a 0.48 0.59 0.51 0.38 0.73 1.00

(7) p a a 0.41 0.35 0.46 0.34 0.46 0.44 1.00

(8) p - a 0.42 0.40 0.38 0.39 0.49 0.50 0.54 1.00

Note: Analytic samples consist o 12,000 7th-, 9th-, and 11th-grade respondents sampled rom surveys administered between spring 2003 and spring 2005.Weighted data.

tAble b5e m a h v m a a a aa a , ma am , 4- a

i mo a

i m 1 2 3 4

10 s ca d a . . . a a a ? 0.74 0.05 0.01 0.01

13 s ca t a . . . w . . . a m a ? 0.62 0.07 0.00 0.05

11 s h t a . . . w a j ? 0.56 0.02 0.17 0.07

14 s h t a . . . a a a j ? 0.67 0.10 0.02 0.03

52 h m ca pa . . . a a w k? 0.00 0.81 0.01 0.01

55 h m ca pa . . . w a m a ? 0.06 0.51 0.20 0.01

53 h m h pa . . . a a a j ? 0.11 0.79 0.00 0.02

54 h m h pa . . . a m wa ? 0.10 0.77 0.03 0.00

9 s pa d mak a / a ? 0.14 0.16 0.48 0.05

15 s pa d a ? 0.16 0.11 0.50 0.15

56 h m pa d a m ? 0.17 0.21 0.48 0.03

56 h m pa d mak a m ? 0.10 0.05 0.37 0.03

50 p h d ? 0.03 0.01 0.07 0.63

51 p h d ? 0.02 0.04 0.18 0.68

Note: Analytic samples consist o 2,000 th-grade respondents sampled rom surveys administered between

description: tags: REL 2007034

Documents

Transcript of description: tags: REL 2007034