Devereux Early Childhood Assessment for Infants and ... · The DECA-I/T standardization sample...

Devereux Early ChildhoodAssessment for Infants

and ToddlersTechnical Manual

by Gregg Powell, Mary Mackrain, and Paul LeBuffe

Kaplan Early Learning CorporationLewisville, NC

©2007 The Devereux FoundationAll rights Reserved

Published by Kaplan Early Learning Corporation, Lewisville, NCISBN number 10: 0-88076-682-4

ISBN number 13: 978-0-88076-682-1Item number: 14779Printed in the USA

iPreface

Prefacehis manual presents information on the development and

standardization of the Devereux Early Childhood Assessment for Infants and Toddlers (DECA-I/T), as well as the results of studies examining its reliability and validity. This information is provided to help the user becomebetter informed about the strengths and proper uses of the DECA-I/T.

Complete information on the administration, scoring and interpretation ofthe DECA-I/T may be found in the DECA-I/T User’s Guide.

Many individuals have contributed to the DECA-I/T. Their names may befound in the acknowledgement section of the DECA-I/T User’s Guide.

The authors welcome feedback on the DECA-I/T as well as opportunities forcollaboration. They may be reached at the Devereux Early ChildhoodInitiative of the Devereux Foundation.

444 Devereux DriveVillanova, PA 19085Telephone: 610/542-3109E-Mail: [email protected] Page: www.devereuxearlychildhood.org

T

iiiTable of Contents

Table n of n ContentsPreface.....................................................................................................................................................................i

Table of Contents ...........................................................................................................................................iii

List of Tables and Figures...........................................................................................................................v

Chapter 1: Development and Standardization ..................................................................................1Development of the DECA-I/T Items ......................................................................................................1

National Standardization .............................................................................................................................2

Representativeness of the DECA-I/T Standardization Sample........................................................2

Age and Gender......................................................................................................................................3

Geographic Region .................................................................................................................................3

Race...........................................................................................................................................................4

Socioeconomic Status ..........................................................................................................................6

Organization of Items into Scales ............................................................................................................6

Norming Procedures.....................................................................................................................................10

T Scores ..........................................................................................................................................................11

Chapter 2: Reliability.................................................................................................................................. 13Internal Reliability ......................................................................................................................................13

Standard Errors of Measurement ...........................................................................................................15

Test-Retest Reliability ...............................................................................................................................16

lnterrater Reliability...................................................................................................................................20

Stability of DECA-I/T Ratings ................................................................................................................23

Summary ........................................................................................................................................................25

ivDevereux Early Childhood Assessment for Infants and Toddlers – Technical Manual

Chapter 3: Validity........................................................................................................................................27Content Validity ...........................................................................................................................................27

Criterion Validity .........................................................................................................................................28

Contrasted Groups............................................................................................................................. 29

Examination of Potential Adverse Impact on Minority Children ............................................31

Individual Prediction ..........................................................................................................................35

Construct Validity ...................................................................................................................................... 38

Convergent Validity ............................................................................................................................38

Protective Factors Study .................................................................................................................40

Summary ........................................................................................................................................................43

References .......................................................................................................................................................45

Tables n and n FiguresTablesTable 1.1 DECA-I/T Standardization Sample Characteristics:

Age and Gender.............................................................................................................................3

Table 1.2 DECA-I/T Standardization Sample Characteristics: Geographic Region and Age ........................................................................................................4

Table 1.3 DECA-I/T Standardization Sample Characteristics: Geographic Region and Race ......................................................................................................5

Table 1.4 Rotated Factor Analysis Results for the DECA-I Scales..................................................8

Table 1.5 Rotated Factor Analysis Results for the DECA-T Scales .................................................9

Table 2.1a Internal Reliability (Alpha) Estimates for DECA-I Scales by Rater............................14

Table 2.1b Internal Reliability (Alpha) Estimates for DECA-T Scales by Rater ...........................15

Table 2.2a Standard Errors of Measurement for the DECA-I Scale T Scores by Rater...........16

Table 2.2b Standard Errors of Measurement for the DECA-T Scale T Scores by Rater ..........17

Table 2.3a Characteristics of the DECA-I Test-Retest Reliability Sample....................................17

Table 2.3b Characteristics of the DECA-T Test-Retest Reliability Sample ...................................18

Table 2.4a Test-Retest Reliability Coefficients for DECA-I Scores Obtained at a 24- to 72-Hour Interval...................................................................................................19

Table 2.4b Test-Retest Reliability Coefficients for DECA-T Scores Obtained at a 24- to 72-Hour Interval...................................................................................................19

Table 2.5a Characteristics of DECA-I Interrater Reliability Sample .............................................20

Table 2.5b Characteristics of DECA-T Interrater Reliability Sample..............................................21

Table 2.6a Interrater Reliability Coefficients for DECA-I Scores..................................................22

Table 2.6b Interrater Reliability Coefficients for DECA-T Scores .................................................22

Table 2.7a DECA-I Pretest and Posttest Mean Scale T-Scores and Standard Deviations—Parent Raters.............................................................................23

vTables and Figures

Table 2.7b DECA-I Pretest and Posttest Mean Scale T-Scores and Standard Deviations—Teacher Raters .........................................................................23

Table 2.7c DECA-T Pretest and Posttest Mean Scale T-Scores and Standard Deviations—Parent Raters.............................................................................24

Table 2.7d DECA-T Pretest and Posttest Mean Scale T-Scores and Standard Deviations—Teacher Raters .........................................................................25

Table 3.1a Characteristics of the DECA-I Validity Study Sample ...................................................29

Table 3.1b Characteristics of the DECA-T Validity Study Sample...................................................30

Table 3.2a Mean T Scores and Difference Statistics for DECA-I Validity Study .......................31

Table 3.2b Mean T Scores and Difference Statistics for DECA-T Validity Study......................32

Table 3.3a DECA-I Scale Scores: d-Ratios Comparing Minority and Non-Minority Children.......................................................................................................33

Table 3.3b DECA-T Scale Scores: d-Ratios Comparing Minority and Non-Minority Children.......................................................................................................34

Table 3.4a Actual and Predicted Group Membership for the DECA-I Validity Study.................36

Table 3.4b Actual and Predicted Group Membership for the DECA-T Validity Study ................37

Table 3.5 Convergent Validity Results (DECA and DECA-T) .............................................................39

Table 3.6 Sample Characteristics for the DECA-IT Protective Factor Study............................40

Table 3.7 Mean Temperament and Regulatory Index Scores for Risk and Protective Factor Groups in DECA Protective Factor Study .........................................42

FiguresFigure 1.1 Poverty Level of DECA-I/T Sample.........................................................................................6

Figure 1.2 Infant Mean Total Raw Scores by Age.................................................................................10

Figure 3.1 Mean Temperament and Regulatory Index Scores for Risk andProtective Factor Groups.........................................................................................................42

viDevereux Early Childhood Assessment for Infants and Toddlers – Technical Manual

1

1Development and Standardization

Development n and n

StandardizationDevelopment of the DECA-I/T Items

s with the original Devereux Early Childhood Assessment (DECA)(LeBuffe & Naglieri, 1999) multiple approaches were used to develop theinitial set of items for the Devereux Early Childhood Assessment for Infantsand Toddlers (DECA-I/T).

First, we reviewed the literature on resilience and noted behavioral descriptions of resilient children (Egeland, 1997; Gordon-Rouse, 1996;Masten and Coatsworth, 1998; Werner and Smith, 1982, 1992, and 2001;Werner, 1990 and 2000). During this process we reviewed existing measures of infant and toddler social and emotional health.

We also conducted focus groups with parents and teachers of infants andtoddlers, early care and education professionals, and mental health profes-sionals. In the focus group sessions, parents and professionals were askedto describe the behaviors of children that “were likely to do well” or indi-cated that the child was “doing well” in regards to social and emotionalhealth. Conversely, parents and early care and education professionalswere also asked to describe behaviors that indicated that the child was “likely to have problems.” Behavioral descriptions were used to generaterating scale items.

The items were written as directly observable behaviors requiring little or noinference on the part of the observer. Careful attention was also paid topotential psychometric qualities such as reliability and validity as well asease of use of the scales. Finally, throughout all phases of item development,the reading level of the items and Rater directions were carefully consideredso that the overall readability of the text would be as easy as possible.

The item development phase resulted in a pool of 112 items, which servedas the starting point in the development of the DECA-I/T. By conducting a pilot study in the spring of 2005 with 251 participants at 12 sites nation-ally, it was possible to examine the usefulness of the initial set of items and

A

2Devereux Early Childhood Assessment for Infants and Toddlers – Technical Manual

their inter-relationships. Children with identified special needs (behavioral,social, emotional) scored statistically significantly lower on protective factorsthan those children who were not identified as having social emotional concerns. There was also a statistically significant difference between infantand toddler scores.

The results of the pilot study were used to prepare a form consisting of 68 items to be used in the national standardization study described below.

National StandardizationThe DECA-I/T was standardized in a way to ensure the sample would closely represent the United States population on important demographiccharacteristics. The data collection procedures also ensured that a widevariety of children were included for the generation of norms. We collecteddata from a variety of settings across the United States. Infant and toddlerearly care and education professionals from childcare settings provided theteacher ratings and will be referred to in all tables as Teacher Raters. Parent(and/or other family member) ratings were obtained not only from thesesame settings, but also in response to recruitment efforts. To ensure the confidentiality of their responses, parents who chose to participate wereable to: 1) place their completed rating form in a sealed envelope to be sentto Devereux for processing or 2) anonymously fill out the standardizationform online. The online form was identical to the handwritten copy.

Representativeness of the DECA-I/T Standardization SampleThe DECA-I/T standardization sample consisted of 2,183 infants and toddlers between 4 weeks and 3 years of age (45% infants and 55% toddlers). For this sample an infant was defined as being from 4 weeks upto 18 months and a toddler as being from 18 months up to 3 years of age. Early care and education professionals provided ratings on 52% ofthese children; parents provided ratings on the remaining 48% of the children. As shown in Table 1.1, the DECA-I/T standardization sampleclosely approximated the population of the United States with respect togender. The desired characteristics of the standardization sample werebased on the Statistical Abstract of the United States 2006 125th edition:The National Data Book by the U.S. Department of Commerce, Economicsand Statistics Administration, Bureau of the Census. In the tables that follow, the total numbers of children included may not sum to 2,183 due to missing data.

Age and GenderTable 1.1 presents the numbers and percentages of infants and toddlers bygender. The number of infants was 987 and the number of toddlers was1,196. These results show that each age was sufficiently sampled. The dataalso show that the percentages of males and females in the standardizationsample as a whole, as well as at each age, very closely approximated theproportions of the U.S. population.

Geographic RegionWe collected data from 99 sites in 29 states in the four geographic regions:Northeast, Midwest, West, and South. Table 1.2 shows the numbers andpercentages of children for each age and the total sample for each of the four geographic regions. On average, the regional distribution of theDECA-I/T standardization sample was within 6% of the U.S. population forchildren 4 weeks to 3 years old. These data show that the DECA-I/T standardization sample closely approximated the regional distribution ofthe U.S. population.


Table 1.1

DECA-I/T Standardization Sample Characteristics: Age and Gender

Males Females Unknown Total

Age n % n % n % n %

Infants (1-18 Months) 484 49.0% 483 48.9% 20 2.0% 987 45.2%

Toddlers (18-36 Months) 591 49.4% 595 49.7% 10 0.8% 1196 54.8%

Total Sample 1075 49.2% 1078 49.4% 30 1.4% 2183

U.S. % 51.0% 49.0%

Note. The U. S. population data are based on ”Resident Population, by Sex and Age: 2006 Table No. 16,” Statistical Abstractof the United States 2006 125th edition: The National Data Book by the U.S. Department of Commerce, Economics andStatistics Administration, Bureau of the Census, 2006. Washington, DC: Author.


RaceTable 1.3 provides the DECA-I/T standardization sample composition bygeographic region and race. Based on information provided on the ratingforms, the children were classified according to the five major race categories used by the U.S. Bureau of the Census: Native American,Asian/Pacific Islander, African American, Hispanic, and Caucasian. The DECA-I/T rating forms also allowed the Rater to describe the race of the child as “Mixed Race” or “Other.”

The data in Table 1.3 indicate that the racial composition of the total DECA-I/T standardization sample closely approximated that of the U.S. population (total exceeds 100% due to some respondents making multiple selections). Additionally, sample percentages within each regionwere also similar to the actual population percentages found in each geographic region.

Table 1.2

DECA-I/T Standardization Sample Characteristics:Geographic Region and AgeNortheast Midwest West South Unknown Total

Age n % n % n % n % n % n %

Infants (1-18 Months) 165 16.7% 283 28.7% 230 23.3% 302 30.6% 7 0.7% 987 45.2%

Toddlers (18-36 Months) 243 20.3% 301 25.2% 275 23.0% 366 30.6% 11 0.9% 1196 54.8%

Total Sample 408 18.7% 584 26.8% 505 23.1% 668 30.6% 18 0.8% 2183

U.S. % 18.0% 21.8% 26.2% 33.9%

Note. The U. S. population data are based on “Resident Population, by Sex and Age: 2006 Table No. 34,” Statistical Abstractof the United States 2006 125th edition: The National Data Book by the U.S. Department of Commerce, Economics andStatistics Administration, Bureau of the Census, 2006. Washington, DC: Author.


Tabl

e 1.

3

DEC

A-I

/T S

tand

ardi

zati

on S

ampl

e Ch

arac

teri

stic

s:

Geog

raph

ic R

egio

n an

d Ra

ceAs

ian

Nat

ive

Pacif

icAf

rican

Mix

edN

otTo

tal

Amer

ican

Isla

nder

Amer

ican

His

pani

cCa

ucas

ian

Oth

erRa

ceRe

port

edSa

mpl

e

nRo

w %

nRo

w %

nRo

w %

nRo

w %

nRo

w %

nRo

w %

nRo

w %

nRo

w %

nRo

w %

U.S.

%

Nor

thea

st2

0.5%

71.

7%55

13.5

%7

1.7%

307

75.2

%4

1.0%

23

5.6%

30.

7%40

818

.7%

18.1

%

Mid

wes

t12

2.1%

10.

2%52

8.9%

244.

1%40

669

.5%

111.

9%68

11.6

%10

1.7%

584

26.8

%21

.8%

Wes

t26

5.1%

81.

6%3

0.6%

116

23.0

%26

452

.3%

91.

8%71

14.1

%8

1.6%

505

23.1

%26

.2%

Sout

h21

3.1%

81.

2%84

12.6

%18

026

.9%

328

49.1

%9

1.3%

31

4.6%

71.

0%66

830

.6%

33.9

%

Not

Rep

orte

d1

5.6%

00.

0%2

11.1

%3

16.7

%11

61.1

%0

0.0%

15.

6%0

0.0%

180.

8%

Tota

l62

2.8%

241.

1%19

69.

0%33

015

.1%

1316

61.7

%33

1.5%

194

8.9%

281.

3%21

83

Adju

sted

Sam

ple

(w

/o “

Oth

er”,

“M

ixed

Race

”, a

nd “

Not

Rep

orte

d”

Tota

l62

3.2%

241.

2%19

610

.2%

330

17.1

%13

1668

.3%

1928

U.S.

%2.

2%3.

4%16

.2%

12.2

%71

.3%

Not

e. T

he U

. S. p

opul

atio

n da

ta a

re b

ased

on

“Res

iden

t Pop

ulat

ion,

by

Sex

and

Age

: 200

6 Ta

ble

No.

22,

” St

atist

ical

Abs

tract

of th

e U

nite

d St

ates

200

6 12

5th

editi

on: T

he N

atio

nal D

ata

Book

by th

e U

.S. D

epar

tmen

t of C

omm

erce

, Eco

nom

ics

and

Stat

istic

s A

dmin

istra

tion,

Bur

eau

of th

e C

ensu

s, 2

006.

Was

hing

ton,

DC

: Aut

hor.


Socioeconomic StatusDetermining the number of children receiving either subsidized childcare or enrolled in TANF assessed the socioeconomic status of the DECA-I/Tstandardization sample. Of the entire sample of over 2,183 children, 29% were either receiving subsidized childcare or public assistance. Thisvery closely approximates the 27% of infants and toddlers living in poverty(Children’s Defense Fund, 2005). Figure 1.1 depicts the breakdown of theDECA-I/T sample compared to Census Data (2006).

Organization of Items into ScalesUtilizing the standardization data set, we attempted to organize the DECA-I/T items into statistically and logically derived scales. The ProtectiveFactor Scales were identified through the use of exploratory item factoranalysis. We applied this method to the entire set of protective factor items. Careful examination of the factorial results suggested that no singlesolution would work across the developmental age span and there neededto be two forms (infant and toddler), which became the Devereux Early Childhood Assessment for Infants (DECA-I) and the Devereux EarlyChildhood Assessment for Toddlers (DECA-T).

Figure 1.1

Poverty Level DECA-I/T Sample

Living in Poverty

28.9%

71.1% 73.0%

27.0%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

Yes No

Sample Census


Next, we conducted a series of analyses to determine which items should be deleted to obtain the best configuration of scales, for infants and toddlersseparately. We based the decisions to delete items on the following goals:1) to identify the best factor solution from psychometric and interpretabilityperspectives, 2) to shorten the two forms of the DECA-I/T as much as possible without compromising breadth of coverage, and 3) to ensure that the constructs are measured reliably by the scales. Although this leftanchor items common to both scales, it also permitted differing items thatwere developmentally appropriate either for the infant or the toddler scale.The final results of these analyses are provided in Table 1.4 for the infantform and Table 1.5 for the toddler form.

The DECA-I/T Infant form (DECA-I) ended up with 33 items and the Toddlerform (DECA-T) with 36 items reflecting positive behaviors (strengths) typically seen in resilient children. Factor analysis elicited a strong two factor solution for the infants and a strong three factor solution for the toddlers. The infant and the toddler factor solutions, with the individual itemdescriptions were sent to the National Advisory Team and the DECIResearch Advisory Board (see Devereux Early Childhood Assessment forInfants and Toddlers: User’s Guide, Appendix D) to advise on naming theprotective factor scales. Additionally our literature review provided furtherguidance in selecting these scale titles. There was strong agreement withboth expert opinion and literature which resulted in the titles and theirdescriptions for the Infant Form as listed below:

Initiative (18 items) assesses the infant’s ability to use independentthought and actions to meet her or his needs.

Attachment/Relationships (15 items) assesses the mutual, strong, longlasting relationship between the infant and significant adults such asfamily members, and teachers.

A Total Protective Factors scale, which is a composite of the above twoscales, provides an overall indication of the strength of the infant’s protective factors.

The DECA-I/T Toddler Form (DECA-T) is comprised of the following scales:

Attachment/Relationships (18 items) assesses the mutual, strong, longlasting relationship between the toddler and significant adults such as family members, and teachers.

Initiative (11 items) assesses the toddler’s ability to use independentthought and actions to meet her or his needs.

Self-Regulation (7 items) assesses the toddler’s ability to gain control ofand manage emotions, and sustain focus and attention.

A Total Protective Factors scale, which is a composite of the abovethree scales, provides an overall indication of the strength of the toddler’s protective factors.


Table 1.4

Rotated Factor Analysis Results for DECA-I ScalesFactors

Item # Item In A/R

1 try to do new things .733 imitate actions of others .725 keep trying when unsuccessful .727 show interest in what others were doing .709 notice changes in surroundings .70

11 adjust her/his energy level to the type of play .6913 act happy when praised .6715 explore surroundings .6517 express her/his dislikes .6519 reach for a familiar adult .6120 respond to her/his name .6022 react to another child's cry .6021 keep trying to obtain a toy .5926 act in a way that make others smile or show interest .5927 easily go from one activity to another .5828 seek attention when a familiar adult was with another child .5830 enjoy being around other children .5729 look to familiar adult when exploring her/his surroundings .5232 act happy with familiar adults .7631 show pleasure when interacting with adults .7333 accept comfort from a familiar adult .7223 smile at familiar adults .7125 act happy .6924 respond positively to adult attention .6818 smile back at a familiar adult .6716 calm down with help from a familiar adult .6614 make eye contact with others .6412 act in a good mood .6310 seek comfort from familiar adults .618 show affection for a familiar adult .606 enjoy being cuddled .594 enjoy interacting with others .522 respond when spoken to .50

Note: Only Loadings of .50 or above are reported.(In = Inititiative, A/R = Attachment/Relationships)


Table 1.5

Rotated Factor Analysis Results for DECA-T ScalesFactors

Item # Item A/R In SR

2 show affection for a familiar adult .7621 easily go from one activity to another .756 act happy with familiar adults .74

22 show pleasure when interacting with adults .7315 smile back at a familiar adult .71

4 seek comfort from familiar adults .6936 express avariety of emotions (e.g., happy, sad, mad) .6617 reach for a familiar adult .66

5 makes needs known to a familiar adult .6625 accept comfort from a familiar adult .6611 act happy when praised .6513 make eye contact with others .6524 makes others aware of her/his needs .62

7 show interest in her/his surroundings .6018 respond to her/his name .598 respond when spoken to .58

14 enjoy being cuddled .581 enjoy interacting with others .529 show concern for other children .76

10 try to comfort others .7526 play make-believe .6729 try to clean up after herself/himself .6728 show preference for a particular playmate .6619 react to another child’s cry .6416 ask to do new things .6431 play with other children .6012 participate in group activities .5632 try to do things for herself/himself .5527 follow simple directions .5523 handle frustration well .7234 accept another choice when the first choice was not available .68

3 adjust to changes in routine .6733 calm herself/himself .6130 easily follow a daily routine .5835 have regular sleeping pattern .5621 easily go from one activity to another .56

Note: Only Loadings of .50 or above are reported.(A/R = Attachment/Relationships, In = Inititiative, SR = Self-Regulation)


The individual DECA-I/T protective item factor loadings were obtainedusing principal components extraction with varimax rotation. The Kaiser(1960) rule (i.e., eigenvalues greater than 1.0) and scree plots were initially used to assist in determining the existence of a factor. Subsequently,an iterative procedure was used, based on absolute value of standardizedfactor loadings of .4, until potential subscales were completely resolved with unique factor loadings. These factor analytic results showed that eachof the subscale factors is comprised of items with substantial loadings on the subscale on which they are placed, and which did not have substantialloadings on the subscales on which they were not placed.

Norming ProceduresThe first step in preparing the norms was to determine if any trends existed in the data. We examined the children’s Total Protective FactorsScale Raw Score means and standard deviations for age, Rater, and gender differences. Figure 1.2 suggests age trends in the total raw score forthe infant (1 to 18 months). There were no age trends for the toddlers.Separate norms tables (see Appendix A and B of the DECA-I/T User’sGuide) were developed for 1 up to 3 months, 3 up to 6 months, 6 up to 9 months, and 9 up to 18 months (there were no differences for the 9 up to 12, 12 up to 15, and 15 up to 18 months scores) for the infant form and18 up to 36 months for the toddler form.

Figure 1.2

Infant Mean Total Raw Scores by Age

020406080

100120

1-3Month

6-9Months

12-18Months

MMeeaan n

T T ootta a

l lSScc o o

r re e

1-3

120100806040200

3-6 6-9 9-12 12-18Age in MonthsM

ean

To

tal R

aw S

core

We also found the need to construct norms by Rater (parents/family members and early care and education professionals) because initial analyses showed significant differences in the scores. These differenceswere most likely due to the different environments in which these differentRaters see the children. After determining that Rater norms would be constructed, we examined the distributions of raw scores for normality. The cumulative frequency distributions for the derived scales (see Tables 1.4and 1.5 on pages 8 and 9) all approached normality but were slightly positively skewed. For this reason we decided to compute the separatenorms tables using classical normalization procedures.

To accomplish this, we fit the obtained frequency distribution for each scale to normal probability standard scores using Blom’s (Blom, 1958) algorithm of

where r is the rank of the score and w is the sum of weight. These procedures were followed for all of the protective factor scales.

T ScoresWe computed standard scores separately for each of the scales based ontheir individual raw score distributions. We determined the standard scorescorresponding to the percentiles for which they are theoretically associatedbased on the normal curve. T scores for each scale were set at a mean of50 and a standard deviation of 10. We selected this metric because of itsfamiliarity to professionals, its previous use with the DECA, and because itfacilitates interpretation of the results and comparison with T scores fromother similar scales.

In order to maintain this metric for Total Protective Factors scores, and in order to equalize the weights due to the different numbers of items per subscale, a second normalization was required. That is, we addedtogether the T scores for the subscales (two for the Infant form and three forthe Toddler form). This sum was then normalized using Blom’s algorithm (Blom, 1958), and transformed with the T score formula of (10 x Z ) + 50.


3

81

4

r

w

−

+

3

13Reliability

Reliabilityhe reliability of assessments like the DECA-I/T is defined as,

“the consistency of scores obtained by the same person when reexaminedwith the same test on different occasions, or with different sets of equivalentitems, or under other variable examining conditions” (Anastasi, 1988, p. 102). We assessed the reliability of both the DECA-I and the DECA-Tusing several methods. First, we computed the internal reliability coefficientsfor each scale. Second, we assessed standard error of measurement (SEM).Third, we assessed the test-retest reliability of each scale. Finally, we determined the interrater reliability for each scale.

Internal ReliabilityInternal reliability (also known as internal consistency) refers to the extent to which the items on the same scale or assessment instrument measure the same underlying construct. High internal reliability, which is desirable,indicates that the items assess the same characteristic of the child (i.e., construct) and, therefore, truly comprise a single scale. In contrast, low internal reliability indicates that the items measure a variety of different child characteristics and, therefore, do not comprise a single scale.

We determined the internal reliability of each scale and for each form usingCronbach’s alpha (Cronbach, 1951). In practice, this statistic can vary from.00 (low) to .99 (high). The internal reliability coefficients (alphas) werebased on the DECA-I/T standardization sample and estimates for eachwere calculated separately for each Rater (parent/family member or earlycare and education professional) and are presented in Table 2.1a (Infants)and Table 2.1b (Toddlers).

The results in these tables indicate that both the DECA-I and the DECA-Thave high internal reliability. For the infant form (DECA-I) the Total ProtectiveFactors Scale alpha for both Parent Raters (.90 to .94) and Teacher Raters(.93 to .94) met or exceeded the .90 minimum for a total score suggestedby Bracken (1987) in each age group. In addition, these values met the“desirable standard” described by Nunnally (1978, p.246). The same wastrue for the toddler form (DECA-T) with Parent Raters at .94 and TeacherRaters at .95.

T

2


The internal reliability coefficients for the DECA-I scales (Initiative and Attachment/Relationships) were also high. These ranged from a low of .80 (1 to 3 Months Attachment/Relationships Parent Rater) to a high of.93 (1 to 3 Months Attachment/Relationships Teacher Rater). The medianreliability coefficient across both scales was .87 for Parent Raters and .90for Teacher Raters. These median values met or exceeded the .80 minimumfor scale scores suggested by Bracken (1987).

Table 2.1a

Internal Reliability (Alpha) Estimates for DECA-I Scales by Rater

Raters

Scale Parents Teachers

1-3 Months

Initiative .87 .87

Attachment/Relationships .80 .93

Total Protective Factors .90 .93

3-6 Months

Initiative .86 .91



6-9 Months

Initiative .90 .89



9-18 Months

Initiative .87 .90



15Reliability

The internal reliability coefficients for the DECA-T remaining scales(Attachment/Relationships, Initiative and Self-Regulation) were high as well.These ranged from a low of .79 (Self-Regulation Parent Rater) to a high of.94 (Initiative Teacher Rater). The median reliability coefficient across thesethree scales was .87 for Parent Raters and .90 for Teacher Raters. Thesemedian values also met or exceeded the .80 minimum for scale scores sug-gested by Bracken (1987).

Standard Errors of MeasurementThe standard error of measurement (SEM) is another index of the reliabilityof test scores. It is an estimate of the amount of error in the observed score,expressed in standard score units (i.e., T scores). We obtained the SEM foreach of the DECA-I/T Scale T scores directly from the internal reliabilitycoefficient (r) using the formula,

where σ is the theoretical standard deviation of the T score (10) and theappropriate reliability coefficient (r) is used (Atkinson, 1991). The SEM

for each DECA-I and DECA-T scale according to Rater are presented inTable 2.2a and Table 2.2b. The SEMS varied with the size of the internal reliability coefficient reported in Tables 2.1a and 2.1b—the higher the reliability, the smaller the standard error of measurement.

Table 2.1b

Internal Reliability (Alpha) Estimates for DECA-T Scales by Rater

Raters



Initiative .92 .94

Self-Regulation .79 .83


1MSE rσ= −


Test-Retest ReliabilityThe correlation between scores obtained for the same child by the sameRater on two separate occasions is another indicator of the reliability of an assessment instrument. The correlation of this pair of scores is the test-retest reliability coefficient (r), and the magnitude of the obtained valueinforms us about the degree to which random changes influence the scores(Anastasi, 1988).

Table 2.2a

Standard Errors of Measurement for the DECA-I Scale T Scores by Rater

Raters


1-3 Months

Initiative 3.61 3.61

Attachment/Relationships 4.47 2.65

Total Protective Factors 3.16 2.65

3-6 Months




6-9 Months




9-18 Months




Reliability

Table 2.2b

Standard Errors of Measurement for the DECA-T Scale T Scores by Rater

Raters




Self-Regulation 4.58 4.12


Table 2.3a

Characteristics of DECA-I Test-Retest Reliability SampleRaters

Characteristic Parents Teachers

Size of Sample (n) 20 23

Age (Months)

Mean 7.9 9.3

SD 3.8 3.8

Gender

Boys 40% 44%

Girls 60% 56%

Race

Native American 5% 5%

Asian/Pacific Islander 5% 5%

African American 5%

Hispanic 5% 5%

Caucasian 80% 75%

Mixed Race 5% 5%

Other

17


To investigate the test-retest reliability of the DECA-I/T, a group of parents(n=20 for DECA-I and n=22 for DECA-T) and a group of teachers (n=23) for DECA-I and n=20 for DECA-T) rated the same child on two differentoccasions separated by a minimum of 24 hours and a maximum of 72 hours. Descriptive information on the children rated in this study is provided in Table 2.3a and Table 2.3b.

Table 2.4a presents the results of Test-Retest Reliability Study for the DECA-I. All of the correlation coefficients were statistically significant (p < .001), which indicates the scales have very good test-retest reliability.Overall, parents were more consistent in their evaluation of the children’sbehavior across time. For parents, the higher correlation was found on theInitiative Scale (.94), and the lower on the Attachment/Relationships Scale(.86). The higher correlation for early care and education professionals wasfound on the Attachment/Relationships and Total Protective Factor Scales(.84) and the lowest on the Initiative Scale (.83).

Table 2.3b

Characteristics of DECA-T Test-Retest Reliability SampleRaters



Age (Months)

Mean 26.1 26.2

SD 4.8 4.6

Gender

Boys 50% 50%

Girls 50% 50%

Race

Native American

Asian/Pacific Islander

African American 5% 5%

Hispanic

Caucasian 90% 95%

Mixed Race

Other 5%

19Reliability

Table 2.4b presents the results of Test-Retest Reliability Study for the DECA-T. All of the correlation coefficients were also statistically significant(p < .001), which indicates the scales have very good test-retest reliability.As was the case with infants, parents were somewhat more consistent intheir assessment of toddlers than teachers, although both raters are highlyreliable. For parents, the highest correlation was found on the InitiativeScale (.99), and the lowest on the Self-Regulation Scale (.92). The highestcorrelation for early care and education professionals was found on theInitiative Scale (.98) and the lowest on the Self-Regulation Scale (.72).

Table 2.4a

Test-Retest Reliability Coefficients for DECA-I ScoresObtained at a 24- to 72-Hour Interval

Scale Parents Teachers Overall

Initiative .94*** .83*** .87***

Attachment/Relationships .86*** .84*** .83***

Total Protective Factors .91*** .84*** .85***

Table 2.4b

Test-Retest Reliability Coefficients for DECA-T Scores Obtained at a 24- to 72-Hour Interval

Scale Parents Teachers Overall

Attachment/Relationships .97*** .96*** .97***

Initiative .99*** .98*** .98***

Self-Regulation .92*** .72*** .85***

Total Protective Factors .99*** .91*** .97***

*** (p < .001)

*** (p < .001)


Interrater ReliabilityThe correlation between scores obtained for the same child at the same timeby two different Raters is another indicator of the reliability of an assessmentinstrument. The magnitude of the obtained value informs us about thedegree of similarity in the different Raters’ perceptions of the child’s behavior.

A set of ratings included two independent ratings of the same child completed on the same day. The ratings were provided by either two earlycare and education professionals or two parents (or other family members).

Table 2.5a

Characteristics of DECA-I Interrater Reliability SampleRaters



Age (Months)Mean 9.9 9.8SD 4.5 4.3

GenderBoys 21 (46.7%) 34 (54.0%)Girls 24 (53.3%) 29 (46.0%)

RaceNative AmericanAsian/Pacific IslanderAfrican American 8 (17.8%) 18 (28.6%)Hispanic 4 (8.9%) 5 (7.9%)Caucasian 33 (73.3%) 40 (63.5%)Other

21Reliability

Two different comparisons were made: 1) Teacher Rater-Teacher Rater and2) Parent Rater-Parent Rater. We collected Teacher Rater-Teacher Rater pairs of ratings on 63 infants and 60 toddlers. We also collected Parent Rater-Parent Rater pairs of ratings on 45 infants and 49 toddlers.Demographic information on the children rated is provided in Table 2.5aand Table 2.5b.

Table 2.6a presents the results of this study for the infants. The interrater reliability coefficients for Parent Rater pairs and Teacher Rater pairs whosaw the child in the same environment were both high and statistically significant (all p < .01). The coefficient for Total Protective Factors was .68 for Parent Raters and .72 for Teacher Raters. These results indicate thatdifferent pairs of parents or teachers rate the same child very similarly on the DECA-I when observing the child in the same environment.

Table 2.5b

Characteristics of DECA-T Interrater Reliability SampleRaters




GenderBoys 37 (75.5%) 26 (43.3%)Girls 12 (24.5%) 34 (56.7%)

RaceNative AmericanAsian/Pacific Islander 4 (6.7%)African American 3 (6.1%) 24 (40.0%)Hispanic 2 (3.3%)Caucasian 46 (93.9%) 30 (50.0%)Other


Table 2.6b presents the results of this study for toddlers. As was the casewith infants, all of the interrater reliability coefficients were high and statistically significant (all p < .01). The interrater reliability coefficients for the three protective factor scales ranged from .62 for Parent Raters on the Attachment/Relationships scale to .72 for Parent Raters on the Self-Regulation scale. The median reliability coefficient across both Parent and Teacher Raters on the three protective factor scales was .68.Parent and Teacher Raters obtained coefficients of .70 and .74 respectivelyon the Total Protective Factors scale. Again these results demonstrate that different pairs of parents or teachers rate the same child very similarly onthe DECA-T when observing that child in the same environment.

Table 2.6a

Interrater Reliability Coefficients for DECA-I Scores Scale Parent-Parent Teacher-Teacher

Initiative .76** .64**

Attachment/Relationships .59** .71**

Total Protective Factors .68** .72**

** (p < .01)

Table 2.6b

Interrater Reliability Coefficients for DECA-T Scores Scale Parent-Parent Teacher-Teacher

Attachment/Relationships .62** .71**

Initiative .64** .66**

Self-Regulation .72** .71**

Total Protective Factors .70** .74**

** (p < .01)

23Reliability

Stability of DECA-I/T RatingsThe correlation coefficients reported for the test-retest reliability studies indicate that the pairs of raters ranked the infants and toddlers similarly.However, the coefficients do not indicate the actual similarity in the scores.Tables 2.7a through 2.7d provide the pretest and posttest mean scale T-scores and standard deviations received by the infants and toddlers in thetest-retest study as rated by parents and teachers.

Table 2.7a

DECA-I Pretest and Posttest Mean Scale T-Scores and Standard Deviations—Parent Raters

Scales Pretest PosttestMean (SD) Mean (SD)

Initiative 54.5 (9.3) 52.8 (8.9)

Attachment/Relationships 52.1 (7.9) 52.3 (8.6)

Total Protective Factors 53.8 (8.7) 52.7 (9.3)

Table 2.7b

DECA-I Pretest and Posttest Mean Scale T-Scores and Standard Deviations—Teacher Raters


Initiative 57.4 (8.4) 56.2 (7.3)




For parents, when rating infants, on average, the absolute value of the test-retest difference on the Initiative and Attachment/Relationship scaleswas less than one T-score point (0.92). The Total Protective Factors scale test-retest absolute value difference for parents was also approximately oneT-score point (1.1). The results for Teacher Raters were very similar. On theInitiative and Attachment/Relationship scales, the mean absolute value ofthe test-retest difference was one T-score point (1.00). The absolute value ofthe test-retest difference for the Total Protective Factors scale was also aboutone T-score point (1.1). These results demonstrate that the DECA Infant Formratings are very stable across a one- to three-day interval for both parentand teacher raters.

Tables 2.7c and 2.7d present the pretest and posttest mean scale T-scoresand standard deviations received by toddlers in the test-retest study as ratedby parents and teachers. Pairs of parent raters differed, on average, by lessthan three T-score points (2.5) across the three protective factor scales as well as the Total Protective Factor Scale (2.6). Teachers were even moreconsistent in their ratings, differing by an average of average of 0.8 T-scorepoints on the three protective factor scales and 0.9 T-score points on theTotal Protective Factors scale. These results demonstrate that the DECAToddler Form ratings are very stable across a one- to three-day interval forboth parent and teacher raters.

Table 2.7c

DECA-T Pretest and Posttest Mean Scale T-Scores and Standard Deviations—Parent Raters



Initiative 50.2 (10.7) 48.3 (9.1)

Self-Regulation 51.6 (9.7) 48.9 (10.0)


25Reliability

Table 2.7d

DECA-T Pretest and Posttest Mean Scale T-Scores and Standard Deviations—Teacher Raters



Initiative 45.5 (7.1) 45.6 (8.4)

Self-Regulation 45.3 (9.5) 45.2 (10.2)


SummaryThe results of the internal consistency, test-retest, and interrater reliabilitystudies indicate that the DECA-I/T is a reliable tool for assessing infant andtoddler protective factors. The results of the internal consistency studydemonstrated that the DECA-I/T meets the desirable standards that measurement and testing professionals have recommended. The test-reteststudy showed that Raters give very similar ratings on the same child acrossrelatively short periods. This indicates that the DECA-I/T is not easilyimpacted by random changes, but tends to provide a consistent assessmentof the child within a single setting, and stability of multiple assessments overtime. The results of the interrater reliability studies demonstrate that pairs ofparents and teachers have similar perceptions of both infants and toddlers.These results should assure parents and early care and educational professionals alike that the DECA-I/T is a reliable assessment package thatcan be used with confidence.

27

33

Validity

Validityhe validity of a test “concerns what the test measures and how

well it does so” (Anastasi, 1988, p. 139). More specifically, validity studiesinvestigate the evidence that supports the conclusions or inferences that aremade based on test results and the interpretive guidelines presented in thetest manual. According to the Standards for Educational and PsychologicalTesting (APA, 1985), validity evidence can be conceptualized as related tocontent, prediction (criterion), and construct. We investigated the validity ofthe DECA-I/T in regard to each of these three areas, and convergence inthe case of the DECA and DECA-I/T.

Content ValidityContent validity assesses the degree to which the domain measured by thetest is represented by the test items. With respect to the DECA-I/T, contentvalidity addresses how well the protective factor items represent the entiredomain of within-child behavioral characteristics related to resilience ininfants and toddlers.

As detailed in Chapter 1 of this manual, the content of the DECA-I/T wasbased on a thorough review of the resilience literature related to young children; results of national focus groups conducted with parents, teachersand infant and early childhood mental health professionals; and carefulreview of other infant and toddler social and emotional instruments. Thisresulted in a large initial pool of 112 distinct strength-based behaviors. The authors and DECI (Devereux Early Childhood Initiative) staff criticallyreviewed this set of potential items. Specifically, they were asked if they thought any content pertaining to within-child protective factors forinfants and toddlers was missing. The consensus was that there was amplecoverage of content with no skills/topics missing.

The 112-item protocol was piloted with a small national sample of 251 children prior to standardization. This protocol was further refined based on feedback from DECI staff and the National Advisory Team as well aspilot study results. The final standardization form consisted of 68 items and was sent out nationally.

T


The standardization data set was further reduced by ridding the sample ofcases that had critical information missing, such as the date of birth of thechild. The final data set was 2,183 children. Utilizing this final data set andthe analytic techniques described in Chapter 1, a large number of the itemswere eliminated, resulting in a final 2-factor solution with 33 items for theDECA-I and a 3-factor solution with 36 items for the DECA-T. It is note-worthy that the items and scales on both the DECA-I and the DECA-T have striking similarities to the within-child protective factor scales on the DECA.All three scales include the constructs of Initiative and Attachment/Relationships. In addition, the construct of Self-Regulation on the DECA-T isquite similar to the construct of Self-Control on the DECA. The overlap andsimilarities also signify an important developmental trajectory that the scalesfollow from infancy through the preschool age. The similarity of the factorstructure and scale content on the DECA and the DECA-I/T, despite the fact that they were developed with two entirely different samples, lends credence to the importance of these constructs in the social and emotionaldevelopment of children from birth through age five.

Criterion ValidityCriterion validity measures the degree to which the scores on the assessmentinstrument predict either 1) an individual’s performance on an outcome orcriterion measure, or 2) the status or group membership of an individual.

Protective factors buffer children against stress and adversity, resulting inbetter outcomes than would have been possible in their absence. Oneimportant outcome for young children is social and emotional health.Consequently, children with high scores on the DECA-I/T Protective FactorScales should have greater social and emotional health than children withlow scores on these scales.

To test this hypothesis, we obtained DECA-I and DECA-T ratings on twosamples of infants and toddlers. The “Identified” sample had known emotional and behavioral problems. These children met at least one of the two following criteria: 1) they had been referred to a mental health professional due to social and emotional challenges, or 2) they had beenasked to leave a childcare setting due to their behavior.

We also obtained DECA-I and DECA-T ratings for a matched comparisongroup of typical infants and toddlers, the “Community” sample. Matchingvariables included age, gender, and race. Table 3.1a and 3.1b providedescriptive information on the samples for the DECA-I and the DECA-Tshowing that the two groups were demographically similar.

29Validity

Contrasted GroupsThe contrasted groups approach to assessing criterion validity examinesscale score differences between groups of individuals who differ on someimportant variable. Multivariate Analysis of Variance (MANOVA) proce-dures were used to contrast scale scores for the identified and communitysamples. Preliminary tests of homogeneity of variance and normality wereconducted, and no adverse violations of assumptions were found.Subsequently, independent t-tests were used to compare the Total ProtectiveFactors scores for the two groups.

Table 3.2a presents the results of this study with the DECA-I and documentsthat there were significant and meaningful differences between the“Identified” and the “Community” samples on all three scales. The meanstandard score differences and other results reported in Table 3.2a indicatethat the ratings of the two groups differ significantly despite the similarity indemographic characteristics (p < .01).

Table 3.1a

Characteristics of the DECA-I Validity Study SampleIdentified Sample Community Sample

Characteristic % %


Age (Months)

Mean 10.6 11.7

SD 5.2 4.5

Gender

Boys 8 53.0% 8 53.0%

Girls 7 47.0% 7 47.0%

Race*

Native American 1 6.7% 1 6.7%

African American 3 20.0% 4 26.6%

Hispanic 4 26.6% 5 33.3%

Caucasian 6 40.0% 7 46.6%

Missing 3 20.0%

*Totals do not add up to 100% due to multiple race


Similarly, Table 3.2b presents the results of this study with the DECA-T anddocuments that there were significant and meaningful differences betweenthe “Identified” and the “Community” samples on all four scales. The meanstandard score differences and other results reported in Table 3.2b strong-ly indicate that the ratings of the two groups differed significantly despite thesimilarity in demographic characteristics (p < .01).

Besides being statistically significant, the means of the two groups on eachinstrument and on each scale differed by approximately one standard deviation (d-ratios range from .75 to 1.52). The d-ratio is a measure of the size of the difference between the mean scores expressed in standarddeviation units. Widely accepted guidelines for interpreting d-ratios (Cohen,1988) in comparing two groups indicate that the magnitudes of .2, .5, and.8 are interpreted as small, medium, and large, respectively. Therefore, theeffect sizes in Tables 3.2a and 3.2b would all be characterized as largeexcept Initiative on the toddler scale, which would be characterized as amedium effect size. These findings provide evidence of the validity of theDECA-I/T scales in discriminating between groups of infants and toddlerswith and without social and emotional concerns.

Table 3.1b

Characteristics of the DECA-T Validity Study SampleIdentified Sample Community Sample

Characteristic % %



GenderBoys 43 62.3% 38 55.1%Girls 26 37.7% 31 44.9%

Race*Native American 8 6.7% 2 2.9%African American 9 20.0% 7 10.1%Hispanic 15 26.6% 19 27.5%Caucasian 45 40.0% 43 62.3%Asian/Pacific Islander 1 20.0% 1 1.4%Other 3 4.3% 2 2.9%

*Totals do not add up to 100% due to multiple race

31Validity

Table 3.2a

Mean T Scores and Difference Statistics for DECA-I Validity Study

Identified Sample Community Sample(n=15) (n=15)

Initiative

Mean 45.3 54.4

SD 8.6 11.8

F Value 6.20***

d-Ratio .89

Attachment/Relationships

Mean 41.6 54.4

SD 9.6 7.2

F Value 18.89***

d-Ratio 1.52

Total Protective Factors

Mean 42.9 53.2

SD 9.3 9.6

t Value* 3.71**

d-Ratio 1.09

* t-test for independent means** p < .01*** p < .001

Examination of Potential Adverse Impact on Minority ChildrenThe contrasted group approach can also be used to show that groups thatdiffer on a variable thought to be irrelevant to the purpose of the instrumentdo not differ on scale scores. To evaluate the appropriateness of the DECA-I/T for use with minority children, we compared the mean scores ofAfrican American and Caucasian children and Hispanic and Caucasianchildren in the standardization sample. The goal was to determine if thesegroups of children received similar ratings on the DECA-I/T. To assess the difference in ratings we compared the means using the d-ratio statistic.It should be noted that d-ratios following non-significant hypothesis testsshould be interpreted as being not statistically significantly different fromzero (Sawilowsky & Yoon, 2002).


Table 3.2b

Mean T Scores and Difference Statistics for DECA-T Validity Study

Identified Sample Community Sample(n=69) (n=69)


Mean 42.6 50.3

SD 9.4 9.6

F Value 31.01***

d-Ratio .81

Initiative

Mean 42.7 50.4

SD 10.7 9.7

F Value 37.42***

d-Ratio .75

Self-Regulation

Mean 41.0 50.5

SD 9.7 9.7

F Value 35.25***

d-Ratio .98


Mean 40.9 50.5

SD 8.9 9.8

t Value* 7.07**

d-Ratio 1.03

* t-test for independent means** p < .01*** p < .001

33Validity

Table 3.3a presents the results of these analyses for the DECA-I. As shownin Table 3.3a, 19 of 12 of the mean score differences were negligible. Two of the remaining three mean score differences would be characterizedas “small” and one “medium.” The average d-ratio when comparing scoresearned by African American and Caucasian children was .09. The averaged-ratio when comparing scores earned by Hispanic and Caucasian childrenwas .26.

Table 3.3a

DECA-I Scale Scores: d-Ratios Comparing Minority and Non-Minority Children

African-American Hispanic vs. Caucasian vs. Caucasian

Teacher Raters

Initiative .08 .11



Parent Raters

Initiative .19 .21




Table 3.3b presents the results of these analyses for the DECA-T. As shownin Table 3.3b, 9 of 16 of the mean score differences were negligible. Six of the remaining mean score differences would be characterized as“small” and one as “medium.” The average d-ratio when comparing scoresearned by African American and Caucasian children was .20. The averaged-ratio when comparing scores earned by Hispanic and Caucasian children was .24.

Table 3.3b

DECA-T Scale Scores: d-Ratios Comparing Minority and Non-Minority Children

African-American Hispanic vs. Caucasian vs. Caucasian

Teacher Raters


Initiative .33 .14



Parent Raters


Initiative .06 .17



35Validity

Individual Prediction The criterion validity of a test can also be determined by examining the ability of scale scores to predict accurately group membership for individ-ual study participants. Therefore, we investigated the extent to which the DECA-I/T scale scores accurately predicted membership in either the identified or the community sample.

For each scale, we predicted that individuals with a T score of less than orequal to 40 would be members of the identified sample, and those withscores above 40 would be members of the community sample. (Recall thatT scores of 40 and below on the protective factor scales indicate Areas ofNeed.) We then compared the accuracy of these predictions with actualgroup membership. Tables 3.4a and 3.4b present the results of this study for infants and toddlers respectively.

There are a number of ways to evaluate the accuracy of the predictionsshown in Tables 3.4a and 3.4b. The first is to examine the sensitivity of thescale scores. Sensitivity is defined as the percentage of individuals in theidentified sample who would be predicted by the scale T scores to be partof that group (i.e., who obtained scale T scores of less than or equal to 40).As shown in Table 3.4a, for the 15 infants in the identified sample, theAttachment/Relationships scale had the highest sensitivity at 46.7%. TotalProtective Factors correctly predicted 40% of the identified sample andInitiative 26.7%.

Another common measure of the accuracy of scale predictions is specificity, or the percentage of individuals in the Community Sample thatwould be predicted to be in that group (i.e., have scale scores of greaterthan or equal to 40). The results in Table 3.4a indicate that the specificity of the DECA-I is quite high. For each scale, 86.7% of the infants in theCommunity Sample had scale T scores greater than 40.

Sensitivity and specificity can be combined to yield the total correct predictions. This is calculated by dividing the number of individuals in thetwo correct prediction categories (Identified Sample infants with T scores lessthan or equal to 40, and Community Sample infants with T scores greaterthan 40) by the total number of individuals in both samples. Using the TotalProtective Factors scale T scores we would achieve a total correct predictionof 63.3%.

Another way to evaluate the predictive validity of an assessment is to examine the positive predictive value, which is the percentage of individu-als receiving a positive score (in the case of the DECA-I, a T score of lessthan or equal to 40) that are part of the identified sample. An examinationof Table 3.4a reveals that six of the eight infants receiving a T score of less


Table 3.4a

Actual and Predicted Group Membership for the DECA-I Validity Study

Identified Sample Community Sample

Characteristic n % n %

Actual Group Membership 15 15

Predicted Group Membership

Initiative

≤40 4 26.7% 2 13.3%

> 40 11 73.3% 13 86.7%


≤40 7 46.7% 2 13.3%

> 40 8 53.3% 13 86.7%


≤ 40 6 40% 2 13.3%

> 40 9 60% 13 86.7%

than or equal to 40 were from the identified sample, resulting in a positivepredictive value of 75%. In other words, based on the results of this study,75% of the time, when an infant receives a Total Protective Factors score thatis in the Area of Need range, that child will be found to have significantbehavioral challenges.

The results for toddlers, shown in Table 3.4b, are even stronger. Sensitivityranged from 40.6% for Attachment/Relationships to 56.6% for TotalProtective Factors. Sensitivity ranged from 79.7% for Attachment/Relationships to 87% for Initiative. Using the Total Protective Factors scale,the total correct classification in this study was 70.3%. Finally, the positivepredictive value for toddlers based on the Total Protective Factors scale was 78%.

37Validity

The results of the individual prediction study indicate that the DECA-I/T canprovide very useful information in identifying infants and toddlers who havesignificant behavioral challenges. It should also be noted that sensitivity and specificity cannot both be maximized—they are a trade off. As oneincreases sensitivity, one necessarily decreases specificity and vice versa.Therefore, the authors of a test have to choose which category of correctprediction to maximize. In the DECA-I/T, we have chosen to maximizespecificity. What this means is that comparatively few young children whodo not have significant emotional and behavioral challenges will achieve

Table 3.4b

Actual and Predicted Group Membership for the DECA-T Validity Study

Identified Sample Community Sample

Characteristic n % n %

Actual Group Membership 69 69

Predicted Group Membership


≤40 28 40.6% 14 20.3%

> 40 41 59.4% 55 79.7%

Initiative

≤40 34 49.3% 9 13.0%

> 40 35 50.7% 60 87.0%

Self-Regulation

≤40 34 49.3% 13 18.8%

> 40 35 50.7% 56 81.2%


≤40 39 56.6% 11 15.9%

> 40 30 43.4% 58 84.1%


low scores on the DECA-I/T. That is, there will be very few false positivefindings. This will help ensure that mental health promotion resources, whichare often quite scarce, will be devoted to those infants and toddlers that truly need them. Caution should be used, since some children will still need additional support and should not be ignored. This underscores theimportance of not rigidly using the recommended cut-score of 40 and considering many sources of information in making important decisionsabout the child.

Construct Validity Convergent ValidityConstruct-related validity ascertains the degree to which the assessmentinstrument measures the theoretical construct of interest. One of the prima-ry methods of contributing evidence of construct validity is to demonstrateconvergence with previously established constructs of a similar nature. In thecase of the DECA-I/T, the construct generally pertains to the extent to which the DECA-I/T captures evidence of resilience versus some other characteristic of infants and toddlers.

Typically, convergence evidence is demonstrated by high correlationsbetween scores on the instrument under investigation and scores on previ-ously established measures of the same construct. Because of the overlap inages between the DECA (LeBuffe & Naglieri, 1999) and the DECA-T, andthe similarity of the factors, it was possible to test the convergence of theconstructs on the two measures.

A sample (n=35) of toddlers of age 2 years to 3 years old was assessedusing both the DECA and the DECA-T. The DECA scales (Initiative, Self-Control, Attachment, and Total Protective Factors) and the DECA-Tscales (Attachment/Relationships, Initiative, Self-Regulation, and TotalProtective Factors) are conceptually the same. Table 3.5 presents the resultsof this study.

These results were examined and all found to be statistically significant, aswould be expected if there were convergence evidence. Thus, these findingsprovide evidence that DECA-T does measure protective factors similarly aswhen compared to the DECA. Furthermore, due to anchor and similar coreitems, albeit differing factor analytic structure, there is a level of confidenceregarding convergence evidence of the DECA-I as well.

39Validity

Table 3.5

Convergent Validity Results (DECA and DECA-T)DECA-T

A/R In SR TPF

AT Pearson Correlation .877(**) .725(**) .646(**) .844(**)N 35 34 33 33

In Pearson Correlation .687(**) .831(**) .639(**) .799(**)N 35 34 33 33

SC Pearson Correlation .637(**) .605(**) .868(**) .712(**)N 35 34 33 33

TPF Pearson Correlation .848(**) .851(**) .769(**) .907(**)N 35 34 33 33

DEC

A

** Correlation is significant at the 0.01 level (2-tailed).

DECA-T FactorsA/R = Attachment/Relationships / In = Initiative / SR = Self-Regulation / TPF = Total Protective Factors

DECA FactorsAT = Attachment / In = Initiative / SC = Self-Control / TPF = Total Protective Factors

The subscales (Attachment/Relationships, Initiative, Self-Regulation) of theDECA-T should not be orthogonal, and all the more so negatively related,which would be counter evidence to their relationship to the same resilien-cy construct. Nevertheless, they should not be perfectly related to each other,which would be counter evidence that they are independent factors of the same construct. Therefore, they should present intercorrelations that are positive and moderately high in magnitude. The intercorrelations in Table 3.5 provide ample evidence of this interrelationship between the subscales.


Protective Factor StudyAn alternative approach to establishing construct validity is to demonstratethat the assessment instrument yields data that are consistent with predic-tions derived from the theory underlying the instrument. This approach wasalso used in demonstrating the construct validity of the DECA-I/T. Protectivefactors are defined as “characteristics that are thought to moderate or bufferthe negative effects of stress and result in more positive behavioral and psychological outcomes in at-risk children than would have possible in theirabsence” (Masten & Garmezy, 1985). Therefore, for similar levels of stressor risk, children with high protective factors as measured by the DECA-I/Tshould have more positive behavioral outcomes. To test this hypothesis,measures of early childhood risk factors and family stress were obtained on56 infants and 109 toddlers. Demographic information on this sample isprovided in Table 3.6.

Table 3.6

Sample Characteristics for the DECA-IT Protective Factor Study

n %

Age

Infants 56 35

Toddlers 109 65

Gender

Male 94 57

Female 71 43

Race

American Indian/Alaska Native 3 2

Asian 5 3

Black/African American 14 8

Native Hawaiian 1 1

White 137 83

Other 4 2

Hispanic Ethnicity 18 11

41Validity

A commonly used approach to measuring stress and risk in children andfamilies is to inventory the major life events that the child has experiencedsuch as death of a parent, homelessness or major illness. An alternativeapproach to measuring stress and risk is to assess daily hassles, which arerepetitive difficulties in daily living such as transportation problems, familyconflict or financial difficulties. Both approaches were used in this study.Parents and family caregivers who provided the DECA-I/T ratings on the165 participants in this study were also asked to complete a questionnairethat included demographic information about the child and his/her family,as well as information about daily hassles and major life events which maycontribute to the child's level of risk. The parents answered each question“Yes” or “No.” The daily hassles and major life events questions on this formwere based on a previously used “Preschool Major Life Events Checklist”((adapted with permission from the Life Events Checklist (Work, Cowen,Parker & Wyman, 1990) and the Sources of Stress Inventory (Chandler,1981) and a “Preschool Daily Hassles Checklist” adapted with permissionfrom the Daily Hassles Scale (Kanner, Coyne, Schaefer & Lazarus, 1981)).

Parents and family caregivers also rated their infant or toddler on theTemperament and Atypical Behavior Scale (TABS; Neisworth, Bagnato,Salvia & Hunt, 1999). The TABS is a questionnaire on which parents ratethe presence of 55 behaviors in their child, marking either “Yes” or “No.”Parents are also invited to mark “Need Help” for behaviors that they feelpose a problem. The TABS is composed of four scales, including Detached,Hyper-sensitive/Active, Underreactive, and Dysregulated. Raw scores arecalculated on each of the four subscales by adding the number of itemsmarked “Yes” and/or “Need Help.” The TABS also includes a summaryscale, the Temperament and Regulatory Index (TRI), which is based on thesum of the item scores. The TRI was used as the dependent variable in thisstudy. The TRI has a mean of 100 and a standard deviation of 15. It shouldbe noted that on the TRI, lower standard scores are associated with moreproblematic behaviors and higher scores are desirable.

Data for infants and toddlers were combined for all analyses in this study.First, the raw scores from the family risk measure were converted to TotalRisk Index T-scores. A median split of the Total Risk Index T-scores was usedto assign the 165 participants in the study to a High Risk Group or a LowRisk Group. Similarly, a median split of the Total Protective Factors T-scorewas used to assign participants to a Low Protective Factors Group or a HighProtective Factors Group. This procedure resulted in four groups: (a) HighRisk-High Protective Factors (n = 40); (b) High Risk- Low Protective Factors(n = 42); (c) Low Risk-High Protective Factors (n = 39); and (d) Low Risk-Low Protective Factors (n = 44). The relationships of Total Risk and TotalProtective Factors scores to the Temperament and Regulatory Index scoresare presented in Table 3.7 and Figure 3.1.


Table 3.7

Mean Temperament and Regulatory Index Scores for Risk and Protective Factor Groups in

DECA Protective Factor Study Low Risk High Risk

Low Protective Factors 96.7 88.0

High Protective Factors 106.3 94.9

Figure 3.1

Mean Temperament and Regulatory Index Scores for Risk and Protective Factor Groups

70

80

90

100

110

High Risk Low Risk

TRI Score

High ProtectiveFactorsLow ProtectiveFactors

43Validity

Consistent with resilience theory, the High Risk-Low Protective Factor grouphad the lowest mean score (x = 88.0). The High Risk- High Protective Factorgroup’s mean score (x = 94.9) was 6.9 points, or nearly half a standarddeviation higher. The Low Risk-High Protective Factor group had the highestmean score, 106.3 more than a full standard deviation higher than theHigh-Risk-Low Protective group mean. These results were examined usinga 2-way ANOVA (Analysis of Variance). Main effects of both Total Risk (F = 19.4, p < .0001 and Total Protective Factors (F = 13.1, p < .00)were found, and there was no interaction (F = 0.4, p = .54). These find-ings indicate that Protective Factors, as measured by the DECA-I/T, doindeed moderate risk. For children at both levels of risk, higher protectivefactors were associated with better outcomes than low protective factors.These findings provide evidence that the DECA-I/T does indeed measureprotective factors related to resilience in infants and toddlers.

SummaryThe studies reported in this chapter, when taken as a whole, provide evidence that the DECA-I/T provides useful information about the social andemotional strengths of infancts and toddlers that can be used to inform practice. In the cross-sectional research studies reported here, protectivefactor scale scores were significantly associated with either the presence orabsence of social emotional concerns for children from both a statistical andpractical perspective. Parents and early care and education professionalsnow have a psychometrically sound measure, which can be used to supporta strength-based system for all infants and toddlers. With the ability to recognize risk and support key protective factors, children will have anincreased potential for school and life success.

The authors of the DECA-I/T and their colleagues at the Devereux EarlyChildhood Initiative are planning a series of multi-year longitudinal researchstudies to clarify further the role of identified protective factors, as measuredby the DECA-I/T, in buffering children from risk and in fostering healthysocial and emotional growth. Until those results are available, however, the DECA-I/T can be utilized with confidence based on the studies reported here.

45

3

References

ReferencesAnastasi, A. (1988). Psychological Testing (6th ed.). New York:Macmillan.

Atkinson, L. (1991). Three standard errors of measurement and theWechsler Memory Scale-Revised. Psychological Assessment, 3, 136-138.

Blom, G. (1958), Statistical Estimates and Transformed Beta-Variables.New York, NY: Wiley.

Bracken, B. A. (1987). Limitations of preschool instruments and standardsfor minimal levels of technical adequacy. Journal of PsychoeducationalAssessment, 5, 313-326.

Chandler, L.A., (1981) The source of stress inventory. Psychology in the Schools, 18 (2), 164-168.

Children’s Defense Fund (2005). The State of America’s Children:Yearbook 2005 Washington, DC: Author.

Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences(2nd ed.). San Diego, CA: Academic Press.

Committee to Develop Standards for Educational and Psychological Testing of the American Educational Research Association, the AmericanPsychological Association, and the National Council on Measurement inEducation (1985). Standards for Educational and Psychological Testing.Washington, DC: American Psychological Association.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

Egeland, Byron. (1997). Risk and resilience in infants and young children.Paper presented at the annual meeting of the American PsychologicalAssociation. Chicago, IL.

Farber, E. A. (1987). Invulnerability among abused and neglected children. In E. J. Anthony and B. J. Cohler (Eds.), The Invulnerable Child(pp. 253-288). New York: Guilford Press.

Gordon-Rouse, Kimberly. (June 1996). Infant and toddler resilience:knowledge, predictions, policy and practice. Paper presented at the Head Start National Research Conference. Washington, DC.

Kaiser, H. F. (1960). The application of electronic computers to factoranalysis. Educational and Psychological Measurement, 20, 141-151.

Kanner, A.D., Coyne, J.C., Schaefer, C., & Lazarus, R.S. (1981).Comparison of two modes of stress management: Daily hassles and upliftsversus major life events. Journal of Behavioral Medicine, 4 (1), 1-37.

LeBuffe, P. A. and Naglieri, J. A. (1999). Devereux Early ChildhoodAssessment Technical Manual. Lewisville, North Carolina, Kaplan Press.

Masten, A., & Garmezy, N. (1985). Risk, vulnerability, and protective factors in developmental psychopathology. In B. Lahey, & A. Kazdin,(Eds.). Advances in Clinical Child Psychology. New York: Plenum Press.

Masten, Ann & Coatsworth, J. (1998). The Development of Competence inFavorable and Unfavorable Environments: Lessons Learned From Researchon Successful Children. American Psychologist. 53, 205-220.

Neisworth, J.T., Bagnato, S.J., Salvia, J., & Hunt, F.M. (1999). TABSManual for the Temperament and Atypical Behavior Scale Early ChildhoodIndicators of Developmental Dysfunction. Baltimore: Paul H. BrooksPublishing Co., Inc.

Nunnally, J. C. (1978). Psychometric Theory (2nd ed.). New York:McGraw-Hill.

Sawilowsky, S. S., & Yoon, J. (2002). The trouble with trivials (p > .05).Journal of Modern Applied Statistical Methods, 1(1), 143-144.

U.S. Department of Commerce, Economics and Statistics Administration,Bureau of the Census (2006). Statistical Abstract of the United States2006: The National Data Book (125th ed.). Washington, DC: Author.

Werner, E. E. (1990). Protective factors and individual resilience. In S. J. Meisels & J. P. Shonkoff (Eds.), Handbook of Early ChildhoodIntervention (pp. 97-116). New York: Cambridge University Press.

Werner, E. E. (2000). Protective factors and individual resilience. In S. J.Meisels & J. P. Shonkoff (Eds.), Handbook of Early Childhood Intervention,Second Edition (pp. 115-132). New York: Cambridge University Press.

Werner, E. E., & Smith, R. S. (1982). Vulnerable but Invincible: A Longitudinal Study of Resilient Children and Youth. New York: McGraw-Hill Book Company.

Werner, E. E., & Smith, R. S. (1992). Overcoming the Odds: High RiskChildren from Birth to Adulthood. Ithaca: Cornell University Press.

Werner, E. E., & Smith, R. S. (2001). Journeys from Childhood to Midlife:Risk, Resilience and Recovery. Ithaca: Cornell University Press.

Work, W.C., Cowen, E.L., Parker, G.R., & Wyman, P.A., (1990). Stress resilient children in an urban setting. Journal of Primary Prevention, 11(1), 3-17.

Devereux Early Childhood Assessment for Infants and Toddlers – Technical Manual46

Devereux Early Childhood Assessment for Infants and ... · The DECA-I/T standardization sample...

Documents

Transcript of Devereux Early Childhood Assessment for Infants and ... · The DECA-I/T standardization sample...