CELF-5 Reliability & Validity - pearsonclinical.com.au Files/CELF-5... · CELF-5A&NZ Reliability...

1 Copyright © 2017. Pearson, Inc., or its affiliates. All rights reserved.

CELF-5A&NZ Reliability and Validity

CELF-5A&NZ

Reliability & Validity

Angela Kinsella-RitterConsultant Speech PathologistPearson Clinical Assessment

19th September 2017

#CELF5ANZ#CELF5

Research Overview

1.Multiple research phases both in USA, Australia and New Zealand

2.Over 4,000 students tested in standardisation and related reliability and validity studies in the US‒ Students tested from March through December 2012‒ Over 450 SLPs across the U.S. participated in

standardisation testing

3. For the standardisation of the CELF-5A&NZ we worked with 184 speech language pathologists and collected data with a sample of more than 1,000 students.

‒ Testing occurred between April 2016 and January 2017

2



Multiple Bias Studies

Multiple phases of objective and subjective reviews of administration directions, cues, test items, and te st formats

• Assessment/bias experts examined test items for potential bias related to• Socioeconomic status• Race/Ethnicity• Gender• Culture• Region

• Clinicians in the field provided feedback about students’ responses and engagement in test tasks

• Statistical analysis of bias verified or refuted subjective bias concerns

3

A Diverse Standardisation Sample: Age and Gender

4



A Diverse Standardisation Sample: Parent Education Level

5

A Diverse Standardisation Sample: Region

6



Sample Distribution

7

The normative sample was not specifically stratified based on indigenous status;the recruitment of Aboriginal and Torres Strait Islander and New Zealand Māoriparticipants was incidental and in line with the general Australian and NewZealand population. A total number of 30 Aboriginal and Torres Strait Islandersand 28 New Zealand Māori were included in the normative sample; 4.2% of theAustralian sample and 25.5% of the New Zealand sample respectively. Thesenumbers were included to correspond with the 5% representation of Aboriginaland Torres Strait Islander and 25% New Zealand Māori children within theAustralian and New Zealand population respectively. (CELF-5A&NZ Examiner'sManual, p. 17)

A Diverse Standardisation Sample: Ancestry/Ethnicity

8



Raw Score Means and SDs

9

Table 7.2 Australia and New Zealand Test Raw Score Means and Standard Deviations

SC=sentence comprehension, LC=linguistic concepts, WS=word structure, WC=word classes, FD=following directions, FS=formulated sentences, RS=recalling sentences, USP=understanding spoken paragraphs, WD=word definitions, SA=sentence assembly, SR=semantic relationships, PP=pragmatics profile, RC=reading comprehension, and SW=structured writing.

Raw Score Means and SDs (cont.)

10

Table 7.2 Australia and New Zealand Test Raw Score Means and Standard Deviations



Reliability 101

1.How confident are you in the accuracy of a test s core?

2.Reliability = accuracy, consistency and stability of test scores across situations

3.True Score = Observed Score + Error‒Errors are systematic and random‒ Random errors are statistical fluctuations in the measured data due to the

precision limitations of the data collection process such as variability across examiners and students’ performance.

‒ Systematic errors , by contrast, are reproducible inaccuracies such as slight differences in scoring FS (among examiners and or from one test time to the next).

11

Internal Consistency &Test-retest

1. Internal consistency: estimates how consistently the items of the test measure one construct (homogeneity)‒Split-half method (Spearman-Brown correction): correlation between

the total scores of two half-tests

2.Test-retest stability: correlation between test a nd retest scores. ‒Time interval between the test and retest is as short as possible.

12



Test Reliability Coefficients

CELF-5A&NZ TestAverage Reliability Coefficients

(rxx) across target ages

Sentence Comprehension .82 GoodLinguistic Concepts .87 Good

Word Structure .86 GoodWord Classes .88 Good

Following Directions .88 GoodFormulated Sentences .85 GoodRecalling Sentences .92 Excellent

Understanding Spoken Paragraphs .81 Good

Word Definitions .85 GoodSentence Assembly .91 Excellent

Semantic Relationships .88 GoodPragmatics Profile .99 Excellent

Reading Comprehension .82 GoodStructured Writing .85 Good

13

Table 8.1 Australia and New Zealand Internal Consistency Rel iability Coefficients of Test Scores

Index Score Reliability Coefficients

CELF-5A&NZ Index ScoresAverage Reliability

Coefficients (across target ages)

Core Language Score .95 Excellent

Receptive Language Index .93 Excellent

Expressive Language Index .94 Excellent

Language Content Index .94 Excellent

Language Structure Index .95 Excellent

Language Memory Index .94 Excellent

14

Table 8.1 Australia and New Zealand Internal Consistency Rel iability Coefficients of Composite Scores



Reliabilities for Clinical Groups

Clinical Group

Test

Language Disorder (n=166)

Learning Disability

(Reading & Writing) (n=69)

Autism Spectrum Disorder

(n=66)

Average r xx

Sentence Comprehension

.94 Excellent -- .96 Excellent .95 Excellent

Linguistic Concepts .96 Excellent -- .98 Excellent .97 Excellent

Word Structure .93 Excellent -- .94 Excellent .94 Excellent

Word Classes .96 Excellent .92 Excellent .97 Excellent .95 Excellent

Following Directions .96 Excellent .90 Excellent .98 Excellent .96 Excellent

Formulated Sentences .97 Excellent .89 Good .96 Excellent .95 Excellent

Recalling Sentences .98 Excellent .92 Excellent .97 Excellent .96 Excellent

15

Table 8.2 Internal Consistency Reliability Coefficients of Te st Scores for Clinical Groups

Clinical Group

Test

Language Disorder (n=166)

Learning Disability

(Reading & Writing) (n=69)

Autism Spectrum Disorder

(n=66)

Average r xx

Understanding Spoken

Paragraphs.81 Good .75 Acceptable .91 Excellent .84 Good

Word Definitions .87 Good .91 Excellent .95 Excellent .92 Excellent

Sentence Assembly

.92 Excellent .94 Excellent .97 Excellent .95 Excellent

Semantic Relationships

.88 Good .89 Good .96 Excellent .92 Excellent

Pragmatics Profile .99 Excellent .99 Excellent .99 Excellent .99 Excellent

Reading Comprehension

.93 Excellent .86 Good .93 Excellent .91 Excellent

Reliabilities for Clinical Groups (cont.)

16

Table 8.2 Internal Consistency Reliability Coefficients of Te st Scores for Clinical Groups



Standard Error of Measurement (SEM) & Confidence Interval

17

Table 8.3 Australian and New Zealand Standard Errors of Measu rement for Test and Composite Scores

Standard Error of Measurement (SEM) & Confidence Interval (cont.)

18

Table 8.3 Australian and New Zealand Standard Errors of Measu rement for Test and Composite Scores



Determining Test Confidence Intervals for Scaled Scores

19

CELF-5A&NZ Examiner's Manual, pp 145; Appendices A & B

Determining Percentile Ranks

and Percentile Rank

Confidence Intervals

20

Appendix DCELF-5A&NZ Examiner's Manual, p. 314



Test-retest Stability

Test Retest Stability (n= 137)

Test-retest interval: 7-46 days

Mean: 19 days

21

Table 8.4 Stability Coefficients for the Test and Composite S cores

Testing for Re-evaluationConsider these factors when making retest decisions:

• Retesting should be conducted when, in the opinion of the examiner, the student is not likely to remember the test items and/or his or her responses when tested previously. ‒ If retesting is required prior to this time, changes in performance should

be interpreted in comparison to mean differences between original and retest scores obtained in the test-retest study (see Chapter 8, Table 8.4 ).

• Retesting should be conducted when the examiner thinks the child has made progress since the previous test administration.

• Retesting can be conducted when the student’s age at testing requires the next-age norms table to score (6 month intervals for 5 and 6 year olds and 12 month intervals for students 7 years and older)

• Retesting can be conducted when other factors negatively affecting the student’s performance (e.g., illness, inattention) cause you to question the accuracy of previous test results.

CELF-5A&NZ Examiner's Manual, pp 33-34. 22



Inter-scorer Agreement

Inter-scorer agreement Reliability

Word Structure .99

Formulated Sentences .95

Word Definitions .91

Structured Writing .96

23

Score Differences

Interpretation of performance1.Examine if difference is statistically significant

‒ Reflection of Standard Error

2. Examine if difference is clinically significant (rare ≤ 10-15%)

24

Discrepancy Comparison Example

CELF-5A&NZ Examiner's Manual, p. 148



Validity 101

1.How well can your test results predict the presence of a disorder (or, predict one’s skill)?

2.Validity is demonstrated through evidence supporting a test’s interpretations and uses. ‒Validity Evidence: the degree to which specific data,

research, or theory supports that: 1. A test measures the concepts it’s supposed to measure2. The test is applicable to its intended population

25

Evidence Based on Test Content

Validity evidence related to test content ‒Content Relevance : when the content areas being

measured are accepted as relating to the proposed construct‒Content Coverage : when the content areas measured by

the test are accepted to be an adequate sampling of these areas (also developmentally appropriate)

Validity evidence includes: literature review; users’ feedback; and expert review.

1. CELF-5A&NZ is designed to reflect the development of language abilities of students ages 5–21 years.

2. The goal of the content was to make sure that the tests and test items on the test were able to adequately sample all the different language domains.

3. Test developers paid particular attention to content that reflects the complex linguistic processes of adolescents and young adults

26



Convergent Validity: Intercorrelation Studies

27

Table 9.4 Australian and New Zealand Intercorrelations of Nor m-Referenced Tests and Composites

Relationship with Other Variables (CELF-4)

28

Table 9.4 Means, Standard Deviations and Correlation Coeffici ents Between CELF-5 and CELF-4 Test and Composite Scores



CELF-5 A&NZ to CELF-4 Australian (cont.)

29

Language Disorder Study

30



Learning Disorder (Reading and/or Writing)

31

Autism Spectrum Disorder with Language Impairment

32



Clinical Applications: HI

33

• There are additional considerations required for assessing children diagnosed with hearing loss.

• An in-depth study was undertaken with this population to investigate CELF–5A&NZ

performance of children with hearing loss who received early intervention.

• The National Acoustic Laboratories collaborated with Pearson Clinical Assessment in Australia to collect new data.

• Consistent with current literature, it was expected that the effect size of hearing loss on language development would be reduced, compared to previous populations who did not have access to early intervention.

• The study aims were to: a. determine CELF–5A&NZ performance of children with hearing loss who have

received early intervention, andb. to outline the standardised administration requirements for this population.


34

• The sample comprised 30 children diagnosed with hearing loss ranging from mild to severe degrees who received hearing aids before 6 months of age

• Another 30 children (13 females, 17 males) with severe or profound hearing loss who first received a cochlear implant before 2 years of age.

• All children had nonverbal cognitive abilities within the normal range.

• All were regular users of personal hearing devices that are routinely managed by audiologists.

• On average, children’s performance were within 1 SD below the normative mean.

• The effect sizes across Core Language Score and Index scores ranged from small to moderate (0.42 to 0.83).

• Index Score Strengths: LSI | Weakness: LMI• Test Scores: WD and SR showed the largest effect sizes (0.93 to 0.95).



Clinical Group Studies: HI

35


36




37

• Across all scores, the effect size of hearing loss on language abilities of children using cochlear implants more than doubled that of children using hearing aids.

• This suggests that the impact increases with the degree of hearing loss.

• Children with hearing loss who use either hearing aids or cochlear implants had Structured Writing scores that did not significantly differ from those of typically hearing peers.

Guide to Administration for Hearing Impaired Populations CELF-5A&NZ Examiner’s Manual, p. 248

Sensitivity & Specificity 101

• Sensitivity: The probability that someone who has the “condition” will test positive for it.

• Specificity: The probability that someone who does not have the “condition” will test negative.

Errors

• False Positive: A student who is falsely identified as having a condition or disorder (over-referrals)

• False Negative: A student with a condition or disorder who is not correctly identified by a test (the most serious error).

38



CELF-5A&NZ Diagnostic Accuracy

39

Table 9.8 Australian and New Zealand Clinical Validity Studie s

40

It is important to consider all the information about the student in conjunction with the CELF-5A&NZ results.

Additional information may include:1. Developmental history 2. Medical history 3. Academic history4. Comparison of the student’s academic progress to peers of a similar

cultural background with English as a second language 5. Parent interview 6. Observations 7. Input from other professionals8. Language sampling9. Dynamic (how easily a student learns new tasks in the learning

environments), criterion-referenced, or norm-referenced assessments.

CELF-5A&NZ Results



CELF-5 vsCELF-5A&NZ in reports

Typing superscript in Word

Enable superscript: Click the x₂ button in the Font group on the Home tab.Select the text that you want to format as superscript.Do one of the following: On the Home tab, in the Font group, click Subscript. Or press CTRL+=. To undo the formatting, click the Superscript button again, or repeat the keyboard shortcut. (or Control Z)

Typing Superscript Text in Mac OS XPull down the “Format” menu and go to “Font”Select the “Baseline” submenu and choose either “Superscript”Type the desired text to be superscripted, then go back to the same menu and choose “Use Default” to return to normal baseline text or uncheck “Superscript” box.

42

Angela Kinsella-Ritter

Consultant Speech Pathologist

[email protected]

D: 02 9454 2209

M: 0408 511 110

Client Services

1800 882 385 (Australia)

0800 942 722 (NZ)

www.pearsonclinical.com.au

Contact Details

43

CELF-5 Reliability & Validity - pearsonclinical.com.au Files/CELF-5... · CELF-5A&NZ Reliability...

Documents

Transcript of CELF-5 Reliability & Validity - pearsonclinical.com.au Files/CELF-5... · CELF-5A&NZ Reliability...