Parsimonious Prediction of Wechsler Memory Scale,

This article was downloaded by: [University Library Utrecht]On: 23 September 2013, At: 06:10Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: MortimerHouse, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Clinical and ExperimentalNeuropsychologyPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/ncen20

Parsimonious prediction of Wechsler MemoryScale, Fourth Edition scores: Immediate anddelayed memory indexesJustin B. Miller a , Bradley N. Axelrod b , Lisa J. Rapport a , Scott R. Millis c , SarahVanDyke b , Christian Schutte b & Robin A. Hanks ca Department of Psychology, Wayne State University, Detroit, MI, USAb Psychology Section, John D. Dingell Department of Veterans Affairs MedicalCenter, Detroit, MI, USAc Department of Physical Medicine and Rehabilitation, Wayne State University,School of Medicine Detroit, MI, USAPublished online: 02 Mar 2012.

To cite this article: Justin B. Miller , Bradley N. Axelrod , Lisa J. Rapport , Scott R. Millis , Sarah VanDyke ,Christian Schutte & Robin A. Hanks (2012) Parsimonious prediction of Wechsler Memory Scale, Fourth Edition scores:Immediate and delayed memory indexes, Journal of Clinical and Experimental Neuropsychology, 34:5, 531-542, DOI:10.1080/13803395.2012.665437

To link to this article: http://dx.doi.org/10.1080/13803395.2012.665437

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the Content)contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensorsmake no representations or warranties whatsoever as to the accuracy, completeness, or suitabilityfor any purpose of the Content. Any opinions and views expressed in this publication are the opinionsand views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy ofthe Content should not be relied upon and should be independently verified with primary sources ofinformation. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands,costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly orindirectly in connection with, in relation to or arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Any substantial orsystematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distributionin any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found athttp://www.tandfonline.com/page/terms-and-conditions

JOURNAL OF CLINICAL AND EXPERIMENTAL NEUROPSYCHOLOGY2012, 34 (5), 531542

Parsimonious prediction of Wechsler Memory Scale,Fourth Edition scores: Immediate and delayed memory

indexes

Justin B. Miller1, Bradley N. Axelrod2, Lisa J. Rapport1, Scott R. Millis3,Sarah VanDyke2, Christian Schutte2, and Robin A. Hanks3

1Department of Psychology, Wayne State University, Detroit, MI, USA2Psychology Section, John D. Dingell Department of Veterans Affairs Medical Center, Detroit,MI, USA3Department of Physical Medicine and Rehabilitation, Wayne State University, School of MedicineDetroit, MI, USA

Research on previous versions of the Wechsler Memory Scale (WMS) found that index scores could be predictedusing a parsimonious selection of subtests (e.g., Axelrod & Woodard, 2000). The release of the Fourth Edition(WMSIV) requires a reassessment of these predictive formulas as well as the use of indices from the CaliforniaVerbal Learning TestII (CVLTII). Complete WMSIV and CVLTII data were obtained from 295 individuals.Six regression models were fit using WMSIV subtest scaled scoresLogical Memory (LM), Visual Reproduction(VR), and Verbal Paired Associates (VPA)and CVLTII substituted scores to predict Immediate MemoryIndex (IMI) and Delayed Memory Index (DMI) scores. All three predictions of IMI significantly correlated withthe complete IMI (r = .92 to .97). Likewise, predicted DMI scores significantly correlated with complete DMI(r = .92 to .97). Statistical preference was indicated for the models using LM, VR, and VPA, in which 97% and96% of the cases fell within two standard errors of measurement (SEMs) of full index scores, respectively. Thepresent findings demonstrate that the IMI and DMI can be reliably estimated using two or three subtests fromthe WMSIV, with preference for using three. In addition, evidence suggests little to no improvement in predictiveaccuracy with the inclusion of CVLTII indices.

Keywords: Memory; Wechsler Memory Scale; Short form; Parsimonious prediction; Concordance correlationcoefficient; Immediate memory; Delayed memory.

The release of the Fourth Edition of the WechslerMemory Scale (WMSIV; Wechsler, 2008b)introduces new measures (e.g., Designs, SpatialAddition, Symbol Span) along with the returnof several familiar measures that have roots thatreach back to the inception of the scale from1945 (e.g., Visual Reproduction, Logical Memory,Verbal Paired Associates). The WMSIV has alsointroduced a new degree of flexibility by allowing

This research was supported by a grant from the National Institute on Disability and Rehabilitation Research, Department ofEducation (H133A080044), the Del Harder Foundation, and the Wayne State University Graduate School. The contents of this studydo not necessarily represent the policies of the funding agencies. Justin B. Miller is now at the Semel Institute for Neuroscience andHuman Behavior, University of California, Los Angeles, CA, USA.Address correspondence to Bradley N. Axelrod, John D. Dingell Department of Veterans Affairs, Psychology Section, Detroit, MI,

USA (E-mail: [email protected]).

for substitution of scores from the Second Editionof the California Verbal Learning Test (CVLTII;Delis, Kramer, Kaplan, & Ober, 2000). Specifically,clinicians who chose to utilize the CVLTII as ameasure of verbal list learning instead of VerbalPaired Associates can generate scaled scores bysubstituting the Trials 15 t score and long-delayfree-recall z score from the CVLTII for the imme-diate and delayed memory recall portions of Verbal

2012 Psychology Press, an imprint of the Taylor & Francis Group, an Informa business

http://www.psypress.com/jcen http://dx.doi.org/10.1080/13803395.2012.665437

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013

532 MILLER ET AL.

Paired Associates, respectively. These scores werederived from a subset of 380 participants in thenormative sample who were administered both theCVLTII and the full WMSIV; for greater detailregarding the derivation of these substituted scaledscores, the reader is referred to the technical andinterpretive manual for the WMSIV (Wechsler,2008a).The structure of the WMSIV index scores has

changed in the new edition. Whereas completeadministration of the Third Edition (WMSIII)generated eight separate index scores, some ofwhich have been retained in the Advanced ClinicalSolutions supplement, the standard WMSIV hassimplified this profile and calculates just fiveindexes. Specifically, separate index scores are gen-erated for auditory memory (AMI), visual memory(VMI), immediate memory (IMI), delayed mem-ory (DMI), and visual working memory (VWMI).The most notable change is that there is no longercalculation of a General Memory Quotient. Thenumber of available subtests has also been reducedfrom 17 to 11 for the WMSIV. According to themanual, complete administration of all 11 subtestsin the WMSIV requires approximately 83 minutesat the 50th percentile of general test takers and126 minutes at the 90th percentile in the specialgroup sample. However, our experiences suggestthat these estimates may be low and that more real-istic estimates are that the complete battery takes anaverage of approximately 120 minutes when all sub-tests are administered to individuals with cognitiveimpairment.Research on the Revised (WMSR) and Third

(WMSIII) editions of the Wechsler MemoryScale found that index scores could be pre-dicted using a parsimonious selection of sub-tests (Axelrod & Woodard, 2000; Woodard &Axelrod, 1995). Specifically, three-subtest shortforms of the WMSR resulted in 94% and 97%of the sample obtaining scores within 6 points ofimmediate and delayed summary scores, respec-tively. Similarly, three-subtest short forms for theWMSIII resulted in 92% and 96% of the samplefalling within 6 points of immediate and delayedrecall, respectively.The release of the Fourth Edition (WMSIV)

requires a reassessment of these predictive formulasas well as the substitution of CVLTII indices. Theaim of the present study was to determine the extentto which a parsimonious selection of the 11 avail-able subtests reliably predicted the composite indexscores generated by full administration. Emphasiswas placed on predicting the IMI and DMI fromthree of the four primary subtests used in calcula-tion of the IMI and DMI and those measures that

were carried over from the previous WMS version(i.e., Logical Memory, Visual Reproduction, andVerbal Paired Associates). Although it has beensuggested that short forms may be inappropriatewhen specific measures for brief assessment exist(Kaufman & Kaufman, 2001), the reality remainsthat complete administration of an entire batteryof memory measures is not always feasible for anynumber of reasons (e.g., limitations on resources,patient fatigue, third-party payers). This studysought to provide an option for those situations inwhich a brief assessment is warranted.Of additional interest was the difference in

predicted index scores when CVLTII indices weresubstituted for Verbal Paired Associates. Giventhat the Logical Memory and Visual Reproductionsubtests are nearly identical, and Verbal PairedAssociates is highly similar in the third and fourtheditions, it was predicted that the present studywould yield findings similar to those by Axelrodand Woodard (2000) and Woodard and Axelrod(1995), in that the IMI and DMI could be reliablypredicted using less than the full battery of sub-tests included in the WMSIV. Moreover, it wasexpected that predicted values from these equationswould demonstrate adequate reliability to beconsidered useful in clinical settings. Although itis assumed that the intercorrelations among theindividual subtests and the respective index scoresare quite high, such a finding adds additionalsupport to the assertion that the measure caneffectively be shortened without compromising theintegrity of the IMI or DMI.

METHOD

Participants

Archival data for the present study were obtainedfrom the John D. Dingell Department of VeteransAffairs (VA) Medical Center in Detroit, Michiganand the Rehabilitation Institute of Michigan (RIM)in Detroit. The VA sample was drawn from theclinical archives of persons evaluated on an out-patient basis during the course of routine clinicalcare. The primary diagnostic concerns includeddementia, mood disorders, traumatic brain injury,learning disability, attention deficit hyperactivitydisorder, and other medical concerns, such as cog-nitive declines related to Parkinsons disease andvascular events.The sample from RIM was collected as part

of a research study investigating the ability ofthe WMSIV to differentiate cognitive impair-ment resulting from actual moderate to severe

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013

WMSIV INDEX SCORE PREDICTION 533

traumatic brain injury (TBI) from feigned cognitiveimpairment. This sample included persons withbona fide TBI (n = 60) recruited from thepool of participants currently enrolled in theSoutheastern Michigan Traumatic Brain InjurySystem (SEMTBIS) and healthy adults coachedto feign cognitive impairment (n = 64). For adetailed description of this sample of participants,and the coaching procedures, the reader is referredto Miller et al. (2011). Although cases simulatingcognitive impairment do not represent actual cog-nitive functioning, the present study focused on thepsychometric properties of the index scores ratherthan assessment of actual cognitive functioning.Furthermore, including individuals feigning cogni-tive impairment helps to ensure that an adequaterange of scores is sampled and that index score cal-culation is independent from clinical presentation.The sample included 295 cases; 124 from RIM

and 171 from the VA. Mean age for the total sam-ple was 43.4 years (SD = 13.3; range = 18 to 65),and mean years of education were 12.8 (SD = 2.0;range = 8 to 21). The sample was predomi-nantly men (90.2%) with nearly equal representa-tions of African Americans (48.5%) and Caucasians(49.8%). Fewer than 2% of the sample reportedtheir ethnicity as Arabic, Hispanic, or Latino.

Materials

For all cases, the entire WMSIV and CVLTIIwere administered and scored according tothe standardized procedures. The sample fromRIM also completed several stand-alone symp-tom validity measures (e.g., Test of MemoryMalingering; Tombaugh, 1996) and embeddedmeasures of response bias (e.g., Reliable DigitSpan; Greiffenstein, Gola, & Baker, 1995). The VAsample also completed the full WMSIV as part ofa comprehensive assessment of cognitive function-ing in the context of a standard outpatient clinicalevaluation.

Procedure

This study was reviewed and approved by theHuman Investigation Committee at Wayne StateUniversity and the Veterans Affairs ClinicalInvestigation Committee. Persons recruited for thestudy conducted at RIM were contacted via tele-phone (Southeastern Michigan Traumatic BrainInjury System participants) or responded to printedadvertisements and posted fliers (simulator par-ticipants); all persons in this sample who agreed

to participate were evaluated at RIM or WayneState University. Testing was completed in a sin-gle session, and participants were compensated $30.Because the archival data from the VA samplewere collected as part of routine clinical care, theseindividuals were neither recruited nor compensatedfor participating. All tests were administered bya licensed psychologist, trained neuropsychologytechnician, or advanced graduate students in clin-ical psychology.The WMSIV and CVLTII were administered

independently; the order of test administration var-ied among participants in both samples to pre-clude the introduction of order effect bias. Allmeasures were scored according to standardizedinstructions provided in the test manuals. PrimaryWMSIV index scores were calculated using boththe standard calculation method using all four pri-mary subtests and the substitution of the Total15 t score (CVLTT) from the CVLTII forVerbal Paired Associates (VPA) Immediate andLong-Delay Free-Recall z score (CVLTZ) for VPAdelayed.

Analyses

The primary analytic strategy entailed use of linearregressions with one of two index scores as the out-come and a combination of several subtest scoresas predictors. The primary index scores of interestincluded the Immediate Memory Index (IMI) andthe Delayed Memory Index (DMI). The individ-ual predictors included scaled scores from theVisual Reproduction (VR), Logical Memory (LM),and Verbal Paired Associates (VPA) as well asscaled scores generated from substituting CVLTIIindices for VPA. The Design Memory subtestwas not included as part of the analysis to focusthe present study on those subtests that are mostfrequently administered. Six separate models wereevaluated, three predicting IMI(a) VR and LM(IMI2); (b) VR, LM, and VPA (IMI3W); and (c)VR, LM, and CVLTT (IMI3C)and three com-parable models predicting DMI(a) VR and LM(DMI2); (b) VR, LM, and VPA (DMI3W); and(c) VR, LM, and CVLTZ (DMI3C). Predictedvalues were calculated using the unstandardizedregression coefficients from each resulting equation.Model performance was evaluated by compar-

ing R-squared values, and the contribution ofindividual predictors was assessed based on stan-dardized regression coefficients. Predicted valuesfrom each regression equation were correlatedwith observed index score values using Pearsoncorrelation coefficients to determine the association

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013

534 MILLER ET AL.

between actual and predicted values. The percent-age of predicted index scores that fell within 1 stan-dard error of measurement (SEM) of observedindex scores was computed. As reported in the tech-nical and interpretive manual for the WMSIV, theaverage SEM for the adult battery (i.e., 1669 yearsof age) is 3.53 points for the IMI and 3.71 points forthe DMI (Wechsler, 2008a). Thus, for the presentstudy, the SEM was rounded to 4 points for bothindexes. To provide a clinically relevant metric, thepercentage of scores falling within a full standarddeviation (SD; 15 index points) was also com-puted. The Pearson correlations between predictedWMSIV scores and obtained WMSIV scoreswere compared across the different models, andthe differences in correlations between models wereevaluated using t tests following r-to-z transforma-tions of the correlations.To evaluate the rate of agreement between

the original value and predicted values, LinsConcordance Correlation Coefficient (CCC; King& Chinchilli, 2001; Lin, 1989) was also calculated.The CCC is a method for measuring agreementon a continuous measure obtained by two personsor methods; it measures both precision and accu-racy to determine how far the observed data deviatefrom the line of perfect concordance (i.e., the lineat 45 on a square scatter plot). Lins coefficient isexpressed as the product of the Pearson correlationcoefficient, which is a quantification of precision(i.e., the distance of each data point from the fit-ted model), with the added incorporation of a biascorrection factor (Cb) that measures accuracy (i.e.,the distance of the resulting model from the opti-mal 45 diagonal originating at the origin; King& Chinchilli, 2001; Lin, 1989). Unlike using stan-dard Pearson statistics or paired-sample t tests, theCCC is capable of detecting systematic bias thatmay be present among comparisons (e.g., over orunder prediction) and is thus a preferred statisticfor evaluating agreement between continuous vari-ables. Bias correction factor values that differ from1.00 indicate the presence of bias, with greater devi-ation indicating stronger bias (Lin, 1989). Values ofthe CCC can range from 1.0, indicating poor con-cordance, to 1.0, which would be observed in thepresence of perfect agreement between values.

RESULTS

Descriptive statistics for predicted index score val-ues for all six models are presented in Table 1,and model summary statistics for the three IMIand three DMI prediction equations are presentedin Table 2. All regression coefficients in each of

the models were significant at the p < .001 level.The frequencies of score discrepancies between pre-dicted and actual values based on the SEM andstandard deviation (SD) are presented in Table 3;these scores have been provided in order to eval-uate whether any of the predictive models sys-tematically over- or underpredict observed scores.The IMI3W model was statistically significant,F(3, 288) = 1,832.18, p < .001, with an R2 of.95. Visual Reproduction Immediate (VR1) gener-ated the largest standardized regression coefficient,and LM1 generated the smallest. This model alsogenerated predicted scores that had the tightest dis-tribution of the immediate memory models, withnearly two thirds of scores falling within 1 SEM ofactual values and 96% of scores within 2 SEMs (i.e., 8 index score points). The IMI3C model wasalso significant, F(3, 279) = 634.19, p < .001, withan R2 of .87. Visual Reproduction Immediate againhad the largest coefficient; however, the regressioncoefficient for CVLTT was the smallest in thismodel. In relation to actual values, 44.9% of pre-dicted values fell within 1 SEM and 79.9% ofscores within 2 SEMs of actual values. The mostparsimonious model, IMI2, produced the smallestR2 value of the three immediate memorymodels butwas still statistically significant, F(2, 290) = 817.14,p < .001. This model produced the largest discrep-ancies between observed and predicted values withonly 40% of scores within 1 SEM and 23.3% ofscores differing by more than 2 SEMs. Among indi-viduals with an observed IMI that was within 1 SDof the mean in either direction (n = 125), the aver-age prediction discrepancy for the three models was0.70 points (SD = 5.4, range = 12.1 to 21.7). Forindividuals with an observed IMI between 1 and2 SDs below the mean (n = 86), predicted valuesaveraged 0.20 points lower than observed values(SD = 5.8, range = 16.49 to 17.11). Among indi-viduals with observed IMI values more than 2 SDsbelow the mean (n = 73), the average discrep-ancy was 1.80 points lower than observed values(SD = 5.8, range = 17.54 to 10.64). Nine partici-pants had observed IMI scores that were more than1 SD above the mean.Predicting the DMI yielded similar results to the

three models for IMI, with the DMI3W demon-strating the largest R2. This model was significantoverall, F(3, 287) = 1,727.41, p < .001. Predictedvalues from this model most closely approximatedthe distribution of observed scores, with 71.1% ofscores falling within 1 SEM and fewer than 4%of scores falling more than 2 SEMs from actualvalues. As with the immediate memory models,replacing Verbal Paired Associates Delayed Recall(VPA2) with CVLTZ for the DMI3C produced a

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013


TABLE 1Descriptive statistics for full and short-form predicted WMSIV Immediate and Delayed Memory Indexes

Model M SD Range of difference

Actual IMI 83.0 18.0 NAIMI2 (LM1, VR1) 82.9 16.7 17.5 to 21.7IMI3C (LM1, VR1, CVLTII1)a 83.2 16.9 16.4 to 20.8IMI3W (LM1, VR1, VPA1) 82.9 17.6 10.7 to 14.4

Actual DMI 82.6 16.3 NADMI2 (LM2, VR2) 82.4 15.0 16.1 to 17.5DMI3C (LM2, VR2, CVLTII2)b 82.7 15.0 15.4 to 16.3DMI3W (LM2, VR2, VPA2) 82.6 15.8 10.7 to 11.8

Note. WMSIV = Wechsler Memory ScaleFourth Edition; IMI = Immediate Memory Index; DMI = Delayed Memory Index;LM1 = Logical MemoryImmediate; LM2 = Logical MemoryDelayed; VR1 = Visual ReproductionImmediate; VR2 = VisualReproductionDelayed; CVLTII1=California Verbal Learning TestIIImmediate; CVLTII2=California Verbal Learning TestIIDelayed; VPA1 = Verbal Paired AssociatesImmediate; VPA2 = Verbal Paired AssociatesDelayed.aIMI3C employed a substitution of CVLTII Trials 15 t score for Verbal Paired Associates (Immediate).bDMI3C employed a substitution of CVLTII Long-Delay Free-Recall z score for Verbal Paired Associates (Delayed).

TABLE 2Regression summary statistics for short-form prediction models of Immediate and Delayed Memory Indexes

Predictors Model summary

Model B SEB R R2 SEE

IMI2Logical Memory 1 2.78 0.14 0.54 .92 .85 7.00Visual Reproduction 1 2.61 0.13 0.52Constant 43.76 1.05

IMI3CLogical Memory 1 2.32 0.15 0.44 .93 .87 6.49Visual Reproduction 1 2.48 0.13 0.49Substitution 1 1.13 0.17 0.18Constant 39.48 1.18

IMI3WLogical Memory 1 1.83 0.09 0.35 .98 .95 4.04Visual Reproduction 1 2.25 0.08 0.45Verbal Paired Associates 1 2.16 0.09 0.39Constant 36.77 0.67

DMI2Logical Memory 2 2.76 1.22 0.57 .92 .84 6.49Visual Reproduction 2 2.55 0.12 0.55Constant 46.55 0.99

DMI3CLogical Memory 2 2.39 0.13 0.49 .93 .86 6.10Visual Reproduction 2 2.35 0.13 0.50Substitution 2 0.85 0.13 0.18Constant 44.43 1.02

DMI3WLogical Memory 2 1.98 0.08 0.41 .97 .95 3.75Visual Reproduction 2 2.02 0.07 0.43Verbal Paired Associates 2 1.94 0.08 0.39Constant 40.43 0.63

Note. SEB = Standard error of beta weights; SEE = Standard error of the estimate; IMI = Immediate Memory Index; IMI3C =predicted IMI using LM, VR, and California Verbal Learning Test, 2nd edition (CVLTII); IMI3W = predicted IMI using LM,VR, and Verbal Paired Associates (VPA); DMI = Delayed Memory Index; DMI2 = predicted DMI using LM and VR; DMI3C = predicted DMI using LM, VR, and CVLTII; DMI3W = predicted DMI using LM, VR, and VPA. Substitution refers to useof California Verbal Learning TestII indices in place of verbal paired associates.

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013

536 MILLER ET AL.

TABLE 3Accuracy of predicted WMSIV index scores in relation to the standard error of measurement and standard deviation of the

indexes

Predicted IMI Predicted DMI

Range of discrepancy IMI2 IMI3C IMI3W DMI2 DMI3C DMI3W

Standard error of measurementa

(Predicted > actual) 8 points 12.6 9.9 1.0 9.6 10.3 1.4(Predicted > actual) < 4 points 16.6 18.7 15.8 19.5 14.9 12.7Within 1 SEM 40.0 44.9 65.4 43.8 49.8 71.1(Predicted < actual) < 4 points 19.3 16.3 15.4 15.1 13.9 12.4(Predicted < actual) 8 points 10.8 10.2 2.4 12.0 11.0 2.4

Standard deviation(Predicted > actual) 15 points 1.4 0.7 0.0 0.7 0.0 0.0(Predicted > actual) within 15 points 49.7 47.1 52.7 49.8 50.7 48.8(Predicted < actual) within 15 points 47.6 51.4 47.3 49.1 48.9 51.2(Predicted < actual) 15 points 1.4 0.7 0.0 0.3 0.4 0.0

Note. WMSIV = Wechsler Memory ScaleFourth Edition; IMI = Immediate Memory Index; IMI2 = predicted IMI using LogicalMemory (LM) and Visual Reproduction (VR); IMI3C = predicted IMI using LM, VR, and California Verbal Learning Test, 2ndedition (CVLTII); IMI3W = predicted IMI using LM, VR, and Verbal Paired Associates (VPA); DMI = Delayed Memory Index;DMI2 = predicted DMI using LM and VR; DMI3C = predicted DMI using LM, VR, and CVLTII; DMI3W = predicted DMIusing LM, VR, and VPA.aStandard errors of measurement (SEMs) for IMI = 3.5 and 3.7 points for IMI and DMI, respectively. Cutpoints rounded to nearestwhole points of SEM = 4. Standard deviations (SDs) for IMI and DMI = 15 points.

significant model, F(3, 277)= 563.26, p< .001. TheR2 value for this model, however, was considerablysmaller than the model using VPA2 and slightlysmaller than the R2 value for the IMI modelusing CVLTZ. This finding indicates that substi-tution of the CVLTII Long-Delay Free Recall forVPA2 produces greater variability in subtest scoresthan does the standard calculation of the DMIusing VPA or the substitution method for the IMI.The DMI2 generated the smallest R2 value of allsix models; however, the model was still statisticallysignificant, F(2, 289) = 769.42, p < .001. A totalof 43% of scores from this model were within 1SEM and 78% within 2 SEMs. Among individu-als with an observed DMI that was within 1 SDof the mean in either direction (n = 108), the aver-age prediction discrepancy for the three models was1.3 points (SD = 5.1, range = 16.03 to 15.01). Forindividuals with an observed DMI between 1 and2 SDs below the mean (n = 109), there was no dif-ference on average between observed and predictedvalues (M = 0.0, SD = 4.9, range = 15.41 to17.51). Among individuals with observed DMI val-ues more than 2 SDs below the mean (n = 65),the average discrepancy was 2.4 points lower thanobserved DMI values (SD = 5.0, range = 16.08 to10.53). Nine participants had observed DMI scoresthat were more than 1 SD above the mean.The rate of agreement between observed values

and predicted values generated from each modelwas also high. Each Pearson correlation coefficientcalculated between observed index score values and

predicted values from each model was positive andsignificant (p < .001), indicating a high degree ofprecision in predicted values. The bias correctionfactors used in the calculation of the CCC eachapproximated 1.00, indicating near-perfect accu-racy with very little evidence of systematic biasin any of the six models. Inspection of the gener-ated scatter plots between observed and predictedvalues showed a slight trend of each model to over-predict at the higher end of the score distributionand underpredict at the lower end; this trend ismost pronounced in the two-variable models andthe models that substitute CVLTII variables forVPA. The predictive models using LM, VR, andVPA showed near-perfect agreement. All measuresof agreement are presented in Table 4; Figures 16plot the data along with line of perfect concordance(i.e., the line at 45 on a square scatter plot) and thedatas reduced major axis (i.e., summarized centerof observed data plotted through the intersectionof the means with a slope equal to the ratio of stan-dard deviations) to show the extent of bias presentin each prediction model.Given that the fit of each of the six models

accounted for a significant proportion of the vari-ance in the respective index scores, and each pre-dictive model generated values that were in highagreement with observed values, correlations ofpredicted to actual index scores were compared toeach other using Fischers r-to-z transformation,which translates Pearson correlation coefficients toa common metric to facilitate significance testing

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013


TABLE 4Measures of agreement between observed and predicted index scores

ModelPrecision (Pearson

correlation coefficient)Accuracy

(bias factor)Concordance correlation

coefficient (CCC)

95%Confidenceinterval

IMI2 .92 1.00 .92 [.90, .94]IMI3C .93 1.00 .93 [.92, .95]IMI3W .98 1.00 .97 [.97, .98]

DMI2 .92 1.00 .91 [.89, .93]DMI3C .93 1.00 .92 [.91, .94]DMI3W .97 1.00 .97 [.97, .98]

Note. IMI = Immediate Memory Index; IMI2 = predicted IMI using Logical Memory (LM) and Visual Reproduction (VR);IMI3C = predicted IMI using LM, VR, and California Verbal Learning Test, 2nd edition (CVLTII); IMI3W = predicted IMIusing LM, VR, and Verbal Paired Associates (VPA); DMI = Delayed Memory Index; DMI2 = predicted DMI using LM and VR;DMI3C = predicted DMI using LM, VR, and CVLTII; DMI3W = predicted DMI using LM, VR, and VPA.

4060

8010

012

014

0IM

I

40 60 80 100 120IMI2

Line of perfect concordanceReduced major axis

Figure 1. Scatter plot of observed Immediate Memory Index (IMI) values and predicted IMI values using Logical Memory and VisualReproduction subtests (IMI2).

using a t distribution. This transformation allowsfor determination of the relative benefit of oneprediction equation over the others. For immedi-ate memory scores, the correlation of IMI3W toactual IMI was significantly stronger, t(294)= 5.99,p< .001, than were the correlations of IMI to eitherIMI2 or IMI3C, which did not significantlydiffer from each other, t(294) = 0.83, p = .40.Similarly, actual DMI scores were most highly cor-related with DMI3W, significantly more strongly,t(294) = 6.15, p < .001, than to comparing DMIto either DMI2 or DMI3C, which again did notdiffer from each other, t(294) = 0.83, p = .40. Boththe IMI3W and DMI3W generated significantly

stronger correlations than the other models andshowed the highest rate of agreement with observedscores. All Pearson correlation coefficients betweenobserved and predicted scores are presented inTable 4.To evaluate the influence of test-taking strategy

on the predictive abilities of the generated mod-els, predicted values were compared to originalvalues for the subsample of participants simulat-ing cognitive impairment. As the purpose of thepresent paper was to determine the ability of a sub-set of test scores to estimate the full index scores,it was expected that test-taking strategy wouldhave no influence on predictive accuracy. This

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013

538 MILLER ET AL.

4060

8010

012

014

0IM

I

40 60 80 100 120IMI3C


Figure 2. Scatter plot of observed Immediate Memory Index (IMI) values and predicted IMI values using Logical Memory, VisualReproduction, and California Verbal Learning Tests, 2nd Edition subtest scaled scores (IMI3C).

4060

8010

012

014

0IM

I

40 60 80 100 120IMI3W


Figure 3. Scatter plot of observed Immediate Memory Index (IMI) values and predicted IMI values using Logical Memory, VisualReproduction, and Verbal Paired Associates subtest scaled scores (IMI3W).

expectation was confirmed by paired-sample t tests,which were nonsignificant for IMI comparisons (allp values > .38). However, DMI predicted values forthis subsample were all significantly higher than

the actual DMI values (all p values .02) byan average of 1.5 points (DMI3C, M = 79.5,SD = 14.4) and 1.7 points (DMI3W, M = 79.7,SD = 14.9; DMI2, M = 79.7, SD = 14.3).

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013


4060

8010

012

014

0D

MI

60 80 100 120 140DMI2


Figure 4. Scatter plot of observed Delayed Memory Index (DMI) values and predicted DMI values using Logical Memory and VisualReproduction subtests (DMI2).

4060

8010

012

014

0D

MI

40 60 80 100 120 140DMI3C


Figure 5. Scatter plot of observed Delayed Memory Index (DMI) values and predicted DMI values using Logical Memory, VisualReproduction, and California Verbal Learning Tests, 2nd Edition subtest scaled scores (DMI3C).

Although statistically significant, the differencesof less than 2 points are not clinically meaning-ful, as demonstrated by negligible effect sizes foreach of the three comparisons (Cohens ds < .12).Looking at the distribution of predicted scores

among simulators based on the SEM reaffirms thisobservation as evidenced by the percentage of pre-dicted DMI values that were greater than observedscores by 2 or more SEMs (14.1%, 10.9%, and1.6% for the DMI2, DMI3C, and DMI3W,

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013

540 MILLER ET AL.

4060

8010

012

014

0D

MI

40 60 80 100 120 140DMI3W

Reduced major axis Line of perfect concordance

Figure 6. Scatter plot of observed Delayed Memory Index (DMI) values and predicted DMI values using Logical Memory, VisualReproduction, and Verbal Paired Associates subtest scaled scores (DMI3W).

respectively). The proportion of predicted DMIvalues that were underestimated by the predictiveformulas by more than 2 SEMs was considerablysmaller (DMI2 = 4.7%; DMI3C = 6.3%; DMI3W = 1.6%). Among nonsimulators, a similar pro-portion of overprediction by 2 or more SEMswas observed (DMI2 = 8.3%; DMI3C = 10.1%;DMI3W = 1.3%).

DISCUSSION

The findings provide strong support for estimat-ing WMSIV index scores using parsimonioussubsets of the most frequently administered sub-tests. The three primary subtests of the WMSIVthat are most familiar and most frequently used(Logical Memory, Visual Reproduction, andVerbal Paired Associates) serve remarkably wellto estimate both the Immediate Memory Index(IMI) and Delayed Memory Index (DMI) of theWMSIV. Furthermore, the present findingsdemonstrate that the IMI and DMI can be reliablyestimated with a level of predictive accuracy thatis considered acceptable for use in clinical practice.Estimations that used the CVLTII substitutiongenerally performed less well than did estimationsusing the WMSIV subtests. Although the mostaccurate model uses three of the four original

WMS subtests used to calculate the IMI and DMI,the present findings also show that using just two ofthe subtests will still yield index scores that closelyapproximate the actual values that would have beenobtained if all four subtests were administered.Such findings likely have particular relevance forclinicians facing increasing pressures and restric-tions on testing practices, or for those working withpatient populations who may fatigue easily and areunable to endure an extensive cognitive battery.Comparing the different short forms finds that

the two-subtest versions and the versions thatadded CVLTII substituted scores are essen-tially interchangeable. Specifically, using LogicalMemory and Visual Reproduction alone to pre-dict the Immediate Memory and Delayed Memoryindex scores yields the same predictive accu-racy as the models that added indices from theCVLTII. Therefore, the addition of CVLTIIindices contributes no additional predictive accu-racy over using just Logical Memory and VisualReproduction.In contrast, the addition of Verbal Paired

Associates in addition to Logical Memory andVisual Reproduction significantly improvedestimations of IMI and DMI. This finding isanticipated because using Logical Memory, VisualReproduction, and Verbal Paired associatestogether approximates most closely to the original

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013


sample from which the Immediate and DelayedMemory indexes were constructed. The predictiveaccuracy of the three-subtest version also raisesquestions regarding the utility of the Designssubtest included in the full WMSIV. The presentfindings suggest that Designs plays a minimalrole in the actual immediate or delayed memoryindex scores, given that short forms using LogicalMemory, Visual Reproduction, and Verbal PairedAssociates together account for approximately95% of variability in these indices. What remainsunclear, however, is the extent to which the Designssubtest contributes to the auditory and visualmemory indexes. Although the Designs subtestmay contribute little to the immediate and delayedmemory indexes, it may be an integral componentof these other indexes. For clinicians who are onlyinterested in the IMI and DMI, however, omissionof the Designs subtest is particularly appealing inthat it removes the subtest with the greatest timedemand and the most materials, thus simplifyingadministration.Although the predicted DMI values were statis-

tically higher than the actual values in the sam-ple of simulators, it is important to note that themagnitude of the actual differences (i.e., less than2 points) is not clinically meaningful. Additionally,it is important to note that these values still showeda strong positive relationship with actual values,which all fall at the extreme end of the distribu-tion. Consistent with the existing literature, thepredictive accuracy of these models deterioratesthe further out scores get from the center of thedistribution (Axelrod & Woodard, 2000; Spinkset al., 2009). For individuals who obtained anactual immediate memory index within two stan-dard deviations of the mean in either direction,predicted values were within one index point onaverage. Predicted values for individuals with anactual immediate memory index that fell more thantwo standard deviations below the mean, however,generated predicted values that differed by nearlytwo index score points on average. A similar pat-tern was observed among delayed memory indexscores, in which individuals who obtained an actualdelayed memory index within two standard devia-tions of the mean generated predicted values thatwere generally within one index point. Individualswith an actual delayed memory index that wasmore than two standard deviations below the meangenerated predicted values that differed by morethan two index points on average, and althoughthis difference is very small and not clinicallymeaningful, it is still of theoretical interest. Thus,clinicians faced with a test-taker demonstratingseverely impaired levels of memory functioning as

evidenced by scores that are more than two fullstandard deviations below the mean on individ-ual subtests should use these predictive equationswith this consideration in mind. Similar caution-ary warnings likely apply to the opposite end ofthe distribution as well; however, the proportion ofindividuals scoring more than one standard devia-tion above the mean in the present sample was toosmall to investigate this possibility adequately.From a practical standpoint, clinicians who

wish to adopt the present approach and obtainImmediate and Delayed index scores without hav-ing to administer the entire WMSIV battery cando so by utilizing the coefficients presented inTable 2. Specifically, once the specific model hasbeen selected, the respective unstandardized regres-sion coefficients (B) can be applied by multiply-ing the subtest age-adjusted scaled scores by theirrespective coefficients in Table 2, summing thetotals, and adding the constant. The resulting valueequals the predicted index score, which approxi-mates what would have been obtained had all foursubtests been administered. The predicted valuesthat were generated to approximate index scoreswere obtained from administration of the entireWMSIV battery; also, measures not included inthe original standardization process were used tosatisfy delay intervals. These limitations may haveinfluenced performance on DMI subtests; however,they are unlikely to have had significant influence,because during the standardization process itself,the tests go through myriad changes (i.e., addingitems, dropping items) before the final product ispublished. Another important point to reiterateis that all predictive models utilized age-adjustedscaled scores obtained from the test manual. Useof raw subtest scores in these equations will resultin drastically different values that will not approxi-mate an index score.In comparison to previous short forms of the

Wechsler Memory Scale (e.g., Axelrod &Woodard,2000; Woodard & Axelrod, 1995), the results fromthe present study are very consistent. As with theWMSIII, there is continued evidence that theimmediate memory index score can be reliably pre-dicted from two or three of the available subtests.Although there is no longer a General Memoryindex such as that included in the WMSIII andas studied by Axelrod and Woodard (2000), a sim-ilar composite included in the present study is thedelayed memory index, which also was effectivelypredicted by less than the full battery of subtests.One striking similarity between the present paperand earlier short forms is that the squared multi-ple correlations for each model are nearly identical.Although the specific predictors are different, the

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013

542 MILLER ET AL.

three-variable models fit by Axelrod and Woodardaccounted for approximately 95% of the variancein index scores, whereas the two-variable modelsaccounted for only 87%. This pattern between thetwo- and three-variable models was virtually iden-tical in the present study for those models thatonly included Wechsler Memory Scale subtests(i.e., those not including CVLTII substitution).Moreover, the same shortcomings were also iden-tified in that the predictive formulas lose stabilityat the extreme ends of the normal distribution,which is likely a factor of the predictive method asopposed to a function of the test itself.Comparing the contributions of the individual

subtests between the WMSIII and the WMSIVreveals some specific differences that are impor-tant to note. For example, in each model studiedby Axelrod and Woodard (2000) for the WMSIII,Logical Memory had the largest contribution ofthe included predictors. In the present study, how-ever, this was only the case when Logical Memorywas paired with one other subtest (e.g., VisualReproduction). In each of the three-variable mod-els for the WMSIV, visual reproduction madethe largest contribution to predicted index scores,and it is unclear how this subtest contributed toWMSIII short forms as Visual Reproduction wasnot included in the earlier studies.A unique contribution of the present study was

the inclusion of individuals coached to feign cog-nitive impairment. Theoretically, the generation ofindex scorescomplete or estimated from shortformsshould be independent of test-taking strat-egy as they are a linear composite of observedscores and therefore should not be affected byskill or psychological characteristics such as moti-vation; the calculations should apply regardless oftest-taking strategy. This notion was supported inpart by the fact that the pattern of findings amongthe coached simulators paralleled that observedamong the bona fide patients. At the same time,although the absolute differences were relativelysmall and not clinically meaningful (i.e., less than2 points), it is of some interest that the shortforms tended to slightly overpredict simulators.It is also important to note that the subsam-ple of participants simulating cognitive impair-ment performed quite poorly, which reinforces thenotion that scores from these predictive equationsbecome less reliable at the extreme ends of thescore distributions. Furthermore, a nearly identi-cal proportion of scores were overpredicted in theremainder of the sample for each of the predictivemethods. In that regard, the short forms seem towork in the direction of being robust to feignedcognitive impairment rather that vulnerable to it

and can thus be applied in the context of suspectednegative response bias.The mixed nature of the clinical sample, which

included a variety of neuropsychological andpsychological conditions across a broad range ofage and education, represents both a strength anda weakness of the present design. Also, althoughthere is no reason to expect that gender wouldaffect prediction accuracy, the present sampleoverrepresented men. Future research should verifythat the patterns observed in this study hold trueacross specific subgroups and conditions. Theexclusion of the Designs subtest also representsa weakness of the present study, as its inclusionmay have allowed for even greater precision in theresulting predicted values.

Original manuscript received 7 October 2011Revised manuscript accepted 6 February 2012

First published online 2 March 2012

REFERENCES

Axelrod, B. N., & Woodard, J. L. (2000). Parsimoniousprediction of Wechsler Memory ScaleIII memoryindices. Psychological Assessment, 12, 431435.

Delis, D., Kramer, J., Kaplan, E., & Ober, B. (2000).The California Verbal Learning Test (2nd ed.). SanAntonio, TX: Pearson.

Greiffenstein, M. F., Gola, T., & Baker, W. J. (1995).MMPI2 validity scales versus domain-specific mea-sures in detection of factitious traumatic brain injury.Clinical Neuropsychologist, 9, 230240.

Kaufman, J. C., & Kaufman, A. S. (2001). Time forthe changing of the guard: A farewell to short formsof intelligence tests. Journal of PsychoeducationalAssessment, 19, 245267.

King, T. S., & Chinchilli, V.M. (2001). A generalized con-cordance correlation coefficient for continuous andcategorical data. Statistics inMedicine, 20, 21312147.

Lin, L. I. (1989). A concordance correlation-coefficientto evaluate reproducibility. Biometrics, 45, 255268.

Miller, J. B., Millis, S. R., Rapport, L. J., Bashem, J. R.,Hanks, R. A., & Axelrod, B. N. (2011). Detection ofinsufficient effort using the advanced clinical solutionsfor the Wechsler Memory Scale, Fourth Edition. TheClinical Neuropsychologist, 25, 160172.

Spinks, R., McKirgan, L. W., Arndt, S., Caspers,K., Yucuis, R., & Pfalzgraf, C. J. (2009). IQestimate smackdown: Comparing IQ proxy mea-sures to the WAISIII. Journal of the InternationalNeuropsychological Society, 15, 590596.

Tombaugh, T. (1996). Test of Memory Malingering.North Tonawanda, NY: Multi-Health Systems.

Wechsler, D. (2008a). Technical and interpretive manualfor Wechsler Memory Scale (4th ed.). San Antonio,TX: Pearson.

Wechsler, D. (2008b). Wechsler Memory Scale (4th ed.).San Antonio, TX: Pearson.

Woodard, J. L., & Axelrod, B. N. (1995). Parsimoniousprediction of Wechsler Memory Scale Revised mem-ory indices. Psychological Assessment, 7, 445449.

Dow

nloa

ded

by [U

nivers

ity L

ibrary

Utre

cht]

at 06

:10 23

Septe

mber

2013

Parsimonious Prediction of Wechsler Memory Scale,

Documents

Transcript of Parsimonious Prediction of Wechsler Memory Scale,