Sharon H. Ulanoff, Joan C. Fingon, & Dolores Beltrán

125

Teacher Education Quarterly, Spring 2009

Using Case Studies To AssessCandidates’ Knowledge and Skills

in a Graduate Reading Program

By Sharon H. Ulanoff, Joan C. Fingon, & Dolores Beltrán

Introduction Inthisageofheightenedaccountability,academiaisincreasinglybeingaskedtolinkassessmenttocandidateperformanceoutcomesinmultipleways.Researchdemonstratestheimportanceofaligningassessmentwithcontentstandardsbutcautionsthatitiscriticalthatassessmentsmatchthecontent,coverawiderangeofknowledge,arecognitivelydemanding,andavoidirrelevantmaterials(AERA,2003). Case-based pedagogy is one way to link program content to classroompractice.Muchoftheresearchoncasemethodscallsfortheuseofcasesto“createbridgesacrossthegreatchasmthatdividespolicyfrompractice”(Shulman,2000,

Sharon H. Ulanoff, Joan C. Fingon, and Dolores Beltrán are professors in the Department of Curriculum and Instruction of the Charter College of Education at California State University, Los Angeles.

p.2)inordertohelpteachersunderstandhow practiceis constructed in the classroom.Within case-basedpedagogy,thecasesbecometeachingtoolsthatserveasacontextformakingmeaningofconceptspresentedduringinstructioninavarietyofinstructionalsettings,andthusmakeunderstandingtransparent. Inthisarticleweexaminetheuseofcandidate-authored case studies as a culminating assessmentactivityinoneReadingandLanguageArtsSpecialistCredentialProgram(RRLA)inalarge,urbanpublicuniversitywithadiversestudentpopulationinsouth-

Using Case Studies To Assess Candidates’ Knowledge and Skills

126

ernCalifornia.CandidatesintheprogramearnanM.A.inEducation,OptioninReading,alongwiththestate-issuedspecialistcredential.Weaskthefollowingresearchquestionsinorder toexaminetheuseofcase-basedassessment in theRRLAprogram:

1.Arecasestudiesaneffectivewayforgraduatecandidatestodemonstrateknowledgeandskillslearnedintheprogram?

2.Cancandidatesusecasestudiestodemonstratewhattheyknowaboutservingculturallyandlinguisticallydiverseurbanstudents?

Theoretical Framework Muchhasbeenwrittenabouthowassessmentpracticeshaveandhavenotchangedinthepast100years(Brown,1996;Shepard,2004).Whileexpertsarguethatthereisreciprocitybetweenassessmentandinstruction,theconstructionofschoolpracticeisofteninformedbyoutdatedtheoriesthatdonotconsidernewunderstandingsfromthefield(Brown,1996),frequentlycreatingadisconnectbetweenassessmentandinstruction.Shepard(2004)arguesthat“thecontentofassessmentsshouldmatchchallengingsubjectmatterstandardsandservetoinstantiatewhatitmeanstoknowandlearnineachofthedisciplines”(p.1621)andsheproposesasocial-constructivistconceptofassessmentwheredynamic,ongoingassessmentofferscandidatesexplicitevaluationcriteria,inadditiontosupportandassistanceasfeedbackastheyprogressthroughtheprogram.Thustheinstructionalprogramleadscandidatestowardsthedesiredcompetenciesthattheassessmentseekstomeasure. Theuseofcase-basedmethodologytomeasureteachercompetenceisonewaytolinkassessmenttopractice(Shulman,2002)andtoshiftfromtraditionalmodesofevaluationintoamoredynamicandauthenticreviewoflearning.Whiletheprofessionalfieldsofbusinessandlawhaveusedcasesandcasemethodsforyears,theideaofusingthemineducationhasemergedoverthelast15yearsasapromisingidea.Merseth(1991)arguesthatthecurrentmovetowardacase-basedpedagogyisduetoagrowinginterestinteacherknowledgeandcognitionaswellasanacknowledgementofthecomplexitiesofteaching. Withthisgrowingawareness,theeffortstodefinecasestudieshavegainedprominence.AccordingtoMerseth(1994)acaseis“adescriptivenarrativedocumentthatisbasedonareallifesituationorevent”(p.2).Theauthorfurtherdescribesthecaseashavingthreeessentialelements:afirmbaseinreality,arelianceonresearch,andthedevelopmentofmultipleperspectivesbythosewhousethem.Complicatingtheseeffortsarethevarioususesofcasestudies.Forexample,casescanbeusedasstimulantsforreflection,techniquestoenrichfieldexperiences,andtoolsforprofessionalevaluation.Theycanfurtherservetoframeconversationsbetweenmentorsandnovices,orientindividualstoparticularwaysofthinkingandinitiatediscussionsfromdifferentperspectives(Merseth).Casesmethodsmaybe

Sharon H. Ulanoff, Joan C. Fingon, & Dolores Beltrán

127

usedduringlarge-andsmall-groupdiscussions,role-playing,writtenanalysisorteam-baseddiscussion(Merseth).Additionally,theuseofcasesforassessmentcanhelpteacherstoreflectontheirownpracticeandmakemoreinformeddecisionsregardingthatpractice(Harrington,1995;Merseth,1996;Shulman,1987). Whythegrowinginterestincase-basedmethodology?Whatdocasesorcasestudiesoffer thatdiffers fromother,more traditional researchmethods?Whileopinionsmayvary,thefundamentalresponsestothesequestionsincludecontextandauthenticity. Notonlydothecasesthemselvesprovidecompellingcontextsfordiscussingandmakingsenseofclassroompractice,butthenotionthattheyareauthenticrepresentationsofsuchpracticealsosupportstheiruse.Authenticityenhancestheeffectivenessofcasesbyaddingcontexttotheory(Colbert,Trimble&Desberg,1996).Theparamountconcernistomakecasesrealandtousetrue-lifestoriesthatillustratekeyeducationaltheoriesandbringupissuesthatarecriticaltotheprofessionalgrowthofteachers.Inthiswaycasescanhelpstudentsthinkproductively about concrete experiences and enhance the ability to learn fromtheirownexperiencesastheyemploytheoreticalconceptstoilluminateapracticeproblemorsituationandpracticehowtothinkprofessionallyaboutrealproblemsandsituations(Kleinfeld,1996). Whilemuchcanberelatedtotheuseofcase-basedpedagogyasateachingtoolwherestudentsreadandrespondtocases,itsuseasaformofassessmentwherestudentscreatetheirowncasesasameansofdemonstratingcompetenciesisrela-tivelynew.Whenusingcasestudiesforassessmentitisimportanttoconsidernotonlythebenefitsbutalsothedrawbacks.Thebenefitsofutilizingcasestudiesininstructionincludethewaythatcasesmodelhowtothinkprofessionallyaboutrealproblemsandsituations,helpingcandidatestothinkproductivelyaboutconcreteexperiences(Kleinfeld,1990).Whencasesareusedforassessment,candidatescanbepresentedwithsituationsthatrequirethemtoapplytheirknowledgeandskillstosolverealproblemsinthefield.Tellis(1994)suggeststhatwhilecasestudiesaredynamic,authentic,contextualizedandlinkedtoreality,theyalsopresentthecandidatewithchallengesintermsoftimeexpended,bias,objectivity/subjectivity,andotherobstaclesthatarise,includingblockedaccessandinabilitytobuildrap-portwiththecasestudystudent.Furthermore,eventhoughtheuseofcasestudiesasameansofauthenticassessmentlinkingtheorytopracticeleanstowardamorequalitativeapproachtoassessment,currentandnationaltrendsineducationfavormorequantitativesystematicapproaches.

Context for the Study TheReadingandLanguageArtsSpecialistCredential isastate-authorizedadvanced credential issued to teachers who have a minimum of three years ofteachingexperience.CoursesintheRRLA,whichisaccreditedbytheCaliforniaCommissiononTeacherCredentialing(CCTC)andtheNationalCouncilforAc-

Using Case Studies To Assess Candidates’ Knowledge and Skills

128

creditationofTeacherEducation(NCATE),alsomeetthestandardsoftheInterna-tionalReadingAssociation.Programstandardsemphasizefoundationalknowledge,instructionalstrategiesandcurriculummaterials,assessment,diagnosisandevalua-tion.TheRRLAfurtherincludesafocusonliteracyinstructionforEnglishlanguagelearners(ELLs). Inaddition toallcourseworkin theprogram,candidatesmustsuccessfullycompleteacomprehensiveexam(comps),whichservesasaculminatingactivityfortheprogram.Traditionally,allcandidatesinM.A.programsineducationatthiscampussitforathree-hourcompsexamofferedtwiceayear(duringfallandspringquarter).Duringthisexamcandidateswriteessayresponsestoaseriesofquestionspreparedbyfacultywithintheirspecificgraduateprograms.Atthebeginningofthequarter,candidatesareprovidedwithaseriesofpracticeorstudyquestions.Typically,betweenthetimecandidatesreceivethequestionsandtheactualdateoftheexam,theyformstudygroupsandworktogetherinpreparingresponsestothosequestions.Theyalsomayaskfacultytoclarifyquestionsandreviewtheirresponsesbeforethedateoftheexam.FortheRRLAprogram,candidatesweregiventenstudyquestions,twoofwhichshowedupasmandatoryquestionsontheexam.Thecandidatesansweredtwoadditionalquestions,whichwerechosenfromtherestofthequestions.Eachresponsewasscoredbyaminimumoftwofacultymembers,withsomefacultyscoringresponsestomorethanonequestion.Attimes,somefacultyscoredmorethan40responses. AlthoughtheRRLAfacultyrewrotesomecompsquestionseachyear,thosefacultywho scored the examsbegan tonotice similarities between candidates’responsesduetothefactthatcandidatesstudiedtogetherandoftenmemorizedsimilar,ifnotidentical,responsestothestudyquestions.Becauseofthesimilarityofexamsandageneraldissatisfactionwiththetraditionalmeasureofusingasit-downwrittenexaminationtoassessourcandidates,welookedformoreauthenticandmeaningfulwaystoassessourcandidates’knowledgeandskills. Infallquarter,2004,thefacultydecidedtochangethenatureoftheRRLAcomprehensiveexaminordertomoreeffectivelymeasurecandidates’knowledgeandskillsinconditionswheretheywouldbeabletodemonstratecompetencesitu-atedinthecontextofworkingwith“realstudents.”Wedesignedanexamtogiveourcandidatesdifferentopportunities toexamine,understand,anddemonstratetheirknowledgeofrelationshipsamongtheory,research,practice,anddecision-making.Theideawastosimulateaclinicalexperience,tohavethecandidateactasareadingspecialistanddemonstratetheknowledgeandskillslearnedintheprogram.Inotherwords,weaskedourcandidatesto“applyandshowwhattheyknow.”Weutilizedacommonsix-stepmodelindesigningthecasemethodthataskedcandidatesto:(1)identifytheeducationalissuesinvolved;(2)thinkaboutthecasefrommultiplepointsofview(e.g.,parent,student,teacher,principal);(3)useprofessionalknowledge(e.g.,learningtheory)todiscussthecase;(4)projectcoursesofactionthatmightsolvetheproblem;(5)determinetheconsequencethat

Sharon H. Ulanoff, Joan C. Fingon, & Dolores Beltrán

129

mightfollowfromeachcourseofactiongenerated;(6)afterevaluatingeach,choosecoursesofactiontobefollowedanddecidehowtoevaluatetheeffectivenessoftheplan(Greenwood,1996,p.59). Thus,thenewcompsexamaskedcandidatestoadministerabatteryofas-sessmentstoonechild;analyzetheassessmentresultsintermsofreading,writingandEnglishlanguagedevelopment(ELD);anddeterminethechild’sstrengthsandweaknessesineachareaandwriteanappropriateinterventionplan.Itisimportanttonotethatourcampusnotonlyservesadiversepopulation,butteacherswhoattendourprogramsthemselvesteachanextremelydiverserangeofstudentsincludingaconsiderablenumberofELLs,mostofwhominitiallylearntoreadinEnglish,theirsecondlanguage.Asaresult,ourgraduatesneedtheknowledgeandskillstoworkeffectivelywithELLs. Thereforeweaskedeachcandidatetodevelophis/herownindividuallywrittencasestudyofa“real”strugglingreader.Guidelinesrequiredcandidatestobeginbydescribingthestudent’sbackgroundandthentoconstructthecasestudyusingdatafromtheassessmentsandtheirknowledgeabouteffectiveliteracyinstructioninordertomakerecommendationsforthestudent.Withinthecasestudy,candidatesmustdemonstrateandapplydeepknowledgeofcurriculumandinstructionalap-proachestousewithstrugglingreaders,includingabroadandin-depthknowledgeofinstructionalprogramsandspecializedmaterials. Oncethenewcompsexamwasconceptualized,agreedupon,developed,andimplemented,facultymemberswerefacedwith thechallengeofevaluating thecasestudiesthatcandidatesgenerated.First,weneededtodecideontherubricthatwouldbemosteffectivetoscorethecases,sincetheexistingdepartmentrubricwasnolongerappropriateforthenewexam.Becauseourcandidateswerenolongerwritingexamsbyhandinonesitting,weanalyzedtheissuesthatscoringthecasespresented.Forexample,facultybelievedthatsincestudentswouldhavemoretimetowriteandwouldtheoreticallysubmitmorepolishedword-processedexams,thenewrubricshouldallowforgrammar,spelling,writingstyle,anduseofreferencestobeweighteddifferentlyinthescoring. Theauthorsofthisstudyappliedforandreceivedasmallassessmentgrantinwinterquarter,2005, toaddress issues related toevaluating thecasestudiesdevelopedbyourcandidatesforthecomprehensiveexam.Weusedthegranttodevelop a new scoring rubric (seeFigure One), guidelines for scoring and thegateway requirements (see FigureTwo), which documented the use of currentresearchonreading,writingandELD,APAformat,pagenumbers,andtheuseofwww.turnitin.comtoscanforplagiarism,etc.Thepurposeofthegatewaywastoimmediatelyscreenoutanycasestudiesthathadmissingrequirementspriortothefacultyevaluatorsscoringthem.Examsthatdidnotmeetthegatewayrequirementswerenotacceptedandcandidatesweregiven24hourstocompletetheexam.

Using Case Studies To Assess Candidates’ Knowledge and Skills

130

Figure One: RRLA Comprehensive Exam Scoring Rubric.

Sharon H. Ulanoff, Joan C. Fingon, & Dolores Beltrán

131

Methodology

Setting and Sample DatawerecollectedinoneCharterCollegeofEducation(CCOE)atalarge,diverse,urbanstateuniversityinSouthernCalifornia.Atthetimeofthestudy,theCCOEenrolled15%ofthetotalcampuspopulationof20,000.Thestudentpopu-lationwasapproximately50%Latino,19%Asian-AmericanorPacificIslander,14%White,Non-Latino,1.5%African-American,9%unknown,5.5%non-residentundocumentedand.4%AmericanIndian.Moreover62.3%ofstudentswerewomenandtheaverageageofgraduatestudentswas34. The sample for this study consisted of 110 candidates in three cohorts ofcandidates(graduatestudents)takingthecomprehensiveexamasaculminatingactivity in the previously described RRLA program, which was the 5th largestgraduateprogramoncampus.Allcandidatesheldatleastapreliminaryteaching

Figure Two. Gateway Requirements (Fingon, Ulanoff & Beltrán, 2005).

Candidates must first meet these minimal overall requirements before final evaluationtakesplace.Ifallgatewayrequirementsarenotmetthecandidatewillbegiven24hourstosuccessfullymeettheserequirementsinorderforhis/herworktobeacceptedforthefinalevaluationphase.

_____3hardcopiesofexam _____Classroomobservationform_____Coversheetw/CEIDonexam _____Assessmentsummarydatasheet (listingallassessmentsandresults)_____CEIDonly(noname)onallpages _____Samplesofrequiredassessments andwritingsamples(inAppendices)_____Maximum25pages+appendices _____Referencepage(9minimum requiredreferences)_____Pagesarenumbered _____APAstyle_____Typed,double-spaced,12pt.font, _____Copyofturnitin.comreceipt w/one-inchmargins _____Tableofcontents(w/pagenumbers) _____Oneaudiocassette organizedaccordingtothespecified withassessments sectionsoftheexam

Pleaseplaceall3copiesoftheexam+audiocassette,classroomobservationformandturnitin.comdigitalreceiptcopyinamanilaenvelopeidentifiedonlybyCEID.Nootherpaperswillbeaccepted.Pleasenote:fileuploadedtowww.turnitin.commustbeidenticaltoeachofthe3hardcopies(minustheclassroomobservationformandturnitin.comdigitalreceipt).

Decision: Examcomplete/acceptasis(pleasecheckone):YesoNoo Exammissingsection(s)(seeabove).Reviseandsubmitby____________.

Signatureoffacultymemberacceptingexam:_______________________.

Using Case Studies To Assess Candidates’ Knowledge and Skills

132

credentialand92%ofthemwereteachinginK-12publicschools.Theresttaughtincharterandprivateschools,orservedassubstituteteachers.

Method DatacollectiontookplacefromSeptember2004toDecember2005andwasbothqualitativeandquantitativeinnature.All110candidatescompletedacom-prehensiveexamthatrequiredthemtogenerateacasestudyofastrugglingreader,whowasalsoanELL.Theintentionoftheexamwastohavestudentsdemonstratetheknowledgeandskillsrelatedtotheirfuturerolesasreadingspecialistsinlin-guisticallyandculturallydiverseurbansettings.Theexamwasdeliveredthroughacourse-basedtechnologywebsite(WebCT)andcandidatesweregivenfourweekstocompletetheexam.Allexamsweresubmittedthroughwww.turnitin.comandashardcopies.Inadditiontowritingthecasestudy,candidateswererequiredtomeetaseriesofgatewayrequirements(seeFigureTwo). Ateamofthreefacultymembers(evaluators)scoredeachexam.Thenumberofteamsvariedeachquarterdependingonthenumberofexamsandthisservedtostabilizefacultyworkloadrelatedtoscoring.Onaverageeachevaluatorscored10-15exams.Eachevaluatorinateamreadforoneofthreefoci:reading,writingorELD.Theevaluatorsusedthe4-pointrubric(seeFigureOne)developedforthecasestudyandanoverall“3”ratingwasconsideredpassingormeetingtherequirements.Candidatesneededtoreceiveameanscoreofatleast2.5orbetter,withnoscoreoflessthan2.0inanyonesection,inordertopasstheexam.Onescorebelow2allowedarewriteofthatfocusarea.Itisimportanttonotethatthesameevaluatorsscoredtheexamsatallthreedatapoints,Fall2004—Fall2005.

Data Sources Thequalitativedatasetconsistedofthefollowing:(1)comprehensiveexamscreatedandsubmittedbythecandidates;(2)exitsurveycompletedbythecandidates(seeAppendix); (3) rubrics; (4)evaluatorcomments;and(5)e-mailexchangesbetweencandidatesandfaculty.Quantitativedataconsistedofexamscores,whichwereanalyzedaccordingtofocusarea. Dataanalysiswasbasedonareviewofeachdatasourceandcodingofdata.Patternswereidentifiedastheysurfacedandthesewereusedtofurtheridentifysalientthemes,andcategorieswithinandacrosseachdataset.Examswereassessedbasedonhowwellthecandidatesdemonstratedthefollowing:

•scoring,analyzing,andinterpretingmultiplesourcesofauthenticdatarelatedtothecasestudystudent;

•examiningandusingdataincontext;

•translatingtheoryintopractice;

Sharon H. Ulanoff, Joan C. Fingon, & Dolores Beltrán

133

•engaginginhigherorderthinking,problemsolving,andlearningatahighercognitivelevel;and

•generatingandevaluatingpossiblesolutionsandmakingrecommenda-tionsbasedonresults.

Inessence,theevaluatorswereinterestedinfindingouthowcandidates“applywhattheyknow”andtransformwhatislearnedfromtheexperienceintopractice.Themajorthemesthatemergedfromthedatahighlightedcandidate strengths, the link between theory and practice, and candidate weaknesses. Thesethemesfur-therservedtoidentifymatchesandgapsbetweenprogramcontentandcandidateknowledgeandskills.

Findings Overallfindingsindicatethatwewereabletousethecasestudiestomeasureprogramcontentknowledgeandalsotoidentifythematchesandgapsbetweenprogramcontentandcandidateknowledgeandskills.Atotalof114candidatesattemptedtheexam,butonly110completedtheexam;104candidatespassedtheexam.InFall2004,95%ofcandidatespassedtheexam(42outof44)andtherewere6rewritesononefocusarea.InSpring2005,92%candidatespassedtheexam(48outof52),andtherewereatotalof8rewrites.All14candidates(100%)whotooktheexaminfall2005passedwithouttheneedtorewriteanyonearea.ThecandidateswhofailedtheexaminFall2004andSpring2005receivedlowscoresinallthreefocusareas.Tableoneshowsmeanscoresforallfocusareasaswellasthemeancompositescoreforeachcohort,Fall2004,Spring2005,andFall2005.

Candidate Strengths Candidateswhosuccessfullycompletedtheexamdemonstratedknowledgeandskillsrelatedtotheteachingofreading,writingandELDasmeasuredbythecasestudiesproducedforthecomprehensiveexam.Ascanbeseenfromtableone,therewaslittledifferenceinscoresoverthecourseofthestudy,withcandidatesinSpring2005scoringslightlylowerthanthecandidatesineachoftheothertwoquartersexamined.Ninetypercentofexitsurveyrespondentsinspringandfall2005indicatedthattheyfeltwellorextremelywellpreparedtobereadingspecialists.

Table One. Mean Passing Comprehensive Exam Scores, Fall 2004 to Fall 2005 (n=104).

Reading Writing ELD Mean CompositeScore

Fall2004(n=42) 2.95 3.08 3.17 3.07

Spring2005(n=48) 2.87 2.92 2.62 2.80

Fall2005(n=14) 2.96 3.14 3.18 3.09

Using Case Studies To Assess Candidates’ Knowledge and Skills

134

Uponanalyzingtheexams,rubrics,andevaluatorcommentsfromthreecandidatecohortsoveratwo-yearperiod,wefoundthattherewasgreaterspecificityintheinstructionsgiven to thecandidates, the requirementsof theexam,and facultyexpectations for procedure and content. By Spring 2005, both evaluators andcandidatesmoreclearlyunderstoodthecontentoftheexamaswellaswhatwasrequiredintermsofcollectionandanalysisofthecasestudystudentdata,makingrecommendationsforstudentpractice,andlinkinganalysisandrecommendationstoresearchinthefield.

Facilitating candidate strengths with the gateway and the four-point rubric:Aswetransitionedfromtheoldten-pointrubrictothenewfour-pointrubricforevaluation of the case studies, initially there was a lack of understanding andconsistencyamongfacultyaboutinterpretingandusingtherubric.Thegatewayrequirements(FigureTwo)werecreatedtospellouttheminimalrequirementsintermsofacceptablecontentandformatforcandidates.Sinceonlyexamsthatmetthegatewayrequirementswereaccepted,theevaluatorsscoredonlycompleteex-ams.Inessence,thegatewayrequirementshelpedtoexplainexamproceduresforcandidatesaswellasfacultywhoservedasevaluators.Whilethegatewayrequire-mentsclarifiedproceduralissues(e.g.,howmanypages,theuseofAPAformat,theuseoftheidnumber,thenumberofreferences,etc.)the4-pointscoringrubricclarifiedfacultyexpectationsregardingthecontentoftheexam. ItisimportanttonotethattheRRLAprogramcoordinator,oneoftheauthorsofthispaper,oversawtheexamprocess.Shemetwithstudentstoreviewexpectationsforthecasestudy,theonlinedeliveryandsubmissionsystems,andservedasthecontactpersonforsupportduringtheexam.Thecoordinatororganizedtheblindreviewofexams,includingtheassignmentofexamstofacultyforscoring,anddeterminedthetimelinefortheexam.Shealsocompiledscoresandnotifiedstudentsastotheirpass/failstatus.Thesameprogramcoordinatorservedthroughoutthestudy.

Examining content for demonstration of candidate strengths:Wefoundthatwewereabletousethecasestudiestohelpidentitycandidatestrengthsintermsofthecontentknowledgeandskillsofourcandidates.Thecandidatesconsistentlyprovidedevidencethattheyunderstoodhowtoassessstudentsanduseassessmentresultstomakerecommendationsforeffectiveinstruction,andasnoted,scoreswerefairlystableoverthecourseofthestudy.Forexample,oneoftherequirementsinthecasestudywastoidentifystrategiesthatbestmettheneedsoftheirstudentforremediationintheareasofreading,writing,andELD,basedontheirinterpretationofassessmentresults.TabletwoliststhespecificinterventionstrategiesusedinrecommendationsforthecasestudystudentsinFall2005.Thesestrategiesreflecttheapplicationandunderstandingofcoursecontentand,inmostcases,wereusedappropriatelyinthecontextofthecasestudy.

Sharon H. Ulanoff, Joan C. Fingon, & Dolores Beltrán

135

Table Two. Spring 2005 Comps Case Study Student Strategies.

Reading Writing ELD

DirectedListeningThinking Graphic ReadaloudsActivity(DLTA)* organizers*

DirectedReadingThinking Writers DRTAActivity(DRTA)* workshop*

Storymapping Dialoguejournals Scaffolding*

LanguageExperienceApproach Interactive K-W-L(Whatyouknow,(LEA) writing* whatyouwanttoknow, whatyoulearned)* orK-W-LPlus

VocabularySelfSelection(VSS)* Sharedwriting* Writer’sworkshop

Guidedreading* Authorstudy Dailyreading

DirectedReadingActivity(DRA) Scaffolding Contextualredefinition*

LiteratureCircles Conferencing VSS

ReQuest LEA Guidedreading

Question-Answer-Relationship Storymapping Retelling*(QAR)

Storyboard Wordwalls Grandconversations

Textstructure Wordbuilding Multiculturalliterature

Reader’sTheatre Learninglogs ShelteredEnglishapproach

Retelling Peerediting Semanticfeatureanalysis

Imaging Webbing Showandtell

Reciprocalteaching Wordsorts LEA

Independentreading

Semanticmapping

Cloze

Repeatedreading

Overlapofsomestrategiesoccurinmorethanonearea*=mostfrequentlydescribedasastrategybyastudentinthecasestudy

Using Case Studies To Assess Candidates’ Knowledge and Skills

136

The Link Between Theory and Practice Candidateswererequiredtolinktheorytopractice,usingpublishedresearchtosupporttheiranalysesandrecommendations,whichwasdemonstratedintheiranalysisofthedatafromtheircasestudyandtherecommendationsforpractice,including the strategies listed in table two.The strategies that theychoseweregenerallylinkedtoresearchpresentedintheircoursework.Byandlarge,thecasestudiesprovidedcandidateswithopportunitiestodemonstratetheirknowledgeoftheprofessionalresearchrelatedtoreading,writingandELD. Weanalyzedallcasestudiesdescribedinthispaperandusedthefollowingexcerptstoilluminatehowthecasestudiesenabledcandidatestodemonstratetheirability to linkreading,writing,andELDtheoryandpracticeforculturallyandlinguisticallydiverseurbanstudents.Wethoughtitbesttoprovideexcerptsfromonecasestudythatreceiveda“3,”orpassingrating,intheareasofreading,writ-ingandELDtoprovideaglimpseintoacompletecasestudy.Alloftheexamplescomefromonefemalecandidate’scasestudyofan8-year-oldboynamedJosé(apseudonym),whowasborninMexicoandcametotheUnitedStatesin2004.TheexamwascompletedinSpring2005whenJoséwasinthe3rdgrade.

Reading: Inthisexcerpt thecandidatedeterminedthatoneofJosé’sreadingneedswasintheareaofwordattackskills.ThecandidatedescribedandprioritizedJosé’sneedsbasedonanIndividualizedReadingInventory(IRI)andgavespecificexamplesofmiscuesinanotherportionofthecasestudy.ThecandidatethenprovidedrelevanttheoreticalresearchsourcestosupporttherecommendationsshesuggestedforJoséwhensheofferedanotherapproachtolearningwordattackskills.

A need displayed by José is in the area of word attack skills and many of his miscues on the IRI were substitutions for other words with the same initial sound such as “laughed” for “left”. Although he is in a systematic phonics program there are several other strategies available for decoding and learning unknown words. For example, Cooper (2003) suggests structural analysis because it requires the reader to look at the word for meaningful units or parts in order to decode the word or figure out its meaning. Another area of need which was made evident through the CELDT [California English Language Development Test], IRI, and Slosson [Slosson Intelligence Test] was reading comprehension….

Writing: In this excerpt on writing the candidate made recommendationsregardingJosé’sneedtodevelopagoodattitudeandwillingnesstowriteandtheneedforgoodmodelstoinspirehiswriting.

There has to be enthusiasm for writing. That means a teacher who continually “sells” writing to his or her students, who knows compelling arguments for the importance in writing in today’s society and who makes an event of most writing occasions from writing (Walshe, 1979). According to Walshe and others reading is inextricably linked to writing, just as talk of adults is linked to infant’s acquisition of speech. José would benefit from daily mini-lessons that model writing skills….

Sharon H. Ulanoff, Joan C. Fingon, & Dolores Beltrán

137

ELD:InthisthirdexcerptthecandidatereferredtoJosé’sELDneedsbyreferencinghisemerginglanguageskillssincehisrecentarrivalintheUnitedStates(mentionedpreviously)andtheneedtoincreasehisoverallEnglishlanguagelearning:

Although José has developed Basic Interpersonal Communications Skills (BICS) he has not acquired academic language and does not have “language proficiency or ability to use a language effectively and appropriately throughout the range of social, personal, school, work situations required for daily living in a given society” (Peregoy & Boyle, 2001, p. 29). His vocabulary can be improved by pro-viding additional scaffolding to his lessons with comprehensible input. Interactive opportunities between José and other students will also allow him to strengthen his English and acquire additional BICS and eventually Cognitive/Academic Language Proficiency (CALP).

ThecandidateconcludedthissectionofthecasestudybylinkingmanyofJosé’sareasofneed(readingcomprehension,wordattackskills,andvocabulary)withELDanddocumentingthesesuggestionssupportedintheliterature:

Transferring prior knowledge, translating passages to his native language and reflecting on the text in his native language has the potential for improving his reading comprehension (Jimenez, 1997). The teacher should try to learn about things that are of more interest to José and perhaps call on him for his expertise during appropriate class discussions. It is also important to include multicultural children’s literature, which is representative of his Mexican culture… to develop a sense of belonging.

Requesting additional information:Afinalsection,recentlyaddedtothecasestudy,askscandidatesforadditionalinformationtheywishedtheyhadknownregard-ingtheirstudent(sincetherewerestricttimelimitationsontheexam).Theywereaskedtoelaborateonwhomtheymightaskandtoprovidearationale.Inthisfinalexcerpt,thecandidatedemonstratedherknowledgeandskillsbeyondtheconfinesofstudentandassessor/readingspecialist,allowinganindicationofunderstandingwithinabroaderschoolcommunitycontext.Thecandidaterespondedasfollows:

If I had had additional time to work with this student and this project I would have liked to have interviewed José’s current teacher and asked José’s study habits and what type of vocabulary instruction was currently being provided? I would have liked to know more about his interests and performed an interest inventory, what types of books he liked what sort of things he does in his spare time? I would have liked to have known his grades from Mexico or had spoken to his previous teacher to get more background information about José. I would have also liked to have observed José more in his classroom to get a more realistic picture of who he is in class… and performed other assessments in order to help José … this information could also have been beneficial in individualizing an instructional plan to fit José’s needs.

Using Case Studies To Assess Candidates’ Knowledge and Skills

138

Candidate Weaknesses Postexamsurveyresponsesindicatedthatcandidatesfeltthattheexamwasameasureoftheprogramcontent—98%inSpring2005and100%inFall2005respondedthattheyeitheragreedorstronglyagreedthatthecontentoftheexamwasrepresentativeoftheprogramcontent.However,candidatesstatedthattheyfeltmoreconfidentaboutcontentrelatedtoreadingthaneitherwritingorELDbasedonprogramcourses. Examinationofthedatafromall110candidateswhosuccessfullycompletedtheexambetweenFall2004andFall2005demonstratedcandidateweaknessesinlinkingtheorytopracticeintheareaofELD.Candidatesspecificallystruggledwithidentify-ingwaysinwhichtoscaffoldinstructionforELLs,arguingthat“therewasonlyoneELDclassandIdidn’tthinkthattherewasenoughemphasisonELDineachoftheotherclasses.”SincetheRRLAprogramisdesignedtoinfusecontentrelatedtotheassessmentandinstructionofELLsthroughouttheprogram,commentssuchasthisoneservetohighlighttheneedforfurtherprogramexamination.Thisfindingmir-rorsarecentreportfromtheCaliforniaStateUniversityChancellor’sOffice,andthiscontentisespeciallyimportantinrelationtothepreparationofourRRLAcandidatessincetheyteachinclassroomswithhighnumbersofELLsandneedknowledgeandskillsrelatedtoELDinordertoeffectivelyservetheirstudents. Anotherpatternofcandidateweaknessidentifiedbytheexams,postexamsurveydata,andevaluatorcommentswasfoundintheadministrationandinterpretationofwritinganalyses.Athirdofthefacultywhoevaluatedtheexamscomplainedaboutthelimitationsofrequiredwritingassessments,citingcandidates’lackofproficiencyinthisarea.Thispatternofweaknessgeneratedongoingdebateabouthowbesttoteachandassesscandidateknowledgeandskillsrelatedtoteachingwriting.Onecandidate’spostexamsurveyresponsestated,“Iwouldhavepreferredtohavesome[more] experience using the [required] vocabulary index and the t-unit analysis[writingassessments].”Facultyfeltthatfurtherexaminationofthesedatawouldbehelpfulinprogramevaluationforongoingprogramrevision,includingtherevisionofcoursecontentthataddressesknowledgeandskillsalignedtoprogramstandards.

Conclusions and Implications MojeandWade(1997)arguethatcase-baseddiscussionsingeneralcanhelpteacherstochallengepreconceivednotionsaboutteachingandlearningastheyexploretherelationshipsbetween“…knowledge,abilityandliteracy”(p.705).Asteachersmovebeyonddiscussionstoexaminestudentartifacts(includingassessmentresults)asdirectrecordsoftheclassroomexperience,theseartifactssupportthedevelopmentofacaseasaninterpretiveaccountofthesubject’sstrengthsandweaknesses(Shul-man&Kepner,1999).AstheteachersintheRRLAprogramexaminedthestrengthsandweaknessoftheircasestudystudents,welookatthecasestheydevelopedtomeasuretheirknowledgeandskillsasreadingspecialists.

Sharon H. Ulanoff, Joan C. Fingon, & Dolores Beltrán

139

While some researchhasbeendone in theareaof case-basedassessment,limitedpractice exists using candidate-generated case studies as assessment tomeasurestudentoutcomesingraduateprograms.Uponcloseexaminationofthesecasestudies,weseehowtheydemonstratethewaysinwhichcandidatesprocessinformationandapplyitto“reallife”situations.Mostimportantly,ourstudydem-onstratesthemultipleusesofcandidate-generatedcasestudiesforassessmentastheyprovideevidenceofindividuals’knowledgeandskills,andoftheknowledgeandskillsofthegroupasawhole.Theyfurtheraffordusaglimpseintooverallprogrameffectiveness. Asfacultyresearchers,wehavebeguntoseemorebenefitsfromdoingassess-mentconsciouslyandconscientiously.Thethreeauthorsofthispaperhaveoperatedasasmallteam,systematicallyexaminingcandidates’work.OurcollaborationhasledustonewinsightsandtheparticularactionofimprovingtheRRLAprogram.Wehavecometoviewthegraduatecasestudiesaspartofanon-linear,recursivedataanalysiscyclethatisprocessofnoticing,collectingandthinking(Seidel,1998,p.2).Becauseourdataanalysiscyclebuildsonexistingunderstandingandactiontocreatenewunderstandingandaction,witheachiterationwenoticenewthingsinthedataandreact.Aswehavemovedthroughourprocessandgainedinsightintothepotentialofourgraduatecasestudies,wehavereturnedfrequentlytoourearliersourcestogathernewdataforimprovingourcoursecontent.Becausethecasestudiesareauthenticapplicationscontextualizedwithinsimulatedpracticewith real students, the varied competencies developed in our program becometransparentforprogramfaculty.Likeourcandidatesintheanalysisoftheirownassessmentdata,wehaveaccesstoamorecomplexsetofdatathanwaspreviouslyavailable,allowingustoexaminetheRRLAprogrammoreeffectively. Centraltoourworkistheimportanceoflookingmorecloselyatcandidates’originalworkthatwasproducedinthecasestudies.Inessence,ourfocusasateamexaminingaculminatingactivity(thecomprehensiveexam)shiftedfrommoreforma-tiveassessmentofourcoursecontenttosummativeassessmentofprogram content.Thisshiftoffocuswasalsonoticeableinthefacultywhoscoredthecases,asthematchesandgapsbetweentheexamandprogramcontentbecamemoretransparent.Whileanexternaloroutsideevaluatormighthavebeenusefulforinvestigatingprogramoutcomes,whatemergedfromourstudywastheimportanceoffacultyexaminingauthenticworksamplesasonemeansofmeasuringprogramoutcomes,specificallythewaysinwhichtheRRLAcandidateswereabletodemonstrateunderstandingofprogramcontentanditsapplicationto“reallife”situations. Someoftheunexpectedoutcomesofthisstudyforfacultyincludedabetterunderstandingoftheoverallgraduateprogram.Withtheinceptionoftheuseofthecasestudies,facultymeetingsandconversationsweremuchmorecenteredontheinterrelationshipbetweencourserequirements,programgoals,andourcandidates’overallabilitiestowriteaboutandreferencewhattheyknow.Moreover,theRRLAprogramstandardsallowedustoexaminemorecloselywhichcoursesmightcause

Using Case Studies To Assess Candidates’ Knowledge and Skills

140

informationgapsoroverlapsforstudentsinapplyingtheircourseworktotheprepara-tionofthecasestudy.Ournewunderstandingsledustolaunchacollaborativeefforttoensureastrongeralignmentofstandardsacrosscourses.Closeexaminationofthecasestudiesalsogavefacultymoreinsightintoeachcandidate’sunderstandingoflinkagesbetweentheoryandpracticeintheareasofreading,writing,andELD.Overthecourseoftherefinementofboththeprocessandcontentoftheexam,wehaveregularlysharedourfindingswithprogramfacultyinformalandinformalforums.Asaresultofthesefocusedconversations,facultymembershaveengagedinmoremeaningfuldialogueaboutdesiredexpectationsandhowcoursefeaturesandassignmentssupportbothformativeandsummativeassessment. Otherpositiveoutcomesthatcameoutofthisstudyincludedthesuccessofthenewgatewayrequirementandrubricinestablishingmoreconsistencyinthescoringprocessbothacrosscasesandamongevaluators.Additional interestingevidenceshowedthatwhilecandidateswererespondingtoourcriteria,theyalsohadissuesandconcernsrelatedtocourseandprogramstandardsandexpectations.Clearly,wehavejustbeguntounderstandthevalueofthecasestudiesandwhatotherpowerfulmessagestheyoffer. Ourinquiryinvolvedusinadynamic,ongoingprocessandtheresulthasbeentheemergingsocialconstructionofa“learningculture”(Shepard,2004).Whileourstudyislimitedinscope,thefindingsofferaninterestinganddifferentperspectiveonusesofcasesstudies,specificallycandidate-writtencasesstudiesforassessmentpurposes.Inaddition,thestudy’soutcomesseemtimelyandrelevantincontributingtothegrowingbodyofresearchthatsupportsalternativeassessmentmethods.

ReferencesAERA. (2003). Standards and tests: Keeping them aligned. Research Points, 1(1), 1-4.

RetrievedJune1,2004fromhttp://www.aera.netBrown,A.L.(1996).Theadvancementoflearning.Educational Researcher, 23(8),4-12.Colbert,J.,Trimble,K.,&Desberg,P.(1996).The case for education: Contemporary ap-

proaches for using case methods. Boston:Allyn&Bacon.Greenwood,G.E. (1996).Using thecasemethod to translate theory intopractice. InJ.

Colbert,K.Trimble,&P.Desberg(Eds.).The case for education: contemporary ap-proaches for using case methods. Boston: Allyn&Bacon.

Harrington,H.L.(1995).Fosteringreasoneddecisions:Case-basedpedagogyandtheprofes-sionaldevelopmentofteachers.Teaching and Teacher Education, 11,203-214.

Kleinfeld,J.(1996).Ourherocomesofage:Whatstudentslearnfromcasewritinginstu-dentteaching.InColbert,J.,K.Trimble,&P.Desberg.(Eds.).The case for education: contemporary approaches for using case methods. Boston:Allyn&Bacon.

Kleinfeld,J.(1992).Learningtothinklikeateacher:Thestudyofcases.InJ.H.Shulman(Ed.).Case methods in teacher education.NewYork:TeachersCollegePress.

Kvale,S. (1996).Examinations reexamined:Certificationof studentsor certificationofknowledge?InChaiklin,S.&J.Lave(Eds.).Understanding practice: Perspectives on activity and context (pp.215-240).NewYork:CambridgeUniversityPress.

Sharon H. Ulanoff, Joan C. Fingon, & Dolores Beltrán

141

MersethK.K.(1996).Casesandcasemethodsinteachereducation.InJ.Shulman(ed.)Case methods in teacher education(pp.722-744).NewYork:MacMillan.

Merseth,K.(1996).Caseandcasemethodsinteachereducation.InJ.Sikula(Ed.).Hand-book of research on teacher education(pp.102-119).NewYork:Simon&Schuster,MacmillanHigherEducation.

Merseth,K.K. (1991).Theearlyhistoryof case-based instruction: Insights for teachereducationtoday.Journal of Teacher Education,42(4),243-249.

Moje,E.B.,andWade,S.E.(1997).Whatcasediscussionsrevealaboutteacherthinking.Teaching and Teacher Education, 13, 691-712.

Seidel,J.(1998).Qualitative data analysis. RetrievedSeptember15,2006fromhttp://www.qualisresearch.com/

Shepard,L.A.(2004).Theroleofassessmentinalearningculture.InRuddell,R.B.andN.J.Unrau(Eds.).Theoretical models and processes of reading, 5th edition (pp.1614-1635).Newark,DE:InternationalReadingAssociation.

Shulman,J.H.(2003).Case methods as a bridge between standards and classroom practice. Washington,DC:NationalPartnershipforExcellenceandAccountabilityinTeaching.RetrievedJune2,2004fromhttp://www.ericsp.org/pages/digests/shulman.pdf

Shulman,J.H.(2002).Happyaccidents:Casesasopportunitiesforteacherlearning. PaperpresentedattheannualmeetingoftheAmericanEducationalResearchAssociation,NewOrleans,LA.

Shulman,J.H.withKepner,D.(1999).Theeditorialimperative:Respondingtoproduc-tive tensionsbetweencasewritingand individualdevelopment.Teacher Education Quarterly,26(1),91-111.

Shulman,J.H.,Whittaker,A.,&Lew,M.(2002).Using assessments in teaching for under-standing: A casebook for educators.NewYork:TeachersCollegePress.

Shulman,L.S.(1987).Knowledgeandteaching:Foundationsofanewreform.Harvard Educational Review, 57,1-22.

Tellis,W.(1997).Applicationofacasestudymethodology.The Quantitative Report,3(3).RetrievedonJanuary11,2006fromhttp://www.nova.edu/ssss/QR3-3/tellis2.html

AppendixEDCI 596 Student Comprehensive Exam Exit Survey

(© Fingon, Ulanoff, & Beltrán, 2004)

Quarterenteredprogram_______________________

1.Describeyourcurrentposition.

2.WhatisyourapproximateGPA?

Pleaserateeachofthefollowingona1-5scale,where(1)is“StronglyDisagree,”(2)is“Disagree,”(3)is“Noopinion/neutral,”(4)is“Agree,”and(5)is“StronglyAgree.”

3.ThecasestudytasksandquestionswererepresentativeofthecontentoftheMAintheRRLAprogram.

1 2 3 4 5 Strongly Disagree No-Opinion/ Agree Strongly Disagree neutral agree

Using Case Studies To Assess Candidates’ Knowledge and Skills

142

4.ThesupportIreceivedregardingmyquestionsabouttheexamwashelpfultotheprocess.

1 2 3 4 5 Strongly Disagree No-Opinion/ Agree Strongly Disagree neutral agree5.Ienjoyedthetake-homeformatoftheexam.

1 2 3 4 5 Strongly Disagree No-Opinion/ Agree Strongly Disagree neutral agree

6.Whichofthefollowingwerehelpfultoyouduringtheexam?Checkallthatapply?*

oThecompsmeeting oTheexaminstructions

oThecompsWebCT oTheexam“testpilot”

oThegatewayrequirements oThetechsupport

oThescoringrubric oTheexamtimeline

7.Pleaselistanychallengesthatyouhadtakingtheexam.

8.Pleaselistanypositiveoutcomesyouexperiencedduringtheexam.*

9.Doyouhaveanysuggestionsforimprovingtheexam?

10. HowwelldoyoufeelpreparedtobeaReadingSpecialist?

1 2 3 4 5 Unprepared Minimally Somewhat Well-prepared Extremely Prepared Prepared Well-prepared

11.How likelydoyou think it is that youwill becomea reading specialist in thenearfuture?

Don’tKnow Unlikely SomewhatLikely Likely HighlyLikely

*Questions6and8wereaddedtothesurveyinFall2005.