Formative Evaluation of New Hampshire's Performance … · 2017-06-28 · Formative Evaluation of...
36
Formative Evaluation of New Hampshire's Performance Assessment of Competency Education (PACE) Gene Wilhoit, Center for Innovation in Education Arthur Thacker, HumRRO Paul Leather, New Hampshire Dept. of Education Susan Lyons, Center for Assessment (Access the Evaluation Here)
Transcript of Formative Evaluation of New Hampshire's Performance … · 2017-06-28 · Formative Evaluation of...
FormativeEvaluationofNewHampshire'sPerformanceAssessment
ofCompetencyEducation(PACE)
GeneWilhoit,CenterforInnovationinEducationArthurThacker,HumRRO
PaulLeather,NewHampshireDept.ofEducationSusanLyons,CenterforAssessment
(AccesstheEvaluationHere)
FormativeEvaluationofNewHampshire’sPerformanceAssessmentofCompetency
Education(PACE)PaulLeather,DeputyCommissioner
NewHampshireDepartmentofEducation
2
NHPACE-- PerformanceAssessmentofCompetencyEducation
• PACEisalearningsystemdesignedtocapitalizeonthelatestadvancesinunderstandinghowpeoplelearn.Thegoalistostructurelearningopportunitiesthatallowstudentstograpplewithgainingmeaningfulknowledgeandskillsatadepthofunderstandingthatcantheycantransfertonewreal-worldsituations.
CurrentStatus:• 3yearsofwaiverimplementation,and• ESSAplanningtoresultinWaiverrequest,subjecttoSec.1204implementationbySecretaryDeVos
• OurgoalistoscalePACEacrossourstateofNewHampshireandtoinspireotherstatestoacceptasimilarchallenge.
6/28/17 3
PACE– Whatarethebasics?
• First-in-the-nationUSEDapprovedalternativeassessmentandaccountabilitysystempilot
• Reducedlevelofstandardizedtestingalongwithlocallydevelopedcommonperformanceassessments
• SmarterBalanced/SATgivenonceinelementaryschool,onceinmiddleschoolandonceinhighschool(threegradesinsteadofseven)
• Inallotheryears,administrationofcommonandlocalperformanceassessmentsdevelopedbythedistrictsandvalidatedatthestatelevel
EngagingtheUSDepartmentofEducation
March27,20121st MeetingwithArneDuncan
Sept.23,20142nd MeetingwithArne
March3,2015
Approval!
5
PD DesignTheoryof
ActionImplementation
NHProcess
WhathaschangedwithPACE?
NCLB/Waiver• Alldistrictsheldaccountableatoncetostatesystem
• Statelevelassessment– lasttwoinNHarenationallydeveloped
• AnnualDeterminationsbasedonStateAssessmentssolely
• Thereoftenaretwosystems– stateandlocal
• Educatordevelopmenttiedtoschoolimprovementafterassessmentresultsarereleased
PACE• DistrictsmustmeetguardrailsbeforejoiningPACE
• PACEAssessmentsdevelopedlocally–SystemisStateANDLocal
• Annualdeterminationsbasedonmultiplemeasures
• LocalcommunitiesandeducatorsOWNthesinglesystemofaccountability
• IntensiveEducatorDevelopmentoccursPRIORtojoiningPACE,andcontinuesasPACEinvolvementevolves
6
AssessmentContinuum
Extended Performance
Tasks(SCALE,
EPIC, ILN)
Narrow Assessment Assessments of Deeper Learning
CCSS Assessments
(SBAC &PARCC)
Performance Based Items
& Tasks(MARS, BAM)
Student-Designed Projects
(Envision, NY Performance
Standards Consortium, Singapore, IB)
Traditional Tests
Descriptions
Standardized, multiple-
choice tests of routine skills
Systems of standardized performance
items and tasks (1 day to 1 week) that measure key concepts in
thought-provoking items
that require extended
problem solving
Examples
Standardized tests with m-c & open-ended items + short
(1-2 day) performance tasks of some applied skills
Performance tasks that require
students to formulate and carry out their own inquiries,
analyze & present
findings, and (sometimes)
revise in response to feedback
Longer, deeper investigations,(2-3
months) & exhibitions, including
graduation portfolios,
requiring students to initiate, design, conduct, analyze,
revise, and present their work
in multiple modalities LindaDarling-Hammond
WhatisPACE?– WaterTowerProposal!• TheProblem:Yourtown’spopulationispredictedtoincreaseoverthenext3years.Asoneofthetownplanners,youareaskedtoaddressthisissueintermsofthetown’swatersupply.Inordertomeetthefutureneedsofthetown,youneedtomakeaproposaltoaddawatertowersomewhereontownpropertythatwillbecapableofholding45,000±2,000cubicfeetofwater.Thetownislookingforawatertowertocontainthemostamountofwaterwhileusingtheleastamountofconstructionmaterial.
• StudentTask:Yourjobistoprepareaproposalthatcanbesubmittedtothetownplanningcommittee.Usingyourcalculationsofsurfaceareaandvolumeforthetwodesigns,describeandanalyzethecharacteristicsthatleadyoutoafinalrecommendation.
6/28/17 8
GeometryPACECommonTask
PACE:Local,StateandCommonAssessments
Grade ELA MATH SCIENCE
K-2 Local PBA LocalPBA LocalPBA
3 SmarterBalanced CommonPACEPBA Local PBA
4 CommonPACEPBA SmarterBalanced CommonPACEPBA
5 CommonPACEPBA CommonPACEPBA LocalPBA
6 CommonPACEPBA CommonPACEPBA LocalPBA
7 CommonPACEPBA CommonPACEPBA LocalPBA
8 SmarterBalanced SmarterBalanced CommonPACEPBA
9 CommonPACEPBA CommonPACEPBA CommonPACEPBA
10 CommonPACEPBA CommonPACEPBA CommonPACEPBA
11 SAT SAT CommonPACEPBA
12 Local PBA/CapstoneExhibitions
Local PBA/CapstoneExhibitions
LocalPBA/CapstoneExhibitions
WhoisPACE-- WhichDistricts/Schools?
6/28/17 10
Tier1--Year1(2014-15)
AdditionalTier1--Year2(2015-16)
Tiers2and3–Year3(2016-17)
• Rochester• SanbornRegional
• Epping• Souhegan HS
• Concord• Monroe• Pittsfield• SeacoastCharter
• Allenstown• FallMountain• Plymouth• SAU23No
Haverhill• SAU58Groveton• Manchester• Rollinsford• Ashland• SAU39Amherst
andMontVernon• VLACS
AdditionalTier1--Year3(2016-17)
• SAU35WhiteMountains
(asnegotiatedwithUSED)
OrganizationtoPACEScaling
AdoptedfromNHDOEgraphic
Tier1ImplementingDistricts
• DataCollection• CalibratingandScoring• BodyofWorkProtocols
Tier2PreparingDistricts
• PerformanceAssessmentDevelopment
• ImplementationProcess
Tier3PlanningDistricts
• DevelopCBE• Competencies,Instruction, AssessmentandGrading
PACESystemDesignisbasedonHighPerformingProfessionalLearningCommunities• Whereeducatorsworktogethertocontinuously:
• Reviewcompetencyexpectations, alignedwithcollegeandcareeroutcomes
• Evolveinstructionalpractice tosupportstudentlearningofcompetencies
• Developstrategiestoenhancepersonalizedlearningwherestudentagencyisaprimarystudentoutcome
• ImprovingtheAssessmentsystem tomeasurestudentachievementandgrowthrelatedtocompetencies
• Educatorsreviewstudentworkonperformancetasksregularly
6/28/17 12
Headquarters: 66 Canal Center Plaza, Suite 700, Alexandria, VA 22314-1578 | Phone: 703.549.3611 | www.humrro.org
Formative Evaluation of New Hampshire’s Performance Assessment of Competency Education (PACE)
PresentationforNCSA
Presenters: ArthurThacker,HumRROSunnyBecker,HumRRO
June29,2017
PresentationOverview
• WhatisHumRRO?• BriefReminderofSelectPACECharacteristics• EvaluationDesign• KeyEvaluationFindings• EvaluationRecommendations• TheStoryofPACE
14
HumanResourcesResearchOrganization(HumRRO)• Independent,non-profitU.S.basedorganization• Approximately100employees
• Professionalstaffmostlyeducationresearchers,I-Opsychologists,statisticians,andmeasurementexperts
• Expertise:• Programevaluationsatlocal,state,andnationallevels
EvaluationsofschoolimprovementinitiativesLargescale,longrangeevaluationsofstateandnationaleducationprogramsQualityassuranceforNationalAssessmentofEducationalProgress(NAEP)Independent,real-timereplicationofassessmentresultsasaqualitycontrolcheckforseveralstates
15
SelectPACECharacteristics
• Competencyeducation• Performanceassessmentsintegratedintostudents’day-to-daywork• Tier1,2,and3districts• Commonandlocaltasks• Taskbank
16
EvaluationDesign(1of2)
• April2016– February2017• Formativeevaluation• TheoryofAction• Datasources
• observationsofmajorPACEevents(e.g.,taskdevelopmentandscoringsessions);• interviewswithstudents,parents,teachers,principals,anddistrictleaders;• classroomobservations;• surveysofteachers;• reviewofmaterialsprovidedtoandbyTier1districts;and• analysesofscoredata.
• AccumulatedevidenceregardingeachclaimintheTheoryofActionoverthecourseoftheevaluation.
17
PA
18
PACETheoryofAction
KeyEvaluationFindings(1of3)
• ReportincludesdetailedfindingsoneachclaimwithintheTheoryofAction
• Buy-in• Educatorsdeterminewhatisassessed,howitisassessed,andhowthetasksarescored
• Assessmentsintegratedintoinstruction• ReplacesSmarterBalancedassessmentsandrequiresdeepknowledge
• Collaboration• Cross-districtactivities(taskdevelopmentsessions,professionaldevelopment,scoringsessions,standard-setting,etc.)
• LibGuide (onlinerepository)
19
KeyEvaluationFindings(2of3)
• Teaching&Learning• Increaseddepthofknowledge• “Testingtowhatistaught”vs.“teachingtothetest”• Learningwhiletesting
• Context• Previousexperiencewithcompetency-basededucation• Districtsize• Substantialworkforteachers
20
KeyEvaluationFindings(3of3)
• Continuousprocessimprovement• CurriculumCoordinator• Contentleads• Buddydistrict• Saturdaytaskdevelopmentsession• ProvidingconsultantstosupportPACEcoordinator
21
DataAnalyses
22
47.3
53.555.5
44.2
63.3
57.2
48.9
44.6
52.0
44.046.0
51.0
57.0
48.0 47.0
52.0
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
Grade3 Grade5 Grade6 Grade7
Percen
tageofStude
ntsM
eetin
gorExceedingAchievemen
tLevelBen
chmarks(3
or4
)
Mathematics
PACE2014-2015 PACE2015-2016 SB2014-2015 SB2015-2016
DataAnalyses
23
43.6
49.7
40.1
48.651.3
55.3
43.1 42.4
56.0
63.0
57.0
63.0
57.0
63.0
59.062.0
0
10
20
30
40
50
60
70
Grade4 Grade5 Grade6 Grade7
Percen
tageofStude
ntsM
eetin
gorExceedingAchievemen
tLevelBen
chmarks(3
or4
)
ELA
PACE2014-2015 PACE2015-2016 SB2014-2015 SB2015-2016
DataAnalyses
Math2015 Science2016
Science2016 .487
ELA2016 .317 .637
24
Table 1. Correlation Table Grade 3-4
Science2015 ELA2015 ELA2016
ELA2015 .555
ELA2016 .459 .630
Math2016 .440 .603 .735
Table 2. Correlation Table Grade 4-5
DataAnalyses
Math2015 ELA2015 ELA2016
ELA2015 .635
ELA2016 .520 .619
Math2016 .625 .590 .648
25
Table 3. Correlation Table Grade 5-6
Math2015 ELA2015 ELA2016
ELA2015 .470
ELA2016 .482 .586
Math2016 .558 .513 .531
Table 4. Correlation Table Grade 6-7
DataAnalyses
Math2015 ELA2015 Science2016 ELA2016
ELA2015 .517
Science2016 .523 .448
ELA2016 .557 .483 .574
Math2016 .488 .477 .590 .541
26
Table 5. Correlation Table Grade 7-8
Recommendations(1of2)
OngoingMonitoring1. MonitorandSupportDistrictEngagement
2. EvaluateEffectivenessofCollaborationMethods
Additionaltraining/supports3. ConsiderAdditionalTraining/SupportsforTeachersNotDirectlyInvolvedinCommonTask
Development
4. InfuseEquityandAccommodationsTrainingintoPACEActivities
27
Recommendations(2of2)
Routinizeprogramfeatures6. RoutinizeTimelyReviewsofLocalPerformanceTasks
9. ConsiderSystematicallyRecyclingTasks
10. BeginTrackingPerformancefromYeartoYear
TargetedStudies5. InvestigatetheImpactofReading/WritingRequirementsonAccessibility
7. PlanforFutureResearchontheImpactofPACEonTeachingandLearning
8. EvaluatetheBenefitofTimeinProgramonOutcomes
28
TheStoryofPACE
• Integratedcompetency-basededucationsystemwithperformanceeventsareadramaticshiftfromtraditionalschooling
• Substantialprogresstowardthisgoal• Sustainability• Scalability• Carefulexpansion,districtbydistrict
29
ProgrammaticResponsestoPACEFormativeEvaluation
SusanLyons,Ph.D.CenterforAssessment
CCSSO’sNationalConferenceonStudentAssessment
June29,2017
PACEisaboutclassroomimpact
Lyons•Page31
ComplexPerformanceAssessments
•Buildingeducatorcapacity
•Enhancingstudentengagement
TransformingInstruction
•Increasingrigor•Developingstudentagency
CollegeandCareer
Readiness
•Preparingstudentstoengagemeaningfullyintheirpost-secondaryplans
IncreasingCollegeandCareerReadiness
Supporting All TeachersvConsider Additional
Training/Supports for Teachers Not Directly Involved in Common Task Development
Teachers routinely reported that the process of developing the common tasks greatly improved their own task development process and their approach to assessment. As the program expands, it will be important to maintain that benefit for all participants.
ü 16-17: Hiring of PACE Teacher Leaders within districts to better transmit institutional knowledge to all teachers.
ü 16-17: Investment in online intranet for all PACE teachers to share key documents and resources.
ü 17-18: Expanding high quality performance assessment development training within Tiers 2.
ü 17-18: Developing robust set of common resources for competency-based education and assessment across all three tiers of PACE.
Quality Local AssessmentvRoutinize Timely
Reviews of Local Performance Tasks
It is important to ensure that the tasks and rubrics are of sufficient quality to be used to generate student scores and annual determinations. Teachers report that their skill level in developing these tasks improves with each year of PACE participation…. Instituting a system of regular task review will help ensure that happens.
ü 16-17: The NHDOE is now reviewing one major assessment per competency for each PACE course in all of the participating districts.
ü 16-17: Contract with Stanford to review all local performance assessments.
ü Summer ’17: NH DOE will provide feedback to districts related to their assessments systems and targeted supports for those districts in need of additional guidance.
Evaluating ImpactvPlan for Future
Research on the Impact of PACE on Teaching and Learning
The positive impacts of PACE on teaching and learning should continue to be externally verified beyond this evaluation. This may be part of a future research agenda when it becomes possible to evaluate the predictive strength of PACE results on college and career performance.
ü On-going: Annual evaluation of student performance on standardized assessments. 2016 yielded some early indications of success.
ü 16-17: Providing access to PACE data to internal and external researchers
ü 17-18: Seeking funding from Hewlett to more deeply understand the connection between learning and engagement in complex performance assessments.
ü 17-18: Begin to longitudinally track trends in career and college readiness (e.g., persistence in college).
