FormativeEvaluationofNewHampshire'sPerformanceAssessment

ofCompetencyEducation(PACE)

GeneWilhoit,CenterforInnovationinEducationArthurThacker,HumRRO

PaulLeather,NewHampshireDept.ofEducationSusanLyons,CenterforAssessment

(AccesstheEvaluationHere)

FormativeEvaluationofNewHampshire’sPerformanceAssessmentofCompetency

Education(PACE)PaulLeather,DeputyCommissioner

NewHampshireDepartmentofEducation

2

NHPACE-- PerformanceAssessmentofCompetencyEducation

• PACEisalearningsystemdesignedtocapitalizeonthelatestadvancesinunderstandinghowpeoplelearn.Thegoalistostructurelearningopportunitiesthatallowstudentstograpplewithgainingmeaningfulknowledgeandskillsatadepthofunderstandingthatcantheycantransfertonewreal-worldsituations.

CurrentStatus:• 3yearsofwaiverimplementation,and• ESSAplanningtoresultinWaiverrequest,subjecttoSec.1204implementationbySecretaryDeVos

• OurgoalistoscalePACEacrossourstateofNewHampshireandtoinspireotherstatestoacceptasimilarchallenge.

6/28/17 3

PACE– Whatarethebasics?

• First-in-the-nationUSEDapprovedalternativeassessmentandaccountabilitysystempilot

• Reducedlevelofstandardizedtestingalongwithlocallydevelopedcommonperformanceassessments

• SmarterBalanced/SATgivenonceinelementaryschool,onceinmiddleschoolandonceinhighschool(threegradesinsteadofseven)

• Inallotheryears,administrationofcommonandlocalperformanceassessmentsdevelopedbythedistrictsandvalidatedatthestatelevel

EngagingtheUSDepartmentofEducation

March27,20121st MeetingwithArneDuncan

Sept.23,20142nd MeetingwithArne

March3,2015

Approval!

5

PD DesignTheoryof

ActionImplementation

NHProcess

WhathaschangedwithPACE?

NCLB/Waiver• Alldistrictsheldaccountableatoncetostatesystem

• Statelevelassessment– lasttwoinNHarenationallydeveloped

• AnnualDeterminationsbasedonStateAssessmentssolely

• Thereoftenaretwosystems– stateandlocal

• Educatordevelopmenttiedtoschoolimprovementafterassessmentresultsarereleased

PACE• DistrictsmustmeetguardrailsbeforejoiningPACE

• PACEAssessmentsdevelopedlocally–SystemisStateANDLocal

• Annualdeterminationsbasedonmultiplemeasures

• LocalcommunitiesandeducatorsOWNthesinglesystemofaccountability

• IntensiveEducatorDevelopmentoccursPRIORtojoiningPACE,andcontinuesasPACEinvolvementevolves

6

AssessmentContinuum

Extended Performance

Tasks(SCALE,

EPIC, ILN)

Narrow Assessment Assessments of Deeper Learning

CCSS Assessments

(SBAC &PARCC)

Performance Based Items

& Tasks(MARS, BAM)

Student-Designed Projects

(Envision, NY Performance

Standards Consortium, Singapore, IB)

Traditional Tests

Descriptions

Standardized, multiple-

choice tests of routine skills

Systems of standardized performance

items and tasks (1 day to 1 week) that measure key concepts in

thought-provoking items

that require extended

problem solving

Examples

Standardized tests with m-c & open-ended items + short

(1-2 day) performance tasks of some applied skills

Performance tasks that require

students to formulate and carry out their own inquiries,

analyze & present

findings, and (sometimes)

revise in response to feedback

Longer, deeper investigations,(2-3

months) & exhibitions, including

graduation portfolios,

requiring students to initiate, design, conduct, analyze,

revise, and present their work

in multiple modalities LindaDarling-Hammond

WhatisPACE?– WaterTowerProposal!• TheProblem:Yourtown’spopulationispredictedtoincreaseoverthenext3years.Asoneofthetownplanners,youareaskedtoaddressthisissueintermsofthetown’swatersupply.Inordertomeetthefutureneedsofthetown,youneedtomakeaproposaltoaddawatertowersomewhereontownpropertythatwillbecapableofholding45,000±2,000cubicfeetofwater.Thetownislookingforawatertowertocontainthemostamountofwaterwhileusingtheleastamountofconstructionmaterial.

• StudentTask:Yourjobistoprepareaproposalthatcanbesubmittedtothetownplanningcommittee.Usingyourcalculationsofsurfaceareaandvolumeforthetwodesigns,describeandanalyzethecharacteristicsthatleadyoutoafinalrecommendation.

6/28/17 8

GeometryPACECommonTask

PACE:Local,StateandCommonAssessments

Grade ELA MATH SCIENCE

K-2 Local PBA LocalPBA LocalPBA

3 SmarterBalanced CommonPACEPBA Local PBA

4 CommonPACEPBA SmarterBalanced CommonPACEPBA

5 CommonPACEPBA CommonPACEPBA LocalPBA

6 CommonPACEPBA CommonPACEPBA LocalPBA

7 CommonPACEPBA CommonPACEPBA LocalPBA

8 SmarterBalanced SmarterBalanced CommonPACEPBA

9 CommonPACEPBA CommonPACEPBA CommonPACEPBA

10 CommonPACEPBA CommonPACEPBA CommonPACEPBA

11 SAT SAT CommonPACEPBA

12 Local PBA/CapstoneExhibitions

Local PBA/CapstoneExhibitions

LocalPBA/CapstoneExhibitions

WhoisPACE-- WhichDistricts/Schools?

6/28/17 10

Tier1--Year1(2014-15)

AdditionalTier1--Year2(2015-16)

Tiers2and3–Year3(2016-17)

• Rochester• SanbornRegional

• Epping• Souhegan HS

• Concord• Monroe• Pittsfield• SeacoastCharter

• Allenstown• FallMountain• Plymouth• SAU23No

Haverhill• SAU58Groveton• Manchester• Rollinsford• Ashland• SAU39Amherst

andMontVernon• VLACS

AdditionalTier1--Year3(2016-17)

• SAU35WhiteMountains

(asnegotiatedwithUSED)

OrganizationtoPACEScaling

AdoptedfromNHDOEgraphic

Tier1ImplementingDistricts

• DataCollection• CalibratingandScoring• BodyofWorkProtocols

Tier2PreparingDistricts

• PerformanceAssessmentDevelopment

• ImplementationProcess

Tier3PlanningDistricts

• DevelopCBE• Competencies,Instruction, AssessmentandGrading

PACESystemDesignisbasedonHighPerformingProfessionalLearningCommunities• Whereeducatorsworktogethertocontinuously:

• Reviewcompetencyexpectations, alignedwithcollegeandcareeroutcomes

• Evolveinstructionalpractice tosupportstudentlearningofcompetencies

• Developstrategiestoenhancepersonalizedlearningwherestudentagencyisaprimarystudentoutcome

• ImprovingtheAssessmentsystem tomeasurestudentachievementandgrowthrelatedtocompetencies

• Educatorsreviewstudentworkonperformancetasksregularly

6/28/17 12

Headquarters: 66 Canal Center Plaza, Suite 700, Alexandria, VA 22314-1578 | Phone: 703.549.3611 | www.humrro.org

Formative Evaluation of New Hampshire’s Performance Assessment of Competency Education (PACE)

PresentationforNCSA

Presenters: ArthurThacker,HumRROSunnyBecker,HumRRO

June29,2017

PresentationOverview

• WhatisHumRRO?• BriefReminderofSelectPACECharacteristics• EvaluationDesign• KeyEvaluationFindings• EvaluationRecommendations• TheStoryofPACE

14

HumanResourcesResearchOrganization(HumRRO)• Independent,non-profitU.S.basedorganization• Approximately100employees

• Professionalstaffmostlyeducationresearchers,I-Opsychologists,statisticians,andmeasurementexperts

• Expertise:• Programevaluationsatlocal,state,andnationallevels

EvaluationsofschoolimprovementinitiativesLargescale,longrangeevaluationsofstateandnationaleducationprogramsQualityassuranceforNationalAssessmentofEducationalProgress(NAEP)Independent,real-timereplicationofassessmentresultsasaqualitycontrolcheckforseveralstates

15

SelectPACECharacteristics

• Competencyeducation• Performanceassessmentsintegratedintostudents’day-to-daywork• Tier1,2,and3districts• Commonandlocaltasks• Taskbank

16

EvaluationDesign(1of2)

• April2016– February2017• Formativeevaluation• TheoryofAction• Datasources

• observationsofmajorPACEevents(e.g.,taskdevelopmentandscoringsessions);• interviewswithstudents,parents,teachers,principals,anddistrictleaders;• classroomobservations;• surveysofteachers;• reviewofmaterialsprovidedtoandbyTier1districts;and• analysesofscoredata.

• AccumulatedevidenceregardingeachclaimintheTheoryofActionoverthecourseoftheevaluation.

17

PA

18

PACETheoryofAction

KeyEvaluationFindings(1of3)

• ReportincludesdetailedfindingsoneachclaimwithintheTheoryofAction

• Buy-in• Educatorsdeterminewhatisassessed,howitisassessed,andhowthetasksarescored

• Assessmentsintegratedintoinstruction• ReplacesSmarterBalancedassessmentsandrequiresdeepknowledge

• Collaboration• Cross-districtactivities(taskdevelopmentsessions,professionaldevelopment,scoringsessions,standard-setting,etc.)

• LibGuide (onlinerepository)

19

KeyEvaluationFindings(2of3)

• Teaching&Learning• Increaseddepthofknowledge• “Testingtowhatistaught”vs.“teachingtothetest”• Learningwhiletesting

• Context• Previousexperiencewithcompetency-basededucation• Districtsize• Substantialworkforteachers

20

KeyEvaluationFindings(3of3)

• Continuousprocessimprovement• CurriculumCoordinator• Contentleads• Buddydistrict• Saturdaytaskdevelopmentsession• ProvidingconsultantstosupportPACEcoordinator

21

DataAnalyses

22

47.3

53.555.5

44.2

63.3

57.2

48.9

44.6

52.0

44.046.0

51.0

57.0

48.0 47.0

52.0

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

Grade3 Grade5 Grade6 Grade7

Percen

tageofStude

ntsM

eetin

gorExceedingAchievemen

tLevelBen

chmarks(3

or4

)

Mathematics

PACE2014-2015 PACE2015-2016 SB2014-2015 SB2015-2016

DataAnalyses

23

43.6

49.7

40.1

48.651.3

55.3

43.1 42.4

56.0

63.0

57.0

63.0

57.0

63.0

59.062.0

0

10

20

30

40

50

60

70

Grade4 Grade5 Grade6 Grade7

Percen

tageofStude

ntsM

eetin

gorExceedingAchievemen

tLevelBen

chmarks(3

or4

)

ELA

PACE2014-2015 PACE2015-2016 SB2014-2015 SB2015-2016

DataAnalyses

Math2015 Science2016

Science2016 .487

ELA2016 .317 .637

24

Table 1. Correlation Table Grade 3-4

Science2015 ELA2015 ELA2016

ELA2015 .555

ELA2016 .459 .630

Math2016 .440 .603 .735

Table 2. Correlation Table Grade 4-5

DataAnalyses

Math2015 ELA2015 ELA2016

ELA2015 .635

ELA2016 .520 .619

Math2016 .625 .590 .648

25

Table 3. Correlation Table Grade 5-6

Math2015 ELA2015 ELA2016

ELA2015 .470

ELA2016 .482 .586

Math2016 .558 .513 .531

Table 4. Correlation Table Grade 6-7

DataAnalyses

Math2015 ELA2015 Science2016 ELA2016

ELA2015 .517

Science2016 .523 .448

ELA2016 .557 .483 .574

Math2016 .488 .477 .590 .541

26

Table 5. Correlation Table Grade 7-8

Recommendations(1of2)

OngoingMonitoring1. MonitorandSupportDistrictEngagement

2. EvaluateEffectivenessofCollaborationMethods

Additionaltraining/supports3. ConsiderAdditionalTraining/SupportsforTeachersNotDirectlyInvolvedinCommonTask

Development

4. InfuseEquityandAccommodationsTrainingintoPACEActivities

27

Recommendations(2of2)

Routinizeprogramfeatures6. RoutinizeTimelyReviewsofLocalPerformanceTasks

9. ConsiderSystematicallyRecyclingTasks

10. BeginTrackingPerformancefromYeartoYear

TargetedStudies5. InvestigatetheImpactofReading/WritingRequirementsonAccessibility

7. PlanforFutureResearchontheImpactofPACEonTeachingandLearning

8. EvaluatetheBenefitofTimeinProgramonOutcomes

28

TheStoryofPACE

• Integratedcompetency-basededucationsystemwithperformanceeventsareadramaticshiftfromtraditionalschooling

• Substantialprogresstowardthisgoal• Sustainability• Scalability• Carefulexpansion,districtbydistrict

29

ProgrammaticResponsestoPACEFormativeEvaluation

SusanLyons,Ph.D.CenterforAssessment

CCSSO’sNationalConferenceonStudentAssessment

June29,2017

PACEisaboutclassroomimpact

Lyons•Page31

ComplexPerformanceAssessments

•Buildingeducatorcapacity

•Enhancingstudentengagement

TransformingInstruction

•Increasingrigor•Developingstudentagency

CollegeandCareer

Readiness

•Preparingstudentstoengagemeaningfullyintheirpost-secondaryplans

IncreasingCollegeandCareerReadiness

Supporting All TeachersvConsider Additional

Training/Supports for Teachers Not Directly Involved in Common Task Development

Teachers routinely reported that the process of developing the common tasks greatly improved their own task development process and their approach to assessment. As the program expands, it will be important to maintain that benefit for all participants.

ü 16-17: Hiring of PACE Teacher Leaders within districts to better transmit institutional knowledge to all teachers.

ü 16-17: Investment in online intranet for all PACE teachers to share key documents and resources.

ü 17-18: Expanding high quality performance assessment development training within Tiers 2.

ü 17-18: Developing robust set of common resources for competency-based education and assessment across all three tiers of PACE.

Quality Local AssessmentvRoutinize Timely

Reviews of Local Performance Tasks

It is important to ensure that the tasks and rubrics are of sufficient quality to be used to generate student scores and annual determinations. Teachers report that their skill level in developing these tasks improves with each year of PACE participation…. Instituting a system of regular task review will help ensure that happens.

ü 16-17: The NHDOE is now reviewing one major assessment per competency for each PACE course in all of the participating districts.

ü 16-17: Contract with Stanford to review all local performance assessments.

ü Summer ’17: NH DOE will provide feedback to districts related to their assessments systems and targeted supports for those districts in need of additional guidance.

Evaluating ImpactvPlan for Future

Research on the Impact of PACE on Teaching and Learning

The positive impacts of PACE on teaching and learning should continue to be externally verified beyond this evaluation. This may be part of a future research agenda when it becomes possible to evaluate the predictive strength of PACE results on college and career performance.

ü On-going: Annual evaluation of student performance on standardized assessments. 2016 yielded some early indications of success.

ü 16-17: Providing access to PACE data to internal and external researchers

ü 17-18: Seeking funding from Hewlett to more deeply understand the connection between learning and engagement in complex performance assessments.

ü 17-18: Begin to longitudinally track trends in career and college readiness (e.g., persistence in college).

Formoreinformation:

CenterforAssessmentwww.nciea.org

SusanLyonsslyons@nciea.org