Alone in Kyoto Consulting Project Evaluation of Improving ......2. Evaluation Objectives, Scope and...

ProjectEvaluation–ICREECIIinSamloutDistrict,BattambangProvince–February2018–AiKCConsulting 33

ProjectEvaluationofImprovingChildRightsandEnhancingEducationforChildrenProject(ICREEC)inSamloutDistrict,

BattambangProvinceFebruary–March2018

ReportDate:21stMarch,2018

Alone in Kyoto

Consulting

[email protected](AU)+61404038544,(TH)+66932853435


i.AcknowledgementsTheevaluationwouldliketothankthefollowingparticipantsintheevaluation:

o Allteachers,directors,SSCmembers,andparents,fromthetargetschoolswhograciouslymetwiththeevaluatorandgaveupvaluabletimeinordertocontributetheirperceptions,knowledgeandexperiencetotheevaluation.

o Mostimportantly,allthestudentsfromtheseventargetschoolsvisited;childrenarethefocusofChildFundandKHEN’sworkanditwasaprivilegetospendsomecollectingtheideasandexperiencesfromstudentsinvolvedintheproject.

o KeyofficialsfromtheDepartmentofEducationwhomadetimetomeettheevaluatorandgaveforthopenandvaluableinsights.TheseincludeMr.HingKimhouern,PrimaryEducationOfficeChief,andMs.MeasPirun,DeputyDirectorofPOE(Primarysector)attheProvincialDepartmentofEducation,aswellasMr.TochVuthy,DirectoroftheDistrictDepartmentofEducationinSamloutDistrict.

o AllChildFundstaffinbothPhnomPenhandBattambangoffices,whoseadministrative,logisticalandtechnicalsupportwasoutstanding.Staffwereextremelyefficient,interestedinhelping,andgenerouswiththeirtimetosupporttheevaluation.

o ThestaffatKHENwhodemonstratedanopennessandgenuinedesiretoengagewiththeevaluationinaconstructiveandmeaningfulway.

Specificmentionofappreciationgoestothefollowingindividuals:FromKHEN:

o Mr.KhunBunlee:Managing/ExecutiveDirectoro Ms.AnKimsan:ProgramSupportCoordinatoro Ms.SandosMonika:M&EOfficero Mr.HoemNak:ProjectCoordinatoro Mr.HemRoth:ProjectStaff

FromChildFund:

o Mr.OumVongnarith:HeadofOperationso Mr.ChanNarin:ProgrammeManagero Mr.SanSothearo:EducationSpecialisto Ms.DyMany:BattambangProvincialManagero Ms.LunSorphea:EducationProjectCoordinatoro Mr.PhengSokheap:SponsorshipCoordinatoro Mr.HuomSarey:Driver/OfficeAssistant


TableofContents PageNumberi. Acknowledgements 2ii. ProjectSummary 4iii. Listofacronymsandabbreviations 5iv. ExecutiveSummary 61.Background 72.EvaluationObjectives,ScopeandTimeframe 82.1Objectives 82.2LocationandscopeoftheEvaluation 82.3Timeframe 93.Methodology 103.1Deskreviewofprojectdataandrelevant,externaleducationdata

10

3.2Datagathering 103.3Limitationstotheevaluation 133.4EvaluationFramework 134.FieldWorkFindings 145.EvaluationAnalysis 215.1Impact 225.2Effectiveness 225.3Sustainability 235.4Relevance 255.5Efficiency 255.6Qualityofprojectdesign 265.7Integrityoftheprojectdata 265.8Verifiedreportedresults 275.9Humanimpact 275.10AlignmentwiththeCFSframework 276.Recommendations 287.Annexes 33Annex1-DraftFieldworkSchedule 33Annex2-ExampleToolforChildconsultations 33Annex3-EvaluationSteps 34Annex4–Samplequestionnaire 35Annex5–OriginalTermofReference(abridgedversion) 39Annex6–BeneficiaryActivityMap 42


ii.ProjectSummaryProjectName ProjectEvaluationofImprovingChildRightsandEnhancing

EducationforChildrenProject(ICREEC)inSamloutDistrict,BattambangProvince

StartDate October2014

CompletionDate September2017(+3-monthcosted-extensionuntilDecember2017)

TotalFundingUSD294,989.50

FundingSources EducoDeliveryOrganisation KHEN

ImplementingPartner(s) MinistryofEducationandSports(MoES)–POE,DOE,and25individualstateandcommunityschools

Country/Region CambodiaPrimarySector Education

ProjectGoal That“remote/ruralchildrenandtheircommunitiesintheoutreachareasofSamlout,tovalue,participateinandactivelysupport&advocateforeducationandotherChildRights.”

NumberofProjectObjectives 5ObjectivesAchieved(asdefined

bytheproject’sresultsframework

4Objectives2,3,4&5

ObjectivesUnachieved Objective1couldnotbecorrectlymeasuredandwasclassifiedas‘undetermined’

NumberofProjectOutputs/KeyResults

12

ProjectOutputs/KeyResultsachieved(asdefinedbytheproject’sresultsframework

8Results:1.2,2.1,2.2,2.3,3.2,4.1,5.1,5.2

ProjectOutputs/KeyResultsUnachieved

3Results1.1,3.1&4.3wereevaluatedasnotachievedrelativetoplannedtargetsResult4.3wasevaluatedas‘undetermined’

*Furtherdetailscanbefoundinthemainreport


iii.ListofacronymsandabbreviationsBwD Boywithadisability

CC CommuneCouncil

CCWC CommuneCommitteeforWomenandChildren

CFEC ChildFund-EducoCambodia

CFS Child-friendlyschoolframework

CRV ChildRightsConvention

DFAT DepartmentofForeignAffairsandTrade

DoE DistrictOfficeofEducation

DTMT DistrictTrainingandMonitoringTeam

FGD FocusGroupDiscussions

GwD Girlwithadisability

ICREECII ImprovingChildRights&EnhancingEducationforChildrenProject(ICREEC)PhaseI

KHEN KhmerNGOforEducation

KII KeyInformantInterviews

MoEYS MinistryofEducation,YouthandSports

POE ProvincialOfficeofEducation

PTR Pupil-TeacherRatios

RoGC RoyalGovernmentofCambodia

SSC SchoolSupportCommittee

SDP/SIP Schooldevelopmentplan/schoolimprovementplan

UNICEF UnitedNationsInternationalChildren'sEmergencyFund

UXO Landmine/UXOs(UnexplodedOrdnance)


iv.ExecutiveSummaryTheEducationMinistry’s2016Mid-termreviewstatesthat“eventhoughaccesstoprimaryschoolisrelativelyhigh,childrenarenotlearningsufficiently”,andthisisagoodreflectionofthestateoftheschoolsinSamloutDistrictandthosesupportedthroughthisproject.Theprojecthasbenefittedschoolsandchildreninmultipleways,mostnoticeablyimprovedaccessandchildrenareattendingschool.WhenKHEN’sstartedworkinginSamloutwhentherewereeithernoschoolsinexistence,orverybasiccommunity-runschoolsandthestartinglevelofeducationqualitywasverylow.Theprojectmodeltargetsthoseareasthatareremote,isolatedandun-serviced,andinthiscontext,theachievementsofeasieraccesstoschoolingandbetterresourcesarecommendable.Therearealsomanyexamplesofgoodeducatorperformanceinschools,orappropriateandchild-friendlylearningenvironments.However,theoveralleducationqualityavailableinthetargetschools(materials,teachingcompetency,managementquality,anddurability)isweak.Thelinksbetweenschoolsandcommunitiescouldbestrongerandmoremeaningful,especiallyinthestateschools.AndtheevidencesuggestsachildwillnotreceiveaneducationofthestandarddesiredbyRoGCstandards,andtheirlearningoutcomeswillbeweak.

MajorAchievementsoftheProject KeyStrengthsoftheProject5%increaseinstudentsinstateschools Enrolmentcampaignactivitiesarestrongandincreasing

schoolaccess,especiallyforgirls20%increaseinstudentsincommunityschools GoodexampleofWASHhardwareinschools10%ininthenumberofgirlsinstateschools Childrightsawarenessraisinghasbeeneffectivein

strengtheningchildprotectioninschools/communities27%increaseinthenumberofgirlsincommunityschools

Relationshipsbetweenparents/communitiesand‘Communityschools’ispositiveexample

7of11‘CommunityschoolsformallytransferredtoPOE/DOE–verybigachievement

Relationshipwithgovernmentthattranslateintogovernmenttakingownershipofcommunityschools

Enrolmentcampaignscompletedeveryyear Projectmodeltargetsthoseareasthatareremote,isolatedandun-serviced-verycommendable

100%ofKidsfeelsafeintheirschools Theprojectwasanalysedinfivemajorareas-impact,sustainabilityrelevance,effectiveness,andefficiency-usingasimpletrafficlightsystem,theresultsaresummarisedbelow:

Effectiveness-Satisfactoryperformance(borderline) Efficiency-AcceptableperformanceImpact-Acceptableperformance Sustainability-AcceptablePerformance(borderline)Relevance-Satisfactoryperformance

ICREECIIhascontinuedinvestmentin25ruralschoolsandhasbeenabletograduallyimprovethematerialconditions,whilstalsoundertakingattemptsatstrengtheningandreformingteaching,schoolmanagementandcommunityrelations.Duringthisperiod,therehasbeensignificantreformandadvancementintheeducationsectors,whichhighlightsomeofthedeficienciesintheKHENmodel,butalsotheopportunitiesforthemexploreinordertogrowasaneducation-focusedNGO.Theseadvancementsmakeupthebulkoftheevaluation’srecommendationsforfutureeducation-focusedprojectwork,andareasfollows:

1. Foreducationprojects,shiftthefocusonlearningoutcomesforkids,ratherthanmaterialsgains2. KHENtogainmoreeducationtechnicalexpertiseifitwantstogrowasaneducation-focusedNGO

(kookatorgssuchasSeeBeyondBorderssuccessfulfortechnicalhelp)3. Increaseandtargetinvestmentinpre-primaryandGrades1-3.4. Avoidcascadetrainingmodelsandshifttoamodelbuiltaroundregularsupport,mentoring,

progressiveandintervaled,andin-service.5. Increaseengagementandsupportfromprojectstakeholders,andmanagementandmaintenanceof

results,inordertoimprovesustainability.6. Investinqualitysolutions/activities,evenifmeansfewertargetschools/beneficiaries-Qualityover

Quantity!7. Trytonarrowthatgapbetweenstrong,durableresultsandweak,fragileresults–improvethe

consistencyofresultsbetweenlocationandcontexts


1.BackgroundChildFundCambodiaistherepresentativeofficeofChildFundAustralia–anindependentandnon-religiousinternationaldevelopmentorganizationthatworkstoreducepovertyforchildreninthedevelopingworld.ChildFundCambodia’scommunitydevelopmentprogramsareintheareasofchildprotectionandresilience,qualityeducation,sustainablelivelihoods,improvedlocalgovernance,andyouthempowerment.ChildFundinCambodiaimplementsarangeofprogramsincollaborationwithlocalcivilsocietyorganizations,andinpartnershipwiththerelevantministriesandgovernmentdepartments,intheruralprovincesofSvayRieng,KratiéandBattambang,aswellasPhnomPenh,andfocusesonworkingwiththevulnerable,excludedormarginalizedsectionsofsociety.SpanishNGOEDUCOisaglobaldevelopmentagencywithover25years’experienceinchildrights,particularlytherighttoequitableandqualityeducation.EducodeliversprojectsthatsocialjusticeanddevelopmentprojectsinSpain,Africa,AmericaandAsia,withareachofmorethan550,000childrenand150,000adultsinpromotingjustandequitablesocietiesthatguaranteetheirrightsandwellbeing.Educoenvisagesaworldwhereallchildrenfullyenjoytheirrightsandleadalifeofdignity.EducoisamemberoftheChildFundAlliance,oneoftheforemostinternationalnetworksofNGOSworkinginfavourofchildprotectiononfivecontinents.AmanagementtransferoccurredinJuly2017inwhichEDUCOandChildFundCambodiamergedandinJuly2017andbecameajointentityknownasChildFund-EducoCambodia.TheImprovingChildRightsandEnhancingEducationforChildrenProject(ICREEC)PhaseII,implementedbylocalpartnerKHEN(KhmerNGOforEducation),wasanexistingprojectoriginallystartedbyEducowhilstitwasworkinginBattambangProvinceasanindividualdevelopmententity.KHENisalocalNGOandcertifiedChildSafeOrganisationbasedinBattambangProvince.Theyhaveafocusonchildrights,accessible,inclusive&qualityeducationandchildprotection.Theireducationprojectsfocusonruralandremoteareaswheresignificantbarrierstochildeducationexist.ICREECIIoperatedacross4communesinSamloutDistrict,whichisapproximately100kmfromBattambangTown.SamloutwasaformerKhmerRougestrongholdandstillhaslargeareasoflandcontaminatedbylandmine/UXOs(UnexplodedOrdnance).Accesstobasic(letalonequality)educationislimitedinSamlout,withdistancestoschools,resourceshortages,andweakeducatortrainingandcapacity,allnegativelyimpactingtheeducationsystem.SamloutDistrictDepartmentofEducation(DoE)hasexperiencedresourcesandskillsshortagesovertheyears,whichhasfurtherisolatedtheschoolsinthearea.AdultliteracyinSamloutislow,especiallyamongstolderparents(whoseowneducationwasimpactedthelengthofKhmerRougecontrolinthearea)andthishasaddedanotherbarriertoimprovedschoolinginthearea.ICREECIIcommencedin2014,andaimedtocontinuesupportingandbuilduponthegainsmadeinaPhase11,in25schools(state’schoolsandcommunity/annex’2schools).Theproject’sgoalwasthat“remote/ruralchildrenandtheircommunitiesintheoutreachareasofSamlout,tovalue,participateinandactivelysupport&advocateforeducationandotherChildRights.”Itsobjectivesfocusedonincreasingschoolaccessandattendance,improvingteacherqualityandschoolmanagement,andstrengtheningtheunderstandingandapplicationofchildrightswithinschoolsandcommunities.ItwasduetoconcludeinSeptember2017butundertookathree-monthno-cost-extension,inlargepartduetoimplementationdelaysresultingfromtheEduco-ChildFundmergerandthesupportuncertaintythiscreatedforatime.

1ICREECIIisanextensionofaPhase1Project,whichfinishedsometimein2014.Thisearlierversionhadsimilarfocusareasinfrastructure,teacherquality,school-communityrelations,andchildrights.PhaseIcommencedin1stOctober2012andfinishedSeptember2014(twoyears).2 Schoolsnotyetrecognizedandinthegovernmentsystemandunderthemanagementofanearbystateschool


2.EvaluationObjectives,ScopeandTimeframe2.1ObjectivesTheevaluation’saimwastoconductapostevaluationoftheICREECPhaseIIandtoproduceareportreflectingontheimpactandresultsthatwereachievedintheproject,aswellasidentifyingtheeffectivenessofitsimplementation,risksandchallenges,potentiallessonslearned,andfinallytoputforthrecommendationsandsuggestedsolutionsforsimilarfutureinterventions.TheTermsofReferenceareincludedasAnnex5inthisreport,butinshort,themainareasofinvestigationwereasfollows:Impact

• Identifyandanalyseanysignificantchangetheprojecthasbroughttodisadvantagedchildren.• Assesstheextentofoverallprogresstowardstheintendedimpactoftheproject(assetoutinthe

projectobjectives)• Provideevidenceofchangeandimpactatapersonallevelthroughsuccessstories(shortcase

studies)aboutindividualsinvolvedintheproject.Effectiveness

• Identifyforeachexpectedoutcomeandoutput(result)whathasbeenaccomplishedinrelationtowhathasbeenstatedintheprojectdocument,logicalframeworkandKHENprogrammeplan.

• Hasthemonitoringandresultsbasedmanagementsystem(includingM&EandMEALFramework)contributedtotheeffectivenessoftheproject?Haveprojectstafflearnedfromtheirexperiencesandimprovedtheprojectandtheirpracticeasaresult?

• Determinetheincreaseinthenumberofdisadvantagedchildrengettingaccesstoinclusiveeducationbecauseoftheproject.

• DescribethequalityoflearningenvironmentsandwhethertheyreachMoEYSstandards.• Identifyanystrategiesthathaveprovenparticularlyeffectiveforachievingtheoutcomes.• Describehowtheviewsofchildrenweretakenintoaccountatthedifferentlevelsoftheproject

cycleandnoteanyspecificexamplesofusefulinputSustainability

• Assessthesustainabilityoftheresultsoftheinclusiveeducationwork,including:o whichresultsaremostlikelytobesustained?o whichresultsarethemostfragile?o Whichmajorfactors(ifany)arecontributingto,orhindering,sustainability?

• Describeinwhatwaysandtowhatextenttheinclusiveeducationprojecthasincreasedtheaccountabilityandcapacityofparents,governmentbodiesandtheschoolcommunity.

• Describeandprovideexamplesasevidenceofthelevelsofownershipandinvolvementofdifferentgovernmentbodiesatdifferentlevels,andotherkeystakeholders.

Relevance

• Assesstowhatextenttheprojecthasreacheddisadvantagedchildrenandthemostusefulstrategies,

• AssesstherelevanceoftheprojecttothecontextandchildrightssituationinCambodia.• AssesstherelevanceoftheprojecttotheKHENstrategies.

Efficiency

• Assesshowefficientlyprojectresourceshavebeenused.• ThecooperationandlearningamongpartnersandwithinKHEN.

2.2LocationandscopeoftheEvaluationTheanalysiswillfocusonthegeographiclocationof4communesinSamloutDistrict,BattambangProvince,Cambodia.Theprojectareacoversagroupingof20villages,withapopulationofjustover18,000people(seeTable1below)


Table1.TargetCommunesandassociatedvillages

Thefocustargetgroupwasthe25schoolssupportedintheproject,including14official‘state’schoolsand11‘community/annex’schools,whicharedefinedasschoolsnotyetrecognizedbygovernmentasaformalstateschool,butwhichareunderthesupervisionofanearbystateschool.Thesecommunity/annexschoolsaresemi-formal,remoteandoperateinisolated,ruralareas,andwherethelocalchildrenhavereducedaccesstostateschoolsduetothedistancesanddifficultyoftravel.Theywereeitherinitiatedbycommunities,and/orbyKHEN.Theyoperatewithminimalgovernmentfinancialormaterialsupport,butdoreceiveoversightandsomesupportfromastateschoolthattheyare‘annexed’too.WiththesupportoforganisationslikeKHEN,thecommunityschoolsaimtostrengthenandimprovetoapointwheretheirlevelofinfrastructureandteachingqualityisgoodenoughtobeofficiallyrecognisedasastateschool.

Table2.TargetSchoolstudentdataasof2014Baseline.

Table3.TargetSchoolstudentdataasof2017Endline.4

2.3TimeframeTheevaluationresearch,dataconsolidationandreportingoccurredoveraperiodof13dayscommencingonFebruary16thandfinishingonMarch1,2018.TheImplementationPlanwasasfollows:

3Thedistricthas31registeredvillages,20ofwhicharereachedviathisproject.4Asummaryofthechangesinstudentnumbersofthedurationoftheprojectisincludedinthefindingssection.

Communes Villages TotalPopulation FemalePopulation%%

Sanh 3villages 5,054 2,556KampongL’pov 8villages 7,296 3,735

TaToak 6villages 3,013 1,456

Sung 3villages 2,830 1,401 20villages3 18,193 9,148

StateSchools

TotalStudentNumbers TotalFemales Totalgirlswithadisability Totalboyswithadisability

2,088(1101M-53%) 987(47%) 31(1%) 41(2%)CommunitySchools


444(247M-56%) 197(44%) 5(1%) 18(4%)

StateSchools


2,157(1060M-49%) 1,097(51%) 76(4%) 86(4%)CommunitySchools


557(286M-51%) 271(49%) 11(2%) 24(4%)


Table4.AssignmentImplementationPlan

3.Methodology3.1Deskreviewofprojectdataandrelevant,externaleducationdataKHENProjectdatawascollectedandreviewedpriortothefieldresearch,includingallavailabledesigndocuments,baselineandendlinedata,regularmonitoringreports,andotherreportingmechanismssuchasPowerPointresultssummaries,andindividualcasestudies.Thishelpedprovideabroadoverviewoftheproject’sintentandreportedresults,anditalsoallowedtheconsultanttocreateaconsolidateddatasummarythatwillbeusedandreferencedthroughoutthisreport(thesummarysheetisanaccompanyingdocumenttothisreport,titled‘EvaluationMaster’).ItshouldbenotedthatanumberofKHEN’sdatacollectiontoolswere,understandably,onlyinKhmerandtherewasinsufficienttimewithintheconsultancyorwithinKHENtohavethesetranslatedforreview.Andimportantpointtonote,itwasnotpossibletoconductanydatareviewofstudenttestscoresintheschoolsasthisinformationwasa:notavailableinanymeaningfulwayinschools(thiswillbediscussedmoreinthefindingssection,andb:notbeingcollectedbyKHEN.Although,testscoresinthemselveswerenotpartoftheproject’sevaluation/resultsframework,itwouldhavebeenaveryvaluablepieceofdatatohaveforthisreviewinordertotryandcorrelatetheproject’sinvestmentwithanincreaseinstudentperformance.Otherpertinentinformationreviewedinordertoinformthisanalysisinclude:• MinistryofEducationYouthandSportMid-TermReviewReportin2016oftheEducationStrategicPlan

2014-2018andProjectionto2020(2016).• TheSeeBeyondBorder“AnAssessmentofEarlyGradeTeachingQualityinCambodia”Report(2016).• RoGC&UNICEF’sJointEvaluationofChild-FriendlySchoolPolicyImplementationinCambodia(2016)

3.2DatagatheringAmixofqualitativeandquantitativeresearchmethodswereusedtocollectdata,butallofwhichinvolvedinterviewingsixkeygroups:1.Students 2.Teachers 3.Directors 4.SSCmembers 5.Parents 6.POE/DOEstaffAdesiredinterviewschedulewasagreedinadvancebetweentheconsultant,KHENandChildFundCambodia(includedasAnnex2),butultimately,thefinalarrangementswereleftuptoChildFundandKHEN.Theplaninplacewastoalternatebetweencommunityschoolinthemorningandstateschoolintheafternoon,andconductinterviews/groupdiscussionswithasmanyofgroups1-5aspossible.SeparateindividualsessionswereorganisedwithPOE/DOEattheirrespectiveoffices.Afullscheduleofintervieweesaspertheplanwasnotpossibleintheendduetoamixofinterviewsrunningovertime,certaingroupsrepresentativenotshowingup,andtheneedtoprioritisecertaingroupsoverothersduetothetoneoftheresearch.Asummaryoftheinterviewsinthetablesbellows:

# Deliverables/Outputs Days

February/March

16

17-18

19

20

21

22

23

24-25

26

27

28

1

1. Deskreviewanddevelopingtheframeworkfortheevaluation

2

2. Districtvisittogathertoconductfieldresearchwithschools,POE/DOE,parents,CCsandKHENpartnerorganisation

5

Dataconsolidationandanalysis 1

3. InitialdraftreportingatProvincialLevel 1 4. FinalReportsubmission,including

presentationoffindingsinPhnomPenh3

12


Table5.EvaluationSchedulebydayandtype

Total M F

Director 6 4 2

Teachers 16 3 13

Students 37 18 19

SSCmembers 3 2 1

Parents 5 0 5

POE/DOE 3 2 1

Table6.EvaluationParticipantNumberbySchool5

Table7.EvaluationSchedulebydayandtype Monday Tuesday Wednesday Thursday Friday

AM KHENOffices PhnomRussyCommunity/AnnexSchool

PhumKandalCommunity/AnnexSchool

KHENoffices OuDermchiekCommunity/AnnexSchool

KHENStaffMeeting

Children(18pax)-Games

Teachers(2pax-1xstate,1xcontract)-FGD

KHENM&EOffice,ProgramSupportstaff,andDirector

SSCmembers(2xpax)-FGD


Parents(3pax)-FGD

Parents(2pax-FGD

Director-KII Director-KII Director-KII

Classroomobservation(1hr)

Classroomobservation(30min)

Teacher(1pax)-KII

PM DounTroetStateSchool

AksorTeStateSchool

POEOffice OuNoNorngStateSchool

SungIIStateSchool

Director-KII Children(6pax)-Games

POE-DirectorandDeputy-KII

ContractTeacher-(1pax)-KII

Children(2pax)-FGD

Children(11pax)-Games

-Classroomobservation(1hr)

Director-KII Teachers(3pax)-FGD


SSCMember(1pax)-KII

DOE-DirectorSamloutDistrict-KII

Stateteachers(2pax)-FGD

Libraryandclassroomobservation

Director-KII

5Notethatasthisdocumentmayendupinthepublicsphere,intervieweenameshavebeenwithheldforconfidentiallyreasons.


InterviewsandQuestionnaires:Foradultinterviewees,individualKeyInformantInterviews(KII)andFocusGroupDiscussions(FGD),methodswereused,butbasedonbroad,individualisedquestionnairesthatwaspreparedforeachgroupandadministeredbythebytheconsultantteam6Thequestionnairesweredevelopedforeachgroupandconsistedofmultiplechoiceanswerswithfree-responsesectionforfurtherdetail;thisallowedforusefulinformationnotcapturedinthequestionset,orperhapsnotnecessarilyaprojectfocusandusefulinotherareassuchascross-cuttingissues,tobegathered.Thequestionnairesweredevelopedbasedontheproject’sdesignandthereportedinvolvementofeachgroupincertainactivities;meaningcertainquestionswereincludedtocheckif,andtowhatdegree,peoplewereinvolvedinprojectactivitiesasdescribedbytheprojectdocuments.Numerousquestionswerereplicatedacrossgroupingsinordertotriangulateinformationandalsoexploredifferentperspectives.Thequestionnairesweredesignedwithamaximumnumberofquestionsbutalsowithflexibilitytogooffscriptandfollowtheinterviewlineofenquiry,ifthesituationdemanded.So,multiplequestionsintheendwerenotused,butotherusefulinformationwasgatheredregardless.Gamesandcreativeactivities:Forchildreninterviewees,game-anddrawing-basedexerciseswereprimarilyused(basedontheprincipalsandfoundationsofchild-andyouth-participationtheory)7foryoungergradesandlargergroupsofchildren.Groupdiscussionwerealsousedwithsmallergroupsofthreechildrenorless(thisonlyhappenedonetime).Aseriesofquestionswereaskedusinggames,drawingandgroupworktools,forexamplethisquestiondesignedtoinvestigateissuesofschoolaccessandattendance:Question:Thelasttimeyoumissedschool,whatwasthereason?Activity:Childrenaskedtoplaceastickeronacorrespondingpicturethatillustratesavarietyofcommonissueschildrenwouldfacewithschoolaccessibility.ThetoolswerecreateddesignedtocheckthevoracityofprojectclaimsaswellasgiveinsightsintocriticalissuesthatChildFundandKHENwouldneedtobeawareof,forexample,issuesofsafety,orpoorteacherperformance.8Forthemostpartthesegamesandactivitiesworkedwellandyieldedveryusefulinformation,buttheconsultantdidnotethat,especiallyinthecommunityschools,childrenarenotthatfamiliarwithbeingaskedtoexpressthemselvesdirectlyinacreativewayandtherewereobviousexamplesofchildrendeferringtheiranswertoanotherchild’sanswerasthechildrengotupandengagedwiththeactivity.Forthemostpartthiswasvisibleandmanageable,butitanareaforimprovementwiththesetechniques.Duetotimeconstraints,thetoolswerenotproperlyfieldtestedbeforeuseandthisisnotedbelowin‘limitations’.DirectObservation:thiswasusedtoassessthestateoftheschools,student-teacherrelationship,andtosomeextent,community-schoolrelationship.TheConsultancyteamconducted3classroomobservations,rangingindurationfrom30minutestooveronehours,andwithgrades1,4and5,aswellasgeneralobservationoftheschoolgrounds,director’soffices,unused/emptyclassrooms,playgroundandWASHfacilities.ThebasisoftheobservationwasMoEYS’‘MonitoringandEvaluationChecklistClassroomObservation(forBasicEducation)’.Thisfour-pagechecklistcoversoff58keyindicatorsthatassessalignmentwiththeNationalChild-friendlySchoolsFramework.Thechecklistwasusedasroughguideforreviewingthedynamicsandbehaviourswithintheclassroom,aswellasthegeneralstateoftheschool.However,thechecklistswerenot‘completed’infull,ascertainindicatorscouldnotbetestedwithinthetime,andthetoolitselfrequiresfullcompletionbeforea‘score’canbeproduced.Hence,thechecklistsarenotincorporatedinthedatacollectionofthisreport,buttheinsightsfromtheclassroomobservationsarereferredtolateron.

6Notethatanexternal,independenttranslatorwasonhandtoperformtranslation.Inaddition,ChildFundstaffmember,Mr.PhengSokheap(SponsorshipCoordinator),whowasassignedtomanagelogisticsfortheevaluation,alsoconductedinterviewsandadministeredthequestionnaire,afterhavingbeeninductedinitsuse.7Basedon‘UnderstandingandEnablingChildandYouthParticipation’,ChildFundAustraliaandUniversityofMelbourne,20168 AsampleofthetoolsisincludedinthisreportasAnnex2–thefulllistisavailablewiththefinalreportpackagehandedovertoChildFund.


3.3LimitationstotheevaluationThefollowinglimitationsneedtobeconsideredininterpretingthefindingsoftheevaluation.Theevaluationwasrestrictedtovisitingalimitednumberofschools–7or25(28%)–andafractionoftheteachersinvolved–16of119(13%).Also,onlyfiveparentswereconsultedduetotimeandnon-appearance,whichimpactedtheabilityoftheconsultancytoassessandanalyseschool-communityrelations.Inaddition,thereweretimeconstraintsacrosseverysessionmeaningthatanswersoftenhadtobeacceptedatfacevalue,ratherthanverifiedfurtherthroughsecondaryquestionsinordertoensurenoconfusionormisinterpretation.Thegame-basedresearchwithchildrennotedanareaofweaknessinthatsomechildren,especiallyinthecommunityschools,seemedalittleunsurehowtorespondtobeingaskedtoexpressthemselvessodirectlyinacreativeway.Forthemostpartthiswasvisibleandmanageable,butisanareaforimprovementwiththesetechniques.Duetotimeconstraints,atestingprocesswasnotundertakentoassesstheworkabilityofthetoolswithintheCambodiancontext.Similarly,notimewasavailableforfieldtestingthequestionnairesandtheinterviewteamneededtobeflexiblewiththedatagatheringasaresult.3.4EvaluationFramework9Theevaluationframeworkhasmultipleparts,butatitscoreistheneedtoaddressthekeyevaluationquestionposedbyChildFund(aspertheToR),aswellasinvestigatinganddocumentingtowhatextenttheprojecthasincreasedparticipationinschoolingbychildren,increasedcommunitysupportforeducation,andfinally,improvedlearningoutcomesforstudents.Thebasicevaluationmatrixwillbetoconsolidateandgroupthedatagatheredthroughthefivekeyevaluationareasasdefinedearlier(Impact,sustainabilityrelevance,effectiveness,andefficiency),andthenprovideabroadanalysisforeachofthesesections,aswellasansweringthedifferentquestionsposedineacharea.Thebroadanalysiswillbeintheformofanoverall‘quality’assessment,withadescriptionoftheanalysis(forexamplehighlightingkeyopportunitiesandchallenges),aswellasaclear‘measurement’.Foreaseofuse,toCFEC,KHENandanyotherreader,asimpletrafficlightsystemwillbeemployedtoprovideasassessment‘measurement’.Anexampleofhowthiswilllookisprovidedbelow.Greenforsatisfactoryperformance-(withstrengthshighlightedaswellanyareasforrefinement)

Yellowforacceptableperformance-butneedingimprovementinhighlightedareas

Redforconcerningperformance-inneedofsignificantimprovementinhighlightedareas

Evidencewillbeprovidedforeachassessment,andthesub-questionsposedwithin,willbeanswered.Furthermore,thefollowingareaswillalsobereviewedandashortanalysisforeachprovided:

o Qualityoftheprojectdesigno Qualityoftheprojectdatao Verifyreportedresultsagainstplannedresults-throughfieldresearchcheckandtestthevoracityof

theresultsreportedo Evidenceoftheproject’simpactonpeople/children-capturesthehumandimensionand

particularlytheviewpointandexperienceofchildreninvolved–totelltheirstoryandusethatasapointofreflectionontheoverallsuccess,ornot,oftheproject.

o Theproject’sresultsandsuccessmeasuredagainsttheCambodianGovernment’sChild-friendlyschoolframework(CFS).

9Asummaryoftheevaluation‘steps’isincludedasAnnex4


4.FieldWorkFindingsTheEducationMinistry’s2016Mid-termreviewofitsstrategicplanstatesthat“eventhoughaccesstoprimaryschoolisrelativelyhigh,childrenarenotlearningsufficiently”10,andthisisagoodreflectionofthestateoftheschoolsinSamloutDistrictandthosesupportedthroughthisproject.Theprojecthasbenefittedschoolsandchildreninmultipleways,mostnoticeablyimprovedaccess;thatischildrenareattendingschool,whichisundoubtedlybeneficialinmultipleways.Itisverypossiblethatmanychildrenwouldsimplynotgotoschoolatallifthesefacilitieswerenotthere.Furthermore,KHENasagrassrootsNGO,isoperatinginremote,tryingconditions,incommunitieswhereeducationisnotprovidedorignoredbygovernment,andwitharelativelysmallpooloffinancialandhumanresources.Thelogicoftheirprojectmodelaimsforgradualimprovementsineducationquality–stepbystep–andKHEN’sstartedworkinginSamloutwhentherewereeithernoschoolsinexistence,orverybasiccommunity-runschools(e.g.runoutofaprivatehome),andhencethestartinglevelofeducationqualitywasverylow.Inthiscontext,theachievementsofeasieraccesstoschoolingandbetterresourcesarecommendable.Therearemanyindividualexamplesofeducatorgoodperformanceorattitudesinschools,orappropriatelearningenvironments,andthusthereexistsafoundationuponwhichfutureactivitiescanbeundertakentofurtherimprovetheeducationprospectsofchildren.However,asanindependentreviewoftheeducationqualityavailableinthetargetschools,thisassessmentfindsthequalitytobeatanunsatisfactorylevel.Keyareassuchasmaterials,teachingcompetency,managementquality,andoveralldurabilityoftheschools,areweakandinneedtostrengthening.Andultimately,theevidencesuggestsachildwillnotreceiveaneducationofthestandarddesiredbyRoGCstandards,andtheirlearningoutcomeswillbeweak.Aquicksummaryoftheproject’smajorstrengthsandweaknessislistedbelow,andwillbefurtherdiscussedinthefindingssection.

Table8.ICREECMajorstrengthsandweaknessessummaryStrengths WeaknessesKidsfeelsafeinschools–andstrongawarenessofchildrightsincommunities

Thedurabilityofresults(e.g.materialmaintenance,applyingtraining)isquestionable

Communityschoolsarebenefittingkidsandbetterownershipinthose–SSCseemmoreactive

Inconsistentstatebetweenschools,andbetweencommunityandstateschools

Schoolsarebeingaccessed–kidsareenrolledandattendingschools

Thetechnicalapproachtoimprovingeducationinschools–projectisreliantonPoEcascadetrainingwithlittlefollowuptoimproveeducatorcompetencyinbasicssuchasclassroompractice,curriculumimplementation,studentassessment,etc.

Evidenceofallprojectactivitieshavingbeenimplemented

Basicchildsafetyinschools–preventingchildinjuriesatschool

Manyindividualexamplesofgoodeducatorbehaviours

Overallqualityofteachingislow,highlightedinparticularbythedysfunctionalstudentassessmentpractices

GoodexamplesofWASHinschools–primarilyhardwarebutaofgoodhandwashingbehaviourstoo

Educatormotivation/discipline(causeandeffectofotherissues)

KHENhasgoodpresenceinschoolsandstrongrelationships

Studentparticipationinschoolsissubmissiveandstudent-teacherrelationshipsveryhierarchical

7of11Communityschoolstransferredtostatemanagement-greatachievement

Highriskofunsustainability/ongoingweaknesswithcommunityschoolsgivenqualityofDOEintheDistrict

10MoEYSMid-TermReviewReportin2016oftheEducationStrategicPlan2014-2018andProjectionto2020(2016),pp.7.


Akeyobjectiveoftheprojects(Objective1)wastoimproveaccesstoschoolsforruralchildren.Throughacombinationofdirectobservation,reviewingtheofficialattendancelists(whichareadmittedly,notreliablyorcorrectlyused),andthroughspeakingtochildrenaswell,theconsultancycanverifythataccessandattendance,forthemostpart,seemtobestronginthetargetgroupofschools.Verysimply,therearemorechildreninschoolnowthanatthestartoftheproject,particularlyinthecommunityannexschools.11

Impressively,therehasbeenasignificantincreaseinthenumberofgirlsenrolledinbothschools(seeStatBox.1).Thefiguresalsorevealmorechildrenwithadisabilityareenrolled,butitcan’tbedeterminedifthisisanactualincreaseormoreidentificationofchildrenwithadisability.Eitherwayimprovedunderstandingoftheissueishelpful.75%ofteachersand100%ofdirectorsinterviewedstatedtheywereinvolvedintheenrolmentcampaignsatthestartoftheyear,andsothereisasolidsystemforenrollingruralchildreninschool,andgoodcommitmentfromeducators(directorsandteachers)tothatprocess.ItshouldalsobenotedthatKHEN’ssupportforcommunityschoolscommencedinthePhase1/pilotofthis

projectandthisincludedtargetedenrolmentofruralchildrenoutofschool.

Onthedownside,83%ofteachersstatedthattheenrolmentactivitiesDONOTspecificallytargetchildrenthataremorevulnerabletomissingschool,suchastheveryremoteorverypoor,andthatenrolmentprocessesonlyfocuson‘eligible’children,definedasthosereachingtherightagebrackets.12Regularattendanceisanotherissue.Thiswasalsopartoftheproject’sresultsframeworkbuttheevaluationidentifiedanobviousgapbetween‘official’reportedresults,andavisiblereality.Teachersarekeepingrecords,albeitincorrectly(seephoto1).Theyfollowuponstudentsthatrepeatedlymissschoolbutthemostcommonresponsetothisistoeithertosendalettertotheparents,orrecordtheabsenteeisminthe

student’s‘RecordBook’andagainsendthistotheparents(seeGraph1).Theteacher’sattendancelistsgenerallypresentapositivepictureofattendance,andindeedthechildreninterviewedindicatedthatasstudentstheydon’twanttomissschoolanddosoonlyforvalidreasonssuchasbeingsick(48%)orhavingtohelpfamily(24%).But47%ofchildrenalsoknowachildintheirvillageNOTcomingtoschool,meaningthattherearestillissuesofaccessforsomechildren,andthatschoolstaffareratherpassiveintheirresponsetothis,whichisanareaforfutureimprovement.Positively,thereislittleevidenceofsignificantout-of-pocketexpensesbeingincurredby

11Theenrolmentdatapresentedearlierinsection2.2123-5forpre-primary,and6+forprimary

5%increaseinstudentsinstateschools20%increaseinstudentsincommunityschools10%ininthenumberofgirlsinstateschools27%increaseinthenumberofgirlsincommunityschoolsEnrolmentcampaignsdoneeachyear

StatBox.1

Photo1:Onlyabsentstudentsaremarkeddown-anincorrectprocess-butpositivelytheyarefollowinguponstudentsmissingrepeatedly,asindicatedbythehighlightedline.

Sendparentbook,40%

TellDirector,5%

TellSSC,5%

Callparents,35%

Askotherkidsforinfo,

15%

Graph1:Responsetorepeatedstudentabsence(Teachers)


families,orteachersconductingexcessiveprivatelessons,andmostschools13areopenforbothAM/PMshifts;meaningthatsomecommonaccessbarriersarenotmajorconcernsintheprojectarea.Pre-primarywasonlyavailableinonestateschoolvisited(andalsoinacommunity-runpre-school),likelyindicatingthatkindergarten-agedchildrenaremoredisadvantagedintermsofaccess.Itwasnotpossibletoobtaindataontheactualnumberofkindergarten-agedchildreninthedistrict,butundoubtedlythelackofpre-primaryeducationopportunitiesisaconcern,butalsoanopportunityforfutureactivities.Classroomobservationsbytheconsultancyrevealedthatboththestateandcommunityschoolsoperateinaverytraditionalwayinthatteachersarethedominantforceintheroomandthereisverylittle,meaningfulengagementfromstudents.Teachingmostlyconsisted(asperobservations)offollowingthecurriculumguides,alotofcontentrepetitioneitherontheboardsorinstudentbooks,andaskingstudentstoreadthesectionsfromtheirtextbooks.Thereappearedtobeverylittleopportunityforindependent,self-learning,forstudent’squestionsandsubsequentdiscussion,norforstudentsinputintotheclasses.Only2out6(pre-primarynotincluded),hadanactivechildren’scouncilorchildren’sclub(therewasconfusionaboutthedifferencebetweenthetwo).Twoschoolsstatedthatthechildren’scouncilhadnotbeenrenewedinthenewschoolyearastheyhadconsistedfullyofgrade6students,whohadallgraduatedandlefttheschool.AnotherschoolmentionedthattheyreliedonKHENtofacilitatethisandwiththeproject’sendthecouncilstoppedbeingactive.Astowhatthechildren’scouncildo,teachersanddirectorscouldonlynamethreethings:discusschildrights,learnabouthygiene,organiseclean-upactivitiesinschool.Therewasverylittleevidencethatthecouncil’sthemselvesplayameaningfulroleindecisionsabouttheschoolmanagement,orprovideachild’sperspectiveintoschoolmanagement.Children’sthoughtsandfeelingsabouttheirschoolarenotonlyimportanttoacknowledgeandrespect,butalsoextremelyinsightfulandthususefulforschoolmanagement.Graph2illustratesthewiderangeofinsightstheyhaveabouttheschoolexperience.Thegeneraldismissalofstudentparticipationinschoolmanagementisamissedopportunitybytheschools.ForKHEN,itwouldbebeneficialtoreviewtheirapproachtochildren’sclubsandadoptamorechild-centredmodelthatvalueschildren’sinputandamodelthathasclearlinkstoharnessingtheirknowledgeforschoolmanagement.Pupil-TeacherRatio(PTR)wasapositiveelementinalltheschoolsvisited.Noclassobservedlookedtobesurpassinga1:25(officialpolicy)figure.Thecommunityschools,whicharebynaturesmaller,generallyhadaPTRrangingfrom1:4/5upto1:15.Thestateschoolsthatarelargerandservicesignificantlymorechildren,didhaveclasseswithPTRsofaroundthe1:25,butonthewhole,thisdidnotseemanareaofconcern,andifanything.MostschoolswereabletospreadafullsetofgradesacrossAM/PMshifts,andnothavedoubleup,eveninthecommunityschoolswhereindividualclasssizeswereverysmall.Andonlyonschoolwasutilisingmulti-gradeteaching–acombinedgrade4and5class–whichaccordingtotheteacherwasduetotherebeingaverysmallnumberofG4studentsenrolledintheschool.SomeverysmallPTRsof1:5and1:6werenoticedinstateschools,and,ifthosearethegenuinenumberofeligiblestudentsforthatgrade,thenitispossiblethathavingadedicatedteacherforsuchsmallgroupsisnotthemostefficientuseofresources.Communitysupportforandparticipationinthetargetschoolsappearsgood,butprimarilywithregardstofinancialandmaterialsupport.Everyschoolhasaschool-supportcommittee,and100%ofschoolhad

13Onlyonecommunitypre-primaryschoolwasnotopenbothshifts.

Schoollabour(cutgrass),9%

Trashinthecampus,38%

Studentsfighting,9%

Faraway,3%

TooHot,13%

Groundunkept(prickly

foiliage),6%

Bullying,6%

Holesincampus,3%

DirtyToilets,6%

Lackofwater,6%

Graph2:Leastfavoritepartofschool(perchildren)


receivedsomeform‘contribution’fromthecommunityandstudentparents,beitmoney,orbuildingmaterials,orin-kindlabour.ThesupportfromthelocalCommuneCouncils–80%oftheschoolshavenotreceivedanythingfromtheCC,accordingtodirectors-isinstarkrelieftothatofthecommunity.Bothteachersanddirectors,describedtheSSCsinthecommunityschoolsintermsthatindicatetheyaremorepro-activeandengaged,whichislogicalconsideringtheyoftensprangupasacommunityinitiative.ButtheSSCsseemtoasaresourcemobilizationbody.WhenaskedwhatwasthelastactivitytheSSCwasinvolvedwithattheschool,onlythreeresponsescameback:woodcollection,generallabourandschoolrepairs,andeveningtheground.Again,thisappearstobeamissedopportunitytogetmorepeopleinvolvedwithaspectsoftheschoolsthatareunder-resourced,suchasteachersupportorkid’sclubs.Theengagementwithparents,initiatedfromschoolsisamixedresult;noteacherindicatedorganisinganyformalone-on-onemeetingswithparentsatschool,rathertherearelotsofinformalmeetingsinwhichthegeneraltopicswereeither“studentprogress”or“contributions”.Therearealsogroupmeetings/parentforumsheldonceortwiceayearinwhichstudentprogressmightbediscussed.SimilartotheSSCs,parentsseemtobeoverwhelminglyviewedbyschoolstaffastheretohelpwithtwothings,homework,andhelpingtopayforschoolrepairsorupgrades.Thisnotunexpectedbutaweaknessandmissedopportunityinschoolmanagementapproaches.Aninterestingcontradictioninthefindingsisthat80%ofdirectorsbelievethatparentsare“generallyhappy”withtheschool,buttheyalsoallclaimparentscomplaintoomuchabouttheschool;clearly,thebondsbetweenparentsandschoolmanagementcanbeimproved.Theevaluationwasunfortunatelynotabletospeaktoasmanyparentsaswouldhavebeenideal.However,thesmallsampleinterviewedshowedaninterestingbut(anecdotal)division–theyoungerparentswhocouldreaddemonstratedmoreenthusiasmforschoolmattersandmoreknowledgeabouttheirchildreninschool,thantheolderparents,whoalsohappenedtobeilliterate.Again,thisisnotunexpectedbutimportanttopointout-forfutureactivities-astheyoungerparentscouldbegivenmeaningfulroleswithschools,andtheolderparentscouldbesupportedmoredirectlysothattheycanformastrongerunderstandingandthenconnectionwithwhathappensintheschools,andtheirchildren’seducation.Overallschoolmanagementappearsinconsistent,weakattimes,andinneedofmorestructure.Forexample,thedirectoratDounTroetStateSchooldemonstratedarealenthusiasmforhisjob,wasabletoproducerecordsandmaterialsquickly,thosematerialsweresecureandkeptinanorderlymanner,andthegeneralstateoftheschools’groundsandclassroomindicatedgoodinternalattitudesandpractices(seephoto2).Ontheotherhand,twoofthecommunityschools,whichhavebeentransferredandareofficiallynowstateschools,sharetheirdirectorwithanotherstateschool;thedirectoronlyvisitsforamonthlymeetingwithteachers,oronspecialoccasions(liketheevaluation).Mostschoolscoulddemonstrateaprocessofmonthlymeetingsbetweenthedirectorandteachers,butthiswasconcernedwithurgentissuesandsometraining–itwasnotstructured‘schoolmanagementcommittee’perse.Assuch,teachersseemedtohavelittleinfluenceinschoolmanagementotherthanpassingonfeedbackviainformalmeetingswiththedirectors.Therewasaclearhierarchybetweenstateteachersandcontractteachersandschooldirectorsdidutilisethisasalayerofmiddlemanagement/support:stateteacherswereexpectedtohelpandmentorcontractteachers,particularlyinthecommunityschools.

z

Photo2:DounTroetStateSchool–originallyanannexschoolthematerialinvestmenthasbeenmaintainedandsteadilyexpanded-apositiveexampleofvaluingandtakingadvantageofexternalsupportratherthanlettingitdegrade.


Only20%14ofschoolshadacurrentschooldevelopment(SDP)plan,orschoolimprovementplan(SIP).Onedirectorstatedhewas“waitingforKHENtohelpwithit”.Furthermore,prioritisationofresourcesbyschoolmanagementwasverynarrowandfocusedonthehardwareneedsofaschoolonly(seeGraph3).Itisunclearifthisisaninabilitytodifferentiatebetweenmaterialsandsoftwareneeds(withallschoolsneedingmoresoftsupport);abeliefininvestinginmaterialsleadstoimprovedstudentperformance;alackofconfidenceintheirownideas,orsomethingelse.Whatisclearisthatsupportforeducatorsonhowtoidentify(andthenaddress)criticalneedsversusnon-criticalneedswouldbebeneficialforfutureactivities.Encouragingly,100%ofschooldirectorsarefamiliarwiththeCFSframework,canremembergoingtotrainingonCFS(providedbyPOEandKHEN),and80%haveseenaDTMTCFSchecklist.Butastronganddetailedunderstandingofthepolicyandthenhowtoapplyitmeaningfullyandsuccessfullyseemslacking,withonly40%ofbeingabletonameaspecificexampleofusingtheCFSindailyschoolwork.Directorsconductteacherperformancemanagementviaformalprocessesonly50%ofthetime,and60%ofdirectorsstatedthereisnosystemforfollowingupareasofweaknessidentified,otherthansomethinglike“verballypointingoutweakareasanddiscussingwaystoimprove.”

1420%hadaplan;20%claimeditbeingprepared:60%hasnoplanasyet.

Photo3:DounTroetStateSchool–integratedmodel,wellmaintained

Photo4:SungIIStateSchool–handwashingstationinheritedfromanotherNGO–poorlymanaged–thestationis‘padlocked’frombeingused

Photo5:OuDermchiekAnnexSchool–goodaccessandlatrinestructure,withaclosabledoorandwater,gooddistancefromthemainschoolstructure

Photo6:PhumKandalAnnexSchool–poorstructuresandpoorlymanaged

Graph3:ThelastSDP/SIPitemachievedinschool

Buildingupgrade,33%

Toilet,33%

Garden,17%

Groundleveled,17%


WASHinschoolsisgenerallyanareathatwasverypositivefromschooltoschool.Theruralcommunityschoolshaveaccesstobasicstoiletsandsomeformofwatercollectionandstorage.Waterfiltrationdeviceswereavailableinsomeclassroomsandtherewerefunctioninghandwashingstations,aswellashygienepromotionmaterialsvisible.ThesysteminDounTroetStateSchoolisaveryeffective,integratedmodel,withwatercollectioncapacitysufficienttolastthedryseason,accordingtothedirector.Butaswithanumberofaspectsoftheproject,inconsistencyandpoormaintenanceblightotherschools.Multiplelatrineswereseenwithnoflushingwateravailableonhand,despitetherebeingwaterattheschool,andtheirgeneralcleanlinessandmaintenancewasquitepoorin4outof7schools.Seeselectedphotos3-6abovetoillustratetheinconsistency.Studentswereseenwashinghandsinschoolsandinterviewedteachesspokeofencouragingthisduringbreakstime.Teacherperformanceintheschoolsisalsoamixedsetofresults.Theprojectsponsoredapproximately15contractteacherstobeinvolvedintheproject,butonly50%ofthecontractteachersinterviewedhadparticipatedintheproject,theothershadleftorbeenremoved.15Thenewcontractteachershadreceivednoneofthelevelofsupportastheoriginalsplacingthematasignificantdisadvantagetotheboththefieldresearchquestions,butmoreimportantlytheirworkintheclassroom.Only44%ofteacherscouldproperlynameoneofthesixkeydimensionoftheCFSframework,andonly25%wereabletorememberanyCFStrainingoverthelastyear.83%hadnotseentheG1-G3nationalreadingbenchmarkguides.92%saidtheyhadnomentortogotoforhelp/orwerenothelpinganyotherteacherwithissues,and54%saidtheyonlypreparetheirlessonsforthenextclass,thedaybefore,andtheyusuallyonlyplanforKhmerandmaths.Finally,60%ofteacherskeeptheirmaterialsinanuntidyandinsecuremanner,primarilyinalaptopbag,andtheprevailingapproachisforteacherstokeepstudentrecordsattheirhome,withnoduplicates;raisingthequestionwhatwouldhappentoachildifthoserecordswerelost?

Therewasquiteawiderangeofresponsetothequestionofwhatisthebiggestchallengeintheclassroom(seeGraph4),indicatingthatthereareweaknessesacrossallthebasicsofteachingandclassroommanagement;but‘slowlearner’wasrecordedasthelargestchallenge,indicatingteachersdon’tfeelknowledgeableonhowtodealwithlearningvariability,butalsojusthowtoughthewholesystemisastheyarenotgettingtheadditionalsupport,especiallyin-service,supporttheyneed.54%statedthattheDTMThadnotvisitedinthelastthreemonths.Theskillsgapintheclassroomsisalsoreflectedinresponsesbychildren–95%studentsnamedtheirfavouritesubjectaseitherKhmerormaths,indicatingjusthowdominantthesesubjectsare,andhowlittle,andhowpoorly,othersubjectsaredelivered.Another

veryinterestingobservationfromtheevaluationwasthebehaviourofmoststudentswheninvolvedwithoneoftheresearchtools;inadrawingexercise,whengiventhechoiceofmany,manydifferentcolouredpencils,andasmallnumberofgrey,leadpencils,thechildrenalwayschosetheleadpencilsfirst;possiblyindicatingadiscomfortorunfamiliaritywithcreativeexpression,butanoticeablepatternofbehaviour.Asgloomyasthatsounds,theevaluationactuallyfoundcauseforhopeandpositivity.Despitemostoftheteachersbeingill-trainedandunder-supported,thereweremanyexamplesofreallygoodteachinginstincts,orapplicationofsomethinglearnedatatraining,including:

• themakingofadditionalclassroomlearningmaterialsoutsideofschool;• preparinglessonwellinadvance(oneweekbefore)andforafullweek;

15AnRoGClawrequiringcontractteacherstohavefinishedatleastyear12wasoneofthemajordriversofthisandoutofthehandsoftheproject

Graph4:Biggestchallengeintheclassroom(teachers)

Slowlearners,25%

Materials,25%Parentsupport,

13%

G1kidswithnopre-schoolfoundation,

13%

Discipline,13%

Keepingattentionofkids,13%


• realisingthatstudentsareboredtheystopusingthetextbookandswitchtoanotheractivity;• mixingthelessonswithphysicalactivitytootokeepkidenergised;and,• 36%ofteachers,includingcontractteachers,knowingthatslowlearnersneedmoresupport

(seeGraph5)• schoolscomplyingwithministryregulationsbynotputtingcontractteachersintheimportantG1-3

(forstateteachersaresignificantlymoreskilled).

Alltold,thereweremanyexampleofgoodpracticesandinstinctswithinteachers,theyarejustnotseenconsistentlyenoughfromschooltoschool,orasacompletesetofskillsfromteachertoteacher.Butthefacttheyexistmeanstheycanbefurtherdevelopedandutilisedtocreatepeer-to-peerlearning.Testingintheschoolsispoorandamajorareaofconcernfortheschoolgroups.100%ofteachersaredevelopingtheirowntesttemplatesattheschool,ratherthanusingformalmaterialsoreveninformalmaterialknowntowork.Oneteacherdescribedtheprocessofassigningresultstoareadingexamas‘thebestscoresgothechildrenwhoretellthestoryexactlyasItoldit’.ThelackofawarenessoftheG1-3readingbenchmarksisanimmediateopportunityforfutureworkintheseschools;introducethesetotheschools,traintheteachersanddirectorsonthem,andsupportthemin-situtobeusethemcorrectlyandeffectively.Withregardstoifproject’sactivitieshavehadanyimpactonlearningoutcomesforstudents,theevaluationcannotmakeanydeterminationaboutthisforthereasons:itwasnotpossibletoconductanydatareviewofstudenttestscoresonaccountoftheinformationisnotavailableinanymeaningfulwayinschoolsorwithPOE.Testingmethodsofteachersareweakandveryvariabletothepointthatthescoresthemselveslikelyhaveverylittlerelevancetoastudent’sactualability.Andsecondly,theprojectdidnothavetheresourcestotryandcollectthedataindependentlyfromtheschoolsystem.ThepromotionofchildrightsisanintegralpartofKHEN’sidentityandalongwithastrongchildprotectionapproach(thatcanbeutilisedinschools),hasbeenasuccessoftheproject.Teachers,students,anddirectorsalikedemonstratedgoodknowledgeabouttheconceptsofchildrightsandchildprotection.100%ofstudentsfeelsafeatschool,and76%feelsafeintheirvillage–theother24%identifiedlocalhazardsastheroadorriverasdangers.Themajorityofschoolshasachild-protectionresponseposter,providedbyKHEN,readilyavailableandvisible.Aspositiveasthisis,theresearchalsoshowedthereissignificantinconsistencyintheresponsesbyteachersanddirectorsastohowtheywouldhandleapossiblechildprotectioncase.Only20%ofdirectorsindicatedtheywoulduseKHEN’sreferralsystem,and25%ofteachersstatedtheywouldcontacttheparentsfirst,ifinthecaseoftheparentbeingtheperpetrator,isconcerning.Theseresultsareaslightlyskewedasthenewcontractteachershavenothadthesametrainingasothersanditslikelytheirresponsesareweaker.Theseresultsarealsoatoddswiththeapparenthollownessofthechildren’sclubsandanylackofinputfromthemintoschool-decision-making.However,theknowledgeor

MiniCaseStudyAcontractteacheratOuNoNornStateschool,whohasbeenthereforlessthanthreemonths,whoreceivedonly2daystrainingwiththedirector,andhasyetnotbeenpaid,wastheonlyteachertosaythatshewasinstructingafullrangeofsubjectsthroughtheweek(notjustKhmerandmaths)andwasusingfungamesinclasstohelpthechildrenwhenshecanseetheyaregettingbored/inattentive.Hernaturalinstinctsforteachingweregoodandshehasthepotentialtobeagoodcommunityteacher.

Graph5:Howdoyouapproachslowlearners?(teachers)

Moreattention,

36%

Getstudenthelp,5%

Getstaffhelp,0%Getparent

help,14%

GetSSChelp,0%

Givemorehomework,

36%

Moreclasses,5%

Other,5%


one’srightsisanimportantstartingpoint,andfoundationfromwhichchildrencanmoreconfidentlydemandbetterservicefromtheirschoolsandleaders.Akeyfindingfromtheevaluationistheneedforimprovedchildsafeguardingpracticesintheschoolwithregardstocommonrisks,orinotherwords,theabilitytospotthingsinandaroundtheschoolsthatchildrencouldinjurethemselveson.Theevaluationsawanumberofwayschildrencouldhavebeeninjured,whichschoolstaffdon’tseemtobeabletoidentify(seephotos7,8and9).Thisisarecommendationforthefuture,especiallyfortwochild-focusedorganisations.

Finally,akeyobjectiveofthisprojectwastohaveapproximately60%ofthe11communityschoolsconnectedtotheproject,transferredovertomanagementbytheDOE/CommuneCouncilandrecognisedasofficialstateschools.Thetablebelowsummarisesthatresult.Thecommunityschoolmodelisverygoodconceptuallyforlocalownershipandsustainability,andthetransferof7of11schoolstothestateisveryimpressive.Theevaluationdoeswanttoacknowledgethisachievement,astheschoolsarekeptgoingthroughamixofDOE/communecouncilandcommunityinputs.Inparticular,thelandusedforthecommunitypre-primaryschoolshasbeendonatedbycommunecouncils,andthisisaverysignificantcontributionandbasisforanenduringschool.

Table9.Summaryofcommunity-schooltransferstoPOE/DOE/CCPrimarySchools Pre-primary/CommunityKindergarten3schoolsfullyandformallytransferredtoDOEmanagement2schoolsintheprocessoftransfer.2notreadyfortransfer

All4havebeensuccessfultransferredovertomanagementbytheCommuneCouncil

5. EvaluationAnalysisAsperthetermofreferenceforthisevaluation,therearefivemajorareasofanalysis,asoutlinedbelow.Asimpletrafficmodelwillbeusedtoevaluatetheperformanceineacharea,usingathree-categoryscoringrange:

• Greenforsatisfactoryperformance-withstrengthshighlightedaswellanyareasforrefinement• Yellowforacceptableperformance-acceptablebutneedingimprovementinhighlightedareas• Redforconcerningperformance-inneedofsignificantimprovementinidentifiedareas

Evidenceanddatawillbeincludedtosupporteachassessment,andtohelpanswerthesub-questions.

Photo7:Rustyandsharpoldschoolswing–alsoveryhotintheglaringheatoftheday

Photo9:Childplayingwithbricksleftinschoolyard

Photo8:Heavycementbagsleftinclassroom


(5.1)Impact• Changefordisadvantagedchildren• Progresstowardsintendedimpact

• Changeandimpactatapersonallevel

AcceptablePerformance

SummaryAnalysis:Overalltheevaluationassessesthisas‘AcceptablePerformance’.Thereisclearevidencethatmorechildren,andchildrenwholiveinremoteareas,areinschoolasaresultofthisproject’scollaborationwiththetargetschoolsandDOE.Thisimprovedaccessisofsignificantbenefittochildren,evenifaqualitylearningenvironmentandexperiencecannotalwaysbeguaranteed.Theopportunitiestheygettosocialise,play,learnatschoolarestillveryvaluable.Classobservationsrevealedthatbasiclanguageandmathsskillsarebeingtaught,andevenifthequalityisnotstrong,thisishelpingtopreventthepossibilityofgrowingupilliterate.Theobservationsalsoshowthatchildrenaregenerallyactiveintheclassandwiththeteacher,asmuchastheteacher’smodelofinstructionallows,buttheyareattending,andbytheirownaccountenjoycomingtoschoolandtrynottomissschoolifpossible.So,thechildrenthemselvesvaluetheopportunitytocometoschoolandthisisapositiveimpactonapersonallevel.Therehasalsobeenanimpressiveincreaseinthenumberofgirlsenrolledinbothstateandcommunityschools,andalsothenumberofchildrenclassifiedas‘withadisability’16enrolled–sointhisregardtheprojecthasdoneasolidjobofpositivelyimpactingvulnerablechildren.Theprojecthashadlessimpactarewithregardstolearningoutcomes,andcommunity-schoolrelations.Learningoutcomescouldnotbetestedinanymeaningfulway,asmentionedearlier,sonoassessmentcanbemadeoftheproject’simpactonstudenteducationprogress.However,classroomobservationsofteachingapproachesandteacher’sself-descriptionsofhowtheyprepareanddelivercontent,wouldleadtheevaluationtobelievethatchildrenwouldlikelynotscorewellwhentestedagainstnationalstandards.Andalthoughcommunities’schoolsdohavestronglinkswithparentsandthecommunitiestheyexistin,theirinputisreallylimitedtofinancialandmaterialcontribution,attendingschool‘publicforum’onceortwiceayear,andpossiblyhelpingwithhomework.Therewerefewexamplesofparentsinvolvedinclasses,helpingteachers,helpingintheschoolyard,orbeinginvolvedinanyschoolmanagementdecisions.Insomuchasthey“value,participateinandactivelysupport&advocateforeducationandotherChildRights”theevaluationcan’tsaywithconfidencethishasbeenachievedtohighlevelorconsistentlevel.

(5.2)Effectiveness• Outcomeandoutput(result)accomplishedinrelationtotheprojectdesign• EffectiveM&Esystem/evidenceofstafflearningandimprovedprojectpractice• Thequalityofthelearningenvironmentininclusiveclassrooms/reachedMoEYSstandards.• Effectivestrategiesthathaveprovenparticularlyeffectiveforachievingtheoutcomes.• Howtheviewsofchildrenwerecapturedinprojectcycle

SatisfactoryPerformance(borderline)

SummaryAnalysis:Overalltheevaluationassessesthisas‘SatisfactoryPerformance’butborderingononly‘Acceptable’duetotheresultsachievedfor‘qualityofthelearningenvironment/reachingMoEYSstandards’.Generally,KHEN’ssystemsandprojectcyclemanagementaresolidandarebeingapplied.TheyusedtechnicalapproachesthatwerestandardforCambodia(andarestillusedwidely),forexamplecascade

16Enrolledstudentdata:StateschoolBaseline-31GwD&41BwD CommunityschoolBaseline–5GwD&18BwD StateschoolEndline-76GwD&86BwD CommunityschoolBaseline-11GwD&24BwD


training,butdoneedhelpinthisareatoimproveandcatchuptoadvancementsmadeoverthelastfewyearsineducationprojectwork.Table9:Theproject’sachievementsasperitslogframe(plannedversusactual)PlannedObjectives(#)

ObjectivesAchieved(definedasmeetingindicatorssetoutinthelogframe)

ObjectivesnotAchieved/Undetermined

Comments

5 4 1Objective1couldnotbecorrectlymeasuredandwasclassifiedas‘undetermined’

PlannedOutputs/Results(#)

OutputsAchieved(definedasmeetingtheindicatorssetoutinthelogframe)

OutputsnotAchieved/Undetermined

Comments

12

8-Results:1.2,2.1,2.2,2.3,3.2,4.1,5.1,5.2

4-Results1.1,3.1,4,2&4.3

Result4.2&4.3wasevaluatedas‘undetermined’

TheinformationaboverepresentsasatisfactoryresultandisusefulforanorganisationsuchasChildFund,whichisfundinglocalNGOs,tohaveanoverviewofplannedresultsversusachievements.However,‘achievement’hereissimplydefinedasrealisingthesuccessindicatorssetoutinthelogframe,withoutadeeperreviewofthequalityoftheresults,andsoitoffersonlyalittleanalyticalvalue.Inaddition,ananalysisofthelogicwithinthelogframe,therelevantindicators,andthedatacollectionbehindtheindicators,showsthatstructurallythelogframedesigncouldhavebeenstrongerandthatsomeindicators,inandofthemselves,didnotreallyprovideapropermeasureofsuccess.Twoexamplesinclude:

o Objective3:TofacilitatecollaborationwithP/DOEandSSCstoachievesustainableimprovementinthestandardofschoolmanagementinaccordancewithCFSstandards

o Indicator1:63%ofcommunityschoolsaremanagedbyDOEby2017.Thisindicatordoesnotreallytellusanythingabout‘achievingsustainableimprovement’inschoolmanagement–it’ssimplyacountingofschoolsthathavebeenreclassifiedfromcommunitytostateschool.

o Output2.1:Teachingpracticeshaveimprovedandincorporateachild-centredapproachasmeasuredbytheMoEYSCFSdimensions

o Indicator:75%ofCFS/DOEteachingstandardsareconsistentlymetin14stateschoolsand11community(primary&pre-)by2017.

Thisindicatorwasmeasuredusingself-assessmentformsansweredbyteachersanddirectors,andalthough‘self-assessments’arecommoninCambodia,theyarenotthebesttoolduetoobviousconflictsofinterestinsayingwhataninterviewerwantstohear.Willteachersactuallyadmittonot‘consistently’meetingstandards?Inaddition,thesomeofthequestionscouldhavebeenbettercrafted.Forexample,intheinterviewformswithchildren,itwasasked“Doyouthinkyourteachergivesyouenoughattention?”Willachildknowwhat“enough”isiftheyonlyhaveoneexperiencedonestyleofteaching?KHEN’scommitmenttodatacollectionthoughandtheircoresystemsfordatagatheringareverygoodandeffective,especiallyforalocalNGOoperatingonarelativelysmallbudget.Theyhaveaccessibledataanddatacollectiontools,andaclearoverarchingM&Eframework.ThisseemslikeanareatheyhavereallyinvestedinandareuncompromisingintheirM&Ecommitment,andKHENistobecommendedforthis.Inthisproject,theirbaselineandendlinesurveysweresignificantpiecesofworkandveryimpressive.TherearehoweverafewwaysinwhichtheirM&Esystem,andspecificallydataanalysis,couldbeimproved(seesection5.2/5.3formoredetails).Thequalityofthelearningenvironmentsisinconsistentfromschooltoschoolandthat,resultinginpartfromteachercapacity,isanoverallweaknessoftheproject’seffectiveness.TheevaluationteamusedtheMoEYSCFSchecklistaspartoftheirfieldvisitsandallschoolsfailedmultiplesections;reallyDounTroetStatewastheonlyonethatcameclosetomeetingMoEYSstandards.Thedataistoomuchtoincludehere,butthekeypointisthatschooldirectors,teachersandcommunitymemberallhaveageneralsensethatthe


CFSframeworkexistsandthatitincludestheserequirementofinclusive,safe,friendly,andnurturingschoolenvironments,yettheyarenotskilledenough,orresourcedwellenough,tobeabletocreatethem.TheCFSframeworkisanambitiousagendaandinthecommunityschools,inparticular,itisunrealistictoexpectthemtofullycomply.Butitisalsotruethatthefundamentalslikegoodrecordkeeping,lessonpreparation,neatandtidyclassrooms,andchild-safeenvironmentsarenotbeingmet.Thedeliveryandapplicationoftrainingviathecascadetrainingapproachseemstohavebeenineffectiveasthereappearstobelittleevidenceofthetraininginfluencingthedailysituationinschools.Infutureprojects,trainingactivitiesneedstobestructuredtoincludeopportunitiesforregularapplication,andfollowupsupporttoengrainlearning.Thetwomosteffectivestrategiesintheprojectweretheannualenrolmentcampaigns,supportedbycommunesandDOEstaff,andthecommunityschoolmodel.Bothhavecontributedtoincreasingenrolmentsforruralchildren.Thecommunityschoolmodelintheoryisaverygoodone–itreacheschildrenintheveryisolatedpocketswhootherwisemightbetemptedtoskipschoolbecauseofthedistances,andaimstotransfertheschoolstostatemanagementatsomepoint,whichisastrongconceptoflocalownershipandsustainability.Andthisprojecthasseen9out11communityschools‘transfer’tothestate,whichisgreatsuccess.However,continued,sufficientresourcingoftheschoolsbygovernmentcan’tbeguaranteed(obviously),whichisaclearrisk.Asisthefactthatthecommunityschoolsarenotabletoprovidethesameteachingstandardasstateschools–theyarejustnotresourcedwellenough–andthereisadangerofcreatingagroupofstudentswhowillmoveintootherstateschoolsinlatergradesandbesignificantlybehindtheotherstudents.Thisiswhyafocusonteachingqualityandin-servicesupportforallschools,communityorstate,isreallyimportant.Lastly,regardingchildren’sinputintotheproject.KHENwasverythoroughandsensitivewhencollectingbaselineandendlinedatafromchildrenandistobecommendedforthis.However,theexampleofthechildren’scouncil/childclubs–eithernotfunctioningoractingsolelyasaplacetolearnabout‘childrights’and‘hygiene’-indicatesthatmoreregularandmeaningfulinputfromchildreninduringtheprojectcycleissomethingtoaimforinfutureactivities.

(5.3)Sustainability• Sustainabilityoftheresults(thosemost

likelytobesustained,mostfragile,factorscontributingto,orhindering,sustainability)

• Accountabilityandcapacityofparents,governmentbodiesandschoolcommunity

• Ownershipbyprojectstakeholders

AcceptablePerformance(borderline)

SummaryAnalysis:Overalltheevaluationassessesthisas‘AcceptablePerformance”butborderingon‘ConcerningPerformance’.ThisisthemostchallengingareafortheprojectandKHEN.Themajorissuesinclude:

- Inconsistencyofresultsfromschooltoschoolandteachertoteacher:It’shardtohavelastingimpactinacommunityifresultsaresovariablefromonegroupofstakeholderstoanother.

- Maintenanceofmaterialinputs:linkedtoinconsistency,andthereisawiderangeofhowwellthingslikelibrarycorners,schooltextbooks,andWASHinfrastructure,areorarenotmangedandmaintainedinschools.Thisgapneedstobenarrowed.

- Thedeliveryandapplicationoftraining:thecascadetrainingmodelseemstohavesomewhatineffectiveintheprojectastherewasverylittlespecificrecallbythoseinvolved,meaningtherehasbeenlittlebenefitfromthetrainingtothedailysituationinschools.Itappearsmuchofthetrainingtopicswouldhavetobedeliveredagain,whichisessentiallyaresultthathasnotbeensustained.Futuretrainingneedsopportunitiesforregularapplication,andfollowupsupporttoengrainlearning.


Asmentionedabove,thecommunityschoolmodelisconceptuallyverygoodforlocalownershipandsustainability,andthetransferof7of11schoolstothestateisveryimpressive.Theevaluationdoeswanttoacknowledgethisachievement,astheschoolsarekeptgoingthroughamixofDOE/communecouncilandcommunityinputs.However,theseschoolsarealongwaybehindotherstateschoolsintermsofinfrastructure,materials,teacherqualityandschoolmanagement–forexampletheyhavenopermanentschooldirectorandtheteachersreceiveamonthlyvisitfromthedirectoratthenearbystateschooltheyareannexedto.Thisseemslikeafragilemodelthatcouldcollapse,orcreateaschoolthatdoesnothaveapropermanagementcapacityandthusleadingweakerlearningenvironmentsandoutcomes.TheagreementswithDOE/POEthatgovernthetransferareveryloose,asdescribedbyPOE/DOEdirectly,anddonotcontainspecificprovisionsofmaterialorteachersupport.Thereisnoformal‘communityschool’policyorframeworktoapply,rather,theynowfallundertheCFSframework,butaresignificantlyshortofmeetingminimumstandardsinthisregard.Soasmuchasthetransferitselfisagoodexampleofsustainabledevelopmentandgovernment/communitycollaboration,thelong-termviabilityofthecommunityschools,withoutexternalhelp,isquestionable.Theenrolmentactivitiesareasustainableapproachandeffective.Therewashighcommitmenttothetheseoccurringannually,withgoodinvolvementbyteachers,parentsandexistingstudents.Thesearedeliveredbylocally-conceivedpublicannouncementactivities,whichseemeffective,andeasilyimplementedbydifferentpeople.Onthedownside,thelocalmanagementofkeyprojectinputs,forexamplethelibrarycornersorlatrines,wasinconsistentfromschooltoschool.Inthoseweakerschools,theobviousdriversofthepoorerqualityincluded:

- theabsenceofamanagementsystem–forexample,theschoolswithdirtyandrun-downlatrineshadnoclearprocessandnosystemofsharingresponsibility,suchasaroster,forcleaningthelatrinesdailyandprovidingflushwater,leadingtotheirneglect.

- Andalackofleadershipandownershipbyteachersanddirectors.Graph2–leastfavouritepartofschoolasperchildreninterviewees,revealsthateasilymanageableissuesuchastrashcollectionanddisposal,isnotbeingdonewell.Multipleteachersalsocomplainedthatchildrenaretoo‘naughtyandbreakthingstoomuch’,whichisacommonandconvenientexcuseforinactivity.Thisisnotacrosseveryschoolbutthislackofpositiveandaccountableleadershipatsomeschoolsisanareaneedingaddressinfutureactivities.

Asreflectedinthecommentaboveabouttheinconsistencybetweenschools,thereisinconsistencyinownershipbyprojectstakeholders.Activeandengagedteachers,directors,SSCmembersandparentsdoexist,butnotacrosseachgroupandforeachschool.KHENknowsthevalueofharnessingnaturalleadersor‘models’‘forsoftware-focusedwork,17andthatiswhatitsmissingwithcohortofteachers,directorsandcommunitymembersinSamlout;adedicatedefforttogivesuchpeopleopportunitiestobeinvolvedmore,supportandhelpthemwhentheencounterchallenges,andhelpthemsharetheirdevelopmentwithothers.18Contractteacherswerealsooneofproject’smostfragileelements.Thehighturnoverofthosecontractteacherswhoparticipatedinthethree-yearprojectmeanstheinvestmentinthatareawasnotsuccessfulandunsustained(dueinlargeparttothegovernment’slawchangeregardingeligibility).Theideaofthecontractteacherissoundandhasdemonstratedsuccessinotherprojects,andshouldcontinuetobepursued.However,thefactthattheDOE/schoolsinthisprojectarenotabletoadequatelyreplace,trainandsupportnewcontractteacher,indicatestheneedforarevisedapproachinfutureKHENactivities

17Thereisamodelparentsgroupintheprojectbasedonthatconcept18SeeChildFund/KAPE’sworkontheEasy2LearnModel,orSeeBeyondBordersMentoringprogrammeinitswork,forsuccessfulexamplesofthis.


(5.4)Relevance• ProjectrelevancetothecontextandchildrightssituationinCambodia.• ProjectrelevancetoKHENstrategies.

Satisfactoryperformance SummaryAnalysis:Overalltheevaluationassessesthisas‘SatisfactoryPerformance”.ByRoGC’sownadmission,educationandlearningprogressinCambodiaisstillverychallengedandtherighttoqualityeducationnotrealisedformostchildren.Hence,thisprojectisveryrelevantforCambodia.ItisalsocompletelyinalignmentwithKHEN’sownstrategicpriorities.Itsfirstobjectiveunderitscurrentstrategicplan(2015-2017)isto“contributetoacontinuingimprovementintheeducationandwell-beingofallChildren.”Therearetwoimportantinsightsrelatedtorelevance.Firstly,thetechnicaleducationapproachesusedinthisprojectwereprettystandardforwhenthisprojectwasdesignedandstartedthreeyearsago.Someverypromisingdevelopmentworkineducationhassincetakenplace,partlydrivenbystrongperformancebyNGOs19andpartlybyagenuinereformagendafromtheMinistry.ThatcontextisimportanttounderstandthatKHEN’sapproachinthisprojectwasacceptableforthattime,butlookingforward,itwouldhavetotechnicallyshifttowardadoptingthesesuccessfulinterventions(assuggestedin‘Recommendations).Secondly,partofKHEN’sidentityasanorganisationiscommitmenttochildrightsanditssuccessfulpromotionofthisinitsprojectsandwithprojectstakeholders.Thereisclearevidenceinthisprojectalsoofchildrightawarenessraisingbeingconductedandgoodrecognitionofchildrights.However,whatisnotsoclearisthetangible,personalbenefitthisbringschildreninruralSamloutDistrict.Goodschools,goodteachers,andsafecommunitiesareofmoredirectbenefitforthosechildren.Therighttoaqualityeducationisanimportantmessagetosendthroughaneducationprojectandthatcanbethenbackedupanddemonstratedthroughhighqualityeducationsupportinschoolsandcommunities.Buteducationprojectsshouldfocusonthat,andashappenedinthisproject,activitiesfocusedonlearningthefullsetofchildrightsarenotthewisestuseofresourcesintheopinionofthisevaluation.

(5.5)Efficiency• Efficientuseofprojectresources.• CooperationandlearningamongpartnersandwithinKHEN

Acceptableperformance SummaryAnalysis:Overalltheevaluationassessesthisas‘AcceptablePerformance”.Linkedtoissuesraisedearlieraboutsustainabilityandtechnicalapproaches,theprojectcanonlybeassessedas‘acceptable’,withregardstoefficientuseofresources.Thereisnodoubtsomeactivitiesinthisprojectwillsurvivethetestoftime,butsomewillnot,andsointhatsensetheywerenotaneffectiveuseofresources.KHEN’sinternalbudgetingalsohassomeunusuallineitemsandpossiblyandperhapsamoreefficientbudgetcouldhavebeendeveloped,butthisisnotaseriousconcern.Roughly25%ofthebudgetand15%werespentonhardwareandsoftwarerespectively.Thesearenotbadratios,butinordertomovetowardsbettersustainabilityandmorequalityoverquantity,KHENshouldlooktoinvestasmuchaspossibleintraining,supportandmentoringforthoseeducationduty-bearerswhoultimatelyneedtomanageandmaintainanyfuturematerialsinvestment.

19Asabove


KHENhasgoodrelationshipwiththeschoolsandDOE/POEandthiswasastrengthoftheproject.TheevidencealsosuggeststhatreportingtobothEDUCOandChildFundwastimelyandinfull.Asmentionedinotherareas,therearesomeareasforimprovedreportingbyKHEN,particularlywithregardtotrackingandreportingonoutputsinaclearway.5.6QualityoftheprojectdesignThetechnicalapproachesintheprojectarebasicallysound,asatthetimeofdesign,andallofthemajorstakeholdersinanygoodeducationproject(schools,communities,students,government)areincludedinprojectactivities.Ithasagoodbalancebetweenmaterialinvestmentandsoftwareactivities–definedhereasanyofthetraining,awarenessraisingorpromotionwork.Italsousesthenationalsystemsasabasisforitsdesign,includingactivitiesonCFSframework,withteachers,directorsandschoolclustergroups,andSSCmembers,andplanningtoutilisetheDTMTbodyfortechnicalsupportafollowup.Themajorflawsinthedesignare:

- TooheavyrelianceonPOE/DOEfortrainingviaacascademodelthatonlyhadtheDTMTbodyasitswayofprovidingfollowupsupportaftertraining.DTMTsacrossCambodiaareweakandthisisnotexceptioninSamlout.

- It’sbusyandcrowdeddesignthatspreadstoomanyactivitiesacrosstoomanystakeholders(seeAnnex6–BeneficiaryMap)–thequantityofactivitieshaslikelybeenalargeworkloadandburdenforstaffjusttocoordinate,letalonedelveintothequalityofeach.

- Therewasnoattempttounderstandandtracklearningoutcomes.ThisisaprospectmucheasiernowwiththedevelopmentofthingsliketheG1-3Khmerreadingbenchmarks,sothisunderstandable,butshouldbementionedasapriorityforfutureactivities.

- Andtoomanytokeneventsthatarenotvaluableuseofresource,suchas‘exposurevisits’or‘promotingofeducationworkshops’.

Fromalogframeperspective,theplannedactivitiesintheprojectdoflowwell,sequentiallyandlogically.However,therearetwoareasforimprovement.Firstly,ensurethatthereareclearlinksbetweeneachlayerofthelogframe.Forexample,

o objective,o xnumberofoutputsthatcombinetoachievetheobjective,and;

§ xnumberofactivitiesthatcombinetoachievetheoutputTherewassomedisorderinthelogframewithoutputsgroupedunderthewrongobjective(forexampleresult2.8,wasactuallymorenaturallyconnectedtoobjective3).Secondly,andverysimply,thetemplatetheyusedforthelogframegroupedallobjectivestogether,outputstogether,andactivitiestogether,ratherthanallowingaflowfromobjective,tooutput,toactivity.It’sveryeasytogetlostwiththistemplateasseemtobethecase,butthisiseasilyremedied.5.7IntegrityoftheprojectdataKHENdidanoutstandjobcollectinginformationfromalltheschoolstakeholders,andpossiblytoogood;thatisperhapstheyhadtoobigasamplesizeandcollectedtoomuchinformation.Asimilarorfutureprojectcouldbemoretargetedwiththeinformationsought,andthenrelyonasmallersamplesize.InthiscaseKHENhasadmittedthattheywouldhavelikedtocollectstudenttestscores,butitwasjusttoomuchwitheverythingelsebeingcollected.So,ensuringtheyarecollectingtherightdatashouldbeapriorityforthefuture.Inaddition,focusontheindividualunitofanalysisandnotjusttheaggregatescore.Forexample,inthisproject,aschool’scomplianceagainstCFSstandardswasmeasuredusingaseriesofquestionsbasedontheCFSframework.The‘result’ofthiswasthenreportedasanaveragepercentageforall25schools,ratherthangivingaschool-by-schoolscorebreakdownandanalysis.Measuringandreviewingthecoreunit,inthiscasetheschool,ismoreappropriateinordertolookforwaystotargetactivities,forexamplegivemoresupporttoweakerschoolsorleveragestrongerschoolsforlearningabouthowtheydosothingssowell.Thisisjustarefinementoftheirdataanalysis.


Finally,KHENshouldlearntoadapttheirdataasneeded.Forobjective2,theplannedindicatortargetwasrevealedtobyalreadyachievedthroughthebaselineinformation.Anewindicatortargetwasnotestablishedafterthis,whichwasmissedopportunitytocollectanothertypeofusefuldata.5.8VerifiedreportedresultsThereisclearevidenceofallactivitiesplannedfortheproject,beingundertaken.TheonlyexceptionbeingActivity4.5–aliteracypilotprogramforadults,whichdidnotgoaheadbecauseofthelackofgovernmentsupportforit.KHEN’sreportingofresultswasattimessomewhatconfusedandthenmisleading,primarilyduetoamisunderstandingorinconsistencybetweenthedatacollectedandthedatarequestedintheirlogframe.Forexample,theindicatorforOutput1.1was:

-11communityschools&14statecanbecountedandassessedasmeeting80%ofCFSstandardsby2017KHENReportedthisas72%achieved–onfurtherinvestigationtheevaluationfoundthiswastheaveragescoreachievedforall25schools,whichisinteresting,butnotwhattheindicatoraskedfor;itaskedtomeasurehowmanyschoolsreached80%ormoreoftheCFStargets.Thiswasinfactonly8schools,or32%oftheschools.Thesewerenotdeliberateattemptstomisleadordistortdatabutrathersomemisunderstandingonhowtoprocessthedatatheycollected,andsomeerrorsinsettingupofdatatools.Thesearesmall,veryfixableissuesbutworthnotinginthisreportsothatKHENcancontinuetoimproveitsorganisationalcapacity.5.9HumanimpactThemostobvioushumandimensionandbenefitinthisprojectischildreninschool.Forthemostpart,childrenintheschoolsvisitedseemedhappyandengagedintheclassroom,attentiveandresponsivetoexercisesandteacherrequests,eveninspitetheoften-one-wayroteteachingstylebeingused.Childrenwerenotoverlydistractedbytheconsultancyteamarrivingatschools,andquicklyswitchedtheirattentionbacktotheclass,whichwasbothsurprisingandpleasing.Similarly,whilsttheconsultancyteamswereundertakingin-classroomobservations,thechildrenwerenotnoticeablydistractedbytheforeignpresenceandwereabletomaintainfocusinclass.Childrendemonstratedgooddisciplineinclass,andduringbreakperiods,and100%ofinterviewedchildrenrespondedpositivelytoquestionsaboutlikingschool.Thesefindingswereverypleasing,and,puttingasideallotherissuesaboutteachingandschoolquality,theyareevidencethattheproject’sfocusonincreasingaccesstoaschool,isreallyvaluableforanindividualchild.Thisisimportanttohighlightintheevaluation,thatsimplyput,childrenwerehappytobetheschoolsbeingsupportedbyKHEN.Theotherinterestinghumanelementexperiencedintheresearchwasthatof‘ContractTeachers’.Asmentionedearlier,theirexperienceintheprojectisoneofthemorefragileparts.However,someofthenewcontractteachersinterviewedwerethemostimpressiveteachers(pleaseseemini-casestudyonpage20formoreinfo).Thecontractteachermodelcanbereallyimpactfulatahumanlevel,ifitfindsthosepeoplewithapassionforteaching,ornaturalinstinctforit,andiftheyaresupportedproperlycanbeatremendouswin-win–forboththekidsinclasseswhogetamotivatedteacher,andtheteacherswhoarelookingforwaystousetheirpotentialandenergy.5.10AlignmentwiththeCFSframework20Ingeneral,theprojectmodelisattemptingtoaddressallsix-majordimensionoftheRoGCandsub-pointsineachdimension,buthasinvestedmoreincertainareasthanothers,andhadgreatersuccessincertainareathanothers.Thestrongerareasappliedintheprojectasnotedbytheevaluationare:

20Dimension1(access);Dimension2(effectivelearning);Dimension3(heathysafetyprotection);Dimension4(gender);Dimension5(communityparticipation);Dimension6(schoolmanagement)


StrongerareasofcompliancewithCFS Weakerareas/areasformorefocus

Dimension1 Schoolmapping,datacollectionandenrollment

Dimension2 Child-centrededucation,supportforslowlearners,andschoollibraries

Dimension2 Attractiveclassroomenvironments,andlifeskillsclasses(thecommitmenttolifeskillsisgood,contentneedsimproving

Dimension3 Schoolenvironmentsandbasicchildsafetyinschool

Dimension3 Childprotection,WASHinschools Dimension5 Communitysocialisingwithschool,studentcouncils,communityoutreachbyschools

Dimension4 Gendermainstreaming(viaenrollmentandgoodBoy/girlratios)

Dimension6 Linksbetweenschools,schooldevelopmentplans,schoolassessment,directortraining–alltheseareasareveryimportantandneedsstrengthening,despiteinvestmentinmostofthem

Dimension5 Exhibitingchildren’swork(goodexamplesinsomeschoolsbutnotconsistent),communitysupport

6.RecommandationsAfulldetailedlistofdetailedrecommendationsistabledbelow,andcategorisedintokeyareasforeaseofreview.However,somefundamental,broadrecommendationneedtoberaisedinitiallyandshouldbetakenintoaccountwithanyfutureeducationprojectsbeingconsidered.

8. Foreducationprojects,focusonlearningoutcomesforkids–thisisfirstthinganyeducationproject

shouldaimtodo.TheevaluationacknowledgesthatKHENmightnothavebeenreadyforthisinthepast,butgoingforwardsthisshouldbethecentrepieceofalleducationwork.

9. ifKHENwantstogrowasaneducation-focusedNGO,itshouldgainmoretechnicalexpertiseandanyfutureeducationactivitiesshouldbetechnicallystronger,specificallyinactivitiessuchG1-3readingtools,readingkitsandtraining,teachermentoringandin-servicesupport.Looktoleverageexistingopportunitiesandrelationships,suchasSeeBeyondBorderssuccessfulmentoringapproachinBattambang,orKHEN’slinkswithforeignuniversities,totryandgetprofessionalteachersorsocialworkertocomeandsupportKHEN.

10. Forfutureeducationproject,increaseandtargetinvestmentinpre-primaryandGrades1-3.ThisisbecauseitalignswithMoEYSreformagenda,wheresignificantsuccesshasbeenachievedrecently,butmostimportantly,thesearethecrucialyearsforchild’seducationalprospects;asolidfoundationhereleadtopositivelong-termlearningresults.

11. Revisethetrainingmodelfromcascade,infrequenttraining,toregularsupportandmentoring,progressiveandintervaled,andifpossible,in-service.ThiswillbeabigshiftforanorganizationsuchasKHEN,sopossiblyitsonlypilotedinthenearfuture-sayinasmallnumberofschoolswithinabiggergroup–thismightbethemostpracticalwayconsideringKHEN’sstaffsizeandtheircurrentlevelofeducationalexpertiserightnow.

12. Looktoappropriatelybudgetandresourcethesoftwaresideofprojects.13. Improvetheprospectofprojectactivitiesandoutputsflourishingafterproject’sendbyfocusing

morestronglyonmanagementandmaintenanceofresults,andengagingthesupportofstakeholders–atthecommunitylevelandinofficialpositions.

14. Investinqualitysolutions/activities,evenifmeansfewertargetschools/beneficiaries,andatahighercostforeachbeneficiary-QualityoverQuantity!

15. Beattentiveasanorganisationtothegapbetweenstrong,durableresultsandweak,fragileresultsandtrytonarrowthatgap.


CategorisedandspecificrecommendationsLogframe&Design

1.Morepracticallogframethatflows‘logically’-objective,output,activity–withoutputsthatcombinetoachieveanobjective,andimprovedindicatorsthatmeasurethemostimportantpartofaproject,inthecaseofaneducationproject-learningoutcomes.ExternalsupportpossiblyneededtohelpKHENskillupinthisregard2.Linkactivitiestooutputsinthelogframe-thetemplateusedforthelogframegroupedallobjectivestogether,outputstogether,andactivitiestogether,ratherthanallowingaflowfromobjective,tooutput,toactivity;It’seasytogetlostwiththistypeoftemplateanitsrecommendedtouseanalternativethatgroupsactivitiesandoutputsinordertoenhanceprojectdesignandmanagement

3.RemoveM&Efromlogframetosimplify–justincludeasseparatebudgetline–noneedto‘monitor’andreportonM&E4.Dataanalysisneedstofocusonthekeytargetgroupssuchasschoolsandchildrenandorganisedatatobeabletoreviewtheresultsofthosegroups–sothatgroupswhoneedextrasupportcanbeidentified.Forexample,it’snotusefultoaveragetheresultsofallschools,withoutlookingateachindividualtoassessanypoorperformers.

5.Adaptation–don’tbeafraidtoalteralogframeifneeded–forexample,objective2–thegraduationdataasabaselinealreadyexceededthetargetindicator.Inthiscase,itwouldhavebeenbettertoalterthatindicatorandlookforsomeotherissuemeasure

Budgeting 6.Exploreunit-basedbudgeting-impressionsarethatsomeitemsover-budgetedandsomeunder-e.g.exposurevisit5.5kversusschoolrenovation11k

7.UtiliseasingleM&Ebudgetratherthanaddingtoeachobjective–thisisclearerandmoreefficient

8.Clearlybudgetforperdiemsandhaveaclearpolicyonthistoavoidissueswithgovernmentandtoavoidoverspendonthis,attheexpenseofothermoreimportantactivities.

9.Reviewproportionalspendinginanoverallbudgetlevelandmaximizetheproportionspentonprogramactivities.

ChildProtection&safeguarding

10.CPactivitiesaregoodandquitesophisticated,butperhapsoverlyso,forschoolsattheircurrentstageofdevelopment–itmaybebettertofocusonimprovedunderstandingofsafetyandbasicreferralinschools,ratherthancomplexlegaltraining.

11.Focusonchildsafetyinschools–childprotectionisimportantbutbasicconceptsofsafetyinschoolsarelackingandchildsafetyisthefirststeptobeachieved.

Kalyan/SreytacouldprovideinputintoKHEN’sapproachesandongoingtechnicalsupport.

12.Reviewteacheranddirectortrainingandknowledge–contractteachershavenotbeentrainedonchildprotectionandthereisbigvariationinresponsesonhowtoaddresspotentialcaseofCP–refreshertrainingandfurtherfollowupsupportlooksneeded

Training 13.Linkexposurevisitstofollowuptasksorin-schoolapplications–iftherearenoobligationsafterwardspeopleeasilyforget.Thiscouldbepossiblylinkedtoincentives,forexamplesomesmallrewardorbonusforachievingimprovinglessonplanpreparationfor3monthsinarow.

14.ConsiderusingDounTretSchoolasthesiteforanexposurevisitforotherschools


15.Morefocusonsupportforteacherdevelopmentandlearning-in-servicesupport,individualised,mentoring.

LookatSBBforadvice/inputbasedontheirsuccessfulmentoringprogram.

16.ItwouldbebeneficialtoprovideCFStrainingagain/refresher,butwithadifferenttrainingapproachandtrainer.Thetheoryneedstobelinkedtouseinclassroom-clearteachersdon’tfullyunderstandtheconceptsthereinandhowtoapplythem

17.Uselocaltechnicalknowledgeand/orexternalknowledgetoincreaseinternaltechnicalcapabilities–KhencouldleverageitslinkswithAustralianuniversitiestogainmoretechnicalstrength–forexample,anexperiencedteacher/professortohelpwithteachertraining,orasocialworkertocomeandhelpwithstrengtheningCPservices(notadvisabletogeteducationstudentsastheywon’thavetheexperiencetodealwiththesituationsinruralCambodia).

Educationapproaches

18.Moreinvestmentandprojectfocusonpre-primaryschoolreadinessandG1-3–thiscouldbecomeatechnicalcoreorfocusforKHEN(withmoretrainingandsupportfromathirdparty)

19.KHENtolearnaboutuseG1-3readingbenchmarksandlibrarymanagementapproach,andpossiblypilotthisinanextproject-KHENtotryinsmallnumberofschoolsasapilot(tolearnandsuccessattechnique)andtoalsobeabletocomparedifferencesinschools

20.LooktoinfluenceThursdaylifeskillscourses-introducing‘problemsolvingactivitiesintolifeskillsclassesorintokids’clubs-introducefunactivities–getdirectorsinvolvedtostrengthenadult/studentrelations

21.Looktoencourageandharnessenthusiasmfromcontractteachersthehatanddevelopthem–KAPE/ChildFundE2Lapproachcouldbereviewedasthathasdemonstratedgoodresultssofar.

22.Revisetheapproachtothechildren’sclubsand:- enhanceitschild-centeredness;- altertheaimoftheclubstobesafe,comfortable,informativeandalsofunspaces

forchildren;- improvethestruc

Alone in Kyoto Consulting Project Evaluation of Improving ......2. Evaluation Objectives, Scope and...

Documents

Transcript of Alone in Kyoto Consulting Project Evaluation of Improving ......2. Evaluation Objectives, Scope and...