www.governance.sk

Monitoring and Evaluation of the ‘A Good Start’ Project

Brussels, 4th June 2012

Andrej Salner, Judit Kontseková, Martina Kubánová

www.governance.sk

A Good Start as a monitoring challenge

• pilot – innovative, responsive, flexible• partnership with many actors: - 16 localities in four countries (HU, MAC, RO, SVK)- 12 local implementing partners (NGOs)- Roma Education Fund with three international partners - ISSA (pedagogy), FSG (partnership), SGI (monitoring)- strong partnership with UNDP, World Bank on project monitoring and evaluation

www.governance.sk

What we needed to know and instruments we used

• How to set up the intervention locally – information from the Community Assessment (administered before and updated after to see changes)

• Who are our beneficiaries – Household Survey (at the beginning and at the end) and Children’s Database (using the survey as source of records)

www.governance.sk

What we needed to know and instruments we used

• How are we progressing – Children’s Database, Monitoring Visits, Narrative Reports and Indicator Worksheets

• What have we accomplished and what can we learn – Monitoring System and Qualitative Research (final multi-stakeholder individual and group interviews)

BUT NO ROOM FOR COUNTERFACTUAL IMPACT ASSESSMENT!

www.governance.sk

Community Assessment

• Filling gaps in administrative data to help project design and implementation

• Demographic and socio economic data• Residential segregation and specifics of the Roma communities including language• Accessibility of public services: nursery, kindergarten, counseling for special needs,

child welfare and health services, educational counseling, speech therapy• Existing providers of ECEC and financing• Available places in KG and schools and enrolment rates• Qualifications of the educators and teachers and the availability of training• Infrastructure in existing kindergartens and schools• Roma participation in education decision making at the local level• Main educational problems encountered by disadvantaged Roma children

• Benefit for local partners who gained structured information• Updated at end of project to see changes

www.governance.sk

Household Survey and Children’s Database

• survey at start of project and end of project• carried out by local partners• mapping expectations and experience of parents in bringing up their children, motivations and barriers to early childhood education, socio-economic information

+ collection of data for Children’s Database (family name, children’s names, age, institutions where children are enrolled)

www.governance.sk

Household Survey Specifics

• Local partners’ mentors/facilitators served as enumerators

• Some disadvantages but many advantages, too• Lack of experience, mixed incentives• Access, cost, capacity building

• Translated and piloted (HU, MAC, RO, SVK, Romanes)• 1 028 families of recurrent beneficiaries in 16 localities surveyed, 1 768 children aged 0-7 entered into Children’s Database (additional children added later)

www.governance.sk

Selected Household Survey findings

• 3-7 year old children who stay at home not enrolled lag behind their peers in kindergartens in special skills like reading and writing• More than 85% of AGS parents think that children do better in school, if they attend kindergarten or another educational programme.

www.governance.sk

Selected Household Survey findings

Reasons for enrollment in pre-school:1. the child can learn there - (71% 0-3/ 58% 3-7)2. a chance to play, meeting friends and enjoying kindergarten ...(40% 0-3)/ the same 3-73. succeed better in primary school, (53% 3-7 )4. learning the state languageIn localities, where the mother tongue of the children is different from the state or regional language, this reason dominated, especially in Abranovce (60% of parents) and Zborov (36%) in Slovakia, Craiova in Romania (47%), Delčevo (43%) and Vinica (46%) in Macedonia.

Reasons for non-enrollment in pre-school:

www.governance.sk

Children’s Database

• Online application• Contains information from the Household Survey and selected project monitoring data at the level of recurrent beneficiary household and individual children• Accessible for all project partners BUT with limited access• Names of beneficiaries ONLY visible to users in the given locality to allow them to enter monitoring data

www.governance.sk

Technical background• Ruby on Rails, Linux server, 2 Mb of data

Translation• Simple UI for users with different levels of IT proficiency

Training• in person in each locality (with involvement of UNDP and WB experts), hotline

www.governance.sk

Children’s Database Users

• Local partners in 16 localities• 19 local users of the application worked on data entry• 8 local leaders (verification of certain data) • 41 additional mediators administering questionnaires (not working with the application)

• 8 national coordinators and REF managers • 10 users: REF project management and M&E team• 2 administrators

www.governance.sk

Simple UI

www.governance.sk

Children’s Database

Children’s Database

• project management, M&E team and local partners can access anonymous data from the survey and overview of progress in education, attendance, participation in project activities

www.governance.sk

www.governance.sk

Database Benefits for Local Partners

• Lots of work• Learning process• Partnership

• Information about target localities• Ability to track activities• Tables and graphs for project applications

www.governance.sk

Multi-stakeholder focus groups and interviews

• 6 researchers carried out over 80 individual and group interviews in 16 localities

• parents• mentors, facilitators, project administrators• KG teachers and school teacher• local government, social workers

• joint effort with World Bank

www.governance.sk

Qualitative research – initial impressions

• immense benefits from involving Roma (with or without formal qualifications) in the project at all levels and systematic training

• mediators, facilitators, pedagogical or family assistants, volunteers

• incentive to re-enter formal education• pre-school capacity and cost are real barriers and sometimes, additional barriers are erected to keep Roma out

www.governance.sk

Qualitative research – initial impressions

• attendance of integrated pre-school reduces social distance in otherwise segregated settings• intervention can reduce mistrust and fear of parents regarding pre-school• educational intervention cannot work in isolation from assistance in other areas• parents want to learn and interact

www.governance.sk

Qualitative research – initial impressions

• strong demonstration effects supporting relations with local government, formal institutions and helping sustainability• AGS results built on experienced partners and local links

www.governance.sk

What have we learned about M&E?

• Local NGOs intense involvement in M&E has costs but benefits, too• Do not rely on getting everything right rightaway• Project data collection can compensate for gaps in data availability• Additional information on specifics of surveying marginalised, poor people and using database will be available in AGS Good Practice Guide on Data Collection

www.governance.sk

Thank you for your attention!

Andrej Salner: salner@governance.skMartina Kubánová: kubanova@governance.sk