Presentations from the 3rd PISA for Development International Advisory Group Meeting held in...

3rd meeting of the International Advisory Group Granados Park Hotel – Asunción, Paraguay

30 March – 1 April 2016

PISA for Development

Expected results from the meeting

Expected results from the meeting

• Election of two co-chairs: one representative of the participating countries and one representative of the development partners

• Presentation and review of annual report on project activities, as well as the contribution of OECD and the project to education in the 2030 sustainable development agenda

• Presentations by ETS and The Learning Bar on the progress being made on all strands of the project and the plan for next year followed by discussion with the IAG

• Presentations from participating countries on capacity-building activities, including feedback on the international/NPM meetings, peer-to-peer learning and other arrangements

Expected results from the meeting continued…

• Presentation from Kosovo and GIZ of their experiences of launching PISA for the first time in the country; agenda setting, engagement and communication with external stakeholders prior to the tests; embedding PISA in a broader discussion of the value and standards of assessment in the Kosovo education context in general; and preparing to use the results in a National PISA Report

• Presentation by the OECD on an up-date of the PISA-D engagement and communication strategy

• Presentations from each country on their respective engagement and communication strategies since project launch


• Presentations by OECD and Contractors of analysis and reporting strategy, including dissemination tools and activities for the main study, the analysis and assessment frameworks and the technical report

• Presentation by UIS on the status of system level data in the participating countries and discussion of this, followed by comments from each country

• Presentation by OREALC/UNESCO Santiago on lessons from LLECE related to engagement and communication with stakeholders and dissemination of results, particularly in respect of TERCE, followed by discussion


• Presentation by OECD of the final version of the Terms of Reference for an independent review of the project; the IAG will then discuss and agree upon the timing of the review and the call for tender arrangements

• Presentation by OECD on PISA development strategy, including the planned scaling up of PISA-D, reporting on preparations for PISA 2021 and exploring opportunities for expanding PISA to a wider set of countries, particularly in Latin America

• Conclusions of the meeting, next steps to be taken, agreement of the draft of the summary record and plans for 4th IAG Meeting

Day One sessions

1. Introductions, Purpose of Meeting, Election of Co-Chairs, Project Progress Report, establishment of the TAG and the contribution of the OECD and the project to education in the post-2015 agenda

2. Presentation by the International Contractors of detailed progress report for Strands A, B and C, including the draft frameworks, draft instruments, sampling design and detailed plans for 2016-17

3. Participating countries report on capacity-building activities and outcomes

Day Two sessions 4. Presentation by the PISA national team from

Kosovo and GIZ: engaging with stakeholders and preparing these for PISA, national reports and the dissemination of PISA results

5. PISA for Development engagement and communication strategy – participating countries to report on stakeholder engagement and communication activities

6. Presentation by OECD and contractors of analysis and reporting plan

Day Three sessions

7. Presentation by UIS on the status of system level data in the participating countries and discussion of this

8. Lessons from LLECE – TERCE engagement and communication strategy

9. Agreement of the ToR for an independent review of the project to be conducted in 2017/2018

10. PISA development strategy and outreach to potential PISA countries in Latin America

11. Next steps and meeting conclusion

Key documents in folder • Agenda

• Participants list

• Annual progress report & summary record of last meeting

• Draft cognitive assessment frameworks (reading, maths, science)

• Draft framework for contextual questionnaires

• Descriptions of items and tasks for all 3 domains in the cognitive assessments

• Student questionnaire; teacher questionnaire; school questionnaire

• Translation and adaptation guidelines and procedures

• Sampling design and plans for 2016

• PISA-D engagement and communication strategy

• Draft analysis and reporting plan

• Report on the status of system level data in the participating countries

• Terms of reference for an independent review of the project

• PISA development strategy

Other documents in the room

• Capacity Building Plans from the participating countries

• Presentation on the Kosovo experience

• Country and LLECE engagement and communications presentations

IAG Agenda

• The International Advisory Group is invited to adopt the agenda for its third meeting



PISA-D Annual Report on project activities

Summary of achievements since last IAG

• The project plan for 2015 has been implemented in accordance with the 13 next steps agreed upon at the 2nd IAG meeting. There have been no major deviations.

• The project is on track: major technical milestones have been achieved – frameworks, tests, questionnaires, manual, standards, sampling plan, adaptation and translation guidelines are all in place and the field trial is in sight.

• Plans for Strand C have been agreed and way ahead is now clear for this component.

• Capacity building activities are under way and peer-to-peer learning partnerships are in place and working.

Reasons for success • The commitment and performance of participating countries and their

national teams has been outstanding and momentum is building, expectations are high.

• The international contractors have done a great job and delivered on their terms of reference in exactly the right way.

• Development partners have delivered their support (international and in-country) and technical guidance on time and are helping us to move forward.

• Technical partners, especially UNESCO and UNICEF, have provided guidance and co-operation and facilitated the project’s contribution to wider processes; i.e., Education SDG and 2030 agenda.

• PISA countries and economies (Belgium, Brazil, Canada, Chile, China, Hong Kong, Korea, Iceland, Mexico, Peru, Spain, Uruguay, United States) have been generous and valuable contributors to peer-to-peer learning

Annual report format From 2015 onwards, a progress report:

• actual outputs compared to planned outputs

• summary of the use of funds compared to budget,

• explanation of major deviations from plans,

• assessment of problems and risks,

• assessment of the need for adjustments to activity plans and/or inputs and outputs, including actions for risk mitigation,

• assessment of achievements in relation to project purpose

15

13 next steps agreed upon at 2nd IAG

1. OECD to complete the remaining planned expert papers and to make all documentation available on its website

2. OECD to finalise ToR and tendering documents for international contractor(s) for Strand C

3. OECD to sign participation agreements with all the remaining countries participating in the project – Paraguay and Cambodia

4. Participating countries to finalise Project Implementation Plans

5. Participating countries to finalise outstanding agreements with development partners regarding contributions and support (e.g., international costs, in-country costs and activities)

6. OECD to complete the design of capacity-building plans for all participating countries

13 next steps agreed upon at 2nd IAG 7. OECD to finalise with development partners outstanding agreements for support

to the project – general contributions and country-specific contributions

8. First meeting of Technical Advisory Group in August (webinar)

9. Complete tendering process to commission international contractor (s) for Strand C by OECD

10. Project implementation: Strands A and B -- Technical development – First international/NPM meeting (September 2015)

– Second international/NPM meeting (January 2016)

11. OECD Secretariat with International Contractors to facilitate translation of key materials, such as manuals, into French and Spanish

12. Participating countries to link the communication focal points in each National Centre with the OECD Secretariat to facilitate the implementation of the engagement and communication strategy

13. Third meeting of IAG in March 2016 in Asunción, Paraguay.

i) Expert papers

i) Expert papers (continued)


Review of system-level data in the participating countries

• Written by UNESCO Institute of Statistics (UIS)

• Final draft circulated to the IAG at the end of February 2016 for written feedback

• Final draft will be presented at 3rd IAG meeting, and countries will provide oral feedback based on their review of the report

• OECD will implement all feedback after the 3rd IAG meeting and publish the report mid-2016


The ten indicators that have the most impact on learning in middle-income countries

• Written by Lucia Tramonte

• Explores in more detail than was possible in the Lockheed report on the PISA results of the middle-income countries to identify indicators with the most impact on learning in those contexts

• The 1st draft of the paper has been received by the OECD and is under review

ii) Finalise ToR and tendering documents for Strand C

• OECD finalised ToR and tendering documents for Strand C immediately following the last IAG meeting

• Call for tender for Strand C launched in April 2015

• Review panel took place in June 2015

iii) OECD to sign participation agreements with remaining countries

• OECD signed participation agreements with:

Paraguay in September 2015

Honduras in January 2016

Cambodia in February 2016

• OECD is in the process of signing a participation agreement with Panama for Strand C

• OECD is exploring engagement with Ukraine, supported by Poland, as part of scaling up PISA-D

iii) OECD to sign participation agreements with remaining countries

(cont’d) PISA-D has been launched in Ecuador, Guatemala, Honduras, Paraguay, Senegal & Zambia

Launches planned in: Panama (22 April) & Cambodia (30 May)

iv) Countries to finalise PIPs

• Project Implementation Plans (PIPs) have been completed for all participating countries with the exception of Honduras and Panama

• Honduras’ and Panama ’s PIPs will be completed by August 2016, following completion of their capacity needs analysis and capacity building plans

• Discussions have started for Ukraine to elaborate a PIP for PISA 2018 building on PISA-D PIPs

v) Countries to finalise outstanding agreements with development partners

• All 8 countries have finalised or are in the process of finalising their respective budgets for financing their participation in PISA-D

‒ Countries’ contributions to the international costs of participation (budget managed by the OECD)

‒ In-country costs of participation (budget managed by the countries themselves and, in some cases, their development partners), which is described in the PIPs

vi) OECD to complete the design of CBPs for all countries

• CBPs have been developed for Cambodia, Ecuador, Guatemala, Paraguay, Senegal and Zambia – available on the OECD’s PISA-D website

• Honduras’ and Panama’s CBP will be completed by end of May 2016

• Ukraine will also develop a CBP for PISA 2018 building on the model and tools developed by PISA-D

vii) OECD to finalise with development partners outstanding agreements for support to the project

• All outstanding agreements for support to the project – general contributions and country-specific contributions – have been finalised

• Fund-raising is an on-going process

• Since the 2014-15 progress report, new development partners include:

‒ Sunny Varkey Foundation

‒ Dubai Cares and Microsoft Corporation

viii) First meeting of the TAG in August

• 1st meeting of the PISA-D Technical Advisory Group (TAG) took place on 10 September 2015

Webinar focused on inducting the TAG into the project and a discussion of the initial plans for Strands A and B

• 2nd meeting took place on 19 November 2015

Webinar focused on initial plans for Strand C, PISA Technical Standards and their application in PISA-D, design and sampling for the teacher questionnaire and data administration procedures

ix) Complete tendering process to commission international contractor(s) for Strand C

• Review panel in June 2015 selected Educational Testing Service (ETS) as the preferred bidder

• OECD negotiated with ETS on the basis of the following agreed alterations to the design of Strand C:

‒ Drop science from the out-of-school assessment

‒ Extend the timeline for Strand C to July 2019.

• Contract between OECD and ETS for Strand C agreed in March 2016, to be signed…

x) Project implementation

• ETS, TLB, Westat, Pearson and cApStAn with the Secretariat and countries successfully completed the planned technical development work in the months following the last IAG

• Subject Matter Expert Groups (SMEGs) meeting took place in Mexico from 20-24 July 2015 – consultation with national experts followed

x) Project implementation (continued)

• PISA-D Questionnaire Expert Group (QEG) meeting took place in South Africa from 20-24 August 2015 – consultation with national experts followed

• 1st international/NPM meeting from 28 September to 2 October 2015 in Quito, Ecuador – frameworks and items

x) Project implementation (continued)

• 2nd international/NPM meeting was held in January 2016 at the offices of Westat in Rockville, US

Concentrated on preparing the participating countries for the tasks of adapting, translating, verifying and printing the instruments and managing the survey operations and initial plans for Strand C

xi) Translation of key materials into French and Spanish

• PISA-D manual and the PISA Technical Standards have been translated into Spanish and French translation is under way

• PISA-D framework documents are being translated into French and will also be translated into Spanish

xii) Countries to link communication focal points in National Centre with OECD

• Following the last IAG meeting, each participating country linked its communication focal point in its National Centre with the OECD

• Together, they generated web pages for each country on the OECD’s PISA-D website

xii) Countries to link communication focal points in National Centre with OECD (continued)

• Countries are in the process of implementing their national communication strategies in alignment with the PISA-D project’s overall engagement and communication strategy

xiii) 3rd meeting of IAG in March 2016 in Asunción, Paraguay



PISA for Development use of funds

Expenditure Analysis (EUR)

Expenditures Commitments Total 6 years 2014

Y1

2015

Y2

2016

Y3

2017

Y4

2018

Y5

2019

Y6

A1. International Advisory Group

162 406 62 473 60 000 31 000 30 000 30 000 375 879

A2. Instrument development, field trials, local assessment implementation and related services

56 638 810 460 2 200 000 2 000 000 880 000 389 000 6 336 098

A3. Technical oversight, co-ordination, analysis and reporting

344 454 391 106 440 000 440 000 440 000 215 000 2 270 560

A4. Engagement for peer-to-peer learning and contribution to UN-led post-2015 process

17 070 91 236 100 000 40 000 248 306

A5. Tablets* 300 000

Overhead 147 715 88 236 75 000 25 000 335 951

Total 728 283 1 443 511 3 175 000 2 536 000 1 350 000 604 000 9 836 794

* There is a possibility that the tablets will be donated, in which case this cost would not be factored into the international costs of the project.

Challenges, risks and assumptions 1. The institutional, technical and administrative capacity

and conditions that are needed to successfully implement PISA at country and sub-national levels – risks low

2. The technical challenges are many and various and relate to the central question of how far PISA for Development can go in enhancing and adapting PISA to be more relevant to developing countries while still ensuring that the results of the assessment contribute to the establishment of an international benchmark in the context of PISA – risks are manageable

Challenges, risks and assumptions (cont’d)

3. PISA is changing rapidly and PISA-D needs to ensure that it provides an effective bridge for participating countries into main PISA – risks low

4. In all countries, PISA-D is already helping to strengthen institutions devoted to the production of reliable data and evidence for informing policy design. As the project moves forward, it would be essential to maintain the teams, sustain the support for the project, institutionalise a culture of evaluation and draw support for PISA in the country across the entire political spectrum – risks are manageable

Adjustments: Project Implementation Schedule to 2019

Annual Report on Progress

The International Advisory Group is invited to:

• Note the progress that has been made since March 2015

• Discuss any issues arising from the progress report

• Approve the annual report on progress



Progress Report

PISA for Development IAG Meeting

30 March 2016 Asunción, Paraguay

Ann Kennedy, ETS

Overview of Progress Update

Timeline and Key Phases

Key Activities and Milestones (March 2015-March 2016)

Integrated Assessment Design

Frameworks and Assessment Instruments

Translation and Verification

Target Population and Sampling

Key Activities and Milestones through the next IAG meeting in 2017

44

Timeline Summary

45

2015 2016 2017 2018 2019 Develop assessment

and questionnaire frameworks, test

design and master instruments

Develop survey materials, including

national assessment instruments

Carry out FT activities,

including coding

and data entry

Analyze FT data and prepare for

MS

Carry out MS activities,

including coding

and data entry

Analyze MS data

Report planning and

dissemination

Phase 1

Phase 2

Phase 3

Phase 4

Key Activities and Milestones

March 2015 – March 2016

46

Update: Cognitive Instruments

Frameworks

• Expert group revision of descriptions and sample items (July 2015)

• Expert group webinar (October 2015)

• Country reviews in September (NPM1) and November 2015

Cognitive Item Review and Selection

• Reviews of existing item pools

• Selection of units for the Field Trial (NPM1)

• Country reviews (November 2015)

• Final selection and assignment of units to clusters

• Preparation of units for translation and adaptation

47

Update: Questionnaire Instruments

Frameworks

• Expert Group Meeting (August 2015)

• Definitions of Constructs

• Presentation at the 1st NPM Meeting

• Expert Group Consultation (October 2015)

Questionnaire Item Development

• Selection from Existing Measures

• Development of New Items

• Preparation of Notes for Adaptations

48

Update: Sampling and

Translation/Adaptation

Sampling

• Country Questionnaires

• Information about the Educational System and Eligible Students for PISA-D

• Sampling Tasks

• Test Languages

• Target Population, School Sampling Information (Strands A-B)

Translation

• Translation Plan

• Translation and Adaptation Guidelines

• Release of Strands A-B Cognitive Units and Contextual Questionnaires for Translation/Adaptation

49

Update: SharePoint Site

Announcements, Calendar, Tasks

Documents Libraries (General Access)

• General Documents

• Field Trial Resources and Instruments

• Main Survey Resources and Instruments

Country sub-sites

• Share materials with international contractors in secure site with country-specific permissions

• Highlight specific tasks with deadlines

50

Update: Key Documents

PISA-D NPM Manual Part 1: First Review and Field Trial Preparations Phase

• Communications with Contractors

• Overview of SharePoint Site

• Overview of Tasks and Key Dates for Phase 1

Technical Standards

• PISA 2018 Technical Standards (Strands A-B)

• Strand C Technical Standards (Forthcoming June 2016)

51

Assessment Design

52

Goal of PISA-D

Large scale assessments such as PISA and PISA-D are

designed to inform governments and other key stakeholders on a variety of issues

To meet this goal, each assessment is developed around three key design criteria

• Relevance

• Comparability

• Interpretability

53

Overview Strands A-B

• 15-year-olds in school

• School-based

• Group data collection

• Self-administered tests and questionnaire

• Paper-based

• Reading, Mathematical and Scientific Literacy

• Questionnaires: Students, Schools and Teachers

54

Student

Data Collection Strands A-B

(Group Administration)

55

Cognitive Assessment (Reading, Mathematics and

Science)

Student Questionnaire

School Questionnaire

Teacher Questionnaire

Instrumentation Strands A-B

• Tests: – Paper booklets (rotated cluster design)

– Common booklet parts: general instructions and mathematical formulae

– Clusters of Units/items translated through Word files

• Questionnaires – Paper booklets (Student Q with multiple forms in FT)

– Adapted and translated through Word files

56

Frameworks and

Assessment Instruments

57

General Characteristics

of a Framework

Frameworks should:

• Reflect the current thinking of experts in the field along with the views of the participating countries

• Consider the target population

• Define the construct and discuss how results should be reported

• Guide test development:

• Identify and define key task characteristics

• Recommend the relative importance of these characteristics in the final set of items

• Provide a scheme for mapping these items to the construct

• Guide interpretation of results

A Process for

Developing a Framework

Identify and meet with key experts

Develop a sequence of steps:

• Reach consensus on the survey goals and constraints

• Create a working definition that will explain the construct, its relevance, and how it will be measured.

• Identify and define key characteristics for test development

• Recognize that each task provides a piece of evidence about the knowledge and skills mapping to the construct

• Think about and identify key variables associated with each task that help to explain placement along each scale

• Create or revise a described scale

59

Cognitive Assessment

Frameworks and Item Pools

60

PISA-D Cognitive Frameworks

PISA Reading, Mathematical, and Scientific Literacy Frameworks as the basis

• Attention to consistency of language across domains

• Better inform participating countries about performance at lower levels while maintaining full range of the PISA scale

• Extension of descriptions of proficiency Level 1

• Clarify basic processes of each domain

• Addition of sample items from PISA released item pool

• Consideration of Strand C population

61

PISA-D Expert Groups

Experts from PISA and PISA-D

• Framework revisions

• Item reviews

Composition per domain:

• Chair from ‘main’ PISA 2015/2018

• Content lead from Pearson

• Core expert representative from participating countries

• Extended experts from all participating countries

62

PISA-D Item Selection

Item selection began with understanding the goals and constraints of the assessment: • Provide a link to PISA (using data from PISA 2015

paper-based trend items) • Enhance the lower end of the proficiency scale for

each domain while keeping measurement of the higher levels

• Maintain adequate construct coverage • Each student will respond to 2 hours of cognitive

items • No new item development

63

Assembling the Available Pool of Items

PISA 2015 Trend (items from 2000-2012)

New PISA 2015 (Science only)

PISA for Schools

International Adult Literacy Surveys:

• PIAAC (literacy and numeracy)

• STEP (literacy)

• LAMP (literacy and numeracy)

64

Reviewing and Evaluating the Item Pool

Test developers reviewed items based on:

• Data Quality

• Psychometric quality of items

• Scoring reliability

• Expert reviews

• Construct representation

• Content appropriateness

• Factors that affect difficulty

65

Selecting Items

• Approximately 75 items per domain (5 clusters of 15 items each) for the FT

• Include 50-60% of items from PISA 2015 trend item pool; keep intact units where possible

• Greater distribution of items at or below Level 2

• Follow guidance of frameworks for distribution of items across aspects

66

Goals for Item Distribution and Totals from Existing Item Pools (Strand A)

Level 1 Level 2 Level 3 Level 4 Levels

5-6

PISA 2015 Trend

13 13 7 7 5 Total=45

Other Item Pools

9 9 5 5 3 Total=31

Total 22 22 12 12 8 Total=76

67

Approximately 60% of the total number of items

• Reading and Mathematical Literacy

• Approximately 45 items per domain (3 clusters of 15 items each) for the FT

• Subset of Strand A selection

• Distribution of items at or below Level 3

• Adaptability for tablet-based administration (including computer scoring)

68

Item Distribution and Totals

for Strand C

Reading Framework

and Item Pool

69

Reading Literacy: Framework

Modifications for PISA-D

Framework based on the PISA 2012 Reading Literacy Framework

Incorporated Reading Components (PISA 2012 optional, PIAAC)

Extended the description of Integrate and Interpret process to include comprehension of the literal meaning of text

Added category (1C) to the levels of proficiency to indicate expectations of students performing at the lowest level of the proficiency scale

70

PISA 2009/2015 Processes

with PISA-D reading components

71

Comprehend

literal

information

Access and

retrieve

Integrate and

interpret

Reflect and

evaluate

Reading literacy

Use content primarily

within the text

Draw primarily upon outside

knowledge

Retrieve

information

Form a broad

understanding

Develop an

interpretation

Reflect on and

evaluate content of

text

Reflect on and

evaluate form of

text

Reading Items by Process

72

Table 7: Aspect by number & percentage of items in proposed the selection

Process

PISA-D FT

Strand A PISA 2015

Trend

%

Framework Goal

Number %

Access and retrieve

25 31% 29% 25-30%

Integrate and interpret

34 43% 49% 45-55%

Reflect and evaluate

21 26% 23% 15-25%

Total 80 100% 100% 100%

Reading Items by Situation

73


Situation

PISA-D FT

Strand A PISA 2015

Trend

%

Framework Goal

Number %

Personal 25 31% 27% 36%

Educational 25 31% 28% 33%

Occupational 12 15% 23% 20%

Public 18 23% 22% 11%

Total 80 100% 100% 100%

Mathematics Framework

and Item Pool

74

Mathematical Literacy: Framework


Added categories (1B and 1C) to indicate expectations of students performing at the lowest level of the proficiency scale

Expanded descriptions of the following mathematical processes:

• Formulate Situations Mathematically

• Employ Mathematical Concepts, Facts, Procedures

• Interpret, Apply, and Evaluate Mathematical Outcomes

Added sample tasks to illustrate descriptions

75

Mathematics Items by Process

76


Process

PISA-D FT

Strand A

PISA 2015 Trend

%

Framework Goal

Number %

Formulating situations mathematically

19 25% 29% Approx. 25%

Employing mathematical concepts, facts, procedures

30 39% 43% Approx. 50%

Interpreting, applying, and evaluating mathematical outcomes

27 36% 28% Approx. 25%

Total 76 100% 100% 100%

Mathematics Items by Content

77


Content

PISA-D FT

Strand A PISA 2015

Trend

%

Framework Goal

Number %

Change and relationships

13 17% 27% 25%

Space and shape 16 21% 23% 25%

Quantity 28 37% 25% 25%

Uncertainty and data

19 25% 25% 25%

Total 76 100% 100% 100%

Science Framework

and Item Pool

78

Scientific Literacy: Framework


Added category (1C) to indicate expectations of students performing at the lowest level of the proficiency scale

Expanded descriptions of the following scientific competencies:

• Explain Phenomena Scientifically

• Evaluate and Design Enquiry

• Interpreting Data Scientifically

Added sample tasks to illustrate descriptions

79

Science Items by Competency

80


Scientific Competency

PISA-D FT

Strand A

PISA 2015 Trend

%

Framework Goals

Number %

Explaining phenomena scientifically

41 54% 46% 40-50%

Evaluate and design scientific enquiry

12 16% 19% 20-30%

Interpreting data and evidence scientifically

23 30% 34% 30-40%

Total 76 100% 100% 100%

Science Items

by Types of Scientific Knowledge

81


Knowledge Types

PISA-D FT PISA 2015

Trend

%

Framework Goals

Number %

Content 50 66% 62% 55-65%

Procedural 19 25% 27% 20-30%

Epistemic 7 9% 12% 10-20%

Total 76 100% 100% 100%

Translation and Adaptations

of Assessment Instruments

82

In general, a translation should be fit for purpose

In the assessment context: each item should examine the same skills and invoke the same cognitive processes as the original version, while being culturally appropriate within the target country—this is highly important for high-quality, comparable data

‘Localization’ consists of preparation, translation, adaptation, and validation of the assessment instruments

Training and documentation provided in January 2016 NPM meeting

83

Overview of Translation and

Adaptation for PISA-D

Countries received the following materials:

• Materials Preparation documents for the NPM and Translators

• Presentations for Translator Training Workshop

• Translation and Adaptation Guidelines (Cognitive Items and Contextual Questionnaires)

• Practice Materials

85

Translation and Adaptation Kit

Countries negotiated a Translation Plan with cApStAn based on FT data collection schedule

Most countries will submit translations and adaptations of cognitive materials to cApStAn by April

Adaptations of Background Questionnaires must first be negotiated and approved by TLB prior to translations are submitted to cApStAn for verification

Verification of all instruments is scheduled to be complete by June 2016

Strand C Translation/Adaptation begins May 2016

86

Translation/Adaptation and

Verification Timeline

Sampling

87

Sampling Overview Strands A-B

• Sampling Tasks focus on 1) School sample 2) Within-school sample

• Sample Design presented in January 2016 NPM meeting

– Definition of Target Population

– PISA Technical Standards

– Sample Sizes

• Student Sampling Training will take place during April 2016 NPM Meeting

88

Key Activities and Milestones

Through the Next IAG Meeting

89

Upcoming Activities and Milestones:

Phase 1 (Preparation for the Field Trial)

March – September / December 2016

Preparation and Adaptation of School-Level Materials

Finalize FT instruments

Select FT samples and contact schools

Data Management training (NPM 4/4a)

Coder training (NPM 4/4a)

Test Administrator training (NPM 4/4a)

Preparation of National Codebooks for Data Entry

National FT Training

90

Upcoming Activities and Milestones:

Phase 2 (FT Data Collection)

September / December 2016 – March 2017

FT Data Collection

Data Entry

Coding

91

NPM Meetings

92

Meeting Description Date

4 Training: Coding, Data management, Test Administration July 2016

5 Analysis and interpretation of FT results, plans for analysis of

contextual questionnaires, and preparation for MS data

collection

May 2017

6 MS survey operations, MS student sampling, test

administrator training, scoring, and data management

July 2017

7 Scaling methodology, and data analysis and reporting tools Mar. 2018

8 Analysis and interpretation of MS results and preparations for

reporting and dissemination

July 2018

Questions and Discussion

93

Strand C

94

Objectives

To yield a large enough sample size to test the validity of the questionnaire and the assessment items

To explore various approaches and evaluate various options to arrive at a recommendation for identifying and assessing a nationally representative sample of out-of-school 15-year-olds in future cycles of PISA-D Strand C

95

Assessment Overview Strands A-B

• 15-year-olds in school

• School-based

• Group data collection

• Self-administered tests and questionnaire

• Paper-based

• Reading, Mathematical and Scientific Literacy

• Questionnaires: Students, Schools and Teachers

Strand C

• 14 to 16-year-olds

• Household-based

• Individual data collection

• Questionnaire interview and self-administered tests

• Computer-based*

• Reading and Mathematical Literacy

• Questionnaires: Respondents and Parents

*Paper-based Parent Questionnaire

96

Strand C Target Population

15-year-olds in-school at grades 6 and lower

Out-of-school 15-year-olds

Definition

• “15-year-olds” means 14 to 16 year olds in Strand C

97

Two Selection Approaches for

Strand C (Pilot Project)

Primary component -- Probability sampling (required)

Secondary component – Non-probability sampling (must do for field test)

98



Primary component -- Probability sampling (required)

• To be conducted by all countries

• Generalizable results to target population and estimation of precision

• At each stage, all sample units have a non-zero probability of selection

• Gives all individuals in the target population a chance of selection

• All cases fully worked to standard number of attempts

99



Secondary component -- Non-probability sampling (must do for field test)

• Set of respondents represent themselves

• Link-tracing network sampling

• Snowballing

• Respondent driven sampling

100



Importance of primary component -- Probability sampling

• Critical due to producing results that represent the country

• While non-probability sample represents only itself

101

Sampling Strategy

Strand A

• Administered in schools in group setting

• 1st stage unit = School

• 2nd stage unit = Student

102

Sampling Strategy

Strand A • Administered in schools in group setting

• 1st stage unit = School

• 2nd stage unit = Student

Strand C • Administered in dwellings in individual setting

• Typically, a probability-based area sample is selected

• 1st stage unit = Geographic area

• 2nd stage unit = Dwelling unit (DUs) and Group quarters (GQs) – Create list of DUs/GQs from listing operation

• 3rd stage unit = Person – Create list of persons using a screener questionnaire to arrive at a

list of eligible persons within the DU/GQ

103

Sampling Strategy

Strand A • Administered in schools in group setting • 1st stage unit = School • 2nd stage unit = Student

Strand C • Administered in dwellings in individual setting • Typically, a probability-based area sample is selected

• 1st stage unit = Geographic area • 2nd stage unit = Dwelling unit (DUs) and Group quarters (GQs)

– Create list of DUs/GQs from listing operation

• 3rd stage unit = Person – Create list of persons using a screener questionnaire to arrive at a

list of eligible persons within the DU/GQ

• Next step, the non-probability sample – Link-tracing

104

Two Major Strata

Stratum H: Areas with high concentration of target population

Stratum L: Areas with low concentration of target population

105

Sampling Strategy Overview

106

Sampling Strategy

Primary -- Probability-based area sample • Most efficient designs give higher probability of selection to

areas with the largest target population size

• Clustering reduces cost at expense of increasing variances associated with estimates

• Discuss with countries to tailor an area sample design

• 1st stage unit = Census Enumeration Area (EA)

• Discuss with countries their typical approach to probability sampling

• Determine geographic spread of target population

• Identify areas with large concentrations of out-of-school 15-year-olds

• Assign higher sampling rate in Stratum H

107

Sampling Strategy

Secondary -- non-probability sample • Goals of link-tracing as applied to Strand C Pilot Project

• Obtain more sample yield, beyond yield from probability sample

• Recover something like a probability sample

• Option 1. Among initially selected households (seeds) through probability sampling, ask the respondent to recruit k subjects from the target population within the EA

• Continue for s waves

• Option 2. Start with a convenience sample of households in the EA

108

Out-of-school sample options

Within each major strata

• Primary -- Probability sample

• Household area sample

• Secondary -- Non-probability sample options – Household area sample with link-tracing

– School frame approach -- Strand A schools

» Recruit through students

» Apply link-tracing within X kms

• Street children

109

In-School lower than 7th

grade

sample options

Within each major strata

• Primary -- Probability-based

• Household area sample

• Secondary -- Non-probability sample options

• Household area sample with link-tracing

• School frame approach – List all schools associated with Strand A schools with 4th,

5th, or 6th grade

– List all students 14 to 16 years old within the schools

– Take-all, or sample, depending on the number of students listed

– Administer test either at school or household

110

Sample Size

Field Test

• n = 1 200

Main Study

• n = 2 700

May need more sample if more than one language administered for the assessment

111

Respondent Student

Data Collection Strands A-B

(Group Administration)

Strand C

(Interview, paper-based)

112

Cognitive Assessment (Reading, Mathematics and

Science)

Student Questionnaire Cognitive Assessment

(Reading and Mathematics)

School Questionnaire

Teacher Questionnaire

Background Questionnaire

Parent Questionnaire

Reading Items by Process

113


Process

PISA-D FT

Strand A

PISA-D FT

Strand C PISA 2015

Trend

%

Framework Goal

Number % Number %

Access and retrieve

25 31% 14 39% 29% 25-30%

Integrate and interpret

34 43% 13 36% 49% 45-55%

Reflect and evaluate

21 26% 9 25% 23% 15-25%

Total 80 100% 36 100% 100% 100%

Reading Items by Situation

114


Situation

PISA-D FT

Strand A

PISA-D FT

Strand C PISA 2015

Trend

%

Framework Goal

Number % Number %

Personal 25 31% 5 14% 27% 36%

Educational 25 31% 12 33% 28% 33%

Occupational 12 15% 10 28% 23% 20%

Public 18 23% 9 25% 22% 11%

Total 80 100% 36 100% 100% 100%

Mathematics Items by Process

115


Process

PISA-D FT

Strand A

PISA-D FT

Strand C

PISA 2015 Trend

%

Framework Goal

Number % Number %

Formulating situations mathematically

19 25% 8 18% 29% Approx. 25%

Employing mathematical concepts, facts, procedures

30 39% 21 47% 43% Approx. 50%

Interpreting, applying, and evaluating mathematical outcomes

27 36% 16 36% 28% Approx. 25%

Total 76 100% 45 100% 100% 100%

Mathematics Items by Content

116


Content

PISA-D FT

Strand A

PISA-D FT

Strand C PISA 2015

Trend

%

Framework Goal

Number % Number %

Change and relationships

13 17% 7 16% 27% 25%

Space and shape 16 21% 10 22% 23% 25%

Quantity 28 37% 19 42% 25% 25%

Uncertainty and data

19 25% 9 20% 25% 25%

Total 76 100% 45 100% 100% 100%

Instrumentation Strands A-B

• Tests: paper booklets, translated through Word files

• Questionnaires: paper booklets, translated through Word files

Strand C

• Tests: computer-based, authored, translated through xliff files

• Questionnaires: computer-based, authored, translated through an authoring tool; paper-based (parent)

117

Common Materials (same translations)

Unique Strands A-B Materials Unique Strand C Materials

Questions and Discussion

118


PISA for Development Contextual Questionnaires for

Educational Prosperity

J. Douglas Willms University of New Brunswick

and The Learning Bar

Strand B – The Learning Bar

April 2014 – PISA-D Technical Workshop

• Seven core themes

• Willms and Tramonte working paper

May 2014 – IAG Paris

• ToR Meeting: discussion of life-course approach for contextual framework

Dec 2014 – UNESCO workshop on inequalities

Mar 2015 – IAG Paris

• Draft contextual framework

Feb – May 2015 – country visits – Educational Prosperity model

• Ecuador, Guatemala, Senegal, Paraguay, Zambia

Aug 2015 – QEG meeting: Educational Prosperity and draft questionnaires

Sep – Oct 2015 – Quito NPM – further consultation on questionnaires and

framework

Nov 2015 – Paris summit - Finalized Questionnaires

Jan 2015 – Washington – NPM – Adaptations and Translation, Country Options

Mar-Apr 2016 – Asuncion NPM – CF, Qs, Translation process complete

Most countries have evaluation systems that

involve testing students at certain grades during

primary and secondary schooling.

Many of these countries also participate in large-

scale assessment programs, such as the

Programme for International Student Assessment

(PISA).

• An underlying assumption of these evaluations is that students’ test scores are mainly the result of the efforts of teachers and principals.

• However, students’ results in local, national and international assessments are the cumulative result of children’s environments during the prenatal period, their learning experiences at home and in centers during the pre-school period, and their family and school experiences after they enter kindergarten.

• Consequently, the results are often used inappropriately to develop educational policy and to hold educators accountable.

Educational Prosperity:

A Life-Cycle Approach

• Educational Prosperity includes a core set of metrics for success at six key stages of development across the life-course from conception to adolescence.

• These metrics include a set of key outcomes for each developmental stage, called ‘prosperity outcomes,’ and a set of family, institutional, and community factors, called ‘foundations for success,’ which drive the prosperity outcomes.

• The approach considers four ways that success accumulates over the life-span.

Success accumulates in four ways:

1. Biological embedding

2. Foundations for success

3. Cumulative effects

4. Selection

1. Biological Embedding Differential social experiences get under the skin in early life and, through their effects on developing neurobiological pathways, affect later trajectories in human health, learning, and behaviour?

Differential social experiences Socioeconomic gradients are not evident in most social outcomes, they are also evident in children’s early experiences; for example,

Breast-feeding

Smoking during pregnancy

under the skin A key development in the field of human development is the work defining neuro-biological development and its effects on health, learning, and behaviour.

Brain development from conception to age one is rapid and extensive, much more so than previously believed, and is heavily influenced by the infant’s environment (Carnegie Corporation of New York, 1994).

• A newborn has billions of neurons, which, during the course of development, form connections called synapses. These synapses are formed in response to environmental stimuli, and while this is occurring, many of the neurons that are not being used are pruned away.

• This process of synapse formation and neuron pruning is often referred to as the “wiring” or “sculpting” of the brain. Moreover, there are critical periods, especially during the first three years, when particular areas of the brain are sculpted

• Longitudinal studies that have followed children who have received intensive interventions aimed at increasing stimulation and providing parent training and support have demonstrated long-lasting effects on their social, behavioural, and educational outcomes

2. Foundations for Success

In addition to the effects that are biologically embedded, children’s outcomes are directly affected by the foundations for success at each stage of development.

For example, during the formative period from birth to age 2, children’s development is affected by parents’ engagement with the child and intra-family relations. From age 2 to age 5, children’s development is affected by these factors as well as their child care experiences at home and in early childhood centers.

Foundation of Success factors are:

Potent (strong effects on outcomes)

Pervasive (effect a range of

outcomes)

Proximal (has a direct effect on the outcomes)

The timely transition from learning-to-read to reading-to-learn, which for most children occurs at about age 8 or 9, is essential to academic success, school attainment and well-being.

During the primary school years, from kindergarten to Grade 3, considerable emphasis is placed on the development of reading skills. Of course, children learn subject-matter content and acquire many skills during the learning-to-read period.

But after Grade 3 there is a tacit assumption that children can read fluently and comprehend curricular materials in subject domains such as health, social studies, and science. The curriculum changes: students are expected to learn the languages of subject domains and use that language to think critically, solve problems, and create new knowledge. The demands for strong reading skills increase as students make their way into the higher grades. Students who lack the fundamental reading skills fall further and further behind.

The critical transition from learning-to-read to reading-to-learn

Confident Learners A whole-school literacy program for vulnerable

schools

‘The simple view of reading’ (Rose, 2006) has two critical, complementary dimensions:

Code-related skills – the ability to recognize and understand particular words, and

Language skills – being able to understand and interpret spoken

and written language.

A Deficit Approach

A Pathway Approach

Code-Related Skills

Concepts

About Print

Letter Knowle

dge

Phonological Awareness

Supra- Phonemic Awareness

(Large Units)

Phonemic Awareness

(Small units)

Reading Fluency

Accuracy:

Word Decoding

and Spelling

Accuracy:

Word Recognition and Spelling

Speed

and

Prosody

Language Skills

Vocabulary Receptive

Language

Expressive

Language

Written

Language

3. Cumulative Development

Children develop their skills in a cumulative process as they make the transition from one stage to the next. The skills they attain at one stage are an asset that they can use to develop skills at the next stage.

For example, children’s acquisition of language skills at age 2 is a strong predictor of their pre-literacy skills at age 5.

12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

Child's Age (Months)

0

100

200

300

400

500

600

700

800

Vo

cab

ula

ry S

ize (

Wo

rd

s)

Average Growth

Trajectory

Females

Males

Huttenlocher et al., 1991

The Early Years Evaluation in Uruguay

The Early Years Evaluation assesses skills in

five domains • Awareness of Self and Environment - a child's

understanding of the world and his or her ability to make connections with home and community experiences.

• Social Skills and Approaches to Learning - a

child’s attentiveness during classroom activities and his or her ability to interact with peers while respecting the classroom rules.

• Cognitive Skills - a child's basic math and pre-reading skills and his or her ability to solve problems.

• Language and Communication - a child's

understanding of spoken language and his or her ability to express thoughts and feelings.

• Physical Development

Fine motor - a child's ability to perform small

movements that require hand-eye coordination. Gross motor - a child's ability to perform large

movements that involve arms, legs, and body.

The Early Years Evaluation assesses skills in

five domains

In the Colonia pilot study sample (N = 3,425), the reliability coefficients (Cronbach’s alpha) of the EYE-TA subtests were as follows:

Awareness of Self and Environment: 0.91

Social Skills and Approaches to Learning: 0.93

Cognitive Skills: 0.93

Language and Communication: 0.93

Physical Development: 0.90

The Latin American version of the Early Years Evaluation will be used with all kindergarten children in Uruguay.

4. Selection When students are successful at one stage of

development, their life-course can be altered if

they are selected into certain classes, school

programs or schools.

For example, children who have strong reading

and language skills are more likely to be streamed

into classes or school programs where they benefit

from positive peer interactions, a higher quality of

instruction, and other factors that enable them to

develop their skills at a faster pace.

Children who experience learning difficulties at a

particular stage are more likely to be streamed into

lower ability classes and have less access to the

factors that improve their skills.

The Role of Contextual Questionnaires in


http://www.google.ca/url?sa=i&rct=j&q=secondary+schools+denmark&source=images&cd=&cad=rja&docid=x2-D9cn_WMTUoM&tbnid=EGiAaImSKB0wjM:&ved=0CAUQjRw&url=http://www.bu.edu/admissions/apply/international/secondary-school-credentials/&ei=b3ijUaaZBMH54AOP8YDoBg&bvm=bv.47008514,d.dmg&psig=AFQjCNFikuSUQ8mQ82ohXZxHoFUpckCYeA&ust=1369753921414868

The goals of PISA for Development

1. Increase countries use of PISA assessments for monitoring progress towards nationally-set targets for improvement.

2. Analyse factors associated with student learning outcomes, particularly for poor and marginalized populations; and

3. Build capacity for tracking international education targets in the post-2015 framework being developed within the UN’s thematic consultations.

Educational Prosperity builds on key findings

of PISA Educational Prosperity provides an opportunity for countries to

build on the knowledge acquired with PISA over the six cycles since 2000.

1. An understanding of why students with similar levels of socioeconomic status achieve higher levels of performance in some schools and school systems compared with others.

- the quest for ‘school effects’

- input-process-output models

2. Identify subject-matter values and attitudes related to performance that may lead students to follow a particular career path; and

3. Measure general ‘global competencies’ or 21st Century skills that are related to success in the workplace after students leave school.

The contextual questionnaires for

PISA for Development will …

1. Provide valid and reliable measures of a set of prosperity outcomes and foundations for success.

The goal is not to measure 20 or 30 factors that are directly or indirectly related to student performance. Rather, it is to measure in greater depth a small set of outcomes relevant to participating countries, and the key factors that drive these outcomes.

2. Provide an extended measure of socioeconomic status and poverty that can be used alongside other demographic measures to assess equality and equity.

3. Enable countries to use the data from PISA for Development in a straightforward and explicit way to inform policy and practice at all levels of the school system.

Contextual questionnaire content for the

Field Trial

Principles for the development of measures

1. Inclusion of core content from main PISA

2. Content that is high priority for participating countries.

3. Measures that are consistent with the Educational Prosperity framework.

4. Readability – Grade 5 or lower and words with ‘age-of-acquisition’ of age 10 or lower.

5. Tractable measures with direct links to policy and practice.

Contextual questionnaire content for the

Field Trial School questionnaire – 28 questions

focusing mainly to student demographics and the foundations for success. Incudes detailed measures of material resources and inclusive environments.

Teacher questionnaire – 36 questions pertaining mainly student demographics and the foundations for success. Incudes detailed measures of quality instruction and inclusive environments.

Student questionnaire – 83 questions rotated over three booklets. It includes several questions on students’ language and their socioeconomic status.

Trend questions

New questions

Capacity Building Activities

• 4th PISA-D objective: country capacity in assessment and analysis strengthened


Preparation for participation and support during implementation:

Technical support and assistance

Project implementation

plan

Capacity building plan

Capacity needs analysis


OECD support for and collaborative working to produce:

Policy dialogue

Dissemination of results

Preparation of a country

report

Country-specific

data analysis

Preparation of in-country stakeholders


• International/NPM meetings and follow-up

• Manuals, guidelines, procedures – translations of these

• Peer-to-peer learning and support

• Additional training and capacity building workshops and events, engagement and communication efforts

• Country visits by OECD and contractors

• Support for country specific analysis of results and collaboration with OECD over the writing of a national report and dissemination activities, including support for policy dialogue

International/NPM meetings

Meeting Venue Dates Content Peers

2015 International Advisory Group

meeting (3 days)

Paris, France 11-13 March 2015 Project planning and

thematic discussions.

Belgium

Spain

International/NPM#1/Capacity Building

meeting (4-5 days)

Quito, Ecuador 28 September – 2

October 2015

PISA cognitive and

contextual

frameworks,

characteristics of the

available item pools.

Chile

Korea

Mexico

Peru

Uruguay


meeting(4-5 days)

Rockville, MD,

United States

25-29 January

2016

Adaptation, translation

and verification of

survey materials and

sampling.

Canada

US


meeting (3 days)

Asunción,

Paraguay

30 March – 1 April

2016

Project planning and


Kosovo


meeting (4-5 days)

Asunción,

Paraguay

4-8 April 2016 Student sampling and

field trial survey

operations.

Brazil

International/NPM meetings Meeting Venue Dates Content Peers


meeting (4-5 days)

Livingston, Zambia 4-8 July 2016 Scoring and coder

training and data

management for the

field trial.

China

International/NPM#4a/Capacity Building

meeting for Strand C only (2-3 days)

Barcelona, Spain 1-3 November

2016

Quality control sample

selection forms, and

quality assurance

procedures, data

management

(software, codebook,

etc.)

Spain

2017 International Advisory Group meeting

(3 days)

Cambodia 17-19 May 2017 Project planning and


Iceland


meeting (4-5 days)

Cambodia 22-26 May 2017 Analysis and

interpretation of field

trial results and

preparation for main

study.

Korea


meeting (4-5 days)

Princeton 4-8 July 2017 Student sampling and

main study survey

operations.

Canada

US

International/NPM#6a/Capacity Building

meeting Strand C only (2-3 days)

Panama city,

Panama

14-18 January

2018

MS interviewer training

and data management

training.

International/NPM meetings

Meeting Venue Dates Content Peers


meeting (3 days)

Saly, Senegal 14-16 March 2018 Project planning and



meeting (4-5 days)

Saly, Senegal 19-23 March 2018 Data processing,

results, scaling

methodology and

preparation for

analysis.


meeting (8-10 days)

Antigua,

Guatemala

July 2018 Analysis and

interpretation of main

study results, reporting

and dissemination of

results.

Peer-to-peer learning

The goals of peer-to-peer learning in PISA-D are:

• help key staff in PISA-D countries to attain the level of project implementation outlined in PISA’s Standards, capacity building plans and other project documents,

• enhance the management of large-scale assessments in PISA-D countries,

• provide key staff in PISA-D National Centres with opportunities for professional growth and development,

• provide opportunities for collegial sharing and reflective practice,

• accelerate the capacity building objectives of the project.

*Guidelines to support the process*


Peer learning partnerships:

Peers Peer Learners

Korea Cambodia

Uruguay Ecuador

Peru Guatemala

Brazil Honduras

Chile Paraguay

Canada Senegal

Hong Kong/China/Iceland Zambia


• Peer learning partnerships involving exchange

visits, workshops, tele-conferences, email exchanges

• Contributions to international/NPM meetings, IAG meetings, other workshops and training events

• Case studies of participating in PISA, institutional histories of PISA national centres

Additional training and capacity building workshops and

events, engagement and communication efforts

• Project launches and facilitation of project and national

engagement and communication strategies

• With IDB, capacity building workshops for Latin American countries on frameworks and item development; Item Response Theory; secondary analysis of data; and communication and engagement with stakeholders

• Item Response Theory workshop for Senegal

• Capacity building workshops and partnership between Cambodia and Korea Institute for Curriculum and Evaluation

Feedback from the countries

• How many members of your team have benefited from participation in an international NPM meeting or other capacity building event since the last IAG? How did you disseminate what you learnt at these meetings to other members of your team?

• For Ecuador, what was it like to host an NPM meeting? What advice would you give to future hosts?

• For Cambodia (and Korea) what progress have you made with capacity building workshops?

• For the Latin American countries (and IDB) what progress have you made with capacity building workshops?

Feedback from the countries

• How are the peer-to-peer learning partnerships working out?

• Which aspect of the project so far has been most helpful in building your capacity? Has this benefited in any way your national assessment?

• What are you most looking forward to in the next 12 months for capacity building?

• Are there any important areas of capacity building included in your plans that the project or other partnerships not yet addressing or planning to address?

3th PISA-D International Advisory Group Meeting

Capacity Building Activities and

Outcomes

30 March – 1 April 2016 Granados Park Hotel, Asuncion, Paraguay

Structure

• Steering Committee provides strategic direction and oversight

• Technical Task Force performs project tasks and develops capacity that can be used by other projects

• Quality Assurance Department is responsible for all PISA-D activities

Steering

Committee

Task Force

Policy Makers

Develop

ment

Partners

Quality

Assurance

Departmen

t

ERC/RUPP

OECD-

PISA

Peer

Countries

Development

Partners,

Curriculum,

National assessment,

Regional assessment,

Research,

Teacher Training,

Labour,

Private sector (foreign

and domestic)

Structure, cont.

• Steering Committee Memberships: Minister, Secretary of States, Under Secretary of States, Director Generals, Deputy Director Generals, Department Directors, and Development Partners

• Technical Task Force Memberships: Director General (leader), Department Directors, Education Research Council and Development Partners

• Education Quality Assurance Department Members: 12, 7 are full time staff and receiving on the job training

Capacity Building • February 2016: a workshop on PISA-D was held in Phnom Penh,

hosted by Ministry of Education and KICE, supported by UNESCO Bangkok Participants: related departments/institutions, NGOs,

Development Partners, provincial officials and 8 NEQMAP (Network on Education Quality Monitoring in the Asia-Pacific) member countries (Bhutan, Laos, Mongolia, Myanmar, Nepal, Pakistan, Sri Lanka)

The topics: Introduction of PISA and PISA-D, PISA Implementation (Case Study of Korea), PISA results implementation for participating countries (KICE), PISA experience of middle-income countries and its results (OECD), Cambodia Case Study (EQAD), Assessment Framework and Item development (by KICE), Secondary data analysis and reports (KICE)

• Peer-to-peer learning partnerships: KICE and Cambodia are discussing on the activities for the capacity building after the Launching event

Capacity Building, cont.

• For the near future, below topics are be expected to get support from KICE: Item and Questionnaire Developments, Sampling Design, Data Collection and Security Protocols, Data Analysis and Report Writing

• Suggested topics: The use of the findings to inform the policy, curriculum, teaching/learning should be added to be a complete set of assessment cycle

Thank You

Q1. How many members of your team have benefited from participation in an international NPM meeting or other capacity building event since the last IAG? How did you disseminate what you learnt at these meetings to other members of your team?

• 4 members of the team All meetings are subject to a report sent to the Minister of Education, the Secretary-General and the departments concerned by PISA -D. The results of the meetings of the steering committee and technical committees are shared with the technical and financial partners during the meetings of the Steering Committee and the National Technical Committee. Every year, Senegal organizes two meetings of this kind for information on progress made and difficulties noted. Important decisions are selected at the end of these interministerial meetings. Coordination meetings at the Ministry of Education are also sharing opportunities with other directors. Moreover, INEADE the technical team meets regularly to exchange around PISA-D. Roles and responsibilities are distributed among all members of the technical team.

Q2. What progress have you made with capacity building workshops?

• The CBP was finalized interactively with the consultant Satya Brink.

• The summary document prepared by the NPM was sent to the Minister and all training activities were planned.

• The budget for the organization of training sessions is validated by all stakeholders in the Ministry (DAGE - DPRE - DMSG ).

• The INEADE forwarded the terms of reference for the training of 20 members of the technical team at the OECD and DPRE for execution. The team is looking for a consultant to the holding of the first session which covers the IRT and writing items. It is imperative to receive three offers to enable the Department to choose the most appropriate profile. We had planned to hold this session in the month of April 2016 if all constraints are lifted.

Q3. How are the peer-to-peer learning partnerships working out?

• For language reasons, Senegal has chosen Canada as the sponsor in the peer-to -peer learning.

• Meetings of peer-to- peer learning as part of the NPM Peru, Brasil, Belgium, France and Canada were given the opportunity to learn from the experience of these countries.

• In the absence of a budget for this activity, experience tours that would have given the opportunity to understand more about how countries organize their teams and solve problems , peer- to-peer learning has difficulty functioning well. However, Senegal appreciates the openness of the NPM of Canada and its embassy who are willing to accompany us in the PISA -D.

Q4. Which aspect of the project so far has been most helpful in building your capacity? Has this benefited in any way your national assessment?

PISA-D provides lifelong learning in all aspects from planning, implementation and the sharing of results. All activities conducted support the idea that PISA is the most rigorous evaluation. The Senegal boost:

- The collaborative approach in developing the terms of reference of the meetings, feedbacks studies and reports submitted by firms (ETS, Pearsons , TLB, IUS)

- Shared the quality of scientific papers with the teams that are immense treasures and heritage for conducting evaluations (reference frameworks , guides , tools)

- The multiplicity of tools and communication management (Sharepoint , permanent mail reminders)

- The feedback and support provided to countries to enable them to make progress in the assigned tasks (sampling task, action plan, negotiations with financial and technical partners)

Q5. What are you most looking forward to in the next 12 months for capacity building?

Strengthening capacity of Senegal’s PISA-D team in the techniques of multivariate analysis and sharing methodologies and reporting of assessment data (datasvisualization)

Q6. Are there any important areas of capacity building included in your plans that the project or other partnerships not yet addressing or planning to address?

The only area not covered by the project is peer-to-peer learning, which has not yet received funding. We hope to revive the Canadian Embassy or other partner to afford to visit the Canadian experience in PISA . Overall, the plan has good coverage.

Country Report on Capacity

Building Activities

IAG Meeting (30th March, 2016), Asunción, Paraguay

Zambia

Outline of Presentation

• Overview of Progress on PISA – D

• Capacity Building Plans

• NPM Meetings

• International Contractors

• Peer to Peer

• Way Forward

Overview on Progress • Signed Agreement • Launched Project • Paid International Fees • Attended IAG Meetings • Attended NPM Meetings • Visits from Contractors & Secretariat • Video Conferencing, E-mails and Calls with

Secretariat & Contractors • In Country Meetings with Key Stakeholders • Working with Contractors on Adaptation and

Sampling

Capacity Building Plans

• Worked with International Contractor on Capacity Needs.

• Identified Capacity needs were costed and budgeted.

• Worked with Secretariat to align the capacity needs to international project implementation timeline.

• Streamlined capacity needs with those to be covered by the International Contractors.

• Zambia is sourcing funding for activities not covered by International Contractors.

NPM Meetings

• Zambia has attended all the two NPM and technical meetings held so far.

• Participated in the PISA Strand A Assessment Framework development.

• Appreciated considerations on development of Frameworks for Strands A, B & C.

• Training on Sampling design, Adaption and Translation received.

• Interacted with PISA countries, Experts, Developmental Partners, Other Assessment organizations and Institutions.

International Contractors

• During Meetings International Contractors have built capacity.

• International Contractors have visited Zambia to work with local team.

• International Contractors have been guiding Country team through e-mail on various activities through e-mail, Video Conferencing and phone calls.

Peer to Peer • Through meetings such as NPM peer to peer

learning has taken place.

• The hosting of NPM and IAG meetings has facilitated understanding other members situations and how best Zambia can implement the project.

• Involvement non OECD countries who have participated in PISA has been a positive influence.

• Meeting and talking with OECD Countries with experience in PISA has also been beneficial.

Way Forward

• Zambia to secure resources to undertake capacity building activities not covered in the NPM Meetings and International Contractors.

• NC to partner with local institutions who have capacity in some areas of need such as Strand C’s household surveys.

• Zambia will continue cherishing the support of the Secretariat , International Contractors and other partners in the successful implementation of the PISA – D project in the country.



PISA for Development Conclusions Day One

Strong representation at the meeting from:

• Countries signed up (Cambodia, Ecuador, Guatemala, Honduras, Paraguay, Senegal, Zambia) or committed (Panama) to participation in PISA for Development

• Development partners (France, Germany (BMZ/GIZ), IDB, Sunny Varkey Foundation, Japan(JICA), Norway, UK (DFID), World Bank)

• International agencies (UNESCO, UIS, UNICEF, LLECE)

• International contractors.

Conclusions Day One

Conclusions Day One • Two co-chairs elected:

– one representative of the participating countries: Rodolfo Elias (Paraguay)

– one representative of the development partners: Maria Soledad Bos (IDB)

• Annual report on project activities reviewed and approved with suggestions for improving presentation of project progress and highlighting risks (particularly related to risk 4 on project impact), and praised the substantial progress that has been made

• IAG members noted the new development partners and discussed the role of these within the IAG

• IAG members also noted the challenge of countries for financing their participation in international meetings and asked OECD to continue reporting on this

Conclusions Day One

• OECD confirmed that no more countries will join PISA-D

• IAG members noted that Strand C activities are on track despite there not yet being a signed contract with ETS and encouraged OECD to sign the contract as soon as possible

• Members noted the presentations by ETS on Strand A and commended the countries, contractors and Secretariat for the progress that has been made

• IAG members discussed issues related to comparability in the context of PISA-D having no major domain, computer-based vs. paper-based, and trend links

Conclusions Day One

• Members noted the presentations by TLB on Strand B and commended the countries, contractors and Secretariat for the progress that has been made

• There was a discussion of how the framework for the questionnaires are reflected in the items selected for the instruments

• Members noted the presentation by ETS on Strand C and commended the countries, contractors and Secretariat for the progress that has been made in preparing this component

• There was a discussion of the timeline for Strand C and countries noted the short window of time available for the field trial data collection

Presentations from the 3rd PISA for Development International Advisory Group Meeting held in...

Education

Transcript of Presentations from the 3rd PISA for Development International Advisory Group Meeting held in...