OECD Reviews of Evaluation and Assessment in Education: Czech Republic
-
Upload
oecd-education -
Category
Education
-
view
970 -
download
0
description
Transcript of OECD Reviews of Evaluation and Assessment in Education: Czech Republic
OECD Reviews of Evaluation and Assessment in Education: Czech RepublicBy Paulo Santiago, Alison Gilmore, Deborah Nusche and Pamela Sammons
www.oecd.org/edu/evaluationpolicy
Conference, PragueThe main conclusions and policy
recommendationsPresentation by
Paulo SantiagoDirectorate for Education, OECD
15 March 2012
Outline of presentation
Part 1
The OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes and the Review of Evaluation and Assessment in the Czech Republic
Part 2
Conclusions and Recommendations• The Evaluation and Assessment Framework
• Student assessment
• Teacher appraisal
• School evaluation
• Education system evaluation
PART 1
(1) The OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes and;
(2) the Review of Evaluation and Assessment in the Czech Republic
OECD REVIEW - OBJECTIVES
●Purpose: To explore how systems of evaluation and assessment can be used to improve the quality, equity and efficiency of school education.
A Review of national approaches to evaluation and assessment in school education
●The Review:
Synthesises research-based evidence on the impact of evaluation and assessment strategies and disseminate this knowledge among countries.
Identifies innovative and successful policy initiatives and practices.
Facilitates exchanges of lessons and dialogue.
Identifies policy options for governments to consider.
OECD REVIEW - SCOPE
• The Review looks at the various components of assessment and evaluation frameworks. These include:
• Student assessment;
• Teacher appraisal;
• School evaluation;
• Education system evaluation;
• Other types of evaluation (programme evaluation, evaluation of school leadership etc)
• Comprehensive approach: Investigation of each component individually, as well as the coherence of the framework as a whole (including the links between the different components).
OECD REVIEW - KEY ISSUES
Key policy issues for analysis
1.Governance: Designing a systemic framework for evaluation and assessment2.Procedures: Ensuring the effectiveness of evaluation and assessment procedures3.Competencies: Developing competencies for evaluation and for using feedback4.Use of results: Making the best use of evaluation results5.Implementation: Implementing evaluation and assessment policies
OECD REVIEW - BROAD FEATURES
●Oversight by the Group of National Experts (GNE) on Evaluation and Assessmenthttps://community.oecd.org/community/evaluationpolicyinschools
●Countries working collaboratively with the SecretariatCountries do part of the work of the Review, a national co-ordinator is nominated, dialogue within
countries is encouraged
●Countries exchanging lessons and experiences, in particular through GNE meetings
●Collection of views and perspectives from a range of stakeholdersThrough their participation in GNE meetings and their interaction with Country Review expert
teams
●A diverse set of external experts involved in the Country ReviewsInvolvement of top academics and policy makers to bring expertise into the Review
●Ensuring a range of principles are respectedIndependence of analysis; inclusiveness in terms of the views and perspectives which are collected;
extensive reviews of the literature; evidence-based policy advice; site visits to interact with school realities; contextualisation of policy advice.
OECD REVIEW - METHODOLOGY
Analytical strand• Identifying the key questions for analysis and the background information needed
from countries• Reviewing the literature and evidence on the impact of evaluation and
assessment procedures• Gathering data on countries’ policies and practices (Country Background Reports)
Country Review strand• Country Reviews provide specific advice to individual countries. • OECD-led Review Team with external experts• The scope and focus is determined by the country in consultation with the
Secretariat Synthesis report
• Comparative report to analyse policy options and highlight good practices across countries.
Collaboration with other OECD activities and international agencies• INES (in particular NESLI), PISA, TALIS, CERI’s ILE , Longitudinal Information Systems and
Governing Complex Education Systems project, CELE’s on evaluating quality in educational facilities, GOV’s work on Public Sector Evaluation.
• EC, Eurydice, the World Bank, Standing International Conference of Inspectorates(SICI), UNESCO, BIAC, TUAC.
OECD REVIEW - PARTICIPATION
●Twenty four systems are preparing a Country Background Report:
Australia, Austria, Belgium (Flemish Community), Belgium (French Community), Canada, Chile, Czech Republic, Denmark, Finland, France, Hungary, Iceland, Ireland, Korea, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Slovak Republic, Slovenia and Sweden.
●Twelve countries also opted for a Country Review:Australia, Belgium (Flemish Community), Chile, Czech Republic, Denmark, Luxembourg, Mexico,
New Zealand, Norway, Portugal, Slovak Republic and Sweden.
●All OECD Member countries and Observers to the Education Policy Committee are members of the Group of National Experts on Evaluation and Assessment
REVIEW of EVALUATION and ASSESSMENT in the CZECH REPUBLIC
●Objectives: 1. To provide insights and suggestions for policy development in the Czech
Republic;2. To inform the wider international community about (a) the main features of the
Czech Republic’s evaluation and assessment policies; and (b) innovative and effective approaches to evaluation and assessment in the Czech Republic; and
3. To inform the final synthesis report from the project.
●Structure of visit (29 March – 5 April, 2011):• Review team with 4 members: 2 OECD Secretariat members; and researchers
from New Zealand and the United Kingdom;• Interacted with about 200 individuals; had about 45 meetings amounting to
about 50 hours of discussions; visited 6 schools in Prague, Ostrava and Liberec• Discussions were held with a wide range of national, regional and local
authorities; education officials; the Schools Inspectorate; parents’ organisations; representatives of schools and school directors; teacher educators; and researchers with an interest in evaluation and assessment issues.
REVIEW of EVALUATION and ASSESSMENT in the CZECH REPUBLIC
●Relevant features: 1. External perspective: looking from a distance;2. Independent perspective: no review team member has a vested interest in
the system;3. Not an examination: qualitative analysis that seeks to provide an input of a
specific nature to the internal debate;4. Bring together research evidence, data and information available in a broad
comprehensive comparative framework.
●Country Review report released on 30 January, 2012 and available at: www.oecd.org/edu/evaluationpolicy
PART 2
Conclusions and policy recommendations
Conclusions and policy recommendations
The evaluation and assessment framework
The Evaluation and Assessment Framework
Strengths
There are common references at the national level to provide the basis for evaluation and assessment
4-year Long-term policy objectives with indicators; Framework Education Programmes
There are good conditions for adapting learning to local needsSchool Education Programmes: curriculum innovation; collaborative workRegional 4-year Long-term policy objectives
Responsibilities across the evaluation and assessment framework are well articulated
MEYS, CSI, Regions and municipalities, schools
The Evaluation and Assessment Framework
Strengths (continued)
There is a range of initiatives to strengthen evaluation and assessment in the school system
National standardised tests, common part of the school-leaving examination, external school evaluation, school self-evaluation mandatory, national indicators on education
There is an “open door” climate among teachers
There is a good principle of supporting policy work with specific expertise
Institutes with specialised expertise, data collections, OECD projects
There are some reporting requirements:
Education database, inspection reports
The Evaluation and Assessment Framework
Challenges The evaluation and assessment (E&A) framework needs to be completed
and made coherent
o There is no integrated E&A framework
o The E&A framework is incomplete (formative assessment of students, moderation of marks, systematic teacher appraisal, school self-evaluation incipient, no framework for school leadership appraisal)
o Some articulations within the E&A framework are not sufficiently developed
There is little attention to equity and inclusion in the evaluation and assessment framework
The Framework Education Programmes are not perceived as specific enough to guide teaching and assessment
Little shared understanding about what constitutes adequate, good and excellent performance
It is unclear that the students are at the centre of the evaluation and assessment framework
The Evaluation and Assessment Framework
Challenges (continued) There is a narrow understanding of the purposes of evaluation and
assessmentPerceived as instrument to hold stakeholders accountable, to “control” and assess
compliance with regulations There is a need to strengthen competencies for evaluation and assessment
across the system
The articulation between levels of government and the support from the centre are limited
o Concerns about the lack of systematic application of national directions, inconsistency of practices and little capacity or commitment to developing quality frameworks.
o Weak articulations between the different decision-making levels.o Limited provision from the centre of tools and guidelines to assist evaluation and
assessment activities There are challenges in the implementation of some evaluation and
assessment initiatives
The Evaluation and Assessment Framework
Policy recommendations Better articulate learning goals
o Establish clear goals for education and make equity and inclusion more prominent
o Clarify reference points and criteria for quality in evaluation and assessment
Integrate the evaluation and assessment framework
Develop a strategic plan or framework document that conceptualises a complete evaluation and assessment framework and articulates ways to achieve the coherence between its different components
Strengthen some of the components of the evaluation and assessment framework
Teacher appraisal, appraisal of school leaders, formative student assessment, school self-evaluation
Further develop some articulations within the evaluation and assessment framework
The Evaluation and Assessment Framework
Policy recommendations (continued)
Build on some key principles to effectively implement evaluation and assessment
o Place the students at the centre of the evaluation and assessment frameworko Ensure a good emphasis on the improvement function of evaluation and
assessmento Communicate the rationale for evaluation and assessmento Recognise the importance of school leadershipo Establish an implementation strategy
Develop evaluation and assessment capacity across the school system
Improve the articulation between levels of government and assure support from the centre
Conclusions and policy recommendations
Student assessment
Student Assessment
Strengths
Assessment is seen as part of the professional role of teachers
o Teachers play the major role in assessing and reportingo Schools decide and publish assessment criteriao Variety of approaches
An external dimension to assessment was introduced
o National component of school-leaving examinationo Common assignments for the apprenticeship certificateo Addresses the need for checks and balances to ensure reliability in the
application of standards There is an increased focus on student outcomes
o Including a move to identify expected minimum standards of achievement for students at key points in their education
Student Assessment
Challenges Approaches to learning and assessment remain markedly traditional
o Traditional approach to the organisation of classroomso Little emphasis in assessment practices on providing student feedback
Assessment for learning is not systematically used in Czech schools
o Feedback often understood as ‘summative assessment done more often’ Summative assessment of students raises some concerns
o Assessment is often norm-referencedo Achievement mixed with effort and motivation
There is limited consistency of student assessment across schools and classes
The national-level support for teacher-based student assessment is limited
o Lack of guidance to assess against FEPs (e.g. in the form of exemplars), insufficient assessment tools for teachers, limited professional development.
Student Assessment
Challenges (continued)
The national standardised tests entail a range of limitations and risks
o Development of the standards being rushedo May be more appropriately regarded as specifications for the national tests,
rather than standardso Potential negative consequences of high-stakes use: (i) ‘teaching to the test’;
(ii) cross-curricular competencies ignored; (iii) classroom time spent on preparation of the test; (iv) schools selecting the students taking the test.
The assessment of students leads to little interaction among teachers
o Low levels of teacher moderation of student assessments Multiple purposes to school-leaving examinations raise some concerns
o To provide a certificate of achievement to studentso To compare performance of schools: need other sources of information to
achieve this and value-added techniques
Student Assessment
Policy recommendations
Develop educational standards covering the breadth of student learning objectives prior to developing national standardised tests
o There is a need for clear external reference points in terms of expected levels of student performance at different levels of education
oNational tests not to be the vehicle to develop standards Limit the undesired effects of national standardised tests
Reflect further on the purposes of the national tests; should be trialled first; minimise negative impacts
Develop a broad strategy for student assessment and strengthen the role of formative assessment
Assessment for learning; provide tools supporting the assessment of students; broad range of assessment approaches; formative ass.
Build teachers’ capability for student assessment
Student Assessment
Policy recommendations (continued)
Develop a range of tools at the central level to support teacher-based student assessment
e.g. marking rubrics; exemplars illustrating student performance, assessment tools
Put in place moderation processes to ensure the consistency of student summative assessment
Student assessment should be criterion-based rather than norm-referenced
Ensure student assessment is inclusive
e.g. students with special needs, avoid biases by socio-economic background and minority status (e.g. Roma students)
Build capacity of markers of external tests and examinations
Conclusions and policy recommendations
Teacher appraisal
Teacher Appraisal
Strengths
The principle that teachers should be evaluated is widely accepted
o Teacher appraisal takes place in all schools and is an important and normal part of school activities
Teacher appraisal is focused on evaluating classroom teaching
The importance of teacher professional development is recognised in the legislation
Some structures for co-operation and exchange among teachers are in place
o E.g. Subject commissions bringing together all teachers teaching a particular subject; lesson preparation; exchange of pedagogical approaches
Teacher Appraisal
Strengths (continued)
There are plans to develop teaching standards and a new career system for teachers
The link between teacher appraisal and pay increments has potential to incentivise high performance
o Teachers may be awarded additional pay increments and bonuses that are determined by the school leadership team
o However, there are important questions regarding the transparency of how salary rewards are implemented
Teacher Appraisal
Challenges
There is currently no shared understanding of what constitutes high quality teaching
o Lack of a national framework defining standards for the teaching profession
o There are no uniform performance criteria or reference frameworks against which teachers could be appraised
Teacher appraisal is not systematically implemented for all teachers
The quality and extent of teacher appraisal approaches in individual schools depend on the capacity and leadership style of the school principals.
There is little tradition for educational leadership in schools
Many school leaders have not been sufficiently prepared for their wide range of tasks, in particular leading teaching and learning processes in the school.
Teacher Appraisal
Challenges (continued) There are tensions between the accountability and improvement functions of
teacher appraisal
o Teacher appraisal traditionally been conceived as a summative and accountability-oriented process
o There are risks involved in trying to achieve both the accountability function and the improvement function of teacher appraisal in one single process
The link between teacher appraisal and rewards is not transparent
Links between teacher appraisal and professional development could be enhanced
o The provision of prof. development appears fragmented and not systematically linked to teacher appraisal
o There is scope to better link teacher professional development to school development and improvement
Teacher Appraisal
Policy recommendations
Develop a professional profile or standards for the teaching profession
A clear and concise statement of what teachers are expected to know and be able to do is a key element in any teacher appraisal system as it provides a credible reference to make judgements about teacher competence
Strengthen teacher appraisal for improvement purposes (developmental appraisal)
o Strengthening regular school-based formative appraisal with a professional development focus which is separate from the more summative appraisal processes.
o To be validated externally by Schools Inspectorate
Further enhance the role of educational leadership
Teacher Appraisal
Policy recommendations (continued) Consider establishing a system of teacher certification to determine career
progression
Access to each of the key stages of a career could be associated with formal processes of summative appraisal that complement the regular formative appraisal
The different career stages should match the different levels of expertise reflected in teaching standards
Need a stronger component external to the school to validate the process and ensure that practices are consistent across the country
Three core instruments: classroom observation, self appraisal and documentation of practices in a simplified portfolio.
Ensure links between developmental appraisal and appraisal for certification
Ensure appropriate articulation between teacher appraisal and school evaluation
Conclusions and policy recommendations
School evaluation
School Evaluation
Strengths External school evaluation is established
o Clear commitment to external accountability based around school evaluation with a regular cycle of external school evaluations
o External school evaluation valued by stakeholders
The external evaluation model embodies a number of features of best practice
o Well structured and systematico Set of publicly-available criteria for external inspection is drawn up every year o Publication of inspection reportso A range of sources of informationo Thematic reports
Schools facing greater challenges benefit from some follow-up
The CSI undertakes a follow-up inspection to assess whether improvements were undertaken to address the challenges previously identified.
School Evaluation
Strengths (continued)
Classroom observation is part of school evaluation processes
Placing learning and teaching at the heart of the evaluation process sends clear signals about what matters.
There is a new emphasis on schools’ self-evaluation
Has the potential to encourage schools and principals to place a greater emphasis on school improvement and development planning.
School leadership is promoted in school evaluation
There is an explicit recognition that the processes of self-evaluation and external evaluation are hugely dependent on a principal’s capacity to stimulate engagement, to mobilise resources and to ensure appropriate training and support.
School Evaluation
Challenges External school evaluation seems to have limited emphasis on school
improvement
o The accountability function tends to emphasise compliance with legislation rather than the promotion of school improvement.
o Advice is only given to “weaker” schools which are identified as those that do not meet the minimum standards as set by law.
o There is not enough guidance about what will lead to school improvement and little attention is paid to identifying and disseminating best practice
There are a number of limitations in external school evaluation
Difficult to take account of the socio-economic context of the school; not enough emphasis on pedagogical aspects, Criteria used in the CSI external evaluations are not stable enough.
There is little emphasis on student results/progress
Limited ability to assess quality of learning and student progress.
School Evaluation
Challenges (continued) School self-evaluation needs to be strengthened
Its penetration across the school system remains at an early stage of development
The use of data for school development is limited
The evaluation by organising bodies has a limited scope and impact
There is no full recognition of the role of school leaders and their appraisal is limited
oWhile the school principal has a key role in the system and considerable responsibilities, this has not as yet been translated into a dedicated career structure
o Limited preparation for the role of school principalo Little recognition and financial rewardo The evaluation of school principals, conducted by organising bodies, is largely
absent except in terms of the financial aspects of budget management.
School Evaluation
Policy recommendations Strengthen external school evaluation
o Strengthen focus on school improvement and move away from the current “compliance” driven model.
o Provide advice for improvement to all schools evaluated.
o The school evaluation framework, the criteria and questions governing judgements and the methods employed should all focus much more directly on the quality of learning and teaching and their relationship to student outcomes.
Improve the alignment between external and self-evaluation and raise the profile of self-evaluation
Criteria used in both processes sufficiently similar as to create a common language
Give stronger emphasis to the follow-up to external evaluation
Improve the capacity of schools to engage in school evaluation
School Evaluation
Policy recommendations (continued) Improve the instructional leadership skills of school principals
Plan to use data on student results effectively
Any publication of results of school performance in students’ school-leaving examinations and/or national tests should be presented in ways that take account of intake differences including, for example, the socio-economic background of students.
Strengthen the evaluation of school principals
Improve the scope and role of organising bodies in school quality improvement
o The regional and municipal authorities should strengthen their role in supporting school improvement.
o Collaboration and networking amongst schools could be encouraged to help develop and spread good practice and enhance teachers’ professional skills.
Conclusions and policy recommendations
Education system evaluation
Education System Evaluation
Strengths An Education Indicators Framework is established
It involves well-established procedures for data collection in close articulation with schools.
There is a concern to assess the progress of the education system towards pre established objectives
Principle of establishing educational objectives and the subsequent monitoring of the progress towards achieving them
The qualitative analysis of thematic reports provides valuable information for system monitoring
Richness of contextualised qualitative analysis
The participation in international surveys is instrumental for system evaluation
Education System Evaluation
Challenges There is little emphasis on the evaluation of the education system
o Limited policy attention thus far and there is no comprehensive strategic approach to it
o The current narrow approach to system evaluation does not allow a broad enough assessment of the extent to which student learning objectives are being achieved
The absence of student performance data is a major gap in system monitoring
Presently there is no mechanism for the Czech Republic to monitor at a national level the achievement of its students against learning objectives specified in the Framework Education Programmes
There are key information gaps at the system level
There are no measures on students’ socio-economic background; There are gaps in the data collected from schools; Little emphasis on investigating inequities in the system; Limited information on the teaching and learning environment
Education System Evaluation
Challenges (continued) It is not possible to monitor student outcomes over time and across schools
o Mostly due to the absence of national data on student performanceo School-level results of national tests might be disclosed with no account for
schools’ particular contexts. This can considerably distort considerations about the effectiveness of each school
Monitoring at the region and municipality levels is faced with considerable challenges
o Regions have a limited intervention in quality assurance with their main tool being the evaluation of school principals
oNo overview of the different quality assurance systems in the regions, including strategies for school improvement
o Limited articulation between Regions and municipalities System-level information is not fully exploited
Little analysis to inform educational planning; Limited use to inform school management; No comprehensive information system; Systematic sharing of data between schools is limited
Education System Evaluation
Policy recommendations Raise the profile of system evaluation within the evaluation and assessment
framework
The challenge for system-level evaluation is to ensure that the measures of system performance are broad enough to capture the whole range of student learning objectives – policy making at the system level needs to be informed by high quality data and evidence, but not driven by the availability of such information.
Develop national student performance data for system monitoring
o Design national standardised tests for national monitoring and as a pedagogical tool [closely aligned with student learning objectives; can measure only a limited range of student learning objectives; publish test results at the school level is premature]
o Develop strategies to monitor a wider range of curricular areas and broader outcomes [develop sample-based surveys]
o Ensure that national monitoring covers broader outcomes [e.g. higher-order thinking skills and cross-curricular competencies]
Education System Evaluation
Policy recommendations (continued) Prioritise efforts to meet information needs for national monitoring
o Develop measures of the socio-economic background of students o The data collection from schools needs to be improvedo Give more prominence to the analysis of inequities in the systemo Improve the information on the teaching and learning environment
Explore ways to more reliably track educational outcomes over time and across schools
Make meaningful comparisons across schools if test results are published at the school level
Strengthen the role of regions and municipalities in quality improvement Optimise the reporting and use of system-level data
o Strengthen the analysis for educational planning and policy developmento Improve feedback for local monitoringo Integrate available data and facilitate access by key agencieso Facilitate the sharing of student information across schools
Thank you([email protected])
(www.oecd.org/edu/evaluationpolicy)