Art of Evaluation ,Knowledge,Skills &Attitude

Prepared By

Salwa Mahmoud Abd ElwahabUnder supervision

Prf. Dr Tahany El SenosyProfessor of medical surgical nursing

Faculity of nursing Ain Shams University

2010

Outlines: Introduction Specific terminology related to

evaluation Definition of evaluation Types of evaluation Purposes of evaluation Elements Needed for the Construction

of an Evaluation System Standards of Evaluation Guiding Principles for evaluator Steps of evaluation

Outlines contPhases of evaluation Methods of evaluate (knowledge – skills – attitude )

Evaluation models

Objectives :Define evaluation Define types of evaluationList the purposes of evaluationIdentify elements of construction and evaluation system

Identify steps of evaluationExplain phases of evaluationDiscuss models of evaluation

Introduction Evaluation is systematic determination of

merit, worth, and significance of something or someone using criteria against a set of standards.

Evaluation is a learning and action-oriented

management tool and organizational process

for improving current activities and future planning, programming, and decision-making

Introduction cont The knowledge is defined by the Oxford

English Dictionary as expertise, and skills acquired by a person through experience or education There are many methods to evaluate knowledge Interview Questionnaire, Discussion, Survey, Analysis of records and data.

(Lesley, 2001) defines practical nursing skills as hands-on actions that promote the patient's physical comfort, hygiene, and safe medical treatment which are commonly referred to as procedures or psychomotor skills .

There are three types of skills Intellectual skills as analysis, and application, manual and psychomotor skills as lab skills , social and interpersonal skills as communication skills, others as decision making skills and planning skills.

An attitude is a hypothetical construct that represents an individual's degree of like or dislike for an item, we can evaluate attitude througth Likert-type scale, Self report published, Inventories, Semantic, Differential projective technique. Wikipedia Project, (2009)

Introduction cont Art is the process or product of deliberately arranging elements in a way that appeals to the senses or emotions. It encompasses a diverse range of human activities, creations, and modes of expression, including music and literature. The meaning of art is explored in a branch of philosophy known as Aesthetics.

Specific terminology related to evaluation:

Criteria: Statement of needs, rules, standards, or tests that must be used in evaluating a decision, idea, opportunity, program, project, etc., to form correct judgment regarding the intended goal. Criteria is plural of criterion

Evidence: A thing or things helpful in forming a conclusion or judgment:

Judgment: The act or process of judging; the formation of an opinion after consideration or deliberation.

Test: An instrument or tool for obtaining measurement or assessment e.g. an essay

Examination: A formal situation in which students undertake one or more tests under specific rules

Measurement: A quantative process involving the assigning of a number to an individual's characteristics

DefinitionsAccording to American Evaluation Association (2007),

evaluation involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness.

Evaluation: is an integral component of the teaching-learning process that should facilitate student learning and improve instruction. Teachers make judgments about student progress based on information gathered through a variety of assessment techniques.

Definitions cont This information assists teachers in planning and modifying their instructional programs, which in turn helps students learn more effectively. Evaluation is also used for reporting progress to students and parents and for making decisions related to such things as student promotion and awards.

Evaluation must be considered during the planning stage of instruction when learning objectives and appropriately related teaching strategies and methods are chosen. Too often in the past evaluation has been treated as an add-on, something to be dealt with at the end of a unit of study.

Purpose of Student EvaluationIncentive to learn Feedback to student Modification of learning activities Selection of student Success or failure Feedback to teacher School public relations Information for selection and

certification

Types of EvaluationThere are many different types of evaluations

depending on the object being evaluated and the purpose of the evaluation.

1.Formative evaluating 2. Summative evaluation

Formative evaluations is an on-going process, an integral part of the learning process that keeps students and teachers informed of student progress towards program learning objectives. The main purpose of formative evaluation is to improve instruction and student learning ?(Bahola, 1990) .

Types of evaluation cont

Summative evaluations, occurs most often at the end of a unit of study. Its primary purpose is to determine what has been learned over a period of time, to summarize student progress, and to report on progress relative to curriculum foundational objectives to students, parents and educators. It is a judgment of the student's global competence.

Types of evaluation contFormative evaluation includes several evaluation

types: Needs assessment determines who needs the

program, how great the need is, and what might work to meet the need

Evaluability assessment determines whether an evaluation is feasible and how evaluators can help shape its usefulness

Structured conceptualization helps stakeholders define the program or technology, the target population, and the possible outcomes

Implementation evaluation monitors the fidelity of the program or technology delivery

Process evaluation investigates the process of delivering the program or technology, including alternative delivery procedures

Types of evaluation contSummative evaluation can also be subdivided: outcome evaluations investigate whether the

program or technology caused demonstrable effects on specifically defined target outcomes

impact evaluation is broader and assesses the overall or net effects -- intended or unintended -- of the program or technology as a whole

cost-effectiveness and cost-benefit analysis address questions of efficiency by standardizing outcomes in terms of their dollar costs and values

secondary analysis reexamines existing data to address new questions or use methods not previously employed

meta-analysis integrates the outcome estimates from multiple studies to arrive at an overall or summary judgment on an evaluation question

Another method to clarify types of evaluation1. Internal evaluation or self- evaluationAn evaluation carried out by members of

the organization who are associated with the programme, intervention or activity to be evaluated.

2. Ex-ante evaluation or impact assessment

An assessment which seeks to predict the likelihood of achieving the intended results of a programme or intervention or to forecast its unintended effects.

This is conducted before the programme or intervention is formally adopted or started. Common examples of ex-ante evaluation are environmental and/or feasibility studies.

3. Mid-term or interim evaluationAn evaluation conducted half-way through

the lifecycle of the programme or intervention to be evaluated. Monitoring An ongoing activity aimed at assessing whether the programme or intervention is implemented in a way that is consistent with its design and plan and is achieving its intended results.

4. Ex-post or summative evaluationAn evaluation which usually is conducted some

time after the programme or intervention has been completed or fully implemented. Generally its purpose is to study how well the intervention served its aims, and to draw lessons for similar interventions in the future.

5. Meta-evaluation Two processes are often referred to as meta-evaluation: (1) the assessment by a third evaluator of evaluation reports prepared by other evaluators; and (2) the assessment of the performance of systems and processes of evaluation.

6. Formative evaluation An evaluation which is designed to provide some early insights into a programme or intervention to inform management and staff about the components that are working and those that need to be changed in order to achieve the intended objectives. (Information Project, 2006)

Anther methods to clarify types of evaluation 1. Outcome evaluation: An evaluation

which is focused on the change brought about by the programme or intervention to be evaluated or its results regarding the intended beneficiaries.

2. Impact evaluation: An evaluation that focuses on the broad, longer-term impact or effects, whether intended or unintended, of a programme or intervention. It is usually done some time after the programme or intervention has been completed.

3. Performance evaluation: An analysis undertaken at a given point in time to compare actual performance with that planned in terms of both resource utilization and achievement of objectives. This is generally used to redirect efforts and resources and to redesign structures.

Elements Needed for the Construction of an Evaluation System

Planning the evaluation of situation analysis and the identification of priority health problems (context )

Evaluation of the context is concerned with importance of educational programme.

Planning the evaluation of the human and material resources to be used and the elements which included in the programme ( input)

Elements Needed for the Construction of an Evaluation System

It is important to make sure that teachers are competent and comfortable with the teaching methodology to be used or not .

Planning the monitoring of implementation( the educational process)

An evaluation system must be plan how the implementation of the programme is to be monitored.

Planning the evaluation of the learners (the output)

The central component of an evaluation of the learners performance .

Elements Needed for the Construction of an Evaluation System

Standards of EvaluationThe Joint Committee on Standards for

Educational Evaluation (2009) has developed standards for educational programmes, personnel, and student evaluation. The Joint Committee standards are broken into four sections:

Utility : The quality or state of being usefulFeasibility : Capable of being accomplished or

brought about; possiblePropriety : the quality or state of being

appropriate or fittingAccuracy: The ability of a measurement to match

the actual value of the quantity being measured.

The Community Tool Box, (2009)

Steps of Student Evaluation

.

Put the criteria of the educational objectives

Develop and use measuring instrument

Interpret measurement data

Formulate judgments and take appropriate action

Steps of evaluationGet an overview of the programDetermine why you evaluatingDetermine what you need to know and

formulate research questionFigure out what information you need to

answer questionDesign the evaluation Collect the information / dataAnalyze the informationFormulate conclusionsCommunicate results Use results to modify the program.

Phases of evaluationThe evaluation process is cyclical in nature. Each phase is linked to and dependent on the others. It is a continuous process that takes careful planning and systematic implementation not to mention constant review and modification to guide student learning.

Phases of evaluation cont.

In the preparation phase, decisions are made which identify what is to be evaluated, the type of evaluation to be used, the criteria against which student learning outcomes will be judged, and the most appropriate assessment strategies with which to gather information on student progress.

The assessment phase is action-oriented. The teacher identifies appropriate information-gathering strategies, constructs or selects assessment techniques in collaboration with the student, continues to make decisions such as identification and elimination of bias from assessment instruments, and determines where, when and how assessments will be conducted. The teacher collects, organizes and interprets the student information gathered.

During the evaluation phase, the teacher examines the collected student information carefully taking into consideration pertinent points such as the student's particular situation, the curriculum, the time of the year, the variety of resources, etc. to make a judgment on the progress of the student or the level of achievement of student leanings.

The reflection phase allows the teacher to consider possible actions and to make decisions necessary to carry out improvements or modifications to subsequent teaching and evaluation.

Guiding Principles for evaluatorBe selective!Be realistic!Be creative!Be careful! Be balanced! Be holistic! Be human! (Course evaluation

methods, 1999)

Methods of EvaluationTo select most appropriate methods for

evaluation , must answer the following questions:

What types of data will be collected ?From whom or what will data be

collected?How/ when, and where will data be

collected ?By whom will data be collected ?

Qualities of effective tool for

evaluation Validity,This is the most important aspect of a test

and is the extent to which the test measures what it is designed to measure.

Reliability, it defined as the extent to which a set of variables consistent with what it aims to measure

ObjectivityThe state of being objective, just, unbiased and not influenced by emotions or personal prejudices

PracticabilityIt means applicability of a test to be applied; it implies factors as the time taken to conduct the test, the cost of using it and the practicality for everyday use .

Qualities of effective tool for evaluation

Equilibrium : achievement of the correct proportion among questions allocated to each of the objectives.

Equity : extent to which the question set in the examination correspond to the teaching content .

Specificity : quality of a measuring instrument whereby an intelligent student who has not followed the teaching on the basis of which the instrument has been constructed will obtain in result equivalent to that expected by pure chance.

Discrimination : quality of each element of a measuring instrument which makes it possible to distinguish between good and poor students in relation to a given variable.

Efficiency : quality of measuring instrument which insure the greatest possible number of independent answers per unit of time

Time : it is well known that a measuring instrument will be less reliable if it leads to the introduction of relevant factors because the time allowed is too short

Evaluation of knowledge:

The cognitive domain in Bloom's Taxonomy, 2007 comprised knowledge, comprehension, application, analysis synthesis and evaluation. Each of these levels demands a different form of assessment. There are varieties of objectives tests of knowledge that can be used to evaluate the different level of cognitive domain.

Methods of evaluate knowledgeInterview Questionnaire Discussion Survey Analysis of records and dataFocus group Case studyDocumentation review

Evaluation of skillsThere are three types of skills Intellectual skills as analysis, and application)

Manual and psychomotor skills as lab skills

Social and interpersonal skills as communication skills,others as decision making skills and planning skills

Methods of evaluation of skills through:

Rating scales Checklist methodsOSCE

Rating scalesIt is a method to evaluate clinical

performance .Rating scales can be either narrative or numerical in their format:

Narrative (descriptive ) rating scale

Numerical rating scale

Examples of narrative rating scale

Example on rating scale Numerical Pain Scales

OSCE (obstructive structure clinical evaluation)

The candidates rotate through a series of The candidates rotate through a series of stations at which they are asked to carry out a stations at which they are asked to carry out a (usually clinical task)(usually clinical task)

Background about OSCEBackground about OSCEStarted in 1972 by R. Harden and F.

GleesonFirst literature about OSCE 1975Used in undergraduate as well as

postgraduateFormative & summativeUsed in many disciplines

Characteristics of the Characteristics of the OSCEOSCE

It is an assessment approach primarily used to measure clinical competence

Should be planned or structured (predetermined clinical competences)

Examination format or frameworkDifferent types of test method can be

incorporated into itIn most stations students are observed (by one

or more examiners)Scored as they carry out the task or interpret

clinical materials (e.g.laboratory data, X-rays), write notes or answer question

Simulated Patient (examination)

Advantages of the OSCEValid examinationThe examiners can control the

complexities of the examinationUsed as summative as well formativeCan be used with larger number of

studentsReproducibleThe variable of the examiner and the

patient are to a large extent removedpromotes team work

Disadvantages of the OSCEKnowledge and skills are tested in

compartments The OSCE may be demanding for both

examiners and patientsMore time in setting it upShortage of examinersMight be quite distressing to the student

Evaluation of attitudesAttitudes are generally positive or negative views

of a person, place, thing, or event.Components of attitude: Cognitive component refers to ideas that

express relationship between situations as people watch too much TV

Affective component refers to the emotional and feeling accompanied by this idea as feeling of frustration so I watch TV

Behavioral component refers to act towards an object or situation in consistent way

Methods to evaluate attitude

Likert-type scale Self report published Inventories Semantic Differential projective technique

1-Likert-type scale It is the most popular methods of

measuring attitudes by summated ratings. The scale consists of a series of

declarative statements, The subject is asked to indicate whether he agrees or disagrees with each statement. Commonly, five options are provided: "strongly agree," "agree," "undecided," "disagree," and "strongly disagree."

Examples of likert -scaled questions1. How effective was this course in meeting the

objectives that were set? Very effective – effective – neither – ineffective – very

ineffective2. How would you rate the facilities and

resources for learning? Very good – Good – Neither – Poor – Very poor

2-Self report published (Participant Self Evaluation Attitude

Questionnaire)School : I happy at school I get a long with the teachers I get into trouble a lot at school I find school word hard I feel angry at school Future :I am excited about my future People have confidence in me I would love to travel I know I will succeed at school

3-InventoryThe process of making such a list, report, or record.

The items listed in such a report or recordAny list of articles or goods. The law requires that an inventory shall be attached to certain special documents, and in other cases an inventory is found advantageous for reference or comparison

4-Differential projective techniques

A type of psychological test that assesses a person's thinking patterns, observational ability, feelings, and attitudes on the basis of responses to ambiguous test materials. It is not intended to diagnose psychiatric disorders

Models of evaluationDonald Kirkpatrick's , 2008 state 4 stages of

evaluation models (Reaction - Learning - Behavior -Results)

1. Reaction modelThis level measures how those who

participate in the event, react to it, the learner's perception (reaction) to the event.

This level is often measured with attitude questionnaires (smile sheets) that are passed out after most training sessions

2-Learning modelMeans the extent to which participants change

attitudes, improve knowledge, and increase skill as a result of attending the programme

The learning evaluation requires pre testing and post-testing

Evaluating the learning that has taken place is typically focused on such questions as:

1.What knowledge was acquired? 2. What skills were developed or enhanced? 3. What attitudes were changed?

3-Behavior modelThis model involves testing the participant's

capabilities to perform learned skills after the event

Evaluations can be performed formally (testing) or informally (observation).

Behavior data provides insight into the transfer of learning from the workshop to the work environment and the barriers encountered when attempting to implement the new techniques learnt in the programme.

4-Result modelIt measures the training effectiveness,

"what impact has the training achieved?. It is concerned with the impact of the

programme on the wider community Immediate end of course evaluation sheet

largely to measure Reaction Follow up questionnaire sent 6 weeks after

the event concentrated on what participants felt they had learned from the event in terms of increased awareness, development of skills and acquisition of knowledge

Models of Evaluation

First-generation models Second-generation models Third-generation models Fourth-generation models

First-generation models

First-generation models are measurement-oriented models where the evaluation is technical and based on objectively measurable data, e.g., number of students passing registration examinations or mean scores of students from a particular programme. First

generationmodels use measurement (usually testing of individual students) as the means of evaluation and consequently usually do not contribute to programme improvement.Furthermore, results of a first generation evaluation

do not necessarily reflect the quality or appropriateness of a curriculum (Sarnecky, 1990b).

Second-generation model

Second-generation models build on first-generation models and are characterized by their emphasis on description, specifically in relation to programme objectives. These model are a direct response to the popularity of management by objectives.

In second generation models, evaluation is based on describing how well objectives of a programme (or programme outcomes) are met. This usually includes measurement of various aspects of a programme (Sarnecky, 1990a).

Third-generation models

Third-generation models (characterized by the use of high technology) focus on using evaluation as a basis for judgment. Although most evaluations result in some kind of judgement, according to Sarneky (1990a) third-generation models make use of a wider variety of measures and description than first and second-generation models and the focus of analysis of the data is on making judgments and planning interventions. Variations or combinations of the first three-generation

models were traditionally used in evaluation

Fourth-generation modelsThis model is characterized by collaboration,

participation and self-determination and is designed to assist programme stakeholders and to result in empowerment through self-evaluation and reflection.

Use for the purpose of accreditation or professional approval. In spite of the availability of useful third-generation models, it was felt that use of more qualitative, reflective methods was needed to fully understand the quality of educational programmes. The fourth generation models were a response to this perception.