Expert Appraisal

Types of Expert Appraisal

• Some modifications of the instructional material are based on the developer’s hindsight;

• Most should be based on feedback from others. . .

• This process is called “formative evaluation”. . .

• That is, evaluation undertaken for the improvement of the instruction material during its development

• Expert appraisal is one stage of formative evaluation.

• Based on this feedback the material is modified to improve its – appropriateness, – effectiveness, – usability, and – technical quality

feasibility

effectiveness

appropriateness

Instructional Review

language

format

media

Technical Review

Expert Appraisal

Guidelines for conducting expert appraisals

• Use a varity of specialists• Restrict each expert to his/her own field• Present only relevant parts of the instructional

material for evaluation by each expert• Specify the purpose of the review by each expert

by defining the context in which the material is to be used and provide a checklist or questionnaire

Technical review

• Language– To check the appropriateness of the language

for the target trainees and the correctness and clarity of it usage

– See example

• Media– To check the technical quality of the finished

product and the compatibility of the materials and equipment

– Reviewer should be knowledgeable in the specific medium used--and suggest emphasis upon ways of improving the technical quality.

• Format review– Each instruction format has its own standards and

criteria, for example:• Programmed instruction

– Step size, error rate, etc.

• Simulation games– Fidelity, replicability, and play time/prep time ratio

• Given definition of the context and the purpose of the review--appraisal exper should be free to use own procedure for the evaluation

Instructional review

• Appropriatness– The extent to which the objectives and the

content of the instructional material are constant with the goals of the training

– See example

• Effectiveness– The extent to which instructional objectives are

attained by trainees

– See example

• Feasibility– The extent to which the material is usable for

training

– See example

From appraisal to revision• The developer’s own hindsight on what might

have done differently should not be discounted.

• Not all suggestions from all sources are worthy of being acted upon.

• Contradictory suggestion from different experts are often encounter. (“more precise terminology” vs “too much jargon”)

• Expert appraisal does not reduce the number of decisions necessary;

• It merely presents different points of view• Decisions on which suggestions to act on must be

made and then translated into actual modification of the material.

• When a drastic change is proposed--it is a good idea to check back w/major appraisal experts involved in the original evaluation.