Chapter 7 Developing Assessment Instruments

Carolyn Jenkins-Haigler

Background

An “assessment” is not just a test, but a range of items used to gauge a learner’s abilities and progress as well as quality of instruction in addition to evaluating the instructional mediums used. Criterion (or the plural version of criteria) is defined as: Criterion. A standard on which a judgment or decision may be based learners will a characterizing mark or trait answer 85% of quiz Criterion-referenced assessments, also known as objective questions reference assessments, focus on measuring performance items correctly to associated with learner performance and instructional integrity. demonstrate mastery of Benchmarks, or specific levels of reference, are used to gauge the lesson change in performance, attitudes, and other measurable items. It is important to promote learners to evaluate themselves – quality of work and performance “Self-evaluation and self-refinement are two of the main goals of all instruction since they can lead to independent learning.”

Objectives

Describe the purpose for criterion-referenced tests.

Describe how entry behaviors tests, pretests, practice tests, and posttest are used by instructional designers.

Name four categories of criteria for developing criterion-referenced tests and list several considerations within each criterion category.

Given a variety of objectives, write criterion-referenced, objective-style test items that meet quality criteria in all four categories.

Develop instructions for product development, live performance, and attitude assessments, and develop a rubric for evaluating learners’ work.

Evaluate instructional goals, subordinate skills, learner and context analyses, performance objectives, and criterion-referenced test items for congruence.

Types of Criterion-Referenced Tests

Entry Skills Test

Presented to learners before instruction is

provided

Assess prerequisite

skills

Learners may have harder time learning

material if skills are lacking

Pretest

Presented to learners before instruction is

provided

Gauge learner mastery of

material and help instructor

to cater course

material to meet needs of most learner

Practice Test

Presented during the

instructional

Used to facilitate learner

participation during

instruction

Help to gauge learning and

understanding

Post Test

Presented following the instructional

Used to assess all objectives

and skills from the

instructional

Help to evaluate

instructional effectiveness and learner knowledge

Designing a Test

Finding the best testing format to measure different areas/types of learning

Verbal Information Domain ◦ HOW - Learners demonstrate understanding by remembering or not remembering

information◦ WHAT – Direct objective-style test items such as short-answer, alternative response,

matching, and multiple-choice itemsIntellectual Skills Domain

◦ HOW – More difficult and complex to create – gauging knowledge of multi-faceted concepts◦ WHAT – Objective-style test items, creation of a project/product, or

performance/presentation Attitudinal Domain

◦ HOW – Also more difficult and complex instruments to create – focusing on learner preferences and attitudes

◦ WHAT – Direct learner statement, observation, or similar inferencePsychomotor Domain

◦ HOW – Typically require the learner to demonstrate steps to show understanding of a universal concept

◦ WHAT – Rubric, checklist, rating scale, or direct demonstrations

Determining Mastery Levels

In order for learners to “master”, they must achieve a certain level of performance.

Mastery is equivalent to the level of performance normally expected from the best learners.

Statistical

Sufficient opportunities should be provided to perform the skill so that it is nearly impossible for correct performance to be result of chance alone.

As a general principle, mastery level for any performance should be considered with respect to both evaluating the performance at the point in time and enhancing the learning of subsequent, related skills in the course.

The best definition of mastery is the level requires in order to be successful on the job.

Writing Assessment ItemYou should write an assessment item for each objective whose accomplishment you want to measure. Steps to follow when writing a criterion assessment item:

◦ Read the objective and determine what it wants someone to be able to do (i.e., identify the performance).

◦ Draft a test item that asks students to exhibit that performance.◦ Read the objective again and note the conditions under which the

performing should occur (i.e., tools and equipment provided, people present, key environmental conditions).

◦ Write those conditions into your item.◦ For conditions you cannot provide, describe approximations that are as

close to the objective as you can imagine.◦ If you feel you must have more than one item to test an objective, it

should be because (a) the range of possible conditions is so great that one performance won’t tell you that the student can perform under the entire range of conditions, or (b) the performance could be correct by chance. Be sure that each item calls for the performance stated in the objective, under the conditions called for.

If you follow these steps and still find yourself having trouble drafting an assessment item, it is almost always be because the objective isn’t clear enough to provide the necessary guidance.

Types of Test Item Criterion

Goal- Centered

Focusing on the objectives

of the instructional

Learner-Centered

Focusing on the differentiated

needs of learners

Context-Centered

Focusing on the environment in which learning occurs ass well as where direct

application (i.e.

performance) ultimately

occurs

Assessment-Centered

Focusing on all aspects of the assessment design and

creation

Setting Mastery Criteria What is the proper number of items needed to determine mastery of an objective?

How many items must leaner answer correctly to be judged successful on a particular objective?

Types of Items You should select the type of item that gives learners the best opportunity to demonstrate the performance specified in your objective. Possible test items include:• Essay• Fill-in-the-blank• Completion• Multiple-choice• Matching• Product checklist• Live performance checklist

Test Item Format To select the best type of item from among those that are adequate, consider such factors as the response time required by learner, the scoring time required to analyze and judge the answers, the testing environment, and the probability of guessing the correct answer.

Writing Directions

The title suggests the content to be covered.

A brief statement explains the objectives or performance to be demonstrated and the amount of credit that will be given for a practically correct answer.

Learners are told whether they should guess if they are unsure of the answer.

Instructions specify whether words must be spelled correctly to receive full credit.

Learners are told whether they should use their names or simply identify themselves as members of a group.

Time limits, words limit, or space limits are spelled out. Directions for performance and products should clearly describe what is expected of the learners. You should include any special conditions and decide on the amount of guidance you will provide during the assessment.

Evaluating Tests and Test Items

The designer should ensure the following:◦ Test directions◦ Each test item◦ Conditions◦ The response methods◦ Appropriate space, time, and equipment

After writing the test, the designer should administer it to a student or individual who will read and explain aloud what is meant by both the directions and questions and respond to each question in the intended response format.

The designer should keep in mind that tests measure the adequacy of (1) the test itself

(2) the response form

(3) the instructional materials

(4) the instructional environment and situation

(5) The achievement of learners

Developing the Instrument

When assessing performances, products, or attitudes you will need to create an assessment instrument to help you evaluate the performance, product, or attitude. Dick and Carey offer five steps to creating this instrument:

Identify the elements to be evaluated

Paraphrase each element

Sequence the elements on the instrument

Select the type of judgment to be made by the evaluator

Determine how the instrument will be scored

Assessment of Performances, Products, and Attitudes

Writing directions

Developing the instrument

Identify, paraphrase, and sequence elements

Developing the response format:Checklist

Rating Scale

Frequency Count

Scoring procedure

Using Portfolio Assessments

Portfolios are collections of work that together represent learners’ achievements over an extended period of time.

This could include tests, products, performances, essays, or anything else related to the goals of the portfolio. They allow you to assess learners’ work as well as their growth during the process. As with all other forms of assessment, whatever is included in the portfolio must be related to specific goals and objectives. The choice of what to include can be decide on entirely by the teacher, or in cooperation with students.

Assessment of each portfolio component is done as it is completed, and the overall assessment of the portfolio is carried out at the end of the process using rubrics. In addition, learners are given the opportunity to assess their own work by reflecting on the strengths and weaknesses of various components.

Portfolios can also be used as part of the evaluation process to determine what students did and did not learn, and then that information can be used to strengthen the instruction.

Evaluating Congruence in the Design Process

One of the most crucial aspects of the assessment phase of the design process is to be able to evaluate the congruence of the assessment against the objectives and analyses that have been performed.

Remember that this is a systematic approach to instructional design, which means that every step in the process influences subsequent steps. As such, all of your skills, objectives, and assessment items should be parallel.

One way to clearly represent this relationship is to create a three-column table that lists each of the skills from your instructional analysis, the accompanying objective, and the resulting assessment item. At the bottom of the table you would finish up with your main instructional goal, the terminal objective, and the test item for the terminal objective.

Design Evaluation ChartSkill Objective Assessment Item (s)

1 Objective 1 Test Item

2 Objective 2 Test Item

3 Objective 3 Test Item

Instructional Goal Terminal Objective

Test Item

Reference

Dick, W., Carey, L., & Carey, J. O. (2009). The systematic design of instruction. (7th ed., pp. 130-163). Upper Saddle River, NJ: Pearson.

Summary

A criterion-referenced assessment is composed of items or performance tasks that directly measure skills described in one or more behavioral objectives. Leaner-centered assessments are to be criterion-referenced. This type of testing of test is important for evaluating both learners; a process and instructional quality.

Assessment Self-Reflection

Think about how you have used assessments in the past to either gauge your personal knowledge and skill levels or to analyze the aptitudes of your learners.

Do you feel the assessments helped you to gain a better understanding of your knowledge and that of your learners? If so, why, and if not, why not?

THE ENDCAROLYN JENKINS-HAIGLER