criterion-ref-assessment.pdf

1

Criterion-Referenced Assessment

Prepared by: Dr Rob Connoley Educational Developer/Lecturer Teaching & Learning Faculty of Business & Law Deakin University Version 1 - November 2004

2

Introduction The main purpose of this paper is to explain the concept of criterion-referenced assessment and how it differs from norm-referenced assessment. Criterion-referenced assessment is increasingly employed in education practice, particularly in higher education. This is because of its focus on measuring achievement against learning objectives and on assessing changes in performance as a consequence of the student's learning experience. The following describes the concept of criterion-referenced assessment, how it differs from norm-referenced assessment and suggests how you can use rubrics to assist in assessing criterion-referenced assessment activities. You may recall that the previous two papers dealt with the issues of writing learning objectives and formulating learning and assessment activities to meet your learning objectives. Thus, this current paper can be seen as a logical extension to those two papers.

Learning objectives After completing this training package, you should be able to:

• Define criterion-referenced assessment. • Distinguish between criterion-referenced assessment and norm-referenced

assessment. • Explain how to prepare criterion-referenced assessment tasks. • Describe how a rubric can be developed to present assessment criteria and

performance standards to students.

The increasing attention on assessment practices The focus on assessment practices in higher education has grown over the past few years. For instance, in a 2002 study of assessment in Australian universities prepared by the Centre for the Study of Higher Education (CSHE), assessment was considered a central element in the overall quality of teaching and learning. A brief review of some university teaching and learning sites has also revealed most have publications assisting teaching staff on assessment practices. In addition, the Federal Government has established The Carrick Institute for Learning and Teaching in Higher Education for the enhancement of learning and teaching in Australian higher education providers. One of the Institute’s responsibilities will be to improve assessment practices throughout the sector. Furthermore, the University College Dublin reported that there have been major changes internationally in assessment practices over the past 20 years. Some of the major changes noted included the replacement of written exams with more continuous assessment and coursework; a move towards more student involvement and choice in assessment; more explicit course outlines about the expectations in assessment; more frequent use of group assessment; an understanding of process is now seen as, at least, equally important to a knowledge of facts; and a greater focus on what the student will learn rather than what the teacher plans to teach.

3

Types of assessment: Norm-referenced assessment and Criterion referenced assessment Both Norm and Criterion-Referenced assessment refer to the 'type' of assessment. Norm-referenced assessment has been the most common type of assessment used in higher education. A norm-referenced assessment makes judgements about people, expresses students’ scores in rank order, based on a distribution of scores. For instance, the top 10 per cent of the class are awarded high distinction, the next 15 per cent distinction etc. Thus, it relates the achievement of a particular student to the achievements of other students after the teaching is over. Although norm-referenced assessment may be more common in higher education, there is a growing emphasis toward the implementation of criterion-referenced assessment. Criterion-referenced assessment makes judgements about performance, rather than on people. It assessed the extent to which a student has achieved the intended learning objectives and performance outcomes of a subject. Thus, student performance is compared to a previously specified standard of achievement (criteria) and not relative to other students. Under this type of assessment, teachers and students can see where students are succeeding and where they are not. The criteria are usually set before teaching has taken place. It may be necessary to adjust if set inappropriately high or low before reporting final grades (Biggs 2003, p.145).

How to apply criterion-referenced assessment The logic of criterion-referenced assessment is say what you want students to be able to do (see learning objectives), teach them to do it (through lectures, tutorials, and learning activities), and then see if they can do it (Biggs 2003, p.144). Thus, it is about alignment. For criterion-referenced assessment to work, it is necessary to be clear about what your students should be learning in terms of qualities or performance criteria that define the grading categories and then to devise assessment tasks that will tell you how well students meet the criteria (Biggs 2003, p.145). The first task is a matter of setting learning objectives. As noted in the first training package, learning objectives are statements of what you expect your students to learn as a result of studying your unit. In other words, what should they be able to do at the end that they could not do before they commenced the unit. It is a focus on student performance not on how you teach! The use of action verb within Bloom’s taxonomy that has been used to write a learning objective statement inherently establishes the level of performance you are seeking in respect of that objective. Criterion referenced assessment is carried out against previously specified yardsticks ('criteria'). Where a grade is assigned, it is assigned on the basis of the performance standard the student has achieved on each of the criteria. Criteria are the dimensions along which you judge how well a student has carried out the assessment task (yardsticks). For example, an essay may be judged on the following criteria: knowledge of subject material (*), the appropriateness of the literature cited, the quality of the argument, and the quality of expression.

4

Oral presentations may be judged on the following criteria: knowledge of subject material, ability to answer or respond to questions, structure of presentation, use of audio/visual material, pace and timing, and delivery style. (*)The subject material criteria need to be developed from the learning objective. For example: if the essay was based on the learning objective: the student is able to explain the role of the Reserve Bank, then the criteria for assessing a student’s achievement may include: The student’s ability to state the roles accurately. The student’s ability to state the roles in their own words. The student’s ability to apply role to an example. Performance standards are the actual levels which might be achieved by students against the criteria (measurements along the yardsticks). For example, for each of the criteria noted above a range of standards could be developed ranging from poor to excellent. In the above example (*) the standards for each criterion could be: one, two or three roles stated accurately; one, two or three roles paraphrased accurately; and one, two, or three roles illustrated by an example. To help develop performance standards, you will need to write down some ideas as to what is the minimum level of performance to be met. You should then describe the sort of performance that you could best hope to expect in this level unit and for these students. Appendix 1 provides a sample of this for developing performance standards for essays. Having defined these limits, you should then fill in the remaining categories in between. This could then be presented in a rubric (see next section). The University of Queensland’s Teaching and Educational Development Institute has a paper on ‘Assessment criteria, standards and marking criteria’ at http://www.tedi.uq.edu.au/Teaching/assessment/criteria.html. This paper is a good source of reference discussing criteria and standards. You may also wish to examine the following site, which has generic descriptors whenever one is interested in checking the quality of reasoning about a problem, issue, or situation: http://www.criticalthinking.org/university/unistan.html. In addition, a further site provides a suggestion of criteria as they might be applied to essays, whether submitted in tutorials or completed as part of an examination. To obtain a particular class of assessment an essay does not have to fulfil all the criteria listed for that class - for example a beautifully written essay can still fail for being irrelevant - but the list helps to show what is looked for at university level: see: http://www.shef.ac.uk/geography/ug/handbook/assess/marking.html. You should provide students with an explanation of the criteria which are to be used in judging their attempts and the relative importance of the various criteria. You should also provide an indication of the expected standards of performance and grading scale.

5

How to present assessment criteria A rubric, or scoring matrix, is a very useful device for presenting criterion-referenced assessment tasks. A rubric is designed to make grading easier and faster, as well as to provide more useful feedback to students. Especially when grading projects, rubrics offer an easy, fast, and consistent method for grading.

What is a rubric? A rubric is a printed set of scoring guidelines (criteria) for evaluating work (a performance or a product) and for giving feedback. It is a scoring guide or a set of characteristics that are used to judge student performance. A rubric is usually handed out before the assignment begins in order to get students to think about the criteria on which their work will be judged. It is suggested that rubrics improve students' end products and therefore increase learning. When teachers evaluate papers or projects, they know implicitly what makes a good final product and why. When students receive rubrics beforehand, they understand how they will be evaluated and can prepare accordingly. Sharing the rubric and going over it step-by-step will help students to understand the standards by which their work will be judged.

The advantages of using rubrics The following are some advantages to using rubrics:

• Teachers can increase the quality of their direct instruction by providing focus, emphasis, and attention to particular details as a model for students.

• Students have explicit guidelines regarding teacher expectations. • Students can use rubrics as a tool to develop their abilities. • Teachers can reuse rubrics for various activities. An established rubric can be

used or slightly modified and applied to many activities. For example, the standards for excellence in a writing rubric remain constant throughout the year; what does change is students' competence and your teaching strategy. Because the essentials remain constant, it is not necessary to create a completely new rubric for every activity.

How to create a rubric A rubric is constructed by establishing a set of characteristics which are arranged in levels, indicating the degree to which a standard has been met. Criteria define descriptors of ability at each level of performance. Levels referred to are proficiency levels which describe a continuum from excellent to unacceptable. Learning to create rubrics is like learning anything valuable. It takes an initial time investment. Once the task becomes second nature, it actually saves time while creating higher quality student outcomes. The following questions will help you get started:

6

• Determine the concepts to be taught. What are the essential learning

objectives?

• Choose the criteria to be evaluated. Criteria are the dimensions along which you will judge how well a student has carried out the assessment task. Name the evidence to be produced and indicate relevant weighting. Weighted rubrics are useful for explicitly describing to students what concepts take priority over others.

• Develop a grid, which is the physical layout of the rubric. Plug in the concepts

and criteria.

• Share the rubric with students before they begin the assessment task.

• Evaluate the end product. Compare individual students' work with the rubric to determine whether they have mastered the content.

A rubric template is shown in appendix 2. Two samples are also provided (see appendices 3 and 4). The following site provides a means of building your rubric: http://landmark-project.com/classweb/tools/rubric_builder.php3 [Accessed 2 November 2004] The following site requires you to log on first prior to allowing you to build a rubric: http://rubistar.4teachers.org/index.php [Accessed 2 November 2004]

Summary It is important that you should aim to fully align your unit’s objectives, teaching methods, assessment procedures, and educational climate. For this paper it is important to remember the distinction between criterion-referenced assessment and norm-referenced assessment as types of assessment. In addition, and reinforcing the work of the two previous training packages, you should aim to describe all assessment tasks. This means you should:

• describe what learning objectives will be measured by the particular assessment task;

• provide students with clear instructions about the assessment tasks; • describe the conditions under which the students will take the assessment

(such as when, where and how); and • provide the scoring criteria (such as a rubric) that will be used.

Further information about how to develop criterion-referenced assessment can be obtained from the University of Queensland’s Teaching and Educational Development Unit (TEDI), which has some guidelines and templates on how to implement criterion-referenced assessment.

7

References Biggs, J. (2003). Teaching for quality learning at university : What the student does. Second Edition, Society for Research into Higher Education : Open University Press, Buckingham. Centre for the Study of Higher Education. 2002. Assessing Learning in Australian Universities: Ideas, strategies and resources for quality in student assessment, A national project by Richard James, Craig McInnis and Marcia Devlin, www.cshe.unimelb.edu.au/assessinglearning [accessed 8th October 2004] Centre for Teaching and Learning, University College Dublin, Good practices in student assessment, http://www.ucd.ie/teaching/assess/as6.htm [Accessed 20 October 2004] Department of Education, Science and Training. 2004. Fact Sheet: Promoting Excellence in Learning and Teaching, March Family Education Network. Creating Rubrics, http://www.teachervision.fen.com/lesson-plans/lesson-4521.html?detoured=1 [Accessed 2 November 2004] Illinois Online Network (ION), http://www.ion.illinois.edu/IONresources/assessment/rubric.asp [Accessed 11th October 2004] Tognolini, J. 2001. Educational Testing Centre. University of New South Wales http://tlcweb.np.edu.sg/lt/articles/assessmentTognolini.htm, [Accessed 28 October 2004] University of Queensland, Teaching and Educational Development Unit, http://www.tedi.uq.edu.au/Teaching/assessment/cra.html, [Accessed: 19 October 2004], http://www.tedi.uq.edu.au/Teaching/assessment/designing.html, [Accessed 19 October 2004]; http://www.decs.act.gov.au/bsss/critref.htm [Accessed 20 October 2004] University of Tasmania, http://www.utas.edu.au/tl/supporting/assessment/judgement.html, [Accessed 20 October 2004]

8

Assistance Assistance is available on the development of criterion-referenced assessment. My contact details are: Rob Connoley Educational Developer/Lecturer in Teaching & Learning Faculty of Business and Law Deakin University Waurn Ponds VIC 3217 Email: [email protected] Phone: X724755 Rm: ib3.105

9

Appendix 1: Criteria for marking essays and sample performance standards Criteria Sample performance standards 1. Addressing the question

The relevance of the content of the essay to the question or title set. • Good essays select relevant material (knowledge,

concepts, interpretation, theoretical models, others’ perspectives).

• Better ones make it clear throughout how the material is relevant to the question.

2. Using evidence

The use of externally sourced material, such as research findings, facts, quotations, or other forms of information. • Good essays include information from outside sources

that backs up the points made in the essay. • Better ones explicitly highlight or interpret the

evidence to support a more general claim or idea or point being made in the essay.

3. Developing argument

The construction of a coherent and convincing set of reasons for holding a particular point of view; the following of an analytical path leading from a starting point to a concluding point. • Good essays contain expressions of positions on the

issues raised by the essay. • Better ones develop arguments throughout the essay,

with each element building on the last. 4. Critical evaluation

Determining the value, significance, strengths and/or weaknesses of something (e.g., research findings, theory, methodological approach, policy, another’s argument or interpretation). • Good essays contain evaluative assertions or

descriptive points about the strengths and weaknesses of elements referred to in the essay.

• Better ones contain systematic, reasoned explanations for the evaluative points being made.

5. Structuring

The formal arrangement of the essay content into paragraphs. • Good essays have clearly recognisable introductory

and concluding paragraphs, and paragraphs in the main body of the essay each have a clear, single concept or point as their main focus.

• Better essays have a paragraph structure that supports the development of ideas within the essay, so that the structure of the essay is linked to the developing argument.

10

6. Use of language

The use of words, grammar, and punctuation to formulate an utterance appropriate to the purpose and context. • Good essays are free from errors in spelling,

punctuation and grammar, and would be acceptable pieces of writing in the wider world.

• Better essays adopt academic styles and conventions, and approximate to the appropriate academic ‘register’.

Source: http://www.assessmentplus.net/web%20version%20of%20staff%20workshop%20materials/Core%20criteria%20descriptions%20(web).doc [Accessed 2 November 2004]

11

Appendix 2: A rubric template

(Describe here the task or performance that this rubric is designed to evaluate.) Beginning

1

Developing

2

Accomplished

3

Exemplary

4

Score

Criteria (Name)

Description of identifiable performance

characteristics reflecting a beginning level of

performance.


characteristics reflecting

development and

movement toward

mastery of performance.


characteristics reflecting mastery of

performance.


characteristics reflecting the highest level

of performance.

Criteria (Name)



performance.



development and

movement toward




performance.



of performance.

Criteria (Name)



performance.



development and

movement toward




performance.



of performance.

Criteria (Name)



performance.



development and

movement



performance.



of performance.

12

toward mastery of

performance.

Criteria (Name)



performance.



development and

movement toward




performance.



of performance.

Source: Adapted from : http://edweb.sdsu.edu/triton/july/rubrics/Rubric_Template.html [Accessed 2 November 2004]

13

Appendix 3: A sample rubric - Essay

Standard/Benchmark:

Effective Communicator: Students write for a variety of audiences and purposes (i.e. narrative, persuasive, and informative forms) using well organises paragraphs with adequate supporting evidence. Performance Task: Write an essay on an assigned topic or a topic of your choice.

Performance Element

Excellent Good Average Poor Unacceptable

Idea Development (20 points)

Thesis is present and directly supported throughout essay. (18-20 points)

Thesis is present and most points supported. (16-17 points)

Thesis is present but vague; support may stray from thesis. (14-15 points)

Thesis is vague with little support or strays from thesis. (12-13 points)

Lacks thesis or organizing idea. (0-11 points)

Elaboration (50 points)

Well written, fully elaborates points addressed with clear, accurate, and detailed information to support thesis. (45 - 50 points)

Well written, most points elaborated with clear and detailed information to support thesis; may contain minor factual errors. (40 - 44 points)

Adequately written essay; some points elaborated; may contain factual errors or irrelevant information. (35 - 39 points)

Awkward writing style, points are general, factual errors present, and may stray from thesis. (30 - 34 points)

Poor writing style with little or no specific details, off topic, factual errors present. (0 - 29 points)

Organization (20 points)

Clearly organized and remains focused. (18 - 20 points)

Good organization with few statements out of place. (16 - 17 points)

Organization present but awkward; may contain several statements out of place, problems existing with introduction or conclusion; lack of topic sentences. (14 -

Organization is confusing to the reader. Organization may lack proper paragraph construction, transitions, topic sentences, organizing support, or

Little or no structure present. Essay rambles aimlessly from topic to topic or is completely off topic. (0 - 11 points)

14

15 points) relevant information. (12-13 points)

Mechanics (10 points)

Possesses no critical errors interfering with comprehension. (9 - 10 points)

Possesses 1-2 critical errors interfering with comprehension. (8 points)



Possesses 7 or more errors interfering with comprehension. (0 - 5 points)

Source: http://www.rubrics.com/rubrics/Essay.html [Accessed 2 November 2004]

15

Appendix 4: A sample rubric – interpersonal communication skills Standard/Benchmark:

Effective Communicator: Exhibits effective interpersonal communication skills. Performance Task: Conduct a discussion with someone else on a topic of interest.

Performance Element

Level 4 Level 3 Level 2 Level 1

Listening Is always attentive to others when they speak.

Is attentive to others when they speak; is occasionally distracted by others.

Is occasionally inattentive to others when they speak; distracts others' listening.

Is consistently inattentive to others when they speak for a wide variety of reasons.

Interruptions Never interrupts when others are talking.

Interrupts others with their permission; interruption helps delivery of message.

Interrupts others without their permission; interruption interferes with delivery of message and may or may not cause speaker problems.

Is consistently disruptive to others; interruptions interferes with delivery of message and clearly causes speaker problems.

Eye Contact Faces and maintains eye contact with those who are speaking.

Faces and maintain eye contact with those who are speaking; eye contact may be distracting to those who are speaking.

Faces or maintains eye contact with the speaker; occasionally loses or removes eye contact, distracting the speaker.

Does not face or maintain eye contact with the speaker; annoys speaker.

16

Information Seeking Questions

Asks questions when he or she does not understand what is being said; questions are pertinent to main ideas.

Asks questions when he or she does not understand what is being said; some questions may not be pertinent to main ideas.

Asks questions when he or she does not understand what is being said; many questions are not pertinent and do not allow the speaker to effectively communicate main ideas.

Exhibits little or no interest in information being shared; no information seeking questions are asked.

Voice Always speaks in an understandable voice, using clear tone, enunciation, and reasonable pace; message is clearly received.

Speaks in a voice which is generally understandable; sometimes tone, enunciation or pace interferes with message; adjusts delivery when listener does not understand.

Speaks in a voice which is generally understandable; sometimes tone, enunciation or pace interferes with message; needs listener requests to modify delivery of message.

Speaks in a voice which is frequently difficult to understand due to poor tone, enunciation or pace even after listener requests to repeat message.

Understandability Uses words, terms, and examples which others clearly understand.

Uses understandable words and terms, but needs examples to clarify major ideas.

Uses confusing words and terms; uses examples which, to a limited degree, help listener understand.

Uses confusing words, terms, and/or examples which listener does not understand.

Empathy Puts self in others' shoes; accepts and understands the feelings and motivations of others; takes steps to

Puts self in others' shoes; accepts others' feelings, but does not understand others' motivations; is interested in

Accept others' feelings; does not understand others' motivations and shows no or little interest in seeking to understand

Finds it difficult to accept others' feelings; does not understand others' motivations;

17

deepen understanding.

seeking deeper understanding, but does not take active steps.

others. shows no or little interest in others

Source: http://www.rubrics.com/rubrics/Interpersonal_Communication_Skills.html [Accessed 2 November 2004]

criterion-ref-assessment.pdf

Documents

Transcript of criterion-ref-assessment.pdf