#pearsoncite

Measuring Mastery: Best Practices for Assessment in Competency-Based Education

#pearsoncite

Barbara E. Rowan, Ph.D.

Director of Academic Research, Efficacy, & Evaluation

Course Design, Development, and Academic Research, Higher Education Services

Pearson North America

barbara.rowan@pearson.com

#pearsoncite

Agenda

• Brief introduction to today’s student

• Explanation of Competency-based Education

• Definition of competencies

• Measuring mastery of competencies through assessment

@BarbaraRowan

#pearsoncite

Typical College Student

• Not what we think – no longer 18-22

• 38% are over 25 years of age

• 58% work while enrolled in college

• 26% are raising children

• 47% support themselves financiallySource: Lumina Foundation

@BarbaraRowan

#pearsoncite

Competency-based Education

• An answer to:– Stagnant graduation rates

– Increasing prices

– On-demand education

– Need for asynchronous education

@BarbaraRowan

#pearsoncite

What is Competency-based Education?

A change from knowledge acquisition to knowledge application

A change from a single subjective or objective measure to multiple objective and subjective measures

A change from proxy assessment to authentic assessment

A change from norm referenced evaluation to criterion-referenced evaluation

@BarbaraRowan

#pearsoncite

Set term length

Credit Hours

Grades

Traditional

No set term

Mastery of Competencies

Real-world Application

CBE

@BarbaraRowan

#pearsoncite

Department of Education

• March 2013

• Students could receive financial aid if participating in an approved CBE program– Credit hour equivalency

– Direct assessment

@BarbaraRowan

#pearsoncite

Credit Hour Equivalency

• Competencies are course-based, and mapped back to credit hours

• Credit-hour equivalence is used to award student financial aid

@BarbaraRowan

#pearsoncite

Direct Assessment

• Uses a direct measure of student learning

• Disregards credit hours

• DoE has tentatively granted approval of these CBE models

@BarbaraRowan

#pearsoncite

Common Elements of CBE Programs

Competency Framework

Competency Assessments

Demonstration of Competency

@BarbaraRowan

#pearsoncite

Carraccio,Wolfsthal, Englander, Ferentz, & Martin, 2002, p. 362

Competency

Knowledge

Skills

Attitudes

Competence

@BarbaraRowan

#pearsoncite

Competencies

Clearly Defined Measurable

Related to the Knowledge/Skills

Needed for Future Endeavors

@BarbaraRowan

#pearsoncite

Assessment

• Driving force behind a strong CBE program

• Two-step Process– Assessment development and score validation

– Determining cut scores

#pearsoncite

Assessment Development

StandardsPsychometric

EvaluationValidity

#pearsoncite

Assessment Development

Standards

#pearsoncite

Assessment Best Practices

• Each competency must be measured more than one time

• Each competency must be measured more than one way

• Assessments should focus on clusters of competencies as they might appear in real world

#pearsoncite

Assessment Types

Performance-based Assessment (real-world

focus)

Objective Measures of

Competencies

Diagnostic Tests of Knowledge

@BarbaraRowan

#pearsoncite

Assessments

• Assessment maps to competencies

• Assessment measures a student’s knowledge and skills AND how well the student can integrate, synthesize, and use the knowledge and skills necessary to become a community of practice

#pearsoncite

Assessments

The more “real world” the assessment is, the more value the results hold for determining a student’s mastery of the competencies.

#pearsoncite

Assessment Development

@BarbaraRowan

StandardsPsychometric

Evaluation

#pearsoncite

Psychometric Evaluation

• Reliability – indicates consistency of test scores

• Item Analysis – indicates items that are performing poorly

• Rubric Evaluation – to ensure that the rubric is properly evaluating subjective measure

@BarbaraRowan

#pearsoncite

Assessment Development

@BarbaraRowan

StandardsPsychometric

EvaluationValidity

#pearsoncite

Score Validation

Test Content

Relations to Other

Variables

Response Processes

Test Consequences

Internal Test Structure

@BarbaraRowan

#pearsoncite

Framework for Determining Mastery

• Must determine what separates masters from non-masters

• Validity evidence required to support decisions

@BarbaraRowan

#pearsoncite

Cut Scores – Questions to Ask

What do Students Need to Know and Be Able to Do at Each Level?

What Performance Levels will be

Reported?

Are Cut Scores Necessary?

@BarbaraRowan

#pearsoncite

Cut Score Options

@BarbaraRowan

#pearsoncite

Five Basic Steps

Select Judges

Define borderline knowledge and skills

Train the judges

Collect judgments

Choose a cut score

by combining judgments

@BarbaraRowan

#pearsoncite

Choosing a Method

• No one best method

• Choice should depend on what kind of judgments can be gotten and believed

• Based on types of questions in the test (multiple choice; constructed-response; rubric-based

@BarbaraRowan

#pearsoncite

NAEP Suggested Levels

Basic Level

• Partial Mastery of knowledge and skills

Proficient Level

• Solid Academic Performance and Demonstrated Competence over Challenging Subject Matter

Advanced Level

• Signifies Superior Performance on Challenging Subject Matter

@BarbaraRowan

#pearsoncite

Cut Score Process

@BarbaraRowan

Set Provisional Cut Scores

Establish Operational Cut Scores

Document the ProcessEvaluate the Results of

Using the Cut Scores

#pearsoncite

Provisional Cut Scores

• Determined by people who are aware of what students should know and be able to do

• Determine how many score points students have to earn to demonstrate they have reached the level of knowledge and skill for each level

@BarbaraRowan

#pearsoncite

Provisional Cut Scores

• Choose Cut Score Method

Step 1

• Select Judges, that is, 5-15 subject matter experts

Step 2• Train Judges in

the Cut Score Method Chosen

Step 3

@BarbaraRowan

#pearsoncite

Operational Cut Scores

@BarbaraRowan

Is it better to pass a student who should fail?

Or fail a student who should pass?

#pearsoncite

Operational Cut Scores

Test Score Fluctuations

Pass

Fail

@BarbaraRowan

#pearsoncite

Document the Process

@BarbaraRowanRationale

Qualifications

Training

Steps

Data shared with Judges

Judges’ Ratings

Evaluation Forms from

Judges

Rationale for Adjustments

#pearsoncite

Evaluating the Results of Using Cut Scores

• Occurs after at least one round of testing and reporting of results has occurred

• Requires the validation of cut scores

• Ongoing process

@BarbaraRowan

#pearsoncite

Validity Study

• Are the results reasonable?

• Are failure rates far higher or lower than expected?

• Are outstanding students failing to reach the highest level?

• Are poorly prepared students reaching the highest level?

@BarbaraRowan

#pearsoncite

Questions?

@BarbaraRowan