This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 1

Portfolios: A Nontraditional Approach Portfolios: A Nontraditional Approach

to Assessment of Competencyto Assessment of Competency

ADEA CCI Liaisons ConferenceADEA CCI Liaisons Conference

June 23June 23--24, 200824, 2008

Chicago, ILChicago, ILCynthia C. Amyot, BSDH, Ed.D.Cynthia C. Amyot, BSDH, Ed.D.

University of MissouriUniversity of Missouri--Kansas CityKansas City

School of DentistrySchool of Dentistry

Cynthia C. GadburyCynthia C. Gadbury--AmyotAmyot

Professor and DirectorProfessor and Director

Distance Education & Faculty Development Distance Education & Faculty Development

UMKC School of Dentistry, UMKC School of Dentistry,

650 E. 25th Street, KC, MO 64108650 E. 25th Street, KC, MO 64108

Phone: 816Phone: 816--235235--20542054

Email: Email: amyotc@umkc.eduamyotc@umkc.edu

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 2

ObjectivesObjectives

• Provide rationale for the “why” behind using non-

traditional assessment measures in competency-based education.

• Compare and contrast traditional faculty-centered

assessment measures with portfolios as a student-

centered nontraditional assessment measure.

• Discuss a systematic approach (how to) for implementing portfolio assessment along with methods

for evaluation.

• Discuss barriers to implementing portfolio assessment of

student competency and strategies for overcoming potential barriers.

KnowledgeKnowledge--based Societybased Society

• The knowledge worker of the future will be that person who has the capacity to continually learn new concepts and modes of operations through

critical thinking and problem solving

• The knowledge worker will have the “tools” to make evidence-based decisions

• The knowledge to be acquired will be practical in nature and specialized in scope (Drucker, 1994)

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 3

The GOOD News!The GOOD News!

• The knowledge worker’s performance

depends on formal education and

theoretical knowledge.

– Education will become the center of the knowledge society and the school its key institution!

The BAD News!The BAD News!

504United States

505Norway, France

507Austria, Belgium, Iceland

516Sweden

522Japan

523United Kingdom

525Korea

527Ireland

528Australia

529New Zealand

534Canada

546Finland

Average reading literacy scores of 15-year-olds, by country:

2000

NCES, 2006

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 4

The BAD News!The BAD News!

483United States

515-485Iceland, Denmark, France, Sweden, Austria, Germany, Ireland, Slovak Republic, Norway, Luxembourg, Poland, Hungary, Spain

516Czech Republic

523New Zealand

524Australia

527Switzerland

529Belgium

532Canada

534Japan

538Netherlands

542Korea

544Finland

Average math literacy scores of 15-year-olds, by country:

2003

NCES, 2006

The BAD News!The BAD News!

491United States

506-495Sweden, Ireland, Hungary, Germany, Poland, Slovak Republic,

Iceland

509Belgium

511France

513Switzerland

519Canada

521New Zealand

523Czech Republic

524Netherlands

525Australia

538Korea

548Finland, Japan

Average science literacy scores of 15-year-olds, by country:

2003

NCES, 2006

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 5

• The process of increasing metacognition,

self-awareness, and responsibility for

one’s learning is largely atypical in

American education, where students

frequently do only what they need to get by and instructors coerce them with the

threat of poor grades

Burch, 1999

Bottom LineBottom Line

• Era of assessment and accountability in education

• Citizens are concerned over….– Financial well-being

– Economic security• Pressure has increased on those individuals (faculty) and

institutions (education) that might influence the outcome

• Resistance by education/educators is seen as suspect

Campbell et al, 2007; Palomba & Banta, 1999

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 6

Commission on Dental Commission on Dental

AccreditationAccreditation

• Standards for Predoctoral Dental Education

– 2-7 The dental school must define the competencies needed for graduation, which must be focused on educational outcomes.

– 2-8 The dental school must employ student evaluation methods that measure the defined competencies.

Assessment Assessment -- OperationalizedOperationalized

• Assessment– Process that focuses on student learning and involves review

– Reflects on practice in a planned and careful way

• Palomba & Banta believe overarching goal of assessment in higher education should be to enable educators to examine whether the curriculum makes sense in its entirety and whether students, as a result of their collective experiences in education, have the knowledge, skills and values that graduates should possess

Palomba & Banta, 1999

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 7

Traditional Assessment Measures In Dental Traditional Assessment Measures In Dental

EducationEducation

• GPA

• Paper and Pencil Tests

• National Dental Board Examination

• Regional Clinical Exams

CompetencyCompetency

• Competent students have been defined as

those individuals who are qualified to

begin working independently because they

can think critically and apply critical

thinking skills to today’s complex work environment.

Chambers & Glassman, 1997

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 8

So Why Go Beyond Traditional Assessment So Why Go Beyond Traditional Assessment ––

Why NonWhy Non--traditional Assessment Measures??traditional Assessment Measures??

• If the focus of an academic program is on the creation of products or performances, then assessment should draw on these natural results of the educational process (Palomba & Banta, 1999)

Assessment of CompetencyAssessment of Competency

• Palomba and Banta state that authentic (non-traditional) assessment is the process of using student activities or products, as opposed to tests or surveys, to evaluate students’knowledge, skills, and development.

• In academic programs like dentistry that develop complex, integrated skills, this form of assessment is appealing because it requires students to display their skills in a way that is more direct and thorough than that provided by traditional paper and pencil tests.

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 9

Assessment of CompetencyAssessment of Competency

• Research has suggested that the evaluation of competency is best attained through the use of authentic assessment

– Chambers suggests that portfolios are one form of authentic assessment where actual examples of student work is displayed

Chambers & Glassman, 1997Wiggins, 1993

Some Paradoxes in CompetencySome Paradoxes in Competency--

Based Education (JDE, 2008)Based Education (JDE, 2008)

• Study concluded:

– Some dental educators have adopted the learner-centered view of education required of competency-based education, while others

have adopted only some of the language while retaining a teacher- or discipline-centered view

Licari & Chambers, 2008

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 10

Some Paradoxes in CompetencySome Paradoxes in Competency--

Based Education (JDE, 2008)Based Education (JDE, 2008)

• The concept of competency-based

education understood and valued among

administrators and others responsible for

curriculum design and innovation

• Faculty members and students were less

familiar with the concept……

Think Aloud!Think Aloud!

• Rationale for non-traditional, authentic

assessment measures

– Your thoughts, questions????

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 11

Portfolios Portfolios -- OperationalizedOperationalized

• A focused purposeful collection of student work that documents evidence of traditional and nontraditional sources of student learning, progress, and achievement over time (Arter & Spandel, 1992; MacIsacc & Jackson, 1994)

• The construction of a portfolio involves gathering a body of evidence of one’s learning and competence (Lyons, 1988)

– Student-centered versus faculty-centered

Portfolio PedagogyPortfolio Pedagogy

• Portfolio pedagogy is characterized by two processes (Yancy, 1992):– Reflection

– Inquiry

• Reflective portfolio use stimulates students to develop new understanding and appreciation of their experiences, recognize links between different aspects of these experiences and formulate insights to be tested in future actions

• Constructivist Learning Theory (Fosnot, 1997)

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 12

The The ““How ToHow To”” of Portfolio of Portfolio

Assessment of Student Assessment of Student

CompetencyCompetency

ParamountParamount

• Needs a champion(s) at your institution

• Needs the support of administration

– Kneka Smith, M.P.H.Associate Dean for

Education

– John W. Thurmond, D.D.S., M.S., Director, AEGD

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 13

Reliability & ValidityObtain multiple measures of each

competency (Portfolio table)

5.

FairnessOffer choices of ways to demonstrate

competencies (Portfolio table)

4.

ReliabilityEstablish criteria standards and rubrics3.

ValidityDetermine how competencies are

assessed, what courses cover what

(Create a table)

2.

ValidityDetermine the programmatic/clinical

competencies (Determine what students

need to know by the end of the

educational program)

1.

What It ImprovesWhat To DoSteps

ProcessProcess

• Introduce portfolio project early in

curriculum

– Completion of some competencies prior to final semester when portfolio is due

• Deadlines help improve overall quality of

portfolio

• Faculty advising throughout process

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 14

Capstone Project UMKCCapstone Project UMKCDental Hygiene ProgramDental Hygiene Program

• Final grade for portfolio incorporated into

Senior Seminar course (45% of grade)

Development of PortfoliosDevelopment of Portfolios

• Students should choose work which stands as evidence of attainment of the program competencies

– Just as a curriculum vitae or resume, all claims made in the portfolio MUST be supported by evidence

• Faculty have provided a table (Portfolio Table can be found in article…) which lists required items of inclusion and several ideas for additional portfolio entries based on course and clinic assignments, projects, testing, etc.

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 15

UMKC Portfolio FormatUMKC Portfolio Format

• Notebook format

• Includes a Table of Contents

• An introduction is necessary to orient the reader

– Purpose statement

UMKC Portfolio FormatUMKC Portfolio Format

• Notebook divided into sections, one for each competency

• Each section begins with listing the specific competency and an explanation of the items included in that section

• Self-reflection follows at the end of each section discussing how the items included in that section demonstrate attainment of that particular competency – No statements should be made that aren’t supported by evidence in terms of items

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 16

Development of and Demonstration of Development of and Demonstration of

Critical Thinking and Problem Solving SkillsCritical Thinking and Problem Solving Skills

• Reflection requires critical thinking and

problem solving skills

– ADEA CCI

– IOM Report

– Etc.

ReflectionReflection

• Reflection is believed to focus on

professional attributes and offer directions

for improvement by identifying strengths

and weaknesses; Driessen et al., 2005

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 17

Reflection Reflection –– What is gainedWhat is gained

• In one study, mentors saw reflection as a method of identifying causes in order to answer the question of why things are as they are. Driessen et al., 2005

• Mentors see writing reflective reports as

fostering critical attitudes toward one’s own performance and offer directions for improvement; Driessen et al., 2005

How to Stimulate ReflectionHow to Stimulate Reflection

• Stimulate reflection with Favorable

Conditions:

– Good introduction

– Student ownership

– Clear structure

– Appropriate use Wade and Yarbrough, 1996

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 18

UMKC Portfolio FormatUMKC Portfolio Format

• A global self-reflection concludes the portfolio

– overall evaluation of what the student has learned

– how this new knowledge applies to their future goals

– evidence of growth during educational process

– Insight into life long learning

– etc.

Hallmark of CompetenceHallmark of Competence

• Chambers states that one of the

characteristics of a competent individual is

the capacity to accurately assess

competence

Licari & Chambers, 2008

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 19

BarriersBarriers

• Concerns related to validity and reliability

• Time

• Student Resistance

• Faculty Resistance

Validity and Reliability ConcernsValidity and Reliability Concerns

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 20

Grading RubricGrading Rubric

• Allows assessment to be more objective and consistent

• Clarifies assessment criteria in specific terms

• Shows the student how their work will be evaluated and what is expected

• Promotes student awareness

• Provides useful feedback regarding effectiveness of instruction

• Provides benchmarks against which to measure and document progress

4321Components

• llustrates continued development and growth

over time, i.e., ability to read, analyze and

apply scientific literature in decision making

process

• Demonstrates increased use of professional

language over time

• Illustrates heightened professionalism,

humanitarianism and ethical behavior

AllMostSomeNot at

all

1. Is the student able to illustrate growth and

professional development to the reader?

RatingCommentsDescription of Portfolio Traits

Primary Trait Analysis Primary Trait Analysis -- RubricRubric

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 21

Excellent Resource For Creating Excellent Resource For Creating

RubricsRubrics

• By Stevens and Levi

• http://www.introductiontorubrics.com/

-No primary sources listed

-No secondary sources

listed

-No recognizable format style and information partial

-Some primary sources listed

-Some secondary

sources listed

-Includes all relevant info but follows no format

-All possible primary sources listed

-All possible secondary

sources are listed

-Formatting follows assigned style

Preliminary lit search

WeakAverage Excellent

Rubric Rubric –– Another ExampleAnother Example

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 22

Performance AssessmentPerformance Assessment

• Two general conclusions about inter-rater reliability predominate– When tasks are the same for all students and

scoring procedures well defined, inter-rater reliability tends to be high (Dunbar, 1991; Gadbury-Amyot et al., 2003)

– When different students respond to different tasks, select their own task, or produce unique products, inter-rater reliability tends to be low (Koretz et al., 1994)

InterInter--rater Reliabilityrater Reliability

• Faculty accounted for very little variability or error (1.28 percent) in portfolio measurement demonstrating that faculty members were

consistent and calibrated in their use of the scoring rubric. Gadbury-Amyot, 2003

• The fact that very little error is attributed to faculty raters has been reported throughout the portfolio literature in both small- and large-scale studies. Nazier, 1997; LeMahieu, 1995

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 23

Performance Assessment Performance Assessment

• Carefully constructed scoring rubrics and

intensive training session for raters are

essential elements for producing ratings

that are reliable (Brennan, 2000)

Validity and Reliability of Portfolio Validity and Reliability of Portfolio

Assessment of CompetencyAssessment of Competency• “A single score obtained on one occasion on

one test with a single administrator, is not dependable (valid or reliable).”– This is what makes portfolio assessment of

competency such a dependable measure!! (Shavelson and Webb, 1991)

• Gadbury-Amyot et al. (2003)– Only work currently in dentistry looking at this very

important aspect• Generalizability coefficient of .81

– Analogous to reliability coefficient

• Theoretical and empirical evidence for validity using Messick’s unified framework

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 24

Predictive Validity of DH Competency Predictive Validity of DH Competency

Assessment MeasuresAssessment Measures

• Gadbury-Amyot et al. (2005)– Predictive validity of dental hygiene competency

assessment measures on one-shot clinical licensure examinations (CRDTS)

• Explore the ability of four variables (GPA, Clinic GPA, NBDHE, Portfolios) to predict one-shot clinical licensure exam scores.

• Results

– Two-factor solution obtained (DH cognition and DH clinical performance)

– Factor scores used in a linear predictive model to exam ability of predicting outcome on clinical licensure exam

– Shared contribution of both factors only accounted for 13.9%

– Disturbing lack of concordance between previously validated or accepted measures of dh student competency (GPA, NBDHE, Portfolios) and one-shot clinical licensure exam

Time As A BarrierTime As A Barrier

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 25

Student and Faculty Resistance As Student and Faculty Resistance As

A BarrierA Barrier

LargeLarge--Scale Portfolio Scale Portfolio

AssessmentAssessment

• Exemplary Examples

– Truman State University – Portfolio Project

• http://assessment.truman.edu/components/portfoli

o/

– Alverno College – Diagnostic Digital Portfolio

• http://ddp.alverno.edu/

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 26

Accountability Measures for Practicing Accountability Measures for Practicing

DentistsDentists

• Continuing Competency/Quality

Assurance

– Minnesota

• http://www.dentalboard.state.mn.us/

• http://www.dentalboard.state.mn.us/Portals/3/SelfAssessment/SelfAssessment2006.pdf

Objectives RevisitedObjectives Revisited

• Provide rationale for the “why” behind using non-traditional assessment measures in competency-based education.

• Compare and contrast traditional faculty-centered assessment measures with portfolios as a student-centered nontraditional assessment measure.

• Discuss a systematic approach (how to) for implementing portfolio assessment along with methods for evaluation.

• Discuss barriers to implementing portfolio assessment of student competency and strategies for overcoming potential barriers.

This material is the intellectual property

of Cynthia C. Amyot, Ed.D.

(AmyotC@umkc.edu). 27

ReferencesReferences

• Arter, JA & Spandel, V. (1992). Using portfolios of student work in instruction and assessment. Educational Measurement, 11(1), 36-44.

• Boud, D., Keogh, R. and Walker, D. Reflection: Turning Experience into Learning. London: KoganPage; 1985.

• Burch, CD (1999). Inside the portfolio experience: The student’s perspective. English Education, 32(1): 34-39.

• Campbell JP, DeBlois PB, and Oblinger DA. (2007). Academic analytics a new tool for a new era. Educause Review:41-57.

• Chambers, DW & Glassman, P. (1997). A Primer on Competency-Based Evaluation. J Dent Educ, 61(8), 651-666.

• Driessen, E.W., van Tartwijk, J., Overeem, K., Vermunt, J.D. and van der Vleuten, C.P.M. Conditions for successful reflective use of portfolios in undergraduate medical education. Med Educ 2005;39:1230-1235.

• Drucker PF. (1994) The age of social transformation. The Atlantic Monthly, 274 (Nov.):53-80.• Fosnot, CT. (1996). Constructivism: Theory, perspectives, and practice. New York: Teachers

College Press.

• Friedman TL. The world is flat (updated and expanded): a brief history of the twenty-first century. 1st ed. New York: Farrar, Straus and Giroux, 2006.

• Gadbury-Amyot C, Kim J, Palm R, Mills GE, Noble E, Overman P. Validity and Reliability of Portfolio Assessment of Competency in a Baccalaureate Dental Hygiene Program. J Dent Educ, 67(9): 991-1002, 2003.

• Gadbury-Amyot C, Holt L, Overman P, Schmidt C. Implementation of Portfolio Assessment in a Competency-Based Dental Hygiene Program. J Dent Educ, 64(5):375-380, May 2000.

ReferencesReferences

• Lemke M and Gonzales P. (2006). U.S. students and adult performance on international assessment of educational achievement: Findings from the condition of education 2006 (NCES 2006-073). U.S. Department of Education, Washington, DC: National Center for Educational Statistics

• Licari FW & Chambers DW. (2008). Some paradoxes in competency-based dental education. J Dent Educ, 72(1):8-18.

• Lyons, N. (1988). Constructing narratives for understanding: Using portfolio interviews to scaffold teacher reflection. In N. Lyons (Ed.), With Portfolio in Hand: Validating the New Teacher Professionalism. New York: Teachers College Press.

• MacIsacc, D. & Jackson, L. (1994). Assessment process and outcomes: Portfolio construction. New Directions for Adult and Continuing Education, 62, 63-72.

• Palomba, C., & Banta, T. (1999). Assessment Essentials: Planning, Implementing, and Improving Assessment in Higher Education. San Francisco: Jossey-Bass.

• Pink D. (2005) A Whole New Mind: Moving From the Information Age to the Conceptual Age. Riverhead Hardcovers.

• Stevens DD, Levi AJ. (2005). Introduction to rubrics. Stylus Publishing, Sterling, Va.

• Wade, R.C., and Yarbrough, D.B. Portfolios: A Tool for Reflective Thinking in Teacher Education? Teaching and Teacher Educ 1996;12(1):63-79.

• Wiggins, G. (1993). Assessment: Authenticity, context, and validity. Phi Delta Kappan, 75(3), 200-214.

• Yancy, KB. (1992). Teachers’ stories: Notes toward a portfolio pedagogy. In K. B. Yancy (Ed.), Portfolios in the writing classroom: An introduction (pp.12-20). Urbana: National Council of Teachers in English.