Developing Assessment Methods at Classroom, Unit, and Campus-Wide LevelsPresented atThe Robert Gordon UniversityAberdeen, Scotland

March 19, 2004

Presented by

Trudy W. BantaVice ChancellorPlanning and Institutional ImprovementIndiana University-Purdue University Indianapolis355 N. Lansing St., AO 140Indianapolis, Indiana 46202-2896tbanta@ iupui.eduhttp://www.planning.iupui.edu

Outcomes Assessment

The process of providing credible

evidence of the outcomes of

higher education undertaken for

the purpose of improving

programs and services within the

institution.

Banta, T. W.

Plan

Implement

Assess

Improve

ASSESSMENT . . .

“a rich conversation

about student learning

informed by data.”

-- Ted Marchese --

AAHE

Assessment of Individual Student Development

Assessment of basic skills for use in advising• Placement• Counseling

Periodic review of performance with detailed feedback

End-of-program certification of competence• Licensing exams• External examiners

Key Results of Individual Assessment

Faculty can assign grades

Students learn their own strengths and weaknesses

Students become self-assessors

A Second Look

Across students

Across sections

Across courses

Where is learning satisfactory?

What needs to be retaught?

Which approaches produce the most learning for which students?

Group Assessment Activities

• Classroom assignments, tests, projects

• Questionnaires for students, graduates, employers

• Interviews, focus groups• Program completion and placement• Awards/recognition for graduates• Monitoring of success in graduate

school• Monitoring of success on the job

Use of Results of Group Assessment

• Program improvement

• Institutional and / or state peer review

• Regional and / or national accreditation

Some Purposes of Assessment

1. Students learn content

2. Students assess own strengths

3. Faculty improve instruction

4. Institutions improve programs/services

5. Institutions demonstrate accountability

Outcomes Assessment Requires Collaboration

In setting expected program outcomes

In developing sequence of learning experiences (curriculum)

In choosing measures In interpreting assessment findings In making responsive improvements

Barriers to Collaboration in the Academy

1. Graduate schools prepare specialists

2. Departments hire specialists

3. Much of our scholarship is conducted alone

4. Promotion and tenure favor individual achievements -- interdisciplinary work is harder to evaluate

Campus Interest in Assessment

WHAT WORKS in….

increasing student retention? general education? use of technology in instruction? curriculum in the major?

Good assessment is good research . . .

An important question An approach to answer the

question Data collection Analysis Report

-Gary R. Pike (2000)

Most Faculty Are Not Trained as Teachers

FACULTY DEVELOPMENTCan Help Instructors:

Write clear objectives for student learning in courses and curricula

Individualize instruction using a variety of methods and materials

Ask questions that make students active learners

Develop assessment tools that test higher order intellectual skills

Organizing for Assessment

Goal Course Measure Findings Uses

Write Portfolio

Speak Speech

Think Test

Find Information

Project

Taxonomy of Educational Objectives

(Bloom and Others, 1956)Cognitive domain

categories

Knowledge

Comprehension

Application

Analysis

Synthesis

Evaluation

Sample verbs for outcomes

Identifies, defines, describes

Explains, summarizes, classifies

Demonstrates, computes, solves

Differentiates, diagrams, estimates

Creates, formulates, revises

Criticizes, compares, concludes

Learning Outcomes for English Majors

Graduates can:1 Demonstrate how language influences

intellectual and emotional responses2 Apply knowledge of rhetorical context by

writing effectively and appropriately in context

3 Assess critically spoken, written, and visual representations

4 Apply research strategies appropriate to area of study

5 Synthesize diverse issues and responses raised in collaborative discussions of texts

Learning Outcomes in Science

1. Define and explain basic principles, concepts, theories of science

2. Identify characteristics that distinguish math and science from each other and from other ways of obtaining knowledge

3. Illustrate how developments in science can raise ethical issues

4. Solve theoretical or experimental problems in science

5. Evaluate the validity and limitations of theories and scientific claims in interpreting experimental results

6. Evaluate scientific arguments at a level encountered by informed citizens

In each course

Statement

Of

Competence

Teaching/

Learning

Strategy

Assessment

Methods

Critical

Thinking

Group

Project

Written

Paper

Planning for Learning and Assessment

1. Whatgeneraloutcomeare youseeking?

2. Howwould youknow it(theoutcome)if you sawit? (Whatwill thestudentknow orbe able todo?)

3. How willyou helpstudentslearn it?(in classor out ofclass)

4. How couldyoumeasureeach of thedesiredbehaviorslisted in #2?

5. What aretheassess-mentfindings?

6. Whatimprove-mentsmight bebased onassess-mentfindings?

Organizational Levels for Assessment

National

Regional

State

Campus

College

Discipline

Classroom

Student

Direct Measures of LearningAssignments, exams, projects, papers

Indirect MeasuresQuestionnaires, inventories, interviews

- Did the course cover these objectives?- How much did your knowledge increase?- Did the teaching method(s) help you

learn?- Did the assignments help you learn?

GOOD ASSESSMENT INCLUDES BOTH

Fast Feedback(at end of every class)

Most important thing learned Muddiest point Helpfulness of advance reading

assignments for day’s work in class Suggestions for improving class /

assignmentsBateman & Roberts

Graduate School of Business

University of Chicago

Student Suggestions for Improvement

Install a portable microphone Increase type size on

transparencies Leave lights on when using

projector Don’t cover assigned reading in

detail Provide more examples in class

Wingspread Seven Principles for Good Practice in Undergraduate

Education(Chickering & Gamson, 1987)

1. Student-faculty contact

2. Cooperation among students

3. Active learning

4. Prompt feedback

5. Time on task

6. High expectations

7. Respect for diverse talents and ways of learning

Student Learning OrientedCourse Evaluation

1. Learners held high expectations for one another

2. Learners interacted frequently with others

3. Learners participated in learning teams4. Learners respected diverse talents and

ways of learning

-CournoyerAdvances in Social

Work – Fall 2001

Primary Trait Scoring

Assigns scores to attributes (traits) of a task

STEPS Identify traits necessary for success in

assignment Compose scale or rubric giving clear

definition to each point Grade using the rubric

Can Develop a Research Paper

1. Narrows and defines topic

2. Produces bibliography

3. Develops outline

4. Produces first draft

5. Produces final draft

6. Presents oral defense

  

 

Out-standing

Accept-able

Unaccept-able

BibliographyOutstanding – References current,

appropriately cited, representative, relevant

Acceptable – References mostly current, few citation errors, coverage adequate, mostly relevant

Unacceptable – No references or containing many errors in citation format, inadequate coverage or irrelevant

Mapping Course Outcomes to Program Outcomes

Outcomes Course 1 Course 2 Course 3

1 2

3

4

5

6

7

Sophomore Competence in Mathematics(Multiple choice responses & supporting work)

Score3

2

1

0

CriterionClear conceptual understanding, consistent

notation, logical formulation, complete solution

Adequate understanding, careless errors, some logic missing, incomplete solution

Inadequate understanding, procedural errors, logical steps missing, poor or no response

Problem not attempted or conceptual understanding totally lacking

Ball State University

Assessment in Sociology and Anthropology

Focus groups of graduating students Given a scenario appropriate to the discipline, a

faculty facilitator asks questions related to outcomes faculty have identified in 3 areas: concepts, theory, methods.

2 faculty observers use 0-3 scale to rate each student on each question

GROUP scores are discussed by all faculty Murphy & Goreham

North Dakota State University

Assessment of Group Interaction

The Student Participant: Listened to others Actively contributed to discussion Challenged others effectively Was willing to alter own opinion Effectively explained concepts/insights Summarized/proposed solutions

5=Consistently excellent

3=Generally satisfactory1=Inconsistent and/or inappropriate

Capstone Course in Business

Comprehensive case study analysis

Graded using scoring rubric covering writing skills knowledge of historic and current theories

and practice ability to integrate tools and techniques

from business specialties in analyzing the case

Olney & Menger

St. Mary’s University

In a Comprehensive Assessment Program...

INVOLVE Students Faculty Student Affairs

Professionals Administrators Graduates Employers

Student Advisory Council at Montevallo

A way to provide continuous student assessment

Student Recommendations1 Develop a statement of expected

ethical behaviors for students2 Add a second research course with lab3 Increase comparative psychology4 Add terminals for statistics lab5 Increase opportunities for research,

writing, and speaking

Collaboration in Evaluation(Harvard and Samford)

Involve a student team in evaluating a course or program Interaction increases Students develop communication skills

teamwork competence

self-confidence

understanding of others’ perspectives

Involvement in learning increases

PRINCIPLES OF UNDERGRADUATE LEARNING

(PULs)1. Core communication and quantitative

skills2. Critical thinking3. Integration and application of knowledge4. Intellectual depth, breadth, and

adaptiveness5. Understanding society and culture6. Values and ethics

Approved by IUPUI Faculty Council May 1998

Student Electronic Portfolio

Students take responsibility for demonstrating core skills

Unique individual skills and achievements can be emphasized

Multi-media opportunities extend possibilities

Metacognitive thinking is enhanced through reflection on contents

- Sharon J. Hamilton

IUPUI

Virginia Commonwealth University

1. First-year English students wrote 1

response / week for 15 weeks

2. Faculty-student affairs teams read

essays

3. Sample of writers interviewed 18

months later

Responses to findings:

Central advising center, new advising

handbook, multicultural workshops

Involving EmployersCombination of survey and focus groups

for employers of business graduates Identified skills, knowledge, personality attributes

sought by employers Encouraged faculty to make curriculum changes Motivated students to develop needed skills Strengthened ties among faculty, students, employers

- Kretovics & McCambridge

Colorado State University

Colorado State UniversityCollege of Business

Curriculum changes based on employer suggestions:

1 credit added to Business Communications for team training and more presentations

Ethics & social responsibility now discussed in intro courses

New Intro to Business course emphasizing career decision-making

More teamwork, oral & written communication, problem-solving in Management survey courses

- Kretovics & McCambridge

Program Course Measurement Findings Action

Objective Objective Method Needed

Apply CT Use Lab exercises 64% of Students

skills in SW statistical ___________ students ask for

practice analysis to (80% of scored 3.0 more lab

inform students will or better time; 5 days

practice earn 3.0 or on lab of lab

decisions better on lab exercises added

exercises

Columbia College

Authentic Assessmentat

Southern Illinois University - Edwardsville

Business - Case Study Analysis with Memo Education - Professional Portfolio Psychology - Poster on Research Project Engineering - Senior Design Project Nursing - Plan of Care for Patient

Responses to Assessmentat

Southern Illinois University-Edwardsville• Business – More case studies and research• Education – More practice in classroom

management• Psychology – Curriculum change in statistics• Engineering – More practice in writing and

speaking• Nursing – Simulation lab with computerized

patients

Assessment in Fine Arts— Portfolio review and/or audition

for every student every semester by

Panel of faculty, students, community representatives, staff or faculty outside fine arts

Results - Students creativity, conceptualization, technique have improved

— Alec Testa— Eastern New Mexico University

Assessment Essentials

1. Plan for assessment2. Involve all stakeholders3. Develop goals and objectives4. Select or design assessment methods5. Report assessment findings6. Promote the use of assessment findings7. Evaluate the assessment program

- Palomba & Banta

Success Factors1 Committed leadership2 Collaboration between faculty and student

affairs leaders3 Teamwork in planning and implementation4 Supportive campus climate

Concern for students, continuous

improvement5 Involvement in design of assessment6 Results effectively communicated7 Conscientious follow-up8 Persistence

The Future Need for evidence of accountability will

increase More faculty will recognize benefits of

assessment More electronic assessment methods will be

developed More sharing of assessment methods will

take place Faculty will learn more about learning and

student learning will improve