NAVIGATING THE PROCESS OF STUDENT LEARNING OUTCOMES: DEVELOPMENT, EVALUATION, AND IMPROVEMENT

Shannon M. Sexton, Julia M. Williams, & Timothy ChowRose-Hulman Institute of Technology

Introduction & Workshop Goals

RHIT/IRPA - NASPA 2009

Introduction Goals

1. Discuss the process of developing institute student learning outcomes as well as common difficulties.

2. Assist participants in developing their own learning outcomes3. Discuss evidence that can be used to evaluate student learning

outcomes achievement.4. Assist participants in identifying articles of evidence for their

learning outcomes.5. Discuss using an e-portfolio as a means of collecting and rating

evidence.6. Allow participants to rate sample evidence using a sample rating

rubric.7. Discuss how to use the rating results to close the loop and

improve student learning.

Group Activity #1

RHIT/IRPA - NASPA 2009

Define Teamwork Include characteristics of teamwork Include traits of those characteristics Time allotted 10 minutes

About Rose-Hulman Terre Haute, Indiana 1800+ undergraduate

students B.S. degrees in

engineering, science, and mathematics

80%+ engineering students

Accredited through ABET and North Central

RHIT/IRPA - NASPA 2009

CASO

RHIT/IRPA - NASPA 2009

Commission on the Assessment of Student Outcomes

Faculty committee with 1 representative from each academic department

Regular meetings throughout the academic year

Tasked with developing Institute-wide student learning outcomes and a way to measure them

Historic Timeline

RHIT/IRPA - NASPA 2009

1996 – 1998 1999 – 2000 2001 – 2006 2007 – present

Removing the Jargon

RHIT/IRPA - NASPA 2009

Domains = Categories of skills Learning Outcomes = Definition of the

skills students should have Performance Criteria = Characteristics of the learning outcome

Rubrics = Use of a scale to identify traits in a document

Domains and Performance Criteria

8

Domain Student Learning Outcome

Performance Criteria

Technical Skills Outcomes developed by each department.

Levels A – C indicate the level of performance achieved by each student.

Professional Skills

1. Leadership2. Teamwork3. Communication

Levels A – C indicate the level of performance achieved by each student.

World Citizenship 4. Cultural & Global Awareness

5. Ethics6. Service

Levels A – C indicate the level of performance achieved by each student.

RHIT/IRPA - NASPA 2009

Developing Outcomes

RHIT/IRPA - NASPA 2009

Importance of preliminary research Teams of faculty developers Developing in steps & gaining institute

buy-in Importance of measurability

Writing Outcomes – Activity #2

RHIT/IRPA - NASPA 2009

Revise your teamwork definition into a student learning outcome with rubrics

Time allotted 15-20 minutes

10 minutes to stretch your legs

Take a Break

RHIT/IRPA - NASPA 2009

Evidence of Outcomes

RHIT/IRPA - NASPA 2009

Choosing appropriate evidence New vs. Current Importance of Rubric

Collecting evidence How to physically collect & store evidence

Course Mapping Tracking where submissions are coming

from Accountability

Criterion Description and Rubric Teamwork B1: Demonstrate how you reached a

decision as a team.

Primary Traits: A passing submission for this criterion must:

1. Describe the team goal.2. Provide a description of a specific team decision and describe the following:

- the process of making the decision- how multiple team members contributed to the outcome- how team members’ ideas were critically evaluated

3. Show that the ultimate outcome was consistent with the team’s decision-making process.Potential documents: Documents appropriate for this criterion include (but are not limited to): Memo or reflective statement on team process from a lab group design team, debate team, service project.

Additional information:

13

RHIT/IRPA - NASPA 2009

Course Mapping14

Course mapping

RHIT/IRPA - NASPA 2009

Identifying Evidence – Activity #3 Where do you already have existing

assignments that could be used as articles of evidence for the Teamwork outcome? Group Brainstorming Activity

15

RHIT/IRPA - NASPA 2009

Domains and Performance Criteria

16

Performance CriteriaLevel C – associated with awareness, application, or evaluation.Level B – associated with application.Level A – associated with knowledge or comprehension.

A-Level Criteria

B-Level Criteria

C-Level Criteria

RHIT/IRPA - NASPA 2009

Collecting Evidence

RHIT/IRPA - NASPA 2009

Use of an E-portfolio system Faculty input Student input Assessment staff input

Rating Evidence

RHIT/IRPA - NASPA 2009

Rating preparation Administrative side of rating

Paying faculty volunteers, tech support, number of volunteers, etc.

Process of rating days

Portfolio Rating Methodology Evaluation rubric, A, B, C Inter-rater reliability: initial, subsequent Document rating: Pass/Fail/Exemplary Wholistic rating, NOT grading

RHIT/IRPA - NASPA 2009

RHIT/IRPA - NASPA 2009

RHIT/IRPA - NASPA 2009

Rating Documents – Activity #4

RHIT/IRPA - NASPA 2009

Rate 2-3 teamwork documents using the provided rubric.

15 - 20 minutes

Using Rating Results

RHIT/IRPA - NASPA 2009

Identifying constituents How to communicate results to institute What to do with the results

Example Result

RHIT/IRPA - NASPA 2009

2005 2006 20070%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

36%

63%71%

52%

62%

73%65%

74%83%

Historical Portfolio Rating Results

Teamwork Communication Ethics

Mee

ts S

tand

ard

Please contact us for more information after the conference.

RHIT/IRPA - NASPA 2009