Matt Rooney RAL The T2K Beam Window Matt Rooney Rutherford Appleton Laboratory BENE November 2006.
Learning Outcomes: Critical to Assessment Prepared for Office of Dean of Students Assessment...
-
Upload
lilian-mathews -
Category
Documents
-
view
214 -
download
0
Transcript of Learning Outcomes: Critical to Assessment Prepared for Office of Dean of Students Assessment...
Learning Outcomes: Critical to Assessment
Prepared for Office of Dean of Students
Assessment Training
January, 2011
Gail S. Rooney, Ph.D.
Director, The Career Center
University of Illinois at Urbana-Champaign
2
Session Purposes
Introduce a learning framework for student affairs
Gain experience writing learning outcomes tailored to your unique offices
Share sample ideas for engaging in the process of defining outcomes
Share sample evaluation project using learning outcomes
Engage in hands-on learning that you can take back to your offices
3
Session Outline
1. Learning as a framework for student development programs and services
2. Share a process for writing effective learning outcomes
• Defining context, brainstorming outcomes, writing outcomes statements, and considering assessment processes
3. Considerations of a learning outcomes approach for your office
4
Warm Up Activity
Write a learning outcome for a program or service that you might offer.
Rotate, discuss, and rate outcomes
Scale of 1 (not good) to 8 (excellent)
Add the rating scores for a total score
5
Part 1:
A Learning Framework for Student Affairs
6
Learning Framework
To be a part of the core functions of our institution, we need to think of ourselves as learning partners In an educational environment, learning is the
name of the game Student development programming is education As student affairs practitioners, we are facilitating
the development of lifelong and decision-making skills
Learning can be a paradigm for describing the value of our work
7
What is Learning? *
Learning is a complex, holistic, multicentric activity that occurs throughout the lifespan
Learning is more than information transfer – it is identity development (transformation), giving meaning, reflection and application in the context of the individual’s life
* Learning Reconsidered, 2004
8
What are Learning Outcomes? * Learning outcomes focus on intentional student
experiences (activities, courses, programs, services, resources)
What clients should be able to know, do, demonstrate or feel as a result of engaging in a learning experience
Outcomes help us define how clients have changed; they tell us how we are making a difference
* Keeling & Associates, 2007
9
Why Learning Outcomes? *
Outcomes define impact – how the client has changed
Focusing on learning moves beyond “bodies in the door” and “smiles on the way out”
The achievement of learning outcomes (individual success) measures institutional or intervention effectiveness
* Keeling & Associates, 2007
10
How do Learning Outcomes Shift Our Focus? *
Process
Planning activities and things to “do”
“What are we going to do?”
Counting and asking
Outcome
How the client will change as a result of what we do
How will clients be different; how will they learn, change, grow?
Documenting change; showing results
* Keeling & Associates, 2007
11
TCC’s Office-Wide Student Learning Outcomes
Posed two questions:
What do we want undergraduate students to know and be able to do upon graduation regarding personal and professional career development?
When a student graduates from the University of Illinois, what difference should The Career Center have made in their lives?
Gathered input from:
All office staff – counselors, support staff, graduate assistants, undergraduate outreach presenters, undergraduate office assistants
TCC Advisory Committee members from around campus
Questions
back in
Spring 2005
12
Application of Theories, Models, & Standards – TCC’s Office-Wide Student Learning Outcomes Brainstormed responses,
which you see in your handouts
13
TCC’s Office-Wide Student Learning Outcomes
There are no “right” answers… but you may be wondering: What did TCC come up with?
Employed a counseling model
Affective, Behavioral, and Cognitive outcomes
Followed learning domains from professional standards –
Explore self and options, Manage education, Use resources, Gain experience, Communicate accomplishments, Conduct a search
Recognized career development theory
Self, options, decision-making, life-long process
Affective components of encouragement, hopefulness, etc
(See “Samples” Packet, page 2)
14
TCC’s Office-Wide Outcomes: Practice
(See “Samples” Packet, pages 3 - 4)
15
Part 2:
A Process for Writing Effective Learning Outcomes
16
Learning Outcomes Cycle
Defining Context
Brainstorming Outcomes
Connecting Theories / Models /
Standards
Writing Outcome Statements
Prioritizing Focus Areas
Evaluating Interventions
Reconsidering Outcomes,
Evaluations, Interventions
17
Demonstrating a Process
Steps addressed in learning outcomes activities:
Defining context
Brainstorming outcomes
Writing outcomes statements
Connecting theories, models, & professional standards
Additional steps:
Prioritizing focus areas
Evaluating interventions
Re-considering outcomes, evaluations, and interventions
18
Defining Context
What is your program / service? This could be…
Small – a single workshop or activity
Medium – multi-session interactions
Large – all interactions between students and your office
Who are your intended learners? This could be…
Specific – first-year students in a particular minority group
Broad – all community members who may seek services
19
Defining Context – Example 1 Resume Review Services
Intervention: Daily resume review services provided on a drop-in basis to clients who are conducting job and internship searches, as well as by students who need to prepare a resume as a part of an academic class.
Intended Learners: Current students and alumni at the University of Illinois
(See handouts for more detail)
20
Defining Context – Example 2 Exploring Pathways in Career Success
Intervention: EPICS is an interactive, online, modular program that engages students in thinking about their career decision-making early in their college experience.
Intended Learners: Current undergraduate students at the University of Illinois at Urbana-Champaign
(See handouts for more detail)
21
Defining Context – Example 3 Health and Graduate School Fair
Intervention: Health and Grad School Fair is held each fall in a large ballroom. It is a time of information networking and sharing regarding educational opportunities for college graduates.
Intended Learners: Undergraduate students who may or may not have decided to attend grad school.
(See handouts for more detail)
22
Defining Context – Your Turn! Review the tips, examples, and prompts on pages
2 and 3 of your worksheet packet
Share with your partner: the program or service that you will work with
today the intended learners for that program or service
Record your responses on page 4 in your packet
23
Brainstorming Outcomes
What should (intended learners) know, be able to do, be able to demonstrate, value, or feel when they have completed (program / service)?
Or, stated another way…
What difference should (program / service) have made for (intended learners) who participate?
24
Brainstorming Outcomes – Resume Review Services Example What should undergraduate students know, be
able to do, be able to demonstrate, or feel when they have completed a drop-in resume review? Know the characteristics of a well formatted
resume (Know) Demonstrate accomplishments in their descriptions
of past work experiences (Do / Demonstrate) Feel confident in their ability to create a resume
that highlights their strengths and abilities (Feel)
(See handouts for more detail)
25
Brainstorming Outcomes – EPICS Example What should undergraduate students know, be able to do,
be able to demonstrate, or feel when they have completed online modules in the EPICS program?
Understand how exploring their own personal interests is helpful for narrowing down potential major choices (Know)
Identify skills that they would like to use in their major (Do / Demonstrate)
Feel encouraged – that they can choose a major that will be a good fit for them (Feel)
(See handouts for more detail)
26
Brainstorming Outcomes – Health and Grad Fair Example What should undergraduate students know, be able to do,
be able to demonstrate, or feel when they have attended the Health and Graduate School fair?
Understand the requirements for health and graduate school programs (Know)
Identify and implement next steps to prepare for graduate school (Do / Demonstrate)
Feel confident talking with health and graduate school representatives and exploring options (Feel)
(See handouts for more detail)
27
Brainstorming Outcomes – Your Turn! Return to working with your partner
Review the tips, examples, and prompts on page 5 of your worksheet packet
Try brainstorming potential learning outcomes for the intervention and intended learners that you selected
Record your responses on page 6 in your packet
28
Writing Outcomes Statements – First Round (Intended learners) who
(intervention) will be able to
(intended outcome)?
Engage in
Participate in
Complete
Identify Summarize
List Discuss
Describe Explain
Demonstrate{ }
{ }
* Keeling & Associates, 2007
29
Writing Outcomes Statements – Resume Review Services Example Undergraduate students who engage in drop-in
resume review services will be able to articulate a clear target or purpose for their resume.
30
Writing Outcomes Statements – EPICS Example Undergraduate students who complete EPICS
Competency Builder will be able to use resources to strengthen communication skills needed in their potential work environments.
31
Writing Outcomes Statements – Health and Grad Fair Example Juniors and Seniors who attend the Health and
Graduate School fair will be able to identify at least 3 educational options beyond their undergraduate degree.
32
SMART Approach to Writing Outcomes S Specific
M Measurable
A Achievable
R Relevant
T Time-based
33
Writing Outcome Statements –
Your Turn! Return to working with your partner
Review the tips, examples, and prompts on pages 7 and 8 of your worksheet packet
Try writing some outcome statements that connect intervention, intended learners, and intended outcomes that you have brainstormed
Record your responses on page 9 in your packet
34
Writing Outcomes Statements – Second Round
Evaluation
Knowledge
Comprehension
Application
Analysis
SynthesisHigher Cognitive Levels
Lower Cognitive Levels
Bloom’s Taxonomy*Different Levels of Student Understanding & Performance
* Bloom, 1956
35
Writing Outcomes Statements with Bloom – Example Undergraduate students who engage in drop-in
resume review services will be able to:
list appropriate content for the education section of a resume (Knowledge)
use strong, varied action verbs to describe their experiences (Application)
integrate accomplishments / results into descriptions of their experiences (Synthesis)
36
Writing Outcome Statements with Bloom – Your Turn! Return to working with your partner
Review the tips, examples, and prompts on pages 10 and 11 of your worksheet packet
Try writing some outcome statements that connect activity, intended learners, and intended outcomes that you have brainstormed
Record your responses on pages 12 and 13 in your packet
37
Connecting Theories, Models, & Standards Theories / Models
Cognitive Information Processing
Krumboltz’s Learning Theory
Person-Environment correspondence (e.g., Holland’s typology)
Social Cognitive Career Theory
Super’s life-span, life-space Tiedeman’s decision
making model and many more…
Professional Standards National Career
Development Guidelines (NCDG; 1996, 2004)
National Association of Colleges and Employers (NACE) Principles (2006)
Council for the Advancement of Standards in Higher Education (CAS; 2003)
American School Counselor Association’s National Standards for Students (ASCA; 2003)
and many more…
38
Theories & Models – Benefits & DrawbacksBenefits
Helps us understand our assumptions
Provides recognized support for program and service decisions
Provides insights into:
important program and service components
target learners
potential learning outcomes
evaluation strategies
Drawbacks
Can create “blind spots”
May not adequately translate to our unique populations and contexts
At times, challenging to figure out how to translate theory into practice
39
Professional Standards – Benefits & DrawbacksBenefits
Provides recognized practices for program and service decisions
Provides a framework, each through a different lens CAS → system / program
level NCDG & ASCA →
individuals NACE → partnerships
between career practitioners and employers
Drawbacks
Standards speak broadly Need to tailor outcomes to
our unique learners / environments
Can be overwhelming Must prioritize to make it
manageable
40
Connecting Theories, Models, & Standards– Homework!
Review the tips, examples, and prompts on page 14 of your worksheet packet
Brainstorm the theories, models, and standards that influence your practice.
Record your responses on page 15 in your packet
41
Connecting Theories, Models, & Standards – Homework A “take-home” exercise!
How do your learning outcomes currently speak to your theory/model/standards?
Would you change or re-word any of your learning outcomes based on your theory/etc.?
What learning outcomes might you add based on your theory/etc.?
How might your preferred theory/etc. help justify or support your choice of learning outcomes?
We’ll still share examples
42
Part 3:
A Sample Learning Outcomes
Evaluation
43
Learning Outcomes Cycle
Defining Context
Brainstorming Outcomes
Connecting Theories / Models /
Standards
Writing Outcome Statements
Prioritizing Focus Areas
Evaluating Interventions
Reconsidering Outcomes,
Evaluations, Interventions
44
Setting Evaluation Priorities Learning Reconsidered Framework, with activities
to engage staff in thoughtful discussion
Write learning outcomes for specific programs and services
Discuss program evaluation possibilities for the next year
(See “Samples” Packet, pages 5 - 9)
45
Setting Evaluation Priorities, Con’t
Selected three programs for “critical incident” evaluations
Exploring Pathways in Career Success (EPICS)
Resume Review Services
Mock Interview Program
Imagine building a “house of evidence” and each program evaluation is a brick
46
Evaluation Example: Resume Review Services Investigates:
Student learning outcome of effectively communicating accomplishments, as demonstrated through resume writing abilities
Three experimental groups
Participated ONLY in resume review services
Participated in resume review services AND other programs/services
Have NOT used programs/services (control group)
47
Evaluation Example: Resume Review Services Sample consisted of Juniors and Seniors in the
College of Liberal Arts & Sciences who had posted resumes on our online job posting system
Blind review of “self-selected, final product” resumes with pre-determined rubric (See handout)
Compared with ANOVA analysis and Dunnett’s Test of Multiple Comparisons Total rubric score Eight individual components of effective resume
writing
48
Evaluation Example: Resume Review Sample Findings Overall resume quality
In comparison to “No TCC” group: “Resume services and more” group found to have
significantly higher quality resumes (α<.05) “Resume services only” group found to have significantly
higher quality resumes (α<.10)
25.70**
24.60*
22.90
22.00
23.00
24.00
25.00
26.00
No TCC Resume Only Resume Plus
Mea
n S
core
49
Evaluation Example: Resume Review Sample Findings Resume components
Significant differences found between control and experimental groups on three components: education, additional resume sections, clear purpose/target.
3.50**
3.35**
2.80
3.15**
2.852.65
2.80*
2.252.25
1.00
2.00
3.00
4.00
No TCC Resume Only Resume Plus
Me
an
Sc
ore
s
Education
Additional Sections
Clear Purpose /Target
50
Evaluation Example: Resume Review Discussion Items Overall highest ratings
found on resume sections with “fairly standard” content Name / contact
information (3.82/4.00)
Spelling / grammar (3.47/4.00)
Education (3.22/4.00)
Modest overall means on areas that have general tips and guidelines, yet leave ultimate decision-making up to the resume writer. General format (2.90/4.00)
Additional resume sections (2.88/4.00)
Positive / professional tone (2.87/4.00)
Primary experience (2.82/4.00)
51
Evaluation Example: Resume Review Discussion Items Lowest overall mean on
component that requires themes to be threaded through various experiences, requiring high-level analysis and synthesis skills Clear purpose / target
(2.43/4.00)
52
Evaluation Example: Resume Review Discussion Items Overall significantly higher quality for both experimental
groups, in comparison to our control
Significantly higher quality for both experimental groups on the education section of the resume Content was more complete, organized, clear, and well-defined
than control group
53
Evaluation Example: Resume Review Discussion Items Significantly higher quality for “resume services and
more” group on the additional resume sections and clear purpose/target section
These sections demonstrate a “global sense” of career development and focus
Possible that participation in a wider array of career development programs and services is beneficial to assist students in developing skills to effectively communicate a cohesive set of career aspirations through connections among the wide varieties of experiences that make up the content of their resumes
54
Evaluation Example: Resume Review Limitations Small sample (20 resumes in each groups = 60 total)
Possible pre-existing differences in student populations
Who uses the online job posting service that resumes were drawn from? How might they differ from other students?
Students may be receiving resume and career assistance from other sources
Private counseling
Family members
Professional networks
Etc.
55
Evaluation Example: Resume Review Next Steps Recognize the good work that the resume review and counseling
teams have done in working with students (Celebrate successes!)
Consider areas for improvement in staff training, resume workshops, resume reviews, and (where appropriate) counseling appointments
Focus on room for growth in several resume component areas, such as: general formatting, experience sections, tone, clear purpose/target.
Training staff to make referrals to counseling appointments, particularly for students who are having difficulty:
Defining a clear purpose or target
Developing a cohesive, logical flow to the variety of experiences on their resumes
YOUR Evaluation
How will you evaluate your learning outcomes?
Examples?
Brainstorm assessment approaches
Surveys
Documents
Experimental – Pre & Post
Qualitative
Others?
57
Value of Learning Outcomes
What are learning outcomes?
Learning outcomes define impact; they measure institutional or intervention effectiveness
Learning outcomes tell us how the client has changed and how we are making a differenceLearning Reconsidered, 2004
58
Connections Between Learning Outcomes Evaluation and Advocacy Bringing learning outcomes and advocacy
together
Learning outcomes tell us much more about the impact of our programs and services than traditional strategies of participant counts and satisfaction surveys
They show the difference that we make, giving us powerful evidence of the value of our work
59
Learning Outcomes Evaluations: “The Challenges” Pioneering work that may not be fully understood
Requires sustained effort, energy, resources, and leadership over time
Crosses many evaluation fields leading to confusion regarding terms and strategies
Requires developing new creative thinking and technical evaluation skills
Other ideas?