Interpreting Data & “Closing the Loop”
description
Transcript of Interpreting Data & “Closing the Loop”
![Page 1: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/1.jpg)
Interpreting Data & “Closing the Loop”
Margaret Kasimatis, PhDVP for Academic Planning & Effectiveness
![Page 2: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/2.jpg)
ARTICULATEMISSION/
GOALS
IDENTIFYSPECIFIC
OUTCOMES
DETERMINEPRACTICES USED
TO ACHIEVEOUTCOMES
GATHEREVIDENCE
REVIEW &INTERPRETRESULTS
RECOMMENDACTIONS
Location in Assessment Cycle
![Page 3: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/3.jpg)
Step #1: Organize Your Data
• Assemble the following in one place:– All data or possible sources of data– List of learning outcomes/research questions– Curriculum, experiences, practices
• Map data sources to outcomes– Outcomes x Measures Map
• Map results to outcomes– Present in “user-friendly” way
![Page 4: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/4.jpg)
LEARNING OUTCOMES
ASSESSMENT MEASURES
Alumni Survey
National Survey of Student Engagement
(NSSE)
Evaluation of Clinic
Presentations
Evaluation of First-Year
Presentations
Rubric Evaluation of Upper-level Coursework
Ability to apply knowledge in math, science, and engineering X X
Ability to design a system, component, or process to meet desired needs X
Ability to identify, formulate, andsolve engineering problems X X X
Ability to communicate effectively X X X X XAbility to use techniques, skills, and modern engineering tools necessary for engineering practice
X X
Ability to function on multidisciplinary teams X X X
Recognition of need for and abilityto engage in lifelong learning X X
Understanding of professional andethical responsibility X X
Broad education necessary tounderstand the impact of engineering solutions in a global & societal context
X X
Mapping Data Sources to Outcomes:Engineering Example
![Page 5: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/5.jpg)
• It’s best to present results one outcome at a time– It’s not recommended to present results primarily
by measure
• Make sure to present the data in a user-friendly way (i.e., in a way that faculty in your department are comfortable with)
Presenting Results
![Page 6: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/6.jpg)
NSSE: Data on Communication Skills
1
2
3
4
Experience contributed to abilityto write clearly and effectively
Experience contributed to abilityto speak clearly and effectively
Mea
n R
esp
on
se (
1-4
scal
e)
HMC Senior Engineers
AITU Seniors
Liberal Arts Seniors
0
10
20
30
40
50
60
To what extent was an emphasis onwriting skills present while you were at
HMC?
To what extent was an emphasis on oralcommuniction skills present while you
were at HMC?
Pe
rce
nt
Ind
ica
tin
g "
Ve
ry M
uc
h"
Biology
Chemistry
Comp Sci
Engineering
Math
Physics
Alumni Survey: Data on Communication Skills Rubric Evaluation of Student Presentations
1
2
3
4
First-Year Presentation Clinic Presentation
DeliveryLanguageOrganizationContent
Rubric Evaluation of Student Work: Writing Style
1
2
3
4
5
Course #1 Course #2
Mea
n R
ub
ric
Sco
re o
n W
riti
ng
Sty
le (
1-5
scal
e)
![Page 7: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/7.jpg)
ACADEMIC CHALLENGE: LMU Jesuit Master’s NSSETotal
How often did you work harder than you thought you could to meet an instructor's standards or expectations?(Percent of students who said ‘Often’ or ‘Very Often)
FY 62% 55% 54% 52%
SR 54% 59% 60% 56%
How much time did you spend preparing for class (e.g. studying, reading, writing, doing homework, rehearsing,
other academic activities)? (Percent of students who reported spending 11 or more hours per week)
FY 65% 68% 49% 56%
SR 40% 58% 52% 54%
How often did you come to class without completing readings or assignments? (Percent of students who said ‘Often’ or ‘Very Often’)
FY 21% 17% 17% 19%
SR 27% 23% 20% 23%
Describe the extent in the current year to which exams challenged you to do your best work. (mean response on a 7-point scale)
FY 5.47 5.56 5.33 5.42
SR 5.33 5.46 5.43 5.40
![Page 8: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/8.jpg)
Step #2: Interpret the Data
• What type of criterion?
• What is “significant?”
• Are the findings reliable/valid?
![Page 9: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/9.jpg)
What type of criterion?
• Norm-referenced– Need appropriate comparison group
• Avoid percentile rankings
– Good for more nebulous findings
• Absolute standard– Usually more appropriate for
performance-based measures
![Page 10: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/10.jpg)
What is significant?
• Important to test when making comparisons– Numbers that look different may not really
be different
• However, just because a difference is significant doesn’t mean it’s important
![Page 11: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/11.jpg)
Are findings reliable/valid?
• Can we believe student self-reports?
• Are standardized measures more valid?
• Single measure vs. multiple measures
![Page 12: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/12.jpg)
Step #3: Make Recommendations
• Start by considering– Where in the curriculum are outcomes
addressed? And at what level?• Refer to (or create) curriculum map, or
• Inventory for particular outcome
– How? What practices/techniques are used?
– Where are the gaps?
– What can be changed?• In the short-term? In the long-term?
![Page 13: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/13.jpg)
Step #3: Make Recommendations
• Possible types of recommendations:– Changes to pedagogy– Changes to curriculum/programming– Allocation of resources
• Keep in mind that you can’t fix everything at once – so start small
• It’s usually better to modify than add
• Be as specific/concrete as possible
![Page 14: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/14.jpg)
PROPOSED ACTION IMPLEMENTATION
Document (with grades) the importance of presentation clarity
Essay questions added to exams
Engineering Curriculum Committee developed writing rubric to be used to evaluate project report
Make common writing style guidelines available to faculty and students
Department selected Hacker, A Writer’s
Reference
Hacker, A Writer’s Reference required in E4
Hold Clinic team members
accountable for peer review
of team reports
Clinic advisors distributed peer review sheets to
Clinic team members. Students completed the
peer review sheets when they reviewed fellow
team members’ portions of Clinic report .
Step #4: Implement & Document Recommend Actions
Example
![Page 15: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/15.jpg)
Example of Assessment Loop
• Step #1: Articulate Goals/Outcomes– Goal: Graduates will possess strong
communication skills• Outcome: Graduates will be able to write clearly and
effectively
• Step #2: Determine practices used to achieve outcome– Survey of required writing experiences in Core
and academic programs
![Page 16: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/16.jpg)
Sem # BIO CHEM CS ENG MATH PHYS H/SS
One 4 papers
Two Report in E4 Tech report in Phys 28
4 papers
Three 1-2 papers in Bio 101
3 lab reports
Lab notebook & paper in Chem 53
Tech report
in Phys 53
Variable
Amount
Of
Writing
Four 4 lab reports
1-2 papers in Bio 108
Essays/paper
in Chem 52
Lab notebook Exp abstracts
Design doc,
Report, &
User guide
in CS 121
Tech report
in Phys 54
Term paper
in Phys 52
Five 1-2 papers in
Bio 109
3 lab reports
3 reports + paper in 109
Exp abstracts
Abstracts in CS 110
Proposal &
mid-year report OR
Six 1-2 papers
in Bio 113
Seminar paper
Project paper
in Chem 114
Essay in
CS 131
Clinic report Tech report in Phys 134
IE Paper IE Paper IE Paper IE Paper IE Paper IE Paper
Seven Proposal
3 Lab reports
Letter of Intro
Prog report
Proposal +
midyear
report
Proposal +
midyear
report
Seminar
Research
PaperEight Thesis Thesis Clinic
reportClinic report Thesis or
Clin. report
Thesis or
Clin. report
![Page 17: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/17.jpg)
Example of Assessment Loop, cont’d
• Step #3: Conduct measurements– Higher Education Research Institute (HERI)
College Student Survey (CSS)– National Survey of Student Engagement– HMC Alumni Survey
![Page 18: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/18.jpg)
• Step #4: Review & interpret results– Faculty committees
• Assessment• Curriculum• Teaching & Learning
– Full faculty - workshop
Example of Assessment Loop, cont’d
![Page 19: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/19.jpg)
Writing SkillsFrequency data from HERI, NSSE & Alumni Survey
0
10
20
30
40
50
60
70
80
"Much stronger writingskills than when entering
college"
"Institution contributedvery much to ability to
write clearly andeffectively"
"To what extent was anemphasis on writing skillspresent while you were at
HMC?" (5)
HMC Seniors
4-Yr Non-Sectarian
1959-1987Cohort
1988-2001CohortHMC
Seniors
Liberal Arts
AITU Seniors
![Page 20: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/20.jpg)
Seniors’ Perceptions of Writing Skills by Major (from NSSE)
1
1.5
2
2.5
3
3.5
4
Biology CS Engineering Mathematics PhysicalSciencesT
o w
hat
ext
ent
did
exp
erie
nce
s at
you
r in
stit
uti
on c
ontr
ibu
te
to y
our
abil
ity
to w
rite
cle
arly
an
d e
ffec
tive
ly?
![Page 21: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/21.jpg)
Alumni Perceptions of Writing Emphasis by Major (alumni graduating in last 10 years)
0
10
20
30
40
50
60
70
Biology Chemistry CS Engineering Math Physics
Wri
tin
g em
ph
asis
was
"ve
ry m
uch
" p
rese
nt
![Page 22: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/22.jpg)
Sem # BIO CHEM CS ENG MATH PHYS H/SS
One 4 papers
Two Report in E4 Tech report in Phys 28
4 papers
Three 1-2 papers in Bio 101
3 lab reports
Lab notebook & paper in Chem 53
Tech report
in Phys 53
Variable
Amount
Of
Writing
Four 4 lab reports
1-2 papers in Bio 108
Essays/paper
in Chem 52
Lab notebook Exp abstracts
Design doc,
Report, &
User guide
in CS 121
Tech report
in Phys 54
Term paper
in Phys 52
Five 1-2 papers in
Bio 109
3 lab reports
3 reports + paper in 109
Exp abstracts
Abstracts in CS 110
Proposal &
mid-year report OR
Six 1-2 papers
in Bio 113
Seminar paper
Project paper
in Chem 114
Essay in
CS 131
Clinic report Tech report in Phys 134
IE Paper IE Paper IE Paper IE Paper IE Paper IE Paper
Seven Proposal
3 Lab reports
Letter of Intro
Prog report
Proposal +
midyear
report
Proposal +
midyear
report
Seminar
Research
PaperEight Thesis Thesis Clinic
reportClinic report Thesis or
Clin. report
Thesis or
Clin. report
![Page 23: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/23.jpg)
• Step #5: Recommend actions– Universal writing handbook– Faculty workshop on writing instruction– Revision to Core Curriculum
Example of Assessment Loop, cont’d
![Page 24: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/24.jpg)
MULTIPLE CURRICULAR OBJECTIVESIN HORIZONTALLY INTEGRATED
CORE CURRICULUM
Content• disciplinary knowledge • discipline-related techniques
Skills
• oral and written communication • critical thinking• teamwork and collaboration• project management• leadership
Context
• relationship of science or technology with contemporary society
• individual and comparative explorations of cultural identity
![Page 25: Interpreting Data & “Closing the Loop”](https://reader035.fdocuments.net/reader035/viewer/2022062323/5681555d550346895dc32678/html5/thumbnails/25.jpg)
QUESTIONS?