George Kuh SAIR New Orleans, LA September 26, 2010
-
Upload
kylie-campbell -
Category
Documents
-
view
27 -
download
0
description
Transcript of George Kuh SAIR New Orleans, LA September 26, 2010
![Page 1: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/1.jpg)
George Kuh
SAIR
New Orleans, LASeptember 26, 2010
NILOA: Tracking the NILOA: Tracking the Status of Outcomes Status of Outcomes
Assessment in the U.S.Assessment in the U.S.
![Page 2: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/2.jpg)
“Colleges… do so little to measure what students learn between freshman and senior years. So doubt lurks: how much does a college education – the actual teaching and learning that happens on campus – really matter?”
David Leonhardt, NYTimes, Sept 27, 2009
![Page 3: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/3.jpg)
Advance OrganizersAdvance Organizers What kind of information about student
learning and institutional effectiveness is compelling and useful for: (a) guiding improvement efforts?(b) responding to accountability
demands? What can be done to motivate encourage
faculty and staff to use assessment results to enhance student learning?
![Page 4: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/4.jpg)
Advance OrganizersAdvance Organizers
What are the obstacles that must be addressed for IR to help advance the assessment agenda?
What could you and/or NILOA do to overcome these challenges?
![Page 5: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/5.jpg)
OverviewOverview
The current context for assessment
The NILOA agenda: activities and products
Questions to ponder
![Page 6: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/6.jpg)
ContextContext
Global Competitiveness in Degree Attainment
The New Majority and Demographic Gaps
Questionable Levels of Student Performance
In a Most Challenging Fiscal Environment …
We Need Higher Levels of Student We Need Higher Levels of Student Attainment at an Affordable CostAttainment at an Affordable Cost
![Page 7: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/7.jpg)
Assessment 2010Assessment 2010
Greater emphasis on student learning outcomes and evidence that student performance measures up
![Page 8: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/8.jpg)
Assessment 2010Assessment 2010
Greater emphasis on student learning outcomes and evidence that student performance measures up
Demands for comparative measuresIncreased calls for transparency ---
public disclosure of student and institutional performance
Assessment “technology” has improved markedly, but still is insufficient to document learning outcomes most institutions claim
![Page 9: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/9.jpg)
Far too little is known about assessment practices on campuses around the country
![Page 10: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/10.jpg)
NILOA’s mission is to document SLO assessment work, identify and disseminate best practices, and support institutions in their assessment efforts
www.learningoutcomesassessment.orgwww.learningoutcomesassessment.org
![Page 11: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/11.jpg)
FUNDERSFUNDERSLumina Foundation for EducationLumina Foundation for Education
Carnegie Corporation of New YorkCarnegie Corporation of New York
The Teagle FoundationThe Teagle Foundation
NATIONAL ADVISORY PANELNATIONAL ADVISORY PANEL
![Page 12: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/12.jpg)
NILOA ActivitiesNILOA Activities
“Go to” Web sitewww.learningoutcomesassessment.orgwww.learningoutcomesassessment.org
![Page 13: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/13.jpg)
NILOA ActivitiesNILOA Activities“Go to” Web sitewww.learningoutcomesassessment.orgwww.learningoutcomesassessment.orgNational surveys
![Page 14: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/14.jpg)
Survey asked chief academic officers at all accredited 2 & 4 year colleges and universities about their assessment practices.
Survey report “More Than You Think, Less Than We Need” released October, 2009
![Page 15: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/15.jpg)
Report Summary Report Summary
1.1. PerhapsPerhaps more assessment underway than some acknowledge or wish to believe
2. More attention needed to using and reporting assessment results
3. Involving faculty is a major challenge
4. More investment likely needed to move from data to improvement
![Page 16: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/16.jpg)
According to Provosts, what is the driving force for assessment?
a. Institutional Commitment to Improvement
b. Accreditation
c. Faculty & Staff Interest
d. Governing Board Mandate
e. None of the above
“high importance” 85% Regional
80% Specialized
![Page 17: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/17.jpg)
Report Summary Report Summary
1.1. PerhapsPerhaps more assessment underway than some acknowledge or wish to believe
2. More attention needed to using and reporting assessment results
3. Involving faculty is a major challenge
4. More investment likely needed to move from data to improvement
5. Accreditation is a major force shaping assessment
![Page 18: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/18.jpg)
Assessment driven more by accreditation and commitment to improve than external pressures
from government or employers
![Page 19: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/19.jpg)
Accreditation tops the list for uses of assessment data
![Page 20: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/20.jpg)
Regional accreditors cite deficiencies in student learning outcomes assessment with greater frequency Middle States - 2/3 of institutions have follow-up; number one reason being assessmentNEASC - 80% of institutions asked for follow-up on student learning outcomes assessmentHLC - 7 out of 10 institutions are being monitored; the vast majority for student learning outcomes assessment.
![Page 21: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/21.jpg)
Attention to transparency in student learning outcomes assessment is growing.
Middle States, HLC, SACS consider transparency an integrity issue
NEASC increased requests for “candor” by asking for student learning goals and requirements to be published so that they are “readily available”
WASC-ACCJC ask every school to publish their student learning outcomes as eligibility requirement
![Page 22: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/22.jpg)
![Page 23: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/23.jpg)
![Page 24: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/24.jpg)
NILOA 2010 Program-Level SurveyNILOA 2010 Program-Level Survey
All accredited, undergraduate degree-granting 2- and 4-year public, private, and for-profit institutions in the US (n=2,678)
Provosts asked to identify 3-4 program areas to respond
![Page 25: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/25.jpg)
NILOA 2010 Program-Level NILOA 2010 Program-Level QuestionnaireQuestionnaire
Preview survey at: www.niloasurvey.orgwww.niloasurvey.org
![Page 26: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/26.jpg)
NILOA ActivitiesNILOA Activities
“Go to” Web site www.learningoutcomesassessment.orgwww.learningoutcomesassessment.org
National surveys
Web scans: 725 institutions
![Page 27: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/27.jpg)
Exploring the Landscape: What Institutions Post on Their Websites About Student Learning Outcomes Assessment Activities
Natasha Jankowski & Julia Makela
Campuses report doing more assessment than they make accessible on their institutional websites. The typical institution shows only an average of two assessment activities. When information is posted, it is more often intended for internal institutional audiences
![Page 28: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/28.jpg)
Web Scan
Guiding Questions
What does the institution display on its website regarding student learning outcomes assessment?
On which web pages is information about assessment of student learning outcomes located?
![Page 29: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/29.jpg)
Say vs. ShowMore assessment activity was reported by chief academic officers than was available on institution websites
![Page 30: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/30.jpg)
NILOA ActivitiesNILOA Activities
“Go to” Web site
www.learningoutcomesassessment.orgwww.learningoutcomesassessment.orgNational surveysWeb scans: 725 institutionsOccasional papers & products
![Page 31: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/31.jpg)
Occasional Paper #1
Assessment, Accountability, and Improvement
Peter T. Ewell
Assessments of what students learn during college are typically used for either improvement or accountability, and occasionally both. Yet, since the early days of the “assessment movement” in the US, these two purposes of outcomes assessment have not rested comfortably together.
www.learningoutcomeassessment.org/OccasionalPapers.htm9e
![Page 32: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/32.jpg)
Continuous Continuous ImprovementImprovement
AccountabilityAccountability
Strategic dimensions
Purpose Formative (improvement) Summative (judgment)
Orientation Internal External
Motivation Engagement Compliance
Implementation
Instrumentation Multiple/triangulation Standardized
Nature of evidence Quantitative and qualitative Quantitative
Reference points Over time, comparative, established goal
Comparative or fixed standard
Communication of results
Multiple internal channels Public communication, media
Use of results Multiple feedback loops Reporting
Two Paradigms of AssessmentTwo Paradigms of Assessment
Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco.
![Page 33: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/33.jpg)
Looking Back: What’s Been Looking Back: What’s Been Accomplished?Accomplished?Assessment Seen as LegitimateGoals for Learning EstablishedA “Semi-Profession” for
AssessmentMuch Better Instruments and
Methods
![Page 34: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/34.jpg)
Looking Back: What Remains to be Looking Back: What Remains to be Done?Done?Authentic Faculty OwnershipAssessment Still an “Add-On”Use of Information for
Improvement is UnderdevelopedSincere Institutional Engagement
with Accreditors in Assessment
![Page 35: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/35.jpg)
Occasional Paper #2
Three Promising Alternatives for Assessing College Students’ Knowledge and Skills
Banta, Griffin, Flateby & Kahn Of the various ways to assess student learning outcomes, many faculty members prefer what are called “authentic” approaches that document student performance during or at the end of a course or program of study.
www.learningoutcomeassessment.org/OccasionalPapers.htm
![Page 36: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/36.jpg)
Occasional Paper #3
Connecting the Dots Between Learning and Resources
Jane V. WellmanWith all the talk about the need for more accountability, surprisingly little is known about what kind of resources an institution needs in order to produce a given level of student attainment.
www.learningoutcomeassessment.org/OccasionalPapers.htm
![Page 37: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/37.jpg)
Wellman’s Conclusions Wellman’s Conclusions
Intentionality matters as much or more than money alone
Spending on instruction and student services pays off in learning, retention and graduation
Excess units cost institutions money, cost students in time and money, and do not get students to the finish line
![Page 38: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/38.jpg)
Occasional Paper #4
Opening Doors for Faculty Involvement in Assessment Pat HutchingsThe key to effectively using assessment data to improve student learning is to engage faculty in meaningful ways in this critical activity. This paper discusses the challenges inherent in and opportunities for doing so.
www.learningoutcomeassessment.org/OccasionalPapers.htm
![Page 39: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/39.jpg)
Occasional Paper #5
Valuing Assessment: Cost-Benefit ConsiderationsRandy Swing & Christopher CooganHow can a campus know when enough spending is really enough? The authors systematically unpack what needs to be taken into account when allocating resources to the assessment of student learning outcomes.
www.learningoutcomeassessment.org/OccasionalPapers.htm
![Page 40: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/40.jpg)
NILOA ActivitiesNILOA Activities
“Go to” Web site www.learningoutcomesassessment.orgwww.learningoutcomesassessment.org
National surveys Web scans: 725 institutionsOccasional papers & productsInterviews & focus groups with key
actors: -- AAC&U -- ACE -- AIR
![Page 41: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/41.jpg)
Perspectives from Campus Leaders on the Current State of Student Learning Outcomes Assessment
Jillian KinzieThis paper highlights lessons from four focus group sessions with campus leaders--presidents, provosts, academic deans and directors of institutional research from a variety of two- and four-year institutions-- regarding their perspectives on the state of learning assessment practices on their campuses.
Forthcoming Fall 2010
![Page 42: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/42.jpg)
NILOA ActivitiesNILOA Activities
“Go to” Web site www.learningoutcomesassessment.orgwww.learningoutcomesassessment.org
National surveys Web scans: 725 institutions Occasional papers & productsInterviews & focus groups with key
actors: -- AAC&U -- ACE -- AIR -- AccreditorsAccreditors
![Page 43: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/43.jpg)
Occasional Paper #6
www.learningoutcomeassessment.org/OccasionalPapers.htm
Regional Accreditation and Student Learning Outcomes Assessment: Mapping the Territory
Staci Provezis
This paper examines the policies and procedures at each of the seven regional accreditation organizations, as they relate to student learning outcomes assessment.
![Page 44: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/44.jpg)
Connecting State Policies on Assessment with Institutional Assessment Activity
Peter Ewell, Natasha Jankowski & Staci Provezis
Two national surveys—one at the state level and one at the institutional level—show that state policies matter to student learning outcome assessment activities at both public and private institutions. http://www.learningoutcomesassessment.org/NILOAstatestudy2010.htm
![Page 45: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/45.jpg)
Eight “assessment active states:
Georgia Rhode Island
Kentucky South Dakota
Minnesota Tennessee
Oklahoma West Virginia
![Page 46: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/46.jpg)
Institutions in Assessment-Active States:Use general knowledge/ skills measures (CLA, CAAP, MAPP) 58% vs 38%. Use valid samples with national surveys (83% vs 58%) Use employer surveys (76% vs 62%) and employer interviews (57% vs 47%) Have common set of student learning outcomes (83% vs 76%) No differences re public and privates, transparency
![Page 47: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/47.jpg)
Measuring Quality in Higher Education Measuring Quality in Higher Education (Vic Borden & Brandi Kernel, 2010)
Web-based inventory hosted by AIR of assessment resources. Key words can be used to search the four categories: instruments (examinations, surveys, questionnaires,
etc.); software tools and platforms; benchmarking systems and data resources; projects, initiatives and services.
http://applications.airweb.org/surveys/Default.aspx
![Page 48: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/48.jpg)
NILOA Products Coming SoonNILOA Products Coming SoonAssessment and accountability in
community colleges (Seybert & Ewell)
What the field needs from student affairs in assessment (Schuh & Gansemer-Topf)
The role of institutional research in assessment (Volkwein)
From gathering to using assessment results: Lessons from the Wabash National Longitudinal Study (Blaich & Wise)
![Page 49: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/49.jpg)
Ponder this:
1. What is driving assessment of learning outcomes on your campus?
2. How effectively does your institution assess student learning outcomes?
3. What is needed to take assessment to the next level to further both assessment and improvement?
4. How available are your assessment results??
Where are the data posted? Are the data password protected?
![Page 50: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/50.jpg)
Would it help if we agreed on common student learning outcomes (SLOs) for all undergraduates and expressed SLOs in the same language?
Should an evidence guide be developed that institutions and accreditation teams can use to standardize and examine institutional assessment activities?
Should a template be developed that institutions can use to make public information about SLOs, like VSA, but focused on learning?
![Page 51: George Kuh SAIR New Orleans, LA September 26, 2010](https://reader030.fdocuments.net/reader030/viewer/2022032606/56812d5f550346895d926bb8/html5/thumbnails/51.jpg)
DiscussionDiscussion
What are the obstacles that must be addressed to advance your assessment agenda?
What could you and/or NILOA do to overcome these challenges?