I Don’t Do Research . . . But
description
Transcript of I Don’t Do Research . . . But
I Don’t Do Research . . . But
Steve HillerDirector, Assessment and PlanningUniversity of Washington Libraries
I Do Use Research Methods as Part of Our Assessment and Planning Program for:
• Understanding our user communities– How they work – Their library and information needs– How we can make them successful
• Organizational improvement– Improving organizational performance, effectiveness and
efficiency– Delivering services and programs that make a difference
AssessmentMore than Numbers
Library assessment is a structured process: • To learn about our communities• To respond to the needs of our users• To improve our programs and services• To support the goals of the communities
Why Assess?
• Accountability and justification; demonstrating value• Improvement of services • Comparisons with others• Identification of changing patterns• Marketing and promotion • Opportunity to tell our own story• Using data, not assumptions, to make decisions
–Assumicide!
What’s Driving the Agenda• Environmental Changes
– Exploding growth in use and applications of technology– Increased customer expectations for services, including
quality and responsiveness– “Competition” from other sources
• Budgetary Constraints/Reductions– Justification for spending $$$ on libraries– Increasing competition for resources– Budget reductions and reallocations
• Demonstrating Value– Accountability– How do we enable those in our community to succeed
Traditional Library Measures: InputsFocus on how big/how much• Budget (staff, collections, operations)• Staff size• Collection size• Facilities• Other related infrastructure (hours, seats, computers)• Size of user communities and programs ARL “Investment Index” measures inputs related to
expenditures and staff numbers
Traditional Library Measures: Outputs
Focus on usage• Collections (print, electronic, ILL)• Reference services• Facilities (gate counts)• Instruction sessions• Discovery and retrieval• Other Web sessionsMay indicate if “inputs” are used, but doesn’t tell us
what users were able to accomplish as a result.
These Are Self-Reported Statistics Too!
The Challenge for Libraries
• Traditional statistics are no longer sufficient– Emphasize inputs/outputs – how big and how many– Do not tell the library’s or customers’ story– May not align with organizational goals and plans– Do not measure service quality or library impact
• Need better outcome measures that demonstrate difference the library makes and value it adds– To the individual, community and the organization
• “No longer what makes a good library but how much good does the library do” (Peter Brophy)
Assessing and Demonstrating the Library Contribution to the Institutional Mission
• The library’s contribution to learning and research– Student learning (accreditation driven)– Externally funded research and scholarship
• Value of the library to the community– Information resources/collections – Library as place– Current services
• Changes in library and information needs and use• Organizational performance and effectiveness• Collaborations
Good Assessment Starts Before You Begin . . . Some Questions to Ask
• Define the question – What’s Important– What do you need to know, why, and when
• How will you use the information/results• Where/how will you get the information
– Methods used– Existing data– New data (where or who will you get it from)
• How will you analyze the information• Who will act upon the findings
Four Useful Assessment Assumptions
• Your problem/issue is not as unique as you think• You have more data/information than you think• You need less data/information than you think• There are useful methods that are much simpler than
you think
Adapted from Douglas Hubbard, “How to Measure Anything” (2007)
Documenting Library Performance and Impact
• Common library assessment methods– Surveys (satisfaction, needs, importance)– Usage and other library statistics– Qualitative information (interviews, focus groups, etc.)
• Other statistical data– Institutional – Comparator (ARL, ACRL, peer groups, customized)– Government (NCES)
• Collaborations• Value, Impact and Return on Investment
– Lib-Value (IMLS grant to measure value and return on investment in academic libraries)
Choosing the Right Assessment Method
Criteria• Utility• Relevance/Importance• Stakeholder needs• Measurability• Cost• Timely
Tools• Usage data• Surveys (local &
standardized)• Standardized tests• Performance
assessments• Qualitative methods• Rubrics
Presenting Assessment Findings
• Make sure data/results are:– Timely– Understandable– Usable
• Identify important findings/key results– What’s important to know– What’s actionable
• Present key/important results to:– Library administration/institutional administration– Library staff– Other libraries/interested parties/stakeholders
Success with Assessment
• Use multiple assessment methods• Mine/repurpose existing data • Invest in staff training and resources• Focus on the customer and community • Learn from our users• Partner with other campus programs and institutions• Present assessment information so that it is
understandable and usable
A Skeptical View of Metrics
Association of Research Libraries (ARL) and Library Assessment
ARL has played a major role in advancing assessment in academic libraries through:
• ARL Statistics• New measures and standardized methods
– LibQUAL+® user survey, MINES for libraries • Individual library consulting
– Effective, Sustainable and Practical Assessment (ESP) • 42 libraries visited 2005-2010 to evaluate assessment needs and programs
• Library Assessment Conference
ESP Insights • Uncertainty on how to establish and sustain assessment• Staff lack essential assessment/data analysis skills and
knowledge• Lack of focus and assessment priorities; tenuous link to
planning and decision making • Underutilization of campus assessment resources• More data collection than data utilization• Overreliance on surveys for user input• Organizational issues play a significant role in
sustainable assessment
From Institutional Based Assessment to a Community of Practice
THE NEED• Bring together library folks interested in assessment • Focus on effective and practical assessment• Establish an ongoing venue for presentation of library
assessment issues, activities and results• Build a continuing education component (workshops)• Make it fun!AN ANSWER• Library Assessment Conference
– Organized by ARL, U.Va and UW– Biennial conference first held in 2006
Library Assessment Conference Facts
YearConference Site
# Proposals # ConferencePresentations
# Workshops Total registrants
2006Charlottesville
80 paper/panel15 poster
38 paper/panel20 posters3 plenary
Proceedings – 3 lbs.
3 half-day(each repeated)120 participants
220 30 turnaways
2008Seattle
95 paper/panel38 poster
59 paper/panel43 posters5 plenary
Proceedings - 4 lbs.
6 half-day160 participants
375 15 turnaways
2010Baltimore
154 papers55 posters
63 papers 80 posters5 plenary
2 full-day4 half-day
160 participants
47520 turnaways
Using Assessment for Results at the University of Washington or How We Contribute to User
Success
• Assessment program established in 1991– Focus on user needs– Information seeking behavior and use– Patterns of library use– Library contribution to learning and research– User satisfaction with services, collections, overall
• Increasingly tied to strategic goals and priorities• Provides data to improve programs and services and to
demonstrate the library contribution to user success
University of Washington Libraries Assessment Methods Used
• Large scale user surveys every 3 years since 1992 (“triennial survey”)
• In-library use surveys every 3 years beginning 1993• Focus groups/Interviews • User centered design• Observation (guided and non-obtrusive)• Usability• Usage statistics/data miningInformation about assessment program available at:
http://www.lib.washington.edu/assessment/
UW Libraries Triennial Survey
• Started in 1992 with paper; web-based began in 2004• Survey designed by library staff and asks about needs,
importance, satisfaction, use patterns, and impact (comments valuable too)
• Survey all faculty and a sample of students• Survey for each group is different and survey questions
may change over time (although a core set remains the same over time and between groups)
• Survey can help measure effectiveness of existing programs and provide direction for future ones
Longest running cyclical survey in academic libraries
Strategic Priorities 2007-2010
• Expand digital and physical delivery services• Enhance library contributions to research productivity• Raise visibility and effectiveness of librarian liaisons • Inform UW researchers/authors about good scholarly
communications practices• Strengthen library role in undergraduate learning• Reshape library spaces to enhance user experiences• Ensure content needed is accessible and deliverable • Implement new models of service
What We Did 2007-2009
• Began pull and scan service; harmonized ILL• Implemented UW WorldCat as primary access point• Articulated service expectations for librarian liaisons• Expanded scholarly communication efforts• Began revisioning process for undergrad library space• Brought in consultant on teaching and learning• Participated in ARL Library Scorecard Pilot (2009-)• 2009 - 12% budget reduction
– Closed several branch libraries; cut hours; cut 29 positions in 2009– Reduced collections budget; cut serial subscriptions
Libraries 2010 Triennial Survey Highlights
• Record number of faculty and graduate student responses • Satisfaction ratings highest ever for faculty and grads;
slightly lower for undergrads (at all 3 campuses)• Library contributions to teaching, learning, research and
overall success rated very high by faculty/grad students• Substantial increase in use and satisfaction with library
delivery services (ILL, pull and scan)• Online access to and delivery of scholarly information,
especially journals, are driving research and scholarship
UW Libraries Triennial Survey Number of Respondents and Response Rate 1992-2010
http://www.lib.washington.edu/assessment/
2010 2007 2004 2001 1998 1995 1992
Faculty 163439%
145536%
156040%
134536%
150340%
135931%
110828%
Grad/ProfStudents(UWS)
64032%
58033%
62740%
59740%
45746%
40941%
56056%
Undergrads(UWS)
36516%
46720%
50225%
49725%
78739%
46323%
40741%
Overall Satisfaction by Group1995-2010
Faculty 4.254.33 4.33
4.44
4.56Faculty 4.63
Undergrad 3.97
3.99
4.22
4.324.36
Undergrad 4.25Grad 4.18
4.11
4.26
4.34 4.36
Grad 4.43
3.8
3.9
4
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
3.8
3.9
4
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
1995 1998 2001 2004 2007 2010
Library Services and Resources: Overall Importance to Work by Group
(Scale of 1 “Not Important” to 5 “Very Important)
3
3.5
4
4.5
5
Collections Discovery tools Info services and instruction
Physical spaces
Undergrad Grad Faculty
UW Libraries 2010 Triennial SurveyLibraries Contribution to:
(Scale of 1 “Minor” to 5 “Major”)
Mean scores%= those marking 4 or 5
Faculty1634 surveys(39% response)
Graduate Students680 surveys(32% response)
Keeping current in your field 92% 4.67 90% 4.53
Finding information in related fields or new areas
90% 4.56 91% 4.57
Being a more productive researcher 92% 4.63 93% 4.64
Enriching student learning experiencesOverall academic success
77% 4.1892% 4.60
Making more efficient use of your time 87% 4.46 80% 4.21
Importance of Books & Journals by Academic Area (2010, Faculty, Scale of 1 “not important” to 5 “very important)
Books
Journals<1990
Journals>1990
3
3.25
3.5
3.75
4
4.25
4.5
4.75
5
Health Sciences Sci-Eng-Env Hum-Soc Science
Books Journals<1990 Journals >1990
Importance of Books & Older Journals by School
Dentistry Medicine Nursing Pharmacy Public Health Social work3
3.25
3.5
3.75
4
4.25
4.5
Books Journals<1990
Services Satisfaction and Visibility by Group 2007/2010
2010Satisfaction
2010Visibility
2007Satisfaction
2007Visibility
Instruction - S is up, V is downFacultyGradUndergrad (usefulness)
4.454.203.36
34%42%44%
4.273.803.21
52%55%49%
Staff assistance - S is up, V is the sameFacultyGradUndergrad
4.484.304.04
75%75%75%
4.424.063.94
76%75%69%
ILL Books and Journals - S is up, V is upFaculty Grad Undergrad
4.444.454.06
77%81%57%
4.254.193.90
63%61%46%
Remote access to collect/services - ALL GOOD!FacultyGradUndergrad
4.644.654.20
90%90%89%
Subject Librarian Visibility and Satisfaction By Faculty College/School (Balanced Scorecard Metric)
4.90
4.30
4.48
3.85
4.73
3.924.00
4.76
4.00
4.334.28
4.63
4.26
4.88
4.724.71
4.34
3.00
4.25
30% 67%
Satisfaction
Visibility
Built Environments
Education
Business
EngineeringMedicine
I School
PharmacyDentistry
Nursing
Public Affairs
Public Health
Social Work
Environment
Fine Arts
Humanities
Social Sciences
Sciences
Use Patterns: Frequency of In-Library Visits 1998-2010 (Weekly or more often)
Faculty
Faculty
Grad
Grad
Undergrad
Undergrad
20%
30%
40%
50%
60%
70%
80%
1998 2001 2004 2007 2010
Undergraduate Overall Satisfaction 2007-2010
UWT
UWS
UWB
4
4.2
4.4
4.6
4.8
5
4
4.2
4.4
4.6
4.8
5
2007 2010
Undergrad Satisfaction With Facilities
Group work Quiet areas Furniture Security Light & Temperature3
3.25
3.5
3.75
4
4.25
4.5
Seattle Bothell Tacoma
80% of the 400 comments from UWS Undergrads Dealt with Space and Hours
• Open is one thing, space and available computers / tables with laptop plug-ins is whole other issue
• More seating or computer areas, engineer a reduced noise level in Odegaard.
• 1. More space between the computers 2. More quiet study areas 3. Spaces to eat, drink and take breaks
• Suzzallo-Allen. Quiet, neat, clean, cool, beautiful, access to everything I need. Ode, on the other hand . . .
What People Do in Libraries by Group 20082008 In-Library Use Survey: 73% UG, 22% Grad, 5% Faculty
Ask for help Look for ma-terial
Work alone Work in groups Use Lib Computer
Use Own Computer
0%
10%
20%
30%
40%
50%
60%
70%
Undergrad 2008 Grad 2008 Faculty/Staff 2008
Other Relevant Data
• During the past five years at UWS:• Total number weekly hours libraries open dropped 26% • Number of library seats dropped 3% • Enrolment increased by 6%• Gate counts increased by 6% or 250,000 more entrants
How UW Libraries Has Used AssessmentA Few Examples
• Extend hours in Undergraduate Library (24/5.5)• Create more diversified student learning spaces• Enhance usability of discovery tools and website• Provide standardized service training for all staff• Review and restructure librarian liaison program • Consolidate and merge branch libraries• Change/reallocate collections budget• Change/reallocate staffing• Support budget requests to University
Integrated Organizational Performance Model The Balanced Scorecard
• A model for measuring organizational performance developed in the 1990’s by Kaplan and Norton that:– Helps identify the important statistics– Helps ensure a proper balance– Organizes multiple statistics into an intelligible framework
• Clarifies and communicates the organization’s vision• Provides a structured metrics framework for aligning
assessment with strategic priorities & evaluating progress
• ARL Library Scorecard Pilot in 2009-10 with 4 libraries– Johns Hopkins, McMaster, Virginia, Washington
Goals of the ARL Pilot
• Evaluate the Balanced Scorecard a suitable performance model for academic research libraries
• Value as structured process to better integrate and strengthen strategy, planning and assessment
• Encourage cross-library collaboration• Review objectives and measures for commonalities
between libraries
Closing the Loop: Success with Assessment
• Assess what is important• Keep expectations reasonable and achievable • Use multiple assessment methods; corroborate• Mine/repurpose existing data• Focus on users; how they work, find & use information • Use the data to improve and add customer value• Keep staff, customers and stakeholders involved and
informed
Eye to the Future
Measuring performance is an exercise in measuring the past. It is the use of that data to plan an improved future that is all important. Peter Brophy
• Data trends can inform the future• Strategic planning can frame the future• Organizational performance models can align ongoing
operations with future aspirations• Understanding how customers work, how that work is
changing, and ways we can make customers and institutions success are key to the future of libraries
In Conclusion Can You Answer These Questions?
• What do we know about our communities and customers to provide services and resources to make them successful?
• How do we measure the effectiveness of our services, programs and resources from the customer perspective?
• What do our stakeholders need to know in order to provide the resources needed for a successful library?