Lessons Learned from the Assessment Practices of a Sample of Libraries PRESENTED BY: Laura...
-
Upload
giles-caldwell -
Category
Documents
-
view
221 -
download
4
Transcript of Lessons Learned from the Assessment Practices of a Sample of Libraries PRESENTED BY: Laura...
Lessons Learned from the Assessment Practices of a Sample of Libraries
P R E S E N T E D BY: L a u r a G i l -Tr e j o , M P H & M A
S o c i a l S c i e n c e Re s e a r c h C e n t e r
P R E S E N T E D T O : T h e S t a t e w i d e C a l i f o r n i a E l e c t r o n i c L i b r a r y C o n s o r t i u m
Demonstrating the Value of Academic Libraries: Challenges and
Opportunities
Culture of Assessment
Growing recognition of its importance in higher education
What is the value of education?
What is the library’s place in contributing to that value?
WASC Accreditation
Creation of external push for assessment observed across all campuses
Variation exists in the extent to which libraries are being pushed to the same extent that academic departments are
Culture of Assessment Within Libraries
There are diverse attitudes toward assessment within and across campuses
Among those surveyed, there is a belief that the culture of assessment has become more positive as administrators have become convinced of its importance.
Culture of Assessment
Some believe it is important because: They have to do it: it has become critical for survival or
someone is asking for the information directly
They genuinely see assessment as part of ensuring student success
“…we’re very student focused. We want to ensure that [they] are getting the educational resources they need, and that’s the only way you can do that is through some kind of assessment.”
Culture of Assessment
Those who are less enthusiastic or hesitant because they:
fear change fear what the evaluation will tell them
“…but also the people who you say the word and immediately you can see the strain on their face and the stress because they think it’s going to be some sort of negative reaction to them personally. They don’t want to be evaluated personally.”
think it is unnecessary don’t have the resource, skill, and/or time to do it
Culture of Assessment
Factors that shape attitude toward assessment Perceived meaningfulness of assessment activities
Consequences for not doing assessment activities
Not knowing what is meant by assessment exactly
“I don’t think anyone in the library questions the importance of [assessment]. Resistance and challenge comes from ignorance, and we don’t have a magic bullet to close the loop. It is never an easy solution—it is always complex and iterative.”
Approaches to Assessment
There is also diversity as to how assessment is approached.
Those who:
Get it done Get it done, and continue look for better ways to get
it done Want to get it done, but don’t know how Are not sure it can be done rigorously so avoid it all
together.
Definitions of Assessment
Main Themes Lack of a formal definition shared by libraries
Use outside entity standards to inform definitions
Difference between what should be and what is
“…we want to know that the services we offer in the form of collections and our reference services and our interlibrary loan…are helping students to learn and ultimately to graduate. But that’s always been really difficult to make those connections. We end up sticking with satisfaction…we really don’t have a good assessment program.”
Discrepancy/misunderstanding between and within libraries as to what is considered assessment “Within the library there is not agreement as to what is assessment, with one
librarian saying all data collection efforts are assessment and another indicating that assessment is more confined to ‘outcomes.’”
Definition of Assessment
A tool to fit various needs
Encompassing a variety of activities
“I wouldn’t tie to any one form of assessment. Our definition is broad on purpose because we do all kinds of assessment in the building, so it’s any feedback we can get for a service we’ve provided…to both meet the students’ needs and do all the stuff…as well as assessing our services. But we see assessment fulfilling both of those needs.”
Approaches to Assessment
Assessment is happening at multiple layers University Library
Some as part of university-efforts Some coordinated library-wide Some individual
The majority of libraries do not have repositories or keep track of their assessment activities on one hand, while others are publishing the results of their efforts on the other
Potential List of Services Being Assessed
The following services were identified:
ServicesCollectionsE-Resources/ Digital ResourcesInformation Literacy Instruction (unit-bearing, one shot, multi-session, online tutorials, individualized)Study Rooms
Reference Desk
Interlibrary Loan (ILL)/ Get it Now
Integrated Course Design
Laptop Loan Program
Computers/Scanners/Printing/Wireless Access
Physical Space
Faculty/Instructor Services
Archives & Special Collections
Exhibits
Community Outreach
WebsiteAffordable Learning Initiative
Assessment Activities
Service/Resource Use
Did Service/Resou
rce Get Delivered as
Intended
Did Student Receive the Service as Intended?
What Impact Did the
Service/Resource Have on the Student Immediately
Or in the Short Term?
What Impact Did it Have on the Student in
the Long Term?
Does Student Demonstrate
SLO
Assessment as Output
Most common
Usage statistics (quantitative) Question: To what extent are our services being used?
Circulation, E-Resources/digital resources, ILL, Reference, Laptop loan program, Study room reservation, Information literacy instruction
Mandated for several service points Critical to assessment
If done over time using the same definition and data collection systems, it can be used to establish change over time or measure the impact of an outreach program*
Introduction to rapid ILL
Change in the way data is collected/use defined
Assessment as Process
Very common Question: Are our services being delivered/received as
intended? Services: IL instruction, study room, circulation, ILL, etc.
Question: Is the library as a whole or its various components giving the student what he/she achieve SLO.
Most common is the satisfaction survey (LibQual, NSSE, and other independent paper/pencil and web surveys)
Comment boxes where students can provide feedbackFocused group interviewsOther qualitative methods Samples of varying degree of representativeness from
convenience to attempts at randomly samplingBoth faculty and students included in samples
Very often non-users are not used as sample
Assessment as Outcome/Impact
Least common , but viewed as most critical
Some have made steps in that direction with varying degrees of rigor
Question: Does Student X demonstrate some skill at the end of an IL session/workshop/course? Posttest only design (without comparison group)
Rubrics applied to student samples (without control group). Typically students sampled from X number of sessions out of Y
total number of sessions Or, pre- an and posttest after online IL modules
Looking at percent correct on some measure Some efforts to incorporate comparison groups have been
made
Assessment as Outcome/Impact
Question: What impact has this service had on students?
How did it impact you? If you didn’t have the library, how would your academic
experience have been altered? For the better? For the worse?
o Surveyo Anecdotallyo Focused group interviews (student and faculty samples)o One on one interviews
Assessment as Outcome/Impact
Challenges experienced:
Difficult in obtaining control groups
Control and condition groups are typically not randomly assigned
Some tutorials require students to pass before producing a certificate of completion
Rubrics are difficult to develop and time consuming to apply to student products
Challenges finding instructors to participate
For surveys, relying of self reports
Does not incorporate all library services
Assessment as Impact/Output: A Simplified Model of Student Learning
Precollege Student Performance
Support for College Outside of Campus
Quality in Instruction
Engagement in High-Impact
Practices
Student Success
Where does the library fit
in?
Where Does the Library Fit In?
Student Success
Library Services and Resources
Mechanisms
Assessment as Outcome/Impact
Other potential methodology: Merging student outcome data with use data
Can factor out confounding factors Can examine moderating factors Becomes stronger when you start with a cohort and add a
longitudinal component to it (tracking students over time) Allows you to estimate the value of the library on student
outcomes. Permits for large sample sizes Avoid costs of doing surveys The down side?
Start-up is potentially TIME-CONSUMING and EXPENSIVE Relies on statistical expertise that many librarians do not
possess Safeguards must be taken to protect student confidentiality
Merging Student Outcome and Usage Statistics
Other downsides:
Data collected may not gets at actually usage
Sometimes usage gets missed Group study rooms Accessing databases inside the library
Checking out a book
Doesn’t tell us how use is linked to student success
Even if a correlation is present, it doesn’t tell us anythning about the mechanisms.
Merging Student Outcome and Usage Statistics
Being piloted at two universities within the CSU.Already done at University of Minnesota Twin Towers
Potential problem lies with the accuracy of the data and assumptions that are made
Should validate student-use data as collected through library systems with self-reported data to make sure that it approximates reality; otherwise these data are useless
Relies on knowing which student demographic data to include
Relies on being able to bring non-users into the data file Should conduct follow up qualitative studies to focus on
mechanisms Also relies heavily on statistical expertise
Good Assessment Plans:
Leveraging what standards we do possess as a starting point (WASC, ACRL)
Leverage existing campuses resources
Almost all campus respondents indicating having an office of institutional research
IT departments are always helpful Each other
Publications Best Practices/Good models (e.g., assessment team)
Must be meaningfulMust not add to existing workloadsMethodological rigor must match assessment question
Do not let a method be your master
A Tool Box Approach?
Starts with assessment question driving the effort:Who is using our services?Who is not using our services and why?How are our services being used?Are users satisfied with the services they receive? How do we best deliver services to users? Does this
differ by user?Do students have basic competency in some skill that is
required of them at the end of IL Instruction? Does exposure to our services/resources have a
measurable impact on users?
A Tool Box Approach
Selected: Does exposure to our services/resources have a measurable impact on students? Quantitative data Qualitative data Both
A Tool Box Approach
Selected: Does exposure to our services/resources have a measurable impact on students? Selected: Both quantitative and qualitative
What impact/outcome would you like to document? GPA Persistence Time to graduation Faculty scholarship Instruction of faculty Self efficacy Academic achievement (other) Etc….
A Tool Box Approach
Selected: Does exposure to our services/resources have a measurable impact on students? Type of Data Selected: Both quantitative and qualitative
Impact/outcome selected: One Year Retention Potential methods
• Quantitative: Survey : Two time points vs. one time pointCorrelating usage statistics with student outcome statistics
• Samples: First time freshmen • Qualitative
Focused group interviewsKey informant/face-to-face interviews
• Samples: First time freshmen users and non users• Other samples: Student advisors or other educators who work
directly with students.
A Tool Box Approach
For Every Method… Limitations of method How to set up your assessment
Sample survey/interview guides Different sampling approaches Examples of what data files should like Sample data analysis How to report your data
Highlighting new methods of assessing similar questions on library campuses both nationally and locally
Publications using similar methods Places to submit newly discovered methods that address similar
questions. Potentially: Online workshops or training for particular topics Potentially: a moderated community chat room Potentially: access to technical assistance
Reactions, Questions, or Ideas?
Contact Information
Laura Gil-TrejoDirector, Social Science Research CenterCalifornia State University, [email protected]