1 Covering Assessment in LIS Education and in the Profession Megan Oakleaf Peter Hernon Karin De...
-
Upload
lola-forrester -
Category
Documents
-
view
214 -
download
0
Transcript of 1 Covering Assessment in LIS Education and in the Profession Megan Oakleaf Peter Hernon Karin De...
1
Covering Assessment Covering Assessment in LIS Education and in LIS Education and
in the Professionin the Profession
Megan OakleafPeter Hernon
Karin De Jager
Library Assessment ConferenceAugust 2008
2
Panel OverviewPanel OverviewIntroduction
◦ Some examples of what LIS education is doing
◦ Assessment Student outcomes Student learning outcomes
The Assessment ToolkitAssessment in ContextA view from South Africa
Library Assessment Conference 2008 2
3
LIS EducationLIS EducationStatus of using student learning
outcomes should guide programsRole of research at master’s level
◦Research—Application of inquiry process
◦Evaluation—Library centric Examination of program/service for
summative/formative evaluation
◦Assessment—Connects libraries and broader organizations to stakeholder expectations and requirements
Library Assessment Conference 2008 3
4
Key Stakeholders behind AssessmentKey Stakeholders behind Assessment
Government◦Federal◦State
Accreditation◦Regional Accrediting Organizations◦Program “Accreditors”
Library Assessment Conference 2008 4
5
Critical IssuesCritical Issues
How do we build “research” as a most essential activity within LIS education?
How do we build research as a more essential activity among libraries and librarians?
Library Assessment Conference 2008 5
6
PartnershipsPartnerships
An institution
LIS schools/programs
Cross-disciplinary partners
Libr
ary
Cam
pus IR
Library Assessment Conference 2008 6
7
Other PartnershipsOther Partnerships
LIS Education
Libraries within the US
Key professional associations
CEWorkshopsSpeakingScholarship
ISSUES: How to address research and assessment?
Library Assessment Conference 2008 7
8
AGGREGATE STATISTICS ON GROUPS OF AGGREGATE STATISTICS ON GROUPS OF STUDENTS STUDENTS
GRADUATION RATES,GRADUATION RATES,RETENTION RATESRETENTION RATES
TRANSFER RATESTRANSFER RATESEMPLOYMENT RATES FOR A EMPLOYMENT RATES FOR A
GRADUATING CLASSGRADUATING CLASS
Assessment1. Accountability: Meeting institutional mission—
effectiveness and institutional fiscal efficiency
Library Assessment Conference 2008 8
9
Assessment Cycle for Student Assessment Cycle for Student Learning OutcomesLearning Outcomes
InstitutionalMissionVisionValues
Identify outcomes Interpret evidence
Planning (Assessment Plan)
Gather “evidence”
Use the results
Review outcomes
Library Assessment Conference 2008
LibQUAL+ is irrelevant 9
10
Student Learning Outcomes--Student Learning Outcomes--ExamplesExamples
ConceptualLeadership
Critical thinkingProblem solving
Information literacyGlobal citizen
Values (moral, etc.)
SkillsOral/written communication
Foreign language communicationTechnological sophisticationQuantitative reasoning ability
Other
Library Assessment Conference 2008 10
11
WHAT SHOULD STUDENTS LEARN (IN A PROGRAM)? WHAT SHOULD STUDENTS LEARN (IN A PROGRAM)?
HOW WELL ARE THEY LEARNING IT (IN THAT PROGRAM)? HOW WELL ARE THEY LEARNING IT (IN THAT PROGRAM)?
WHAT MEASURES AND PROCEDURES DOES THE INSTITUTION WHAT MEASURES AND PROCEDURES DOES THE INSTITUTION USE TO DETERMINE THAT IT IS EFFECTIVE?USE TO DETERMINE THAT IT IS EFFECTIVE? TO WHAT EXTENT DOES THE INSTITUTION OFFER EVIDENCE TO WHAT EXTENT DOES THE INSTITUTION OFFER EVIDENCE THAT DEMONSTRATES ITS EFFECTIVENESS TO THE PUBLIC?THAT DEMONSTRATES ITS EFFECTIVENESS TO THE PUBLIC?
WHAT DOES THE INSTITUTION PLAN TO DO WITH THIS WHAT DOES THE INSTITUTION PLAN TO DO WITH THIS EVIDENCE TO IMPROVE OUTCOMES? EVIDENCE TO IMPROVE OUTCOMES?
Assessment2. Educational quality and improvement (e.g., student learning)
Library Assessment Conference 2008 11
12
Student learning outcomes exist at the following levels
course
Program
Institutional
Use of rubrics
Peter Hernon 13
Direct MethodsDirect MethodsEmbedded course
assessment (performance on assignments, etc.; minute paper)
Portfolio assessmentPerformance
(internships, practicum, student teaching)
Professional jurors or evaluators
Capstone course/experience
Experimental research designs), with pre- and post-testing
Use of standardized tests
Think-aloud protocolDirected conversationVideotape/audiotape
evaluationAnalysis of
theses/dissertations/ senior papers (content
analysis, interviews, or oral defense)
Peter Hernon 14
Indirect MethodsIndirect Methods
Surveys (self-reporting) and self-assessments
Curriculum and syllabus evaluation
Exit interviewsObservation
Other
15
Simmons’ MLIP Leadership Simmons’ MLIP Leadership ModelModelThe curriculum and assessment
activities are guided by a leadership model, which was adapted from a model developed by the National Center for Healthcare Leadership. The model consists of twenty-five distinct leadership competencies in three broad areas: Transformation, Accomplishment, and People.
http://web.simmons.edu/~phdml/docs/phdmlip_models.pdf
16
Assessment in ContextAssessment in ContextLearning in context is authentic &
meaningful.◦ Students apply skills as they would in the real world.
Learning in context is active.◦ “Students construct meaning and knowledge: they
do not have meaning or knowledge handed to them in a book or lecture. Learning, then, is a process of students ‘making sense’ of how things fit together; factual and procedural knowledge is built along the way” (Shavelson & Baxter, 1996).
Learning in context is open-ended & acknowledges more that one right approach/answer (Shepard, 1996).
Library Assessment Conference 2008 16
17
Example LIS AssignmentExample LIS Assignment
Planning, Marketing, & Assessing Library Services
Assignment Tasks:Locate a new or recently revised library
service & a host librarianFor the service, develop:
◦ Project Management Plan◦ Marketing Plan◦ Assessment Plan
Present final plans to class & host librarian
Library Assessment Conference 2008 17
18
Library Service ExamplesLibrary Service Examples
Virtual/IM reference
Downloadable audio
Gaming programs Single service
pointsInformation
commonsPortals/blogs/wikisLibGuides
DigitizationOrientations &
outreachBook clubs &
summer reading programs
Cafes/coffee bars
Library Assessment Conference 2008 18
19
Assessment Plan OutlineAssessment Plan Outline Service goals & link to strategic plan Literature review Service outcomes Target audience Methods & tools for evidence collection
Recommendations for pilot assessment Analysis of evidence (data plan) How assessors will know the outcome has been
met Result scenarios & decision making indicators Recommendations for reporting Responsible parties Timeline
Library Assessment Conference 2008 19
20
Libraries Impacted (2007-2008)Libraries Impacted (2007-2008)Syracuse University Libraries
Lemoyne College Libraries
SUNY ESF Libraries
SUNY Cortland Libraries
SUNY Brockport Libraries
SUNY Binghamton Libraries
SUNY Upstate Medical University Libraries
Mid-York Library System (NY)
Cazenovia Public Library (NY)
Roanoke City Libraries (VA)
Loyola University Libraries
Cornell University Libraries
University of Utah Libraries
Enoch Pratt Free Library (MD)
Northwestern University Libraries
Supreme Court Library (NV)
Rockefeller University Libraries
UT – San Antonio Libraries
Brandeis University Libraries
26 NYC-area public school libraries
Wantagh Public Library (NY)
Brooklyn Public Library (NY)
Jervis Public Library (NY)
Fayetteville Free Library (NY)
Cazenovia Public Library (NY)
Cicero-North Syracuse HS Library
Fletcher Free Library (VT)
RIT Libraries
US Military Academy Library
Celebration School Library (FL)
Deschutes Public Libraries (OR)
Middlebury College Libraries (??)
University of New Hampshire Libraries
Regent University Libraries
Wake County Public Libraries (NC)
Whitesboro High School Library
Norwood-Norfolk Central HS Library
New England Law Library Consortium
YouthBuild Charter School Library (PA)
Wellesley College Libraries
Library Assessment Conference 2008 20
Onondaga County Public Libraries (NY)
University of Rochester Libraries
Mott Road Elementary School Library (NY)
Vogelson Public Library (NJ)
LeMoyne Elementary School Library (NY)
Groton Elementary School Library (NY)
Paine Memorial Library (NY)
Drew University Libraries
Broome County Public Library
New York University Libraries
Green Mountain Library Consortium
Oneida-Herkimer BOCES School Library System
Schoharie Free Public Library (??)
University of Virginia Libraries
Boston College Libraries
Oneida Castle Elementary School Library
Andrew J. Lanza Library (??)
George C. Marshall European Center for Security Studies
21
Assessment Impact ExamplesAssessment Impact ExamplesLibrarians move forward on projects.
◦ Nearly all librarians say they’ll enact student plans, in part or in whole.
◦ “We have paid thousands to ‘consultants’ who have produced reports that don’t come anywhere near the level of detail and professionalism that these students provided for us gratis. If we were to move on this we could have a family-centered program at the [children’s hospital] that would become a national model.”
–to hospital president and others from chair of pediatrics◦ “If you were wondering if your project was ever touched –
most certainly! Your project has been the backbone of my knowledge and launching point for inquiry. Hopefully in 2-3 months you will see these items [downloadable audio] in the catalog and in our marketing.”
--to student from Wake Public Libraries (NC)
Library Assessment Conference 2008 21
22
Assessment Impact ExamplesAssessment Impact Examples
Students gain professional positions.Student named Federal Library
Technician of the Year.Student recommended as chair of
assessment committee at New England Law Library Consortium.
Library Assessment Conference 2008 22
23
Courses at selected Courses at selected LIS ProgramsLIS Programs
UIUC “Evaluating Programs and Services”
Michigan “Evaluation of Systems and Services”
“Outcome Based Evaluation of Programs and Services”
Rutgers “Evaluation of Library and Information Services & Systems”
Indiana “Evaluation of Resources and Services”
Texas “Administration”
Wisconsin “Information Services Management”
Hawaii “Teaching Information Technology Literacy”
Florida State “Planning, Evaluation & Financial Management”
ECU Theme & component throughout program
Library Assessment Conference 2008 23
Library Assessment Conference 2008 24
From SA point of view - 2 implicit From SA point of view - 2 implicit assumptions:assumptions:
The workplace requires evaluation & assessment activities from librarians
Library schools are teaching some of the competencies required for these activities
Little evidence of either in local practice
Library Assessment Conference 2008 25
ReasonsReasonsNo standardized data collection
required from libraries Inevitable result: not a strong
culture of assessment evident on the SA library scene
If evaluation & assessment not a high priority in libraries - almost self-evidently not high priority in library schools either
Library Assessment Conference 2008 26
Library education in SALibrary education in SASchools/Departments generally small &
threatened with closureReduced in number during the last 10
years from 18 to 12; also more closures in sight
Some that remain have merged with other disciplines in order to survive;
Or evolved other survival strategies; e.g. diversifying into adjacent areas like knowledge or records management, media studies & publishing
Library Assessment Conference 2008 27
Two kinds of qualificationsTwo kinds of qualificationsEnglish speaking universities: mainly
post-graduate Diploma after Bachelor’s degree ◦ To ensure that students have some subject
specialization Other universities: first degree in
librarianship with somewhat less emphasis on subject specialization
2 qualifications initially envisaged as equal (both took 4 years to complete); but
Gradual emergence of 3 year qualification in information studies - much less subject specialization required
Library Assessment Conference 2008 28
ImplicationsImplicationsLibrarians rather technicist in orientation Focus on the practicalities of obtaining,
managing & provision of resources Frequently not enough subject expertise
to be regarded as equals by faculty Tend to concentrate on undergraduate
needs & information literacy of very diverse & frequently underprepared student body.
Library performance measurement may be regarded with suspicion
Fear that own institution ‘might be shown up’ - of lesser quality than others
Library Assessment Conference 2008 29
University of Cape TownUniversity of Cape TownPostgraduate diploma: small course on
performance measurement & evaluation6 teaching periods:
◦ objectives of performance evaluation, ◦ approaches to measuring ◦ few informal case studies & examples of
processes & proceduresEventual need for evaluation skills in
workplaces emphasizedSelf-study projects on e.g. measuring in
ILL depts, assessment of infolit competencies & information needs; statistics for electronic resources & web usability studies
Library Assessment Conference 2008 30
Yet growing demand for evidence of Yet growing demand for evidence of qualityqualitySA Council for HE mandates national
institutional quality audits Libraries to provide evidence of quality
& impact of services on teaching & research ◦ Some assistance from CHELSA
Considerable interest in PM7 in 2007◦ Ca 70 librarians from SA (total of nearly 200)
Influence of LibQUAL+ ◦ Though language & structure very difficult at
institutions where English not first language of student body
Library Assessment Conference 2008 31
Also problem with research in SAAlso problem with research in SASA research output declined since 1990sRespected researchers generally pale &
male & about to retireResults of our LibQual evaluation (2005)
◦ Loud & clear; postgraduates & researchers not happy with Library resources & services
◦ Both faculty & postgraduates (i.e. both current and future researchers) rated all of Information Control below minimum expectations
Serious & sustained interventions required to support & enhance the research enterprise
Library Assessment Conference 2008 32
Novel intervention Novel intervention SA library SA library educationeducation Ambitious Library project to support
researchers in the LibraryConsortium of 3 large academic
libraries funded by CarnegieIntended to catch up with what was
not learnt in library school Program for librarians: ‘total
immersion’ into research enterprise ◦ Monitoring & measuring ALL activities
essential for improvement
Library Assessment Conference 2008 33
No research support without evaluation No research support without evaluation & measurement& measurementTwo-week “Academy” for 6 mid-career librarians
from each institution for 2 (or 3) yearsBest possible researchers talking about their
own research ◦ Wide range of disciplines & from very different
epistemologiesEach participant also to produce potentially
publishable research paper: with data collection, measurement or assessment component
Research involves finding out & counting & measuring to understand what is really going on –
Whether in libraries or elsewhere in the research enterprise
Library Assessment Conference 2008 34
Bibliography Shavelson, Richard J., and Gail P. Baxter. "Linking Assessment with
Instruction." A Handbook for Student Performance in an Era of Restructuring. Eds. Robert E. Blum and Judith A. Arter. Alexandria, Virginia: Association for Supervision and Curriculum Development, 1996. IV-7:1 - IV-7:6.
Shepard, Lorrie A. "Why We Need Better Assessments." A Handbook for Student Performance Assessment in an Era of Restructuring. Eds. Robert E. Blum and Judith A. Arter. Alexandria, Virginia: Association for Supervision and Curriculum Development, 1996. I-2:2 - I-2:7.