www.arl.org

Making Library Assessment Work

ARL 4th Human Resources Management Symposium Washington, D.C.November 9, 2004

Steve Hiller and Jim Self

University of Washington and University of VirginiaAssociation of Research Libraries

www.arl.org

Why Assess?

• Accountability and justification

• Improvement of services

• Comparison with others

• Identification of changing patterns

• Identification of questionable services

• Marketing and promotion

www.arl.org

Good assessment practices

• Focus on the user

• Diverse samples of users

• Fair and unbiased queries

• Measurable results

• Criteria for success

• Qualitative and quantitative techniques

• Corroboration

www.arl.org

Assessment is not…

• Quick and easy

• Free and easy

• A one-time effort

• A complete diagnosis

• A roadmap to the future

www.arl.org

“…but to suppose that the facts, once established in all their fullness, will ‘speak for themselves’ is an illusion.”

Carl BeckerAnnual Address of the President of the

American Historical Association, 1931

www.arl.org

What Does it Mean?Understanding Your Data

• Scan results for basic overview – Frequencies, means, patterns, variation

• Use statistical analysis that make sense• Qualitative information and comparisons provide

context and understanding• Seek internal or external validation

– Within same data sets or others

• Identify what is important and why

www.arl.org

Communicating and Using Results

• Identify key findings, not all results• Mix text, data, and graphics

– avoid jargon– add context

• Know your audiences. Make it understandable • Prioritize potential action items and follow-up• Identify “Handoffs” to those responsible for action• Look for some easy “wins”

– Quick, inexpensive, and noticeable

• Report results

www.arl.org

Effective AssessmentEasier Said Than Done

• Libraries in many cases are collecting data without really having the will, organizational capacity, or interest to interpret and use the data effectively in library planning.

• The profession could benefit from case studies of those libraries that have conducted research efficiently and applied the results effectively.

(Denise Troll Covey, Usage and Usability Assessment: Practices and Concerns, 2002)

www.arl.org

Two Approaches to Assessment

• University of Washington– User needs assessment– Large-scale cyclical surveys and ongoing qualitative

input– Assessment distributed throughout organization

• University of Virginia– Performance and financial standards– Compilation of data from varied sources– Centralized Management Information Services unit

www.arl.org

UW Libraries Assessment Organization

• Library Assessment Coordinator (50%)– Chairs Library Assessment Group (9 members)

– Coordinates and develops broad-based user needs assessment efforts (surveys, focus groups, observation)

– Encourages and supports other assessment work

• Shared and Distributed Assessment ActivitiesUsability (Web Services)

E-Metrics (Assessment, Collection Management Services)

Management information (Assessment, Budget, CMS)

Instruction (Information Literacy, Assessment)

Digital Library (Digital initiatives, Public Services, Assessment)

www.arl.org

UW Assessment Methods

• Large scale user surveys every 3 years (“triennial survey”): 1992, 1995, 1998, 2001, 2004

• In-library use surveys every 3 years beginning 1993• LibQUAL+™ in 2000, 2001, 2002, 2003 • Focus groups on varied topics (annually since 1998)• Observation (guided and non-obtrusive)• Usability• E-Metrics

www.arl.org

Growing Assessment at UWFrom Project-Based to Ongoing and Sustainable

• Libraries’ first strategic plan in 1991 called for survey as part of user-centered services philosophy– Initial large scale library survey done in 1992 as “one-time project”

• Library Services Committee formed in 1993– Conducted in-library use surveys in 1993,1996, triennial survey in 1995

• Library Assessment Group appointed in 1997– Focus groups, observation studies, in-library and triennial surveys

• Collection Management Services, 1997– E-Metrics and collections use

• Library systems, 1997– Usability, Web usage logs

• Library Assessment Coordinator (50%) appointed 1999

www.arl.org

UW Assessment Priorities

• Information seeking behavior and use

• Library use patterns

• Library importance and impact

• User priorities for the library

• User satisfaction with services, collections, overall

• Using data to make informed decisions that lead to library improvement

www.arl.org

How UW Has Used Assessment Information

• Understand that different academic groups have different needs

• Make our physical libraries “student” places• Identify student information technology needs• Move to desktop delivery of resources• Enhance resource discovery tools• Provide standardized service training for all staff• Stop activities that do not add value to users• Consolidate and merge branch libraries

www.arl.org

Branch Library ConsolidationA UW Case Study

• 3 Social Science libraries consolidated in 1994– Described in ARL SPEC Kit

• Review Committee formed in 2002 • Changing use patterns a bigger driver than budget• Review to be objective and data-based• Identify one library to be consolidated into main library

www.arl.org

Performance Measures to Assess Branch Library Viability

• Use – Print items, photocopies, reference questions, gate counts

• Primary user population– Number of faculty and students, change over time

• Facility quality– For users, collections, and staff

• Physical library dependency of primary users

www.arl.org

Data Sources Used

• Library generated use data (including trend data)• Electronic resources use data supplied by vendors• University enrolment data (including trend data)• Interviews, focus groups, survey comments• Facility data• Survey data

– Triennial survey

– In-library use

• Cost data

www.arl.org

Primary User Groups by Branch Library (2003 University data)

232

1137

149

333

837

157

255

336

355

96

335

82

65

163

284

239

179

183

198

369

57

45

20

47

111

86

63

93

106

47

0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300

Architect

Art

Drama

Music

Chem

Fish

Forestry

Math

Physics

Soc Wk

Undergrad Grad Faculty

www.arl.org

Facility Space Quality: Methodology

• Discussed facility issues with unit staff• Reviewed user survey comments from 2001 and 2002• Used previous focus group data for fine arts libraries• Developed list of criteria• A team of 3 walked through each unit• A second walk through was conducted 2 months later• Each member of the team assigned a score of 1 to 5

for quality of staff, collections, and user spaces.• Scores were compared and made consistent.

www.arl.org

Facility Quality

0

1

2

3

4

5

6

7

8

9

10

11

12

13

Arch Art Chem Drama Fish Forestry Math Music Phys Soc W

Users Collections Staff

www.arl.org

Science Faculty Libraries Used Regularly (2001 Survey)

College/Dept’s(Faculty Responses)

“HOME” Library

Natural Sciences/Allen

Other

Chemistry (36)

72% 33% 28%

(Physics)

Fish-Ocean (42)

69% 45% 10% (Engineering)

Forest Resources (28)

79% 57% 14% (Engineering)

Math-Stat (27)

93% 30% 7% (Engineering)

Physics-Astronomy (36)

78% 25% 22% (Chemistry)

www.arl.org

Forestry Faculty and Grad Students Frequency of Library Use

Mode Group 20011x per

week

20041x per

week

2001 2 x+ per

week

20042x+ per

week

Visit in Person

FAC

GRAD

29%

29%

43%

24%

18%

18%

0%

23%

Campus Computer

FAC

GRAD

46%

53%

30%

18%

32%

35%

52%

71%

Off-Campus

Computer

FAC

GRAD

36%

24%

35%

53%

0%

12%

13%

24%

www.arl.org

34

17

12

16

34

13

8

19

26

21

12

14

26

7

8

17

28

13

21

17

14

15

15

17

22

17

8

24

42

13

10

17

12

19

16

15

24

16

21

14

0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

75

80

Architecture Art Chem Drama Fish-Ocean Forestry Math Music Physics Social W

Branch Library Measures Score (Use weighted 2x)

Use Primary User Population Facility Quality Library Dependency

www.arl.org

Merger Time Line

• Review Group formed Spring 2002• Recommendations submitted February 2003

– Merger of Forest Resources Library

– Identification of two other libraries for later merger

• Recommendation accepted June 2003• Joint implementation team appointed September 2003• Forestry faculty and students surveyed and

presentation made February 2004• Forest Resources Library merged into Natural Sciences

Library August 2004

www.arl.org

Triennial Survey Spring 2004Satisfaction All Faculty and Forestry Faculty (1998, 2001, 2004)

Forestry Collections

All Faculty Overall

All Facult Services

Forestry Services

All Faculty Collections

Forestry Overall

4

4.1

4.2

4.3

4.4

4.5

4.6

4.7

4.8

4.9

5

1998 2001 2004

4

4.1

4.2

4.3

4.4

4.5

4.6

4.7

4.8

4.9

5

www.arl.org

The University of Virginia LibraryOrganizational Culture

• Customer Service

• Collecting and using data

• Innovation

• Flexibility

• Learning and development

• Participation and discussion

• Pride

www.arl.org

In the words of our leader…

• Use data to IMPROVE• Services• Collections• Processes• Performance• Etc., etc.

• Don’t use data to preserve the status quo

-Karin Wittenborg University Librarian, University of Virginia June 24, 2004

www.arl.org

University of Virginia LibraryOrganizing for Assessment

• Management Information Services unit– Established in 1996

– Currently 3 staff

– Resource for library management and staff

– Advocates for sustainable assessment

• Centralized data collection, analysis and compilation

• Multifaceted approaches

www.arl.org

Collecting the Data at U.Va.

• Customer Surveys

• Staff Surveys

• Mining Existing Records

• Comparisons with peers

• Qualitative techniques

www.arl.org

Customer Surveys

• Faculty – 1993, 1996, 2000, 2004

• Students – 1994, 1998, 2001, 2005

– Separate analyses for grads and undergrads

www.arl.org

Faculty Surveys 1993-2004University of Virginia Library

Overall Satisfaction by Broad Academic Groups

4.35

4.60

4.42

4.36

4.26

4.48

4.41

4.27

4.35

4.09

4.26

4.41

3.87

3.99

4.25

4.14

3.8

4.8

1993 1996 2000 2004

Year

Ove

rall

Sat

isfa

ctio

n R

atin

g

on

a s

cale

of

1 to

5

Social Sciences

Humanities

Composite

Sciences

www.arl.org

Faculty Priorities1993 to 2004

0

80

1993 1996 2000 2004

Print Journals

27%

ILL/LEO 35%

E-Journals 49%

Books 61%Books

59%

Print Journals

66%

E-Journals 25%

ILL/LEO 17%

E-Ref Tools 32%

E-Ref 31%

47%

75%

www.arl.org

Using Customer Survey Results – UVa

• Additional resources for the science libraries (1994+) Major renovation (2001)

• Revision of library instruction for first year students (1995)

• Redefinition of collection development (1996)• Initiative to improve shelving (1999)• Undergraduate library open 24 hours (2000)• Additional resources for the Fine Arts Library (2000) • Support transition from print to electronic journals (2004)

www.arl.org

Staff Surveys

• Internal Customer Service– 2002, 2003, 2004

– 1 to 5 satisfaction scale

• Worklife Survey– 2004

– Agree or disagree with positive statements

www.arl.org

Internal Customer Service Surveys

• Ratings (1 to 5) of units providing service to other library staff

• Reports to managers and administrators

• Anonymous structured interviews to follow up

• Survey expanded in 2004 to include all library departments

www.arl.org

Worklife Survey

• Areas of inquiry – Job Satisfaction

– Interpersonal Relations

– Communications & Collaborations

– Diversity

– Resource Availability

– Staff Development

– Health & Safety

• Report at Library ‘Town Meeting’

• Focus groups following up

www.arl.org

Data Mining

• Acquisitions

• Circulation

• Finance

• University Records

www.arl.org

Acquisitions Expenditures by FormatUniversity of Virginia Library

$0

$500,000

$1,000,000

$1,500,000

$2,000,000

$2,500,000

$3,000,000

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003

Fiscal Year

Ex

pe

nd

itu

res

Electronic

Print Serials

Print Monographs

Manuscripts

Miscellaneous

Microforms

Video or Film

2004

www.arl.org

University of Virginia Library Serving the Customer

0

100,000

200,000

300,000

400,000

500,000

600,000

Circulation

Reserve

Reference

www.arl.org

University of Virginia Library Serving the Customer

0

300,000

600,000

900,000

1,200,000

1,500,000

1994

/95

1995

/96

1996

/97

1997

/98

1998

/99

1999

/00

2000

/01

2001

/02

2002

/03

2003

/04

E-Journals

Circulation

Reserve

Reference

www.arl.org

Comparisons with Peers

• Within the University

• Within ARL

www.arl.org

Expenditures of UVA Academic Division1989—2003

-50%

0%

50%

100%

150%

200%

250%

Fiscal Year

% C

hang

e si

nce

1989

Other Academic Support (+200%)

Research (+219%)

Total Academic Division (+140%)

Libraries (+81%)

Instruction (+80%)

www.arl.org

Median Faculty SalariesUniversity of Virginia Library

Compared to ARL Median

$30,000

$35,000

$40,000

$45,000

$50,000

$55,000

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004

Fiscal Year

U.Va.

ARL

www.arl.org

Qualitative Techniques

• Focus Groups– Preparation for work life survey

– Follow up to work life survey

• Structured Interviews– Anonymous follow-up to customer service survey

• Open Discussions

www.arl.org

Corroboration

• Data are more credible if they are supported by other information

• John Le Carre’s two proofs

www.arl.org

Analyzing Survey Results

• Two Scores for Resources, Services, Facilities– Satisfaction = Mean Rating (1 to 5)

– Visibility = Percentage Answering the Question

• Permits comparison over time and among groups

• Identifies areas that need more attention

www.arl.org

UVa Reference Activity and Reference Visibility in Student Surveys

2,519

6,008

39.3% Visibility

75.8% Visibililty

63.9% Visibility

1,000

7,000

1993 1994 1995 1996 1997 1998 1999 2000 2001

Fiscal Year

Re

fere

nc

e Q

ue

sti

on

sR

ec

ord

ed

pe

r W

ee

k in

An

nu

al S

am

ple

10%

30%

50%

70%

90%

Re

fere

nc

e V

isib

ilit

y

am

on

g U

nd

erg

rad

ua

te

www.arl.org

The Balanced ScorecardManaging and assessing data

• The Balanced Scorecard is a layered and categorized instrument that– Identifies the important statistics

– Ensures a proper balance

– Organizes multiple statistics into an intelligible framework

www.arl.org

The scorecard measures are “balanced” into four areas

• The user perspective

• The finance perspective

• The internal process perspective

• The future (learning and growth) perspective

www.arl.org

Metrics

• Specific targets indicating full success, partial success, and failure

• At the end of the year we know if we have met our target for each metric

• The metric may be a complex measure encompassing several elements

www.arl.org

Rationale for the BSC:Getting Control of the Data

• Focus

• Balance

• Assessment

• Intelligibility

www.arl.org

The BSC at the U.Va. Library

• Implemented in 2001

• Reports for FY2002 and FY2003

• Reporting results for FY2004

• Metrics for FY2005 in place

• A work in progress

www.arl.org

Metric L.1.A. Impact of training.

• Target1: : Positive scores (4 or 5) on training statements from 80% of respondents in the annual work-life survey.

• Target2: : Positive scores (4 or 5) on training statements from 60% of respondents in the annual work-life survey.

www.arl.org

Metric I.2.A. Internal Communications

• Target1: : Positive scores (4 or 5) on internal communications statements from 80% of respondents in the annual work-life survey.

• Target2: : Positive scores (4 or 5) on internal communications statements from 60% of respondents in the annual work-life survey.

www.arl.org

Metric I.3.B. Staff Rating of Internal Services

• Target1: : A composite rating of 4.00 in the annual internal services survey, with no unit rated below 3.50.

• Target2: : A composite rating of 3.50, with no unit rated below 3.00.

www.arl.org

Metric L.2.A. Job Satisfaction.

• Target1: : Positive scores (4 or 5) on job satisfaction statements from 80% of respondents in the annual work-life survey.

• Target2: : Positive scores (4 or 5) on job satisfaction statements from 60% of respondents in the annual work-life survey.

www.arl.org

Metric L.2.B. Retention Rate of Employees

• Target1: : Retain 95% of employees.

• Target2: : Retain 90% of employees.

www.arl.org

Metric L.2.C. Compare Salaries to Peer Groups

• Target1: : The median faculty salary at the U.Va. Library should rank in the upper 40% of all ARL university libraries.

• Target2: : The median faculty salary at the U.Va. Library should rank in the upper 50% of all ARL university libraries.

www.arl.org

Metric L.2.D. Diversity of Staff

• Target1: : A net increase of at least 4 in faculty/staff diversity, with a net increase of at least 2 in faculty diversity.

• Target2: : A net increase of at least 2 in faculty/staff diversity, with a net increase of at least 1 in faculty diversity.

www.arl.org

Metric L.4.A. Develop a Culture of Assessment

• Target1: : 75% of respondents score 12 or more positive responses on the Culture of Assessment IQ instrument.

• Target2: : 50% of respondents score 12 or more positive responses.

www.arl.org

Moving Forward

• Understand your limitations– Use data wisely and appropriately

– Don’t do more than you can support or utilize

• Don’t expect perfection; strive for accuracy and honesty• Assess what is important to the library and the staff, • Use the data to improve

• But always keep your focus on the user

www.arl.org

For more information…

• Steve Hillerhiller@u.washington.eduwww.lib.washington.edu/assessment/

• Jim Self– self@virginia.edu– www.lib.virginia.edu/mis– www.lib.virginia.edu/bsc

• ARL Assessment Projectold.libqual.org/documents/admin/VPOHillerSelf.pdf