FROM DATA TO ASSESSMENT

Collecting, Presenting, and Using Statistics

Jim SelfUniversity of Virginia VIVA Stats WorkshopFebruary 2, 2010

THIS MORNING’S PROGRAMA few basic statistical conceptsDisplaying the dataExamples from U.Va. and

elsewhereMoving toward assessmentQuestions, comments

WHY MEASURE OR COUNT?To evaluateTo compareTo improve

WHEN NOT TO COLLECT DATA

When someone has already done itWhen the data will not be usefulWhen the results do not justify the

costs

MAKING DATA MEANINGFUL

SummarizeCompareAnalyzePresent

MEASURES OF CENTRAL TENDENCY

MedianMeanModeMidrange

MEASURES OF DISPERSION

RangeVarianceStandard DeviationInterquartile Range

SAMPLINGSelection of members is crucial

Strive for an unbiased sampleUnambiguous, pre-set selection process

Sample sizeLarge enough for appropriate precisionSmall enough to be economical

DETERMINING SAMPLE SIZE

Confidence Interval

Level of confidence

U.VA. STUDENT SURVEYQuestion: Do you own a computer

at U.Va?First Year Students

Yes 99.0%Graduate Students

Yes 88.2%

Do we know if these two groups are different in terms of computer ownership?

95% LEVEL OF CONFIDENCE

Population Size

Sample Size

Percent in Sample Answering “Yes”

Plus orMinusSampleError

Min.Value

Max.Value

Grads 5,400 443 88.2% 2.9% 85.3% 91.1%

1stYears

2,900 102 99.0% 1.9% 97.1% 99.9+%

PREPARATION FORDATA COLLECTION

Clarify your objectivesPrefer the simplest proceduresChoose the appropriate technique

Collect only data you can use

FOUR USEFUL ASSUMPTIONS

Your problem is not unique.

You have more data than you think.

You need less data than you think.

There is a useful measurement that is much simpler than you think.

Hubbard, D. (2007) “How to Measure Anything.”

PROCESSING THE DATA

Check and re-check the numbersSummarizeCompareFind the interesting pointsLook for explanations

COMMON PITFALLS Sampling with bias Surveys with a message Graphs with distorted figures Graphs with varying time frames Atypical projections Inappropriate grouping of data False precision Confusion of correlation and causality

“…BUT TO SUPPOSE THAT THE FACTS, ONCE ESTABLISHED IN ALL THEIR FULLNESS, WILL ‘SPEAK FOR THEMSELVES’ IS AN ILLUSION.”

Carl Becker

Annual Address of the President of the American Historical Association, 1931

DATA REPORTING

Integrate data, graphs, and narrative to tell a succinct and accurate story.

Show the data Give the source of the data Do not decorate graphics, let the data

speak Put your name on the report

UVA LIBRARY SURVEYS Faculty

1993, 1996, 2000, 2004Response rates 59% to 70%

Students 1994, 1998, 2001, 2005Separate analysis for grads and undergradsResponse rates 43% to 63%

LibQual+ 2006Response rates 14% to 24%

Annual Surveys 2008+Student samplesOne third of facultyResponse rates 29% to 47%

FINAL QUESTIONU.VA. LIBRARY SURVEY

Please rate your overall satisfaction with the library.

Not at all Very No

Satisfied Satisfied Opinion

1 2 3 4 5 X

2000 U.VA. FACULTY SURVEYOVERALL RATING

1 2 3 4 5 X

0 5 19 126 154 11

U.VA. FACULTY SURVEYSOVERALL RATING

1 2 3 4 5 X Total

1993 1 5 52 174 93 24 349

1996 1 8 36 146 136 12 339

2000 0 5 19 126 154 11 315

OVERALL SATISFACTIONU.VA. LIBRARY SURVEYS

Category Mean

1993 Faculty 4.09

1994 Undergraduates 4.01

Graduate Students 3.96

1996 Faculty 4.26

1998 Undergraduates 3.90

Graduate Students 3.97

2000 Faculty 4.41

2001 Undergraduates 4.07

Graduate Students 4. 02

2004 Faculty 4.20

2005 Undergraduates 3.94

Graduate Students 4.02

OVERALL SATISFACTIONUNIVERSITY OF VIRGINIA LIBRARY

1993/94

1996/98

2000/01

2004/05 2008/09

Faculty 4.09 4.26 4.41 4.20 4.23

Graduate Students

3.96 3.97 3.97 4.02 4.17

Undergrads

4.01 3.90 3.97 3.94 4.12

OVERALL SATISFACTIONUNIVERSITY OF VIRGINIA LIBRARY1993-2009

1993/94 1996/98 2000/01 2004/05 2008/093.80

4.00

4.20

4.40

4.60

4.09

4.23

4.01

4.12

3.96

4.17

Faculty

Undergrads

Grads

U.Va. Library Surveys 1993-2009 http://www2.lib.virginia.edu/mis/reports/

U.VA. FACULTY AND STUDENTSWHO PHYSICALLY VISIT A LIBRARYAT LEAST ONCE A WEEK

1993/94 1996/98 2000/01 2004/05 2008/090%

20%

40%

60%

80%

100%

79%

26%

69%72%

84%

43%

Faculty

Grads

Undergrads

DATA PRESENTATION ACCORDING TO DILBERT

THE LIBRARY BUDGET AS PIE

DR1% EI, DI

4%

EF, ER, EU9%

FA2%

GF1%

LG2%

LS1%

SE2%

SG77%

SR,SS0%

Sources of Funding FY09

CORROBORATIONData are more credible if they are

supported by other informationJohn Le Carre’s two proofs

ANALYZING SURVEY RESULTS

Two Scores for Resources, Services, FacilitiesSatisfaction = Mean Rating (1 to 5)Visibility = Percentage Answering the Question

Permits comparison over time and among groups

Identifies areas that need more attention

UVA REFERENCE ACTIVITY AND

REFERENCE VISIBILITY IN STUDENT SURVEYS

1,756

6,008

34%Visibility

39% Visibility

75% Visibililty

64% Visibility

1,000

7,000

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004

Ref

eren

ce Q

ues

tion

sR

eco

rded

per

Wee

k i

n A

nn

ual

Sam

ple

10%

Ref

eren

ce V

isib

ility

a

mo

ng

Un

der

gra

du

ate

COLLECTING AND USING DATA AT THE U.VA. LIBRARY

Customer Surveys Staff Surveys Mining Existing Records Comparisons with peers Qualitative techniques Statistical compilations Revealing the budget process and issues

www.arl.org

DATA MININGAcquisitionsCirculationFinanceUniversity Records

COLLECTIONS EXPENDITURES BY FORMAT

FY1993

FY1994

FY1995

FY1996

FY1997

FY1998

FY1999

FY2000

FY2001

FY2002

FY2003

FY2004

FY2005

FY2006

FY2007

$0

$500,000

$1,000,000

$1,500,000

$2,000,000

$2,500,000

$3,000,000

$3,500,000

$4,000,000

University of Virginia Library

Electronic Resources

Print Monographs

Print Serials

Manuscripts

Miscellaneous

Microforms

Video or Film

Fiscal Year

Ex

pe

nd

itu

res

USAGE DATA AT UVA – CIRC, REF, E-JRNLS

1992

/93

1993

/94

1994

/95

1995

/96

1996

/97

1997

/98

1998

/99

1999

/00

2000

/01

2001

/02

2002

/03

2003

/04

2004

/05

2005

/06

2006

/07

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

Circulation

Reference

E-Journal Articles

INVESTMENT AND CUSTOMER ACTIVITYUNIVERSITY OF VIRGINIA LIBRARY1993-2006

Customer Activities

0

600,000

1,200,000

1,800,000

2,400,000

FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06

E-Journals

Circulation

Reference

Acquisitions Expenditures by Format

$0

$1,000,000

$2,000,000

$3,000,000

FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06

Electronic Resources

Print Monographs

Print Serials

TOTAL EXPENDITURES AT UVA 1989—2003

-50%

300%

Fiscal Year

% C

hang

e si

nce

1989 Other Academic Support

(+200%)

Research (+219%)

Total Academic Division (+140%)

Libraries (+81%)

Instruction (+80%)

LOOKING AT LIBQUAL+

22 core questions1-9 scaleRatings of minimum, desired, perceived

Locally selected questions General satisfaction ratings

SELECTING AND DISPLAYING LIBQUAL+

Choose the meaningful data Ignore the chaff

Summarize those data Compare

By category and subgroupsOver timeWith peers

Present findings transparently

LibQUAL+ 2006ARL Composite Faculty

5

6

7

8

9

AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5

5

6

7

8

9

AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5

LibQUAL+ 2006University of Virginia Faculty

5

6

7

8

9

Faculty AS-3 Grad AS-3 Undergrad AS-3 Library Staff AS-3

LibQUAL+ 2006Ratings of AS-3:

Employees who are consistently courteous

Faculty

Grads Undergrads

Library Staff

USING DATA AT UVA

Additional resources for the science libraries (1994+)

Redefinition of collection development (1996)

Initiative to improve shelving (1999)

Clemons Library open 24 hours (2000)

Additional resources for the Fine Arts Library (2000)

Support for transition from print to e-journals (2004)

New and improved study space (2005-06)

Increased appreciation of the role of journals (2007)

Re-design of main floor of Clemons Library (2008)

USING DATA:FROM STATS TO ASSESSMENT

CountingTraditional statistics emphasize inputs:

expenditures, acquisitions, holdings Assessing

More recent efforts assess outputs: service quality

WHY ASSESS?Improve services Justify resourcesCompare with othersIdentify changing needsIdentify questionable servicesPromote services

PARALLEL TRENDSCustomer-Centered

Library

All services and activities are viewed through the eyes of the customers

Customers determine quality

Library services and resources add value to the customer

Culture of Assessment

Organizational environment in which decisions are based on facts, research and analysis.

Services are planned and delivered to maximize positive customer outcomes

EFFECTIVE ASSESSMENT

Is customer-focused Uses representative groups of users Asks fair and unbiased queries Develops criteria for success Uses multiple assessment methods Uses corroboration from other sources Provides results that can be used to

improve libraries

A CULTURE OF ASSESSMENT

Decisions based on facts, research, and analysis

Wide recognition and acceptance of the importance of assessment

Expertise and staff development related to assessment

QUESTIONS AND COMMENTS?

More information: Jim Self self@virginia.edu www.lib.virginia.edu/mis