The Information Literacy Impact Factor: How to Measure Value - Author: Lorna Dodd

Lorna Dodd

College Liaison LibrarianUCD Library | Dublin 4. email: [email protected] Tel: +353 (0)1 716 7074 www.ucd.ie/library

The Information Literacy Impact factor: How to Measure Value

mailto:[email protected]

http://www.ucd.ie/library

Outline

Why conduct a survey of module coordinators?

Review of the Literature

Rationale

How the survey was conducted

Survey results

Implications for the future

Conclusions

Lorna Dodd: LILAC 2013

Why?


Why?Looking Back

& Looking Forward


Looking Back

Large University

No single strategy for the development of Information Literacy

Heavy load of information skills delivery

Driven by individuals

Sometimes evaluated but rarely measured

No clear indication in meeting learning outcomes(Dodd, L. & Kendlin, V., 2010)


Looking Forward…

New Library Strategic Plan 2010-2014

“monitor, measure and evaluate the Library’s teaching and learning strategy and

activities.” (UCD Library, 2010)


Looking forward

Realignment of staffing structure From approx 17 Liaison Librarians to 6 College

Liaison Librarians

Move from module to programme approach

Move from responsive to consultation

Measure impact on student learning


Looking Forward…


Literature Review (1)

A lot available discussing why it is important to measure impact

Much of what has been written focuses on Library use rather than instruction.

(Stone 2011; Poll & Payne 2006; Schilling & Applegate 2012).

Limitations of traditional organisational approaches at measuring success and meeting targets.

Most libraries focus on process and output indicators rather than measuring impact.

Measuring impact on student learning requires systematic evaluation of training

(Markless & Streatfield 2006; Schilling & Applegate 2012))


Literature Review (2) Several problems arise when measuring impact.

Data protection rules Differing value of services for specific groups Difficulty in measuring long-term impact Time consuming nature of measuring impact of Library

instruction.

Important to use results of any study conducted to measure the impact of Library services Benchmarking activities Improving services Justifying resources used for services Campaigning for increased funding.

Common/Successful ways to measure impact of Library instruction Surveys; Pre and post-tests; Self-assessments; Behavioural observations.

(Poll & Payne 2006)



2006 LIR/SCONUL measuring impact initiativeDeveloped impact process

Identified key performance indicators

Used to evaluate the effectiveness of interventions supporting teaching and learning

Research provided information regarding emerging issues throughout the process.

Results included: Improved relationships with academic community A raised profile for Library staff Better understanding of how the Library can support

academic programmes.

(Markless & Streatfield 2006; Blagden, 2005)Lorna Dodd: LILAC 2013


2012 CONUL Information Literacy SurveyGathered feedback on the impact of information literacy instruction

Found library instruction had a positive impact on students’ skill development

Information Literacy often assessed

Targeted respondents

Academics already fully engaged

First year students only

Small sample from each institution

(CONUL Advisory Committee on Information Literacy, 2013)


We needed to:

Make academics aware of new approach

Create a strategy to get academics to embrace new approach

Identify what has been successful to date

Collect feedback from those not actively engaged with the Library

Identify what methods would be useful going forward

Problems:

Measuring impact retrospectively

Perceived impact is subjective

External factors influence development of information skills

More likely to opt for options that are familiar

Rationale


Survey

Seeking information on modules

Two parts:1. Evaluating information skills delivery to date2. Identifying what would be useful in the future

Survey Monkey 11 questions Mainly quantitative Multiple choice questions Measurement/ranking questions


Survey

Introduction:

Level of module – undergraduate/taught postgraduate

If there has been Library instruction

Part 1:

Kind of instruction

Impact of instruction on students’ abilities

Part 2:

Should all students have library instruction

When should library instruction happen?

Which services should be developed

Free Question


Promotion

Approx. 700 module coordinators

Only possible to target academic staff

Not all module coordinators are academic staff and not all academic staff are module coordinators

Targeting email – approx 1,900

College Liaison Librarian contacts

Three weeks


Survey Results

Type of instruction

Impact of instruction

When and who?

Preferred services for the future

Qualitative feedbackLorna Dodd: LILAC 2013

Type of Library Instruction


Impact of Library

Instruction



Mostly Taught Postgraduate: EndNote Special Collections PBL One-on-One Consultation

“Students have always received instruction to date

and so difficult to answer the comparison questions below”



Should ALL students have Library Instruction?


Should Library Instruction happen in every year?


Which Stage?

“Instruction in Year one should be built on in subsequent years

to enhance skills by year”


Preferred Services for the

FutureLorna Dodd: LILAC 2013


Influence of instruction had on preference?

Instruction had

First Choice Second Choice Last Choice

Tour Subject Specific Workshops

Generic workshopTutorialOnline Video

Tour

Lecture Subject Specific Workshops

Generic WorkshopTutorial

Tour/Online Video

Workshop Subject Specific Workshops

Generic WorkshopTutorial

Tour

No Instruction Subject Specific Workshops

Subject Specific Workshops

Tour

“One of the best approaches would be to create materials that are reusable and for any student, regardless of level.

Just because a student moves up a level doesn't mean they've acquired all of the

learning they should have”

“…really good online video materials, from complete intro to specific and

advanced, and available on demand is the way to go”.

“Tutorials (tutors would be trained by Library staff)”


Qualitative Feedback


Comments Themes

20% (53) of those who completed the survey made comments

In line with our new strategy

Online options

Constructive comments & suggestions on current and future services

Complimenting services to date

Comments on the survey


Comments

5 respondents made comments inline with our new strategy Programme Approach Reusable Learning object and Online tools

6 respondents made comments about proposed future instruction


Comments 12 respondents made comments about current

services and changes

Problems with current service:

“…same 'introductory' talks by librarian using lecture slots of various modules for

at least three times (the third time he skipped as he found it ridiculous)

“Students just skip the guest lecture.”


Comments

Benefits of current service:

“…requires personal contact with a friendly face early on.

“The current reduction in library services is a real retrograde step especially with the explosion in available information

sources”


Comments

Over half (28) complimented current strategy

“we really love our librarians; thank you”

‘Campaigning’ for traditional serviceAcknowledging individualsRecognising value of information literacy to students


“…it makes a noticeable difference in the quality of assignment students produce. The Library Tutorial is typically rated as one of the most important lectures they have in that Stage 1 module.”


Implications for the Future


Impact of Instruction

Definite benefit in Library instruction

Active learning environment is preferred

Stronger evidence in the discovery and use of resources

Remains difficult to identify the ‘real’ impact


Timing of Instruction

Most module coordinators feel all students should have Library instruction

Instruction for students at every stage is desirable

Taught Postgraduates Beginning and middle of Semester 1

Undergraduates Stage 1 Stage 3


Preferred Services Active learning sessions tailored to subject needs

most preferred

Tours perceived as ineffective

Questions:

Asked to identify how effective instruction was at improving students’ information literacy skills. When they request instruction, do they usually

consider this? Or do they feel students need to know about the

library rather than what they need in order to transition from 2nd level, develop researching skills etc.


Questions

Did respondents choose workshop environment over online version because they are more familiar?

If so, then why did the majority of those who had no instruction choose workshop?

If delivering workshops on this scale is unsustainable, what can we do ensure effectiveness?

What work do we need to do in terms of marketing?


Conclusion

Shared understanding between Library & Academic in:

Benefit of Instruction Taking a programme approach Ensuring all students get an equal

opportunity (core modules)

Difference in: Most appropriate method of delivery


References

Blagden, P. (2005) ‘The LIRG/SCONUL Measuring Impact Initiative: Overview of phase 1 impact projects’, Library & Information Research (LIR) vol. 29 (91)

Dodd, L. & Kendlin, V. (2010) ‘Damned if we do and Damned if we don’t: How to address sustainability in the delivery of information literacy components in UCD’: Librarians' Information Literacy Annual Conference (LILAC), March 29-31, Limerick, Ireland.

Markless, S & Streatfield D. (2006) ‘Gathering and applying evidence of the impact of UK university libraries on student learning and research: A facilitated action research approach’, Journal of Information Management vol. 26 pp. 3-15

Poll, R & Payne, P. (2006) ‘Impact measures for libraries and information services’, Library Hi Tech vol. 24 (4) pp. 547-562

Schilling, K. & Applegate, R. (2012) ‘Best methods for evaluating education impact: a comparison of the efficacy of commonly used measures in library instruction’, Journal of the Medical Library Association, vol. 100 (4) pp. 258-269

Stone, G. et al (2011) ‘Does Library use affect student attainment? A preliminary report on the library impact data project’ Liber Quarterly vo. 21(1) pp. 5-22

UCD Library (2010) ‘UCD Library Strategic Plan 2010-2014’

http://www.ucd.ie/library/news_publicity/showcase_strategy/


The Information Literacy Impact Factor: How to Measure Value - Author: Lorna Dodd

Education

Transcript of The Information Literacy Impact Factor: How to Measure Value - Author: Lorna Dodd