Impact Studies: Outcome-based Evaluation

Daphne WoodDirector, Planning & Organizational DevelopmentVancouver Public Library

Fooled by Experience

Approach with caution any list that purports to reveal, for instance,

10 common traits of successful companies!

whether it is punctuated with an exclamation point or comes with the seriousness of a legitimate study.

What is outcome-based evaluation?

• Outcome-based evaluation (also known as outcome measurement, results- based evaluation, outcome assessment) is focused on assessing the quality and effectiveness of programs. This is accomplished by project staff tracking both the qualitative and quantitative changes created in the community through their program.

When should it be used?

The outcome-based evaluation approach is most useful when the purpose of the evaluation is to assess a program’s results. – To what extent is the program meeting its goals? – How does the program compare against accepted

standards? – Is the program effective? – How does the program compare with competitive

programs? – Should the program be continued? – Should the program be expanded?

Alligator Pie: Reaching Children Who Can’t Always Reach the Library

Alligator Pie

• Literacy based, professionally developed, and needs appropriate library storytimes to ‐children in childcare programs who do not have the opportunity to attend and benefit from library storytimes

• Special emphasis on reaching Vancouver’s most vulnerable children in at risk ‐neighbourhoods

Alligator Pie

• Three-year pilot program funded through the Vancouver Public Library Foundation by an anonymous donor and the Allan MacDougall Legacy Fund

• Desired outcomes:– 50% of the children attending the Alligator Pie program each

year of the three-year pilot will demonstrate an increase in:• Emergent literacy skills, for example increased vocabulary (learned

new words), eyes following text from left to right, and appropriate vocalization of sounds associated with specific letters.

• Developing social interactions, for example peer to peer eye contact, interaction with the librarians, and social vocalization (laughing, giggling).

Methodology

Results (2014)

• Based on formal and informal observation and assessment, all participants who attended for the duration of the program achieved increases in emergent literacy skills and improvements in positive social interaction.

• Program includes daycares in the Downtown Eastside, Strathcona, and Mount Pleasant neighbourhoods – caregivers report an increase in skills; parents report an increase in confidence

Aboriginal Storyteller in ResidenceSharon Shorty (2015)Photo: Mark Rutledge

• A pilot program from 2009-2011 funded by the Vancouver Foundation and the Vancouver Public Library Foundation.

• In 2012, the VPL Board established the Aboriginal Storyteller in Residence as a core VPL program, re-commencing in 2013.

A unique program every year

Potential Outcomes

Potential Outcomes

Survey results (2014):

In the words of library staff

Through our staff training meeting that Columpa attended I learned more practical tips to help me plan and develop programs; Columpa clarified a lot of questions I had regarding protocol: the importance of "inviting people into your house,” "seeking balance,” going where the communities are, relationship building and longevity versus one-off programs, why singing is important and the role of different songs, the significance of the land, the different role of storytellers in Coast Salish cultures, tips for bridging the oral and written world, and more!

Vancouver Inspiration Pass

SPARC BC evaluation (2014)

• Evaluation period of November 2012 through September 2013

• Evaluate the following outcomes:– Increase accessibility to Vancouver cultural, recreational, and

in-formal learning venues for all Vancouver residents;– Showcase Vancouver’s cultural, heritage, and recreation

venues;– Increase participation in cultural, recreational, and in-formal

learning activities by Vancouver residents who have little or no current participation; and

– Increase the sense of connectedness to community.

Methodology

• Surveys of participants (pass users)• Focus groups of participants (pass users)• Interviews with VPL staff and representatives

of a number of Program partners were conducted to identify challenges and positive outcomes.

Result (participation):

• Almost half of respondents who agreed to report on their income (48%) had low or ‘modest’ household incomes, if modest is defined as less than $60,000. Vancouver was frequently noted in the survey feedback as an expensive city in which to live with many individuals and households experiencing significant challenges in finding sufficient discretionary income to allow them to attend many fee based venues.

Result (connectedness):

• Two in three respondents (67%) reported that, since using the Pass, they know more about what happens in their community, while 86% felt that they have a greater appreciation of the City and its cultural and community life.

• Nine in 10 respondents (90%) either agreed or strongly agreed that the Pass encouraged them to get out and enjoy Vancouver while approximately 1 in 3 (32%) felt that the Pass had changed their view of living in Vancouver.

Of particular note:• Popularity of the program extended the waitlist to 2+

years• Strong desire among program partners to reach those

most socially excluded and economically disadvantaged – current program meeting 48% with household incomes of $60K or less

• VPL implemented change to Pass: admit 4 rather than 6 people; create 40 passes to distribute among community partners to reach most vulnerable individuals

• Evaluation underway!

Access the toolkit

VPL/SPARC Outcome based evaluation toolkit • http://www.vpl.ca/images/uploads/file/pdf/

Outcomes_Based_Evaluation_Toolkit.pdf

Contact: Daphne Wooddaphne.wood@vpl.ca604 331 4006