Smart Commute EvaluationTools, Techniques and Lessons Learned in Monitoring and

Evaluating Workplace-based TDM Programs

Outline1. Background2. Approach3. Monitoring Tools4. Evaluation5. Results6. Challenges7. Lessons8. Next Steps

Ryan LanyonTeam Lead, Smart CommuteMetrolinx (GTTA)October 22, 2008

At-a-Glance Build evaluation into your initial planning Monitor activities, benefits and customer service Scrutinize and understand indicators Learn from results and mistakes Resources available at www.smartcommute.ca Just do it!

Smart Commute works with various stakeholders to relieve traffic congestion, improve air quality and health, and reduce emissions that cause climate change.

Background Smart Commute

Focus on commuters and workplaces Partnership with municipalities Staged development

– 2001: Pilot TMA - BCRTMA– 2004: Pilot regional implementation - Municipalities– 2008: On-going implementation – Metrolinx + Municipalities

Background Metrolinx (Greater Toronto Transportation Authority)

Formed in 2007 Coordinates transportation across GTHA Regional Transportation Plan 6.13 million residents, 3 million commuters

Background Two-tiered program delivery

Metrolinx– Central coordinating body– Centralized service operation e.g. Carpool Zone– TMA Toolkit– Funding agency – 50%

Local Smart Commute / TMA– Delivery agent– Municipality, NGO, Chamber of Commerce/Board of Trade– Find other funding – municipalities, grants, service fees

Background 2001: BCRTMA / NTV 2004: 404-7 2005:

Mississauga NE Toronto

2006: Brampton-Caledon Halton

2007: Central York Durham

2008: Toronto-Central 2009: Airport?

Background Employer program

Improve and encourage commuting options Assessment of current conditions - Baseline Development of a plan Implementation – ‘Commuter benefits program’

– Carpool Zone and ERH– Walking, cycling, transit, carpool, telework programs– Campaigns, contests and special events

Follow-up assessment – Measured against baseline

Approach - Ideal

Goals

ObjectivesRegional Transportation Plan

Municipal Plans Implementation

Strategic Plan Monitoring

Evaluation

Approach - Actual

Goals Objectives

RTP

Municipal PlansImplementation

Strategic Plan

Monitoring

EvaluationFederal Funding Requirements

Implementation

Implementation

Approach 2004-2007 - Contribution Agreement

Negotiated between funder and municipalities 2008 - Monitoring Framework

Established by Smart Commute stakeholders– TMAs and Smart Commute Association (implementers)– Municipalities and Metrolinx (funders)

Multipurpose– Measure activity levels for funders– Track impacts to justify funding– Collect statistics for marketing purposes

Approach Inputs / Activities

Number of businesses– Active– Engaged– Supporting

Number of pamphlets distributed Number of cycling programs Number of media releases Number of commuters reached

Approach Outputs / Benefits / Results

Brand awareness– % of employees aware; change from baseline

Commuter satisfaction– Ratings of Smart Commute program

Reductions– GHG emissions– CAC emissions– Vkt and trips

Commuter cost savings

Monitoring Levels of monitoring

Regional TMA Employer Commuter / customer

Types of monitoring Behaviour change Attitudinal change Customer service

Monitoring Regional

Existing sources– Statistics Canada– Transportation Tomorrow Survey (TTS)

– Conducted by University of Toronto– Detailed behaviour survey– Conducted every five years

– Cordon Count Program– Conducted by municipalities, compiled by University of Toronto– Observed data– Conducted every two to three years

Monitoring Regional

New sources– Commuter Attitudes Survey

– Conducted by Metrolinx– Detailed attitudinal questions– Conducted every two years

– Carpool Zone– Conducted by Metrolinx– Monthly activity survey

– TMAs and Employers– Compiled data from localized sources

Monitoring TMA

Quarterly reports – tied to Metrolinx funding Summary of activities per employers Communications and outreach activity Qualitative results Comparison of activity levels between TMAs

Monitoring Employer – Baseline Module

Baseline employee survey– Standardized across all TMAs and employers– Census survey

Site assessment– Current conditions– Inventory of advantages and deficiencies

Vehicle and occupancy count– Discreet sites– 1,000+ employees

Monitoring Customer – Commuter

Service-related– Monthly Carpool Zone microsurvey– Emergency Ride Home post-ride survey

Employee surveys Campaign-related data collection e.g. Car Free Day

Monitoring

ModeStatistics Canada (2006)

TTS(2006)

CAS(2008)

Surveys(2001-08)

Drive Alone 65.90% 64.7% 48% 75.91%

Transit 19.93% 19.7% 28% 10%

Cycling 0.97% 0.7% 1% 0.89%

Evaluation Benchmarks and Trends

75 employee baseline surveys Statistics Canada Transportation Tomorrow Survey Cordon Counts Commuter Attitudes Survey

Evaluation External

Employer surveys– Biannual survey – program satisfaction rating– Satisfaction with Smart Commute services– Conducted by third party through Metrolinx

Commuter surveys– Employee follow-up surveys – program satisfaction rating– Annual survey – Carpool Zone satisfaction

Internal Smart Commute Technical Committee

Evaluation External Review

Consultant evaluation of operational models

Research Partnerships University of Toronto (Mississauga)

– Factors for successful carpool formation– Further research

Ryerson University– TMA Toolkit assessment

Results Almost 100 members and partners

March 2007 – 50 employers March 2008 – 75 employers June 2008 – 89 employers

Results Transportation indicators

76.4 million vkt 1.3 million trips Equivalent of 10,000 cars Increase in carpooling

– 7% to 12% to 13%

Cars Off the Road

Results Primary Mode Shift

2

5

7

34

50

0 20 40 60

%

1

6

13

28

48

0 20 40 60

%

1

4

12

31

48

0 20 40 60

%

Drive alone

Public Transit

Car/vanpool

Walk/Jog

Bike

2006 20052008

Results Environmental indicators

17,500 tonnes of greenhouse gas emissions– Enough to fill Rogers Centre almost six times

100,000 kg of criteria air contaminants– Equivalent amount of NOx to smoking 3 billion cigarettes

Results Communications and Outreach

400+ stories 75+ million media impressions Brand recognition

– Smart Commute ↑– Carpool Zone ↓

5

9

14

27

17

0 10 20 30

%

Smart Commute

Carpool Week

Clean Air Commute

Carpool Zone Website

Commuter Challenge

Awareness (2008)

Challenges Pressures to implement right away

Show action Pent-up demand More interesting than planning or evaluating

All the pieces not always in place Monitoring not appreciated until results urgently

needed Objectivity

Of course my ‘baby’ was successful!

Challenges Measurement and Evaluation

Employer buy-in is difficult– Some see value in surveys, monitoring– Others feel employees are oversurveyed– Often considered an expensive waste of employee time

Standardization required, but unenforceable– Moving to centralized service provision– Standardization vs. continuous improvement

Data comparison can be unequal– Apples to apples and apples to oranges

Challenges Defining success

Setting targets without enough information– Are examples applicable?

Are targets achievable with the resources alloted? Can these targets be effectively monitored? What happens if we don’t reach these targets?

Lessons Measurement and Evaluation

Some items simple, but time-consuming– Do you really need the data?

Project benefits less easy to isolate– Double-counting also possible through employee surveys

and Carpool Zone surveys– External factors – environmental concerns, gas prices

One clear conclusion: an impact was made. Time and consistency open up opportunities

– Benchmarking

Lessons Measure and Evaluation Access resources at your disposal for help

Universities Municipalities Provincial and Federal Governments

Evaluate what you do and how you do it Give an adjustment period to monitoring

Test! Test! Test! Allow partners time to understand and see value

Lessons Provide incentives for monitoring

Tie completion to access Award and reward success

Measurement and Evaluation Learn from mistakes – you will make them! Learn from poor results

– Was the program effective?– Was the monitoring effective?

Next Steps Continue and improve data collection

Address data management and manipulation Benchmarks and comparisons Strategic planning

Service evaluation Targets

Baseline module refinement

Questions?Ryan Lanyon

Team Lead, Smart CommuteMetrolinx

ryan.lanyon@metrolinx.com416-874-5933