Smart Commute EvaluationTools, Techniques and Lessons Learned in Monitoring and
Evaluating Workplace-based TDM Programs
Outline1. Background2. Approach3. Monitoring Tools4. Evaluation5. Results6. Challenges7. Lessons8. Next Steps
Ryan LanyonTeam Lead, Smart CommuteMetrolinx (GTTA)October 22, 2008
At-a-Glance Build evaluation into your initial planning Monitor activities, benefits and customer service Scrutinize and understand indicators Learn from results and mistakes Resources available at www.smartcommute.ca Just do it!
Smart Commute works with various stakeholders to relieve traffic congestion, improve air quality and health, and reduce emissions that cause climate change.
Background Smart Commute
Focus on commuters and workplaces Partnership with municipalities Staged development
– 2001: Pilot TMA - BCRTMA– 2004: Pilot regional implementation - Municipalities– 2008: On-going implementation – Metrolinx + Municipalities
Background Metrolinx (Greater Toronto Transportation Authority)
Formed in 2007 Coordinates transportation across GTHA Regional Transportation Plan 6.13 million residents, 3 million commuters
Background Two-tiered program delivery
Metrolinx– Central coordinating body– Centralized service operation e.g. Carpool Zone– TMA Toolkit– Funding agency – 50%
Local Smart Commute / TMA– Delivery agent– Municipality, NGO, Chamber of Commerce/Board of Trade– Find other funding – municipalities, grants, service fees
Background 2001: BCRTMA / NTV 2004: 404-7 2005:
Mississauga NE Toronto
2006: Brampton-Caledon Halton
2007: Central York Durham
2008: Toronto-Central 2009: Airport?
Background Employer program
Improve and encourage commuting options Assessment of current conditions - Baseline Development of a plan Implementation – ‘Commuter benefits program’
– Carpool Zone and ERH– Walking, cycling, transit, carpool, telework programs– Campaigns, contests and special events
Follow-up assessment – Measured against baseline
Approach - Ideal
Goals
ObjectivesRegional Transportation Plan
Municipal Plans Implementation
Strategic Plan Monitoring
Evaluation
Approach - Actual
Goals Objectives
RTP
Municipal PlansImplementation
Strategic Plan
Monitoring
EvaluationFederal Funding Requirements
Implementation
Implementation
Approach 2004-2007 - Contribution Agreement
Negotiated between funder and municipalities 2008 - Monitoring Framework
Established by Smart Commute stakeholders– TMAs and Smart Commute Association (implementers)– Municipalities and Metrolinx (funders)
Multipurpose– Measure activity levels for funders– Track impacts to justify funding– Collect statistics for marketing purposes
Approach Inputs / Activities
Number of businesses– Active– Engaged– Supporting
Number of pamphlets distributed Number of cycling programs Number of media releases Number of commuters reached
Approach Outputs / Benefits / Results
Brand awareness– % of employees aware; change from baseline
Commuter satisfaction– Ratings of Smart Commute program
Reductions– GHG emissions– CAC emissions– Vkt and trips
Commuter cost savings
Monitoring Levels of monitoring
Regional TMA Employer Commuter / customer
Types of monitoring Behaviour change Attitudinal change Customer service
Monitoring Regional
Existing sources– Statistics Canada– Transportation Tomorrow Survey (TTS)
– Conducted by University of Toronto– Detailed behaviour survey– Conducted every five years
– Cordon Count Program– Conducted by municipalities, compiled by University of Toronto– Observed data– Conducted every two to three years
Monitoring Regional
New sources– Commuter Attitudes Survey
– Conducted by Metrolinx– Detailed attitudinal questions– Conducted every two years
– Carpool Zone– Conducted by Metrolinx– Monthly activity survey
– TMAs and Employers– Compiled data from localized sources
Monitoring TMA
Quarterly reports – tied to Metrolinx funding Summary of activities per employers Communications and outreach activity Qualitative results Comparison of activity levels between TMAs
Monitoring Employer – Baseline Module
Baseline employee survey– Standardized across all TMAs and employers– Census survey
Site assessment– Current conditions– Inventory of advantages and deficiencies
Vehicle and occupancy count– Discreet sites– 1,000+ employees
Monitoring Customer – Commuter
Service-related– Monthly Carpool Zone microsurvey– Emergency Ride Home post-ride survey
Employee surveys Campaign-related data collection e.g. Car Free Day
Monitoring
ModeStatistics Canada (2006)
TTS(2006)
CAS(2008)
Surveys(2001-08)
Drive Alone 65.90% 64.7% 48% 75.91%
Transit 19.93% 19.7% 28% 10%
Cycling 0.97% 0.7% 1% 0.89%
Evaluation Benchmarks and Trends
75 employee baseline surveys Statistics Canada Transportation Tomorrow Survey Cordon Counts Commuter Attitudes Survey
Evaluation External
Employer surveys– Biannual survey – program satisfaction rating– Satisfaction with Smart Commute services– Conducted by third party through Metrolinx
Commuter surveys– Employee follow-up surveys – program satisfaction rating– Annual survey – Carpool Zone satisfaction
Internal Smart Commute Technical Committee
Evaluation External Review
Consultant evaluation of operational models
Research Partnerships University of Toronto (Mississauga)
– Factors for successful carpool formation– Further research
Ryerson University– TMA Toolkit assessment
Results Almost 100 members and partners
March 2007 – 50 employers March 2008 – 75 employers June 2008 – 89 employers
Results Transportation indicators
76.4 million vkt 1.3 million trips Equivalent of 10,000 cars Increase in carpooling
– 7% to 12% to 13%
Cars Off the Road
Results Primary Mode Shift
2
5
7
34
50
0 20 40 60
%
1
6
13
28
48
0 20 40 60
%
1
4
12
31
48
0 20 40 60
%
Drive alone
Public Transit
Car/vanpool
Walk/Jog
Bike
2006 20052008
Results Environmental indicators
17,500 tonnes of greenhouse gas emissions– Enough to fill Rogers Centre almost six times
100,000 kg of criteria air contaminants– Equivalent amount of NOx to smoking 3 billion cigarettes
Results Communications and Outreach
400+ stories 75+ million media impressions Brand recognition
– Smart Commute ↑– Carpool Zone ↓
5
9
14
27
17
0 10 20 30
%
Smart Commute
Carpool Week
Clean Air Commute
Carpool Zone Website
Commuter Challenge
Awareness (2008)
Challenges Pressures to implement right away
Show action Pent-up demand More interesting than planning or evaluating
All the pieces not always in place Monitoring not appreciated until results urgently
needed Objectivity
Of course my ‘baby’ was successful!
Challenges Measurement and Evaluation
Employer buy-in is difficult– Some see value in surveys, monitoring– Others feel employees are oversurveyed– Often considered an expensive waste of employee time
Standardization required, but unenforceable– Moving to centralized service provision– Standardization vs. continuous improvement
Data comparison can be unequal– Apples to apples and apples to oranges
Challenges Defining success
Setting targets without enough information– Are examples applicable?
Are targets achievable with the resources alloted? Can these targets be effectively monitored? What happens if we don’t reach these targets?
Lessons Measurement and Evaluation
Some items simple, but time-consuming– Do you really need the data?
Project benefits less easy to isolate– Double-counting also possible through employee surveys
and Carpool Zone surveys– External factors – environmental concerns, gas prices
One clear conclusion: an impact was made. Time and consistency open up opportunities
– Benchmarking
Lessons Measure and Evaluation Access resources at your disposal for help
Universities Municipalities Provincial and Federal Governments
Evaluate what you do and how you do it Give an adjustment period to monitoring
Test! Test! Test! Allow partners time to understand and see value
Lessons Provide incentives for monitoring
Tie completion to access Award and reward success
Measurement and Evaluation Learn from mistakes – you will make them! Learn from poor results
– Was the program effective?– Was the monitoring effective?
Next Steps Continue and improve data collection
Address data management and manipulation Benchmarks and comparisons Strategic planning
Service evaluation Targets
Baseline module refinement
Top Related