Formative Evaluation of the Performance Indicators ... · to the Ministry and the field ... entry,...
-
Upload
nguyenquynh -
Category
Documents
-
view
218 -
download
0
Transcript of Formative Evaluation of the Performance Indicators ... · to the Ministry and the field ... entry,...
Formative Evaluation of theFormative Evaluation of thePerformance Indicators MonitoringPerformance Indicators MonitoringSystem (PIMS) for the Smoke-freeSystem (PIMS) for the Smoke-free
Ontario StrategyOntario Strategy
Gala Arh, Robert SchwartzGala Arh, Robert Schwartzand John Garciaand John GarciaOntario Tobacco Research UnitOntario Tobacco Research Unit
Ontario Tobacco Research Unit
The purpose of performance measurementThe purpose of performance measurementis not only to develop performance indicatorsis not only to develop performance indicatorsbut to establish a system to measure resultsbut to establish a system to measure resultswhile improving management (Yang & Jun,while improving management (Yang & Jun,2007)2007)
Ontario Tobacco Research Unit
Learning ObjectivesLearning Objectives
Recognize the context for development ofRecognize the context for development ofOntarioOntario’’s web-based Performance Indicators web-based Performance IndicatorMonitoring System (PIMS)Monitoring System (PIMS)
Understand the process for developing PIMSUnderstand the process for developing PIMSand be aware of the underlying systemand be aware of the underlying systemstructurestructure
Identify early implementation issues,Identify early implementation issues,important to consider when developingimportant to consider when developingsimilar systemssimilar systems
Learn from evaluation resultsLearn from evaluation results
Ontario Tobacco Research Unit
Performance measurement has beenPerformance measurement has beenpromoted as a utilitarian instrument for localpromoted as a utilitarian instrument for localgovernment management since the 1940sgovernment management since the 1940s((StreibStreib & & PoisterPoister, 1999), 1999)
To assist with government management,To assist with government management,performance measurement systems mustperformance measurement systems mustflow strategic information to Ministries toflow strategic information to Ministries toassist in decision making processesassist in decision making processes
Ontario Tobacco Research Unit
Why Develop a Performance MeasurementWhy Develop a Performance MeasurementSystem for Ontario?System for Ontario?
In 2004-05, Ontario implemented the comprehensiveIn 2004-05, Ontario implemented the comprehensiveSmoke-free Ontario StrategySmoke-free Ontario Strategy
$60M annual investment for 2007/08; six fold increase in$60M annual investment for 2007/08; six fold increase inMinistry of Health Promotion funding since 2003/04.Ministry of Health Promotion funding since 2003/04.
Program management responsibilities are greatlyProgram management responsibilities are greatlyexpanded:expanded: Decentralized program/service delivery modelDecentralized program/service delivery model 196 programs funded through local and provincial196 programs funded through local and provincial
transfer payment agencies to deliver cessation,transfer payment agencies to deliver cessation,prevention, and protection programmingprevention, and protection programming
Ontario Tobacco Research Unit
Ministry Requirements for a PerformanceMinistry Requirements for a PerformanceMeasurement SystemMeasurement System
Be work and time savingBe work and time saving Collect program management data usefulCollect program management data useful
to the Ministry and the fieldto the Ministry and the field Provide ready access to current programProvide ready access to current program
data, including aggregate datadata, including aggregate data Provide centralized file management andProvide centralized file management and
online access to online access to workplansworkplans and reports and reports Be readily adaptable and expandableBe readily adaptable and expandable
Ontario Tobacco Research Unit
Designing PIMS to Meet Diverse NeedsDesigning PIMS to Meet Diverse Needs
ProcessProcess RulesRules A glance at what emergedA glance at what emerged Formative evaluation findingsFormative evaluation findings
Ontario Tobacco Research Unit
PIMS Development and ImplementationPIMS Development and ImplementationProcessProcess
1.1. Collaboratively, developed programCollaboratively, developed programindicators.indicators.
2.2. Ministry of Health Promotion approvalMinistry of Health Promotion approvalobtained for all indicators.obtained for all indicators.
3.3. All users participate in work planning, dataAll users participate in work planning, dataentry, data importing and report generationentry, data importing and report generationtraining.training.
Ontario Tobacco Research Unit
PIMS RulesPIMS Rules
User-based and user-friendlyUser-based and user-friendly DynamicDynamic ResponsiveResponsive Work-savingWork-saving Easy to readEasy to read Beyond Indicators: Ample room for proseBeyond Indicators: Ample room for prose
Ontario Tobacco Research Unit
PIMS specificitiesPIMS specificities
280 280 useridsuserids have been defined in PIMS with have been defined in PIMS withmore than 130 distinct user rolesmore than 130 distinct user roles
6 distinct Public Health Unit (PHU) and 176 distinct Public Health Unit (PHU) and 17Transfer Payment Agency (TPA) programsTransfer Payment Agency (TPA) programs
PIMS functionality includes: work planning,PIMS functionality includes: work planning,performance monitoring, interim and finalperformance monitoring, interim and finalreporting, and variable time reportingreporting, and variable time reporting
A total of 1271 indicators in PIMSA total of 1271 indicators in PIMS
Ontario Tobacco Research Unit
Typical content flow between agencyTypical content flow between agencyand MHPand MHP
Ontario Tobacco Research Unit
Data Entry and Importing in PIMSData Entry and Importing in PIMS
Easy to use templatesEasy to use templates Imports data from existing electronicImports data from existing electronic
databases thus circumventing double datadatabases thus circumventing double dataentryentry
Frequency of data entry is program specificFrequency of data entry is program specific
Ontario Tobacco Research Unit
Performance Reporting FunctionalityPerformance Reporting Functionality
Separate reports for each indicator for anySeparate reports for each indicator for anytime periodtime period
Secondary time period reportingSecondary time period reporting Comparison to Tobacco Control AreaComparison to Tobacco Control Area
Network (TCAN) averageNetwork (TCAN) average Comparison to provincial averageComparison to provincial average Bar and line graphs for each indicatorBar and line graphs for each indicator
Ontario Tobacco Research Unit
Semi-annual and Annual ReportsSemi-annual and Annual Reports
All Ministry required reporting is generatedAll Ministry required reporting is generatedthrough PIMSthrough PIMS
Annual and semi-annual reports areAnnual and semi-annual reports areaggregations of monthly reports withaggregations of monthly reports withadditional narrative and uploaded financialadditional narrative and uploaded financialreportsreports
Ontario Tobacco Research Unit
GeertGeert BouckaertBouckaert (1993) argues that in order (1993) argues that in orderto be effective performance measurementto be effective performance measurementsystems need to meet the followingsystems need to meet the followingstandards:standards: Validity: sound, convincing and tellingValidity: sound, convincing and telling Legitimacy: measures are accepted from theLegitimacy: measures are accepted from the
bottom upbottom up Functionality: measures contribute to theFunctionality: measures contribute to the
maintenance or development of an organizationmaintenance or development of an organization We would add utility as wellWe would add utility as well
Ontario Tobacco Research Unit
Formative EvaluationFormative Evaluation
PIMS site solicits ongoing feedbackPIMS site solicits ongoing feedback Periodic web-based survey of all usersPeriodic web-based survey of all users Field visits for feedback and promotionField visits for feedback and promotion Semi-structured interviews with Ministry,Semi-structured interviews with Ministry,
Public Health Unit (PHU) and TransferPublic Health Unit (PHU) and TransferPayment Agency (TPA) usersPayment Agency (TPA) users
Evaluation conducted early in the PIMSEvaluation conducted early in the PIMSdevelopment processdevelopment process
Ontario Tobacco Research Unit
MethodsMethods
Feedback interviews began September 2007Feedback interviews began September 2007 22 interviews completed22 interviews completed Survey disseminated October 2007Survey disseminated October 2007 69 survey respondents of 138 potential69 survey respondents of 138 potential
respondents (50% response rate)respondents (50% response rate)
Ontario Tobacco Research Unit
When asked are the indicators in PIMSWhen asked are the indicators in PIMSclear, 55% found the indicators clear, 33.3%clear, 55% found the indicators clear, 33.3%found the indicators unclear, and 11.7%found the indicators unclear, and 11.7%found the indicators very unclear.found the indicators very unclear.
Ontario Tobacco Research Unit
As a tobacco control manager noted:As a tobacco control manager noted:Unfortunately a lot of the, of the informationUnfortunately a lot of the, of the informationon PIMS is, I want to say on PIMS is, I want to say ““bean countingbean counting””and I think thereand I think there’’s some need fors some need forimprovement in that area, because I thinkimprovement in that area, because I thinksome of that is useless information, it doesnsome of that is useless information, it doesn’’ttmatter how many pamphlets we handed out,matter how many pamphlets we handed out,you know, and ityou know, and it’’s hard when yous hard when you’’re in there in thethrow of things to keep track of those kind ofthrow of things to keep track of those kind ofthings. I think therethings. I think there’’s a lot, in the narratives a lot, in the narrative..
Ontario Tobacco Research Unit
When asked to rate the contribution of PIMSWhen asked to rate the contribution of PIMSas a method of reporting to MHP in terms ofas a method of reporting to MHP in terms ofits efficiency:its efficiency:
11.7% found PIMS very efficient11.7% found PIMS very efficient 53.3% found PIMS efficient 53.3% found PIMS efficient 26.7% found PIMS inefficient26.7% found PIMS inefficient 8.3% found PIMS very inefficient8.3% found PIMS very inefficient
Ontario Tobacco Research Unit
When asked to rate the contribution ofWhen asked to rate the contribution ofPIMS as a method of reporting to MHP:PIMS as a method of reporting to MHP:
18.3% found PIMS very helpful18.3% found PIMS very helpful 55% found PIMS helpful 55% found PIMS helpful 26.7% found PIMS not helpful 26.7% found PIMS not helpful
Ontario Tobacco Research Unit
As one interview participant noted:As one interview participant noted: They [MHP] have the information on aThey [MHP] have the information on a
more regular basis, so I would say that theymore regular basis, so I would say that theyhave, faster knowledge I guess, or morehave, faster knowledge I guess, or moreimmediate knowledgeimmediate knowledge..
Ontario Tobacco Research Unit
Another participant observed:Another participant observed:I think itI think it’’s probably a truer accountability,s probably a truer accountability,not that I was, you know, not being clearnot that I was, you know, not being clearbefore but I think itbefore but I think it’’s just, well its just, well it’’s a shorters a shorteramount of time itamount of time it’’s a clearer, clearer datas a clearer, clearer dataso I think it just makes it a betterso I think it just makes it a betteraccountability.accountability.
Ontario Tobacco Research Unit
Asked, to what extend has PIMS helped yourAsked, to what extend has PIMS helped yourorganization track progress in accomplishingorganization track progress in accomplishingworkplanworkplan targets: targets:
36.7% found PIMS useful 36.7% found PIMS useful 48.3% found PIMS not useful 48.3% found PIMS not useful 15% found it useless15% found it useless
Ontario Tobacco Research Unit
PIMS is PIMS is ““a very good check and balance fora very good check and balance forus internally here.us internally here.”” PIMS helps in PIMS helps in ““makingmakingsure that we stay focusedsure that we stay focused”” reported a PHU reported a PHUinterviewee.interviewee.
Ontario Tobacco Research Unit
When asked to rate the contribution of PIMSWhen asked to rate the contribution of PIMStowards program planning:towards program planning:
3.3% found PIMS very helpful 3.3% found PIMS very helpful 43.3% found PIMS helpful43.3% found PIMS helpful 46.7% found PIMS not helpful46.7% found PIMS not helpful 6.7% found PIMS to be useless6.7% found PIMS to be useless
Ontario Tobacco Research Unit
As one interview participants commented:As one interview participants commented: I think it [PIMS] is a helpful tool and I thinkI think it [PIMS] is a helpful tool and I think
yes, it has affected our decision making inyes, it has affected our decision making interms of, where we put our resources, howterms of, where we put our resources, howmuch wemuch we’’re you know the divide perhapsre you know the divide perhapsbetween advocacy, between prevention,between advocacy, between prevention,cessation, those types of issues I think itcessation, those types of issues I think itdoes, it does make a differencedoes, it does make a difference..
Ontario Tobacco Research Unit
Another interviewee noted that PIMS Another interviewee noted that PIMS ““forcesforcesyou to reflect, forces you to provide thatyou to reflect, forces you to provide thattime.time.”” Suggesting that the process of Suggesting that the process ofproviding data to PIMS is beneficial inproviding data to PIMS is beneficial inprogram planning.program planning.
Ontario Tobacco Research Unit
Early Ministry FeedbackEarly Ministry Feedback
System is intuitive, easy to use. For example,System is intuitive, easy to use. For example,Ministry review comments made directly to workMinistry review comments made directly to workplans and users automatically notified.plans and users automatically notified.
Logic of programs and fit within the SFO StrategyLogic of programs and fit within the SFO Strategyis clearly defined at the program level.is clearly defined at the program level.
Automated centralized file management is anAutomated centralized file management is anasset.asset.
Ontario Tobacco Research Unit
Early Ministry FeedbackEarly Ministry Feedback
Work planning in PIMS is a departure fromWork planning in PIMS is a departure fromconventional work planning; difficult to discernconventional work planning; difficult to discernobjectives/priorities of program activitiesobjectives/priorities of program activities
Explanatory text boxes provided for indicators,Explanatory text boxes provided for indicators,but often not usedbut often not used
Indicators are numerous and require adjustmentIndicators are numerous and require adjustmentto better reflect critical activities, outputs andto better reflect critical activities, outputs andassociated outcomesassociated outcomes
Essential to provide training and on-goingEssential to provide training and on-goingtechnical assistance to both Ministry and TPAtechnical assistance to both Ministry and TPAstaffstaff
Ontario Tobacco Research Unit
Preliminary Evaluation ResultsPreliminary Evaluation Results
Overall compliance with work planning andOverall compliance with work planning andon-going reporting in PIMSon-going reporting in PIMS
Data collected through PIMS largely unusedData collected through PIMS largely unusedby program staff/management as reportingby program staff/management as reportingfunctionality is very newfunctionality is very new
Ontario Tobacco Research Unit
System ImprovementsSystem Improvements
New work plan format for this fiscalNew work plan format for this fiscal Increased search functionalityIncreased search functionality Indicator refinementsIndicator refinements
Ontario Tobacco Research Unit
ConclusionConclusion The development of performance measurement systems isThe development of performance measurement systems is
an ongoing iterative processan ongoing iterative process In terms of validity, the performance measures in PIMSIn terms of validity, the performance measures in PIMS
require refinementrequire refinement Legitimacy has been established for all Legitimacy has been established for all TPAsTPAs by a by a
collaborative process of developing performancecollaborative process of developing performancemeasures. For PHUs this process could be moremeasures. For PHUs this process could be morecollaborative.collaborative.
Functionality for work planning and tracking work planFunctionality for work planning and tracking work plantargets requires improvementtargets requires improvement
And, utilization cannot yet be ascertained as the reportingAnd, utilization cannot yet be ascertained as the reportingfunctionality was rolled out after the evaluationfunctionality was rolled out after the evaluation
Ontario Tobacco Research Unit
Contact InformationContact Information
Gala ArhGala ArhOntario Tobacco Research UnitOntario Tobacco Research Unit(416) 978-8306(416) [email protected]@utoronto.ca
Robert SchwartzRobert SchwartzOntario Tobacco Research UnitOntario Tobacco Research Unit(416) 978-3901(416) [email protected]@utoronto.ca