Measuring Regulatory Excellence: Key Performance Indicators
description
Transcript of Measuring Regulatory Excellence: Key Performance Indicators
![Page 1: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/1.jpg)
2007 Annual Conference
Measuring Regulatory Excellence: Key
Performance IndicatorsBruce G. Matthews, P.Eng.
Professional Engineers Ontario
![Page 2: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/2.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Overview• Project Definition and Scope
• What is a Key Performance Indicator?
• Project Process
• Challenges
• List of Potential KPIs
• Survey re: Relevance and Importance
• Recommended KPIs
• Aftermath
![Page 3: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/3.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Project Definition and Scope
• Project arose from a Strategic Plan Objective: “Increase the public’s confidence in PEO as a regulator”
• Associated Strategic Initiative: “Develop and implement key performance indicators and targets of regulatory effectiveness, benchmarked against other comparable regulatory bodies.”
![Page 4: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/4.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Project Definition and Scope con’t
• Literature review re: best practices
• Review of other regulators’ performance measurement practices
• Survey of stakeholders re: perceptions and expectations
• Assess availability of data for ongoing tracking / reporting of KPIs
![Page 5: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/5.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
What is a Key Performance Indicator?• A metric used to reflect the critical success
factors of an organization
• Can be financial or non-financial
• Nature of KPIs will differ depending upon the nature of an organization
• A KPI is a key part of a measurable objectives (which includes a direction, KPI, benchmark, target, timeframe)
![Page 6: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/6.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Effective KPIs…• Should help define:
– The What – Are we doing the right things? What is our outcome in terms of impact and accomplishments?
– The How – Are we doing things right? Do we have the right inputs in terms of resources and processes?
![Page 7: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/7.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Project Process• Literature Review
• Environmental Scan - Review performance measurement practices of other professional regulators
• Identify potential KPIs
• Survey stakeholders re: relevance and importance of potential KPIs
• Review pros and cons of potential KPIs
![Page 8: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/8.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Literature Review re: Best Practices
• Performance measurement best practices– Balanced scorecard– UK National Consumer Council– “The Regulatory Craft”– “Red Tape to Smart Tape” International
Conference (2002)
![Page 9: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/9.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Balanced Scorecard• Developed by Harvard professors Kaplan
and Norton
• Four categories– Customers– Internal Business Processes– Financial– Organizational Capabilities
• Weakness re: definition of “customer”
![Page 10: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/10.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
UK National Consumer Council
• Three steps to credible regulation– Objectives– Content– Structure and Governance
• www.ncc.org.uk
![Page 11: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/11.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
“The Regulatory Craft”• Book by Malcolm K. Sparrow (Brookings
Institution Press, 2000)
• Controlling risks, solving problems and managing compliance
• Matrix of “Regulatory Competencies” and “Business Results”
• Acknowledges “internal” versus “external” measures
![Page 12: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/12.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Red Tape to Smart Tape Conference
• Sponsored by the Ontario Government in association with the Institute of Public Administration of Canada
• Core concept – Traditional business results measurements do not address public welfare issues and are not suitable for regulatory purposes
• www.smarttape.ca
![Page 13: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/13.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Environmental Scan• Other engineering regulators in Canada
• Other professional regulatory bodies in Ontario and across Canada
• Most report key transactional statistics (e.g., number of licence applications)
• Little evidence of outcome measurement
![Page 14: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/14.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Challenges• What does it mean to be an effective
regulator?
• Inability to benchmark due to lack of reporting by other “comparable regulatory bodies”
• Can’t use measures typical for professional associations
• KPIs must be evaluated in context
![Page 15: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/15.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
List of Potential KPIs• Competence programs
• Intrusions on regulatory mandate
• Complaints about licence holders or unlicensed individuals
• Licensing turnaround times
• Compliance action statistics
• Council evaluation
![Page 16: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/16.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
List of Potential KPIs con’t
• Fairness & dignity survey
• Complaints about PEO
• Employee engagement
• Percentage of licence holders with no reported violations
• Public confidence survey score
• Ratio of administrative overhead to program delivery cost
![Page 17: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/17.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Survey re: Relevance and Importance
• PEO Council, Senior Management and selected external stakeholders
• Relevance – the applicability of a KPI as a measure of PEO’s effectiveness as a regulator
• Importance – the priority that should be given to a KPI as a measure of PEO’s effectiveness as a regulator
![Page 18: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/18.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Survey re: Relevance and Importance
• Survey also included proposed short-term measures for each KPI, intended to cover the introductory transitional period
• Survey results were assessed in view of the best practices review– Need balance of internal and external
measures)– Need to ensure key regulatory competencies
are represented
![Page 19: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/19.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Recommended KPIs• Competence programs
• Complaints about licence holders or unlicensed individuals
• Compliance action statistics
• Employee engagement
• Public confidence survey score
![Page 20: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/20.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Competence Programs• Pros
– Competent members reflect a well regulated profession
– Competency is fundamental to professional regulation and easily understood by the public
• Cons– PEO currently has no competence programs
in place and none planned in the near term
![Page 21: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/21.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Complaints about Licence Holders etc.
• Pros– Easily measured (transaction statistics
already reported)
• Cons– Frivolous or vexatious complaints could skew
the data– Does complaint volume really speak to
regulatory effectiveness?
![Page 22: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/22.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Compliance Action Statistics
• Pros– Clear statement of enforcement activity– Some statistics already reported
• Cons– Need to define what qualifies as a
“compliance action”– Of greater interest to licence holders than to
the public
![Page 23: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/23.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Employee Engagement• Pros
– Regular survey of employees already in place– Survey is widely recognized and used
• Cons– Is PEO willing to take action in deficient
areas?– Are the survey areas relevant to a
professional regulatory organization?
![Page 24: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/24.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Public Confidence Survey Score
• Pros– Solid external measure of how PEO is
perceived– Survey would be conducted by an
independent research firm
• Cons– Frequency of measurement– Who is “the public”?
![Page 25: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/25.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Implementation Issues• Currently no competence programs in
place at PEO
• Measurement data exists for only three of the recommended KPIs
• Public Confidence Survey still in development
• Issues of KPI reporting (timing, frequency, format) still to be determined
![Page 26: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/26.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Aftermath• Project report received and considered by
PEO Council in November 2006
• Concern that recommended KPIs were all “lagging indicators” – there were no “leading indicators”
• Concern that recommended KPIs would drive policy and program development
• Concern about lack of competency programs
![Page 27: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/27.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Aftermath con’t• PEO Council struck a task force to further
examine the question of KPIs for PEO
• Task force report (June 2007) recommends a new project with more research work re: critical success factors before defining KPIs
• Status as of September 2007: awaiting new project charter / terms of reference
![Page 28: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/28.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Speaker Contact Information
Bruce G. Matthews, P.Eng.Manager, Complaints & DisciplineProfessional Engineers Ontario25 Sheppard Avenue West, Suite 1000Toronto, Ontario M2N 6S9T: 416-840-1076F: 416-224-9974E: [email protected]
![Page 29: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/29.jpg)
2007 Annual Conference
Measuring Regulatory Excellence – A Self Audit Tool Focused on Entry to Practice
Jan RobinsonRegistrar & CEO
College of Physiotherapists of OntarioSeptember 8, 2007
![Page 30: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/30.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Interesting times
• accountability
• transparency
• movement of internationally educated professionals
• public perspective
![Page 31: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/31.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Politics in Ontario
![Page 32: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/32.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Ontario Regulators for Access (ORA)
• Info Sharing
• Best practices
• Collaboration
![Page 33: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/33.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
“measures” ➪ government driven
![Page 34: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/34.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
report card to progress report
![Page 35: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/35.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Key Factors Included• # of internationally trained applicants
(annual comparisons)• top five source countries• types of license granted• project initiatives
Focus on “10”
![Page 36: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/36.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Thomson review
![Page 37: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/37.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Key Recommendations
• fair registration practice code
• information for applicants
• criteria for decisions
• reasonable costs
![Page 38: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/38.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
• alternative evidence for unattainable documents
• training
• third party review
• self audits with reporting
• appeal procedures
![Page 39: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/39.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
2002 – ISO 17024
General requirements for bodies operating certification schemes
for persons
![Page 40: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/40.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
ANSI
SCC
![Page 41: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/41.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
2006 Fair Access to Regulated Professions Act
![Page 42: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/42.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Key features
➪ access centre ➪ Fairness Commissioner ➪ audit
![Page 43: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/43.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Consider self audit tool
• gap analysis
• demonstrate commitment
• continuous improvement
• achieve best practice
![Page 44: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/44.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Focus on entry to practice
![Page 45: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/45.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Conversion to standards38 – registration10 - organization
![Page 46: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/46.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Examples• The College has policies and procedures
related to confidentiality and privacy
• The College has clear processes for reviewing, evaluating and making changes to registration requirements
![Page 47: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/47.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Examples• The College has uniform competence
requirements
• The College ensures that individuals with a conflict of interest are not involved in the decision-making process related to an applicant.
![Page 48: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/48.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Format
• standard definition
• evidence
• analysis
• recommendation
• ISO reference
![Page 49: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/49.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Key Findings
• identified no major gaps or concerns
• links between College and third party hard to evaluate
• Identified new areas for policy work activity
![Page 50: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/50.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
• identified areas where evidence weak
• tedious
• how often necessary?
![Page 51: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/51.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Key Outcomes
• confidence in fairness and transparency of process
• evidence of performance all included in one place
• celebration of excellence
![Page 52: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/52.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Next steps
• develop work plan to implement recommendations
• share with government and other regulators
![Page 53: Measuring Regulatory Excellence: Key Performance Indicators](https://reader036.fdocuments.net/reader036/viewer/2022062314/568142ca550346895daf1c0c/html5/thumbnails/53.jpg)
Council on Licensure, Enforcement and RegulationAtlanta, Georgia2007 Annual Conference
Speaker Contact Information
Jan Robinson
Registrar & CEO
College of Physiotherapists of Ontario
e-mail : jrobinson @collegept.org
www.collegept.org