Regulation University: The Consequences of Regulatory Accumulation
CORE Evidence-/based Regulation: A Regulatory Performance Measurement System.
-
Upload
jocelyn-barr -
Category
Documents
-
view
223 -
download
0
Transcript of CORE Evidence-/based Regulation: A Regulatory Performance Measurement System.
COREEvidence-/based Regulation:
A Regulatory Performance Measurement System
Goal
To help State Boards of Nursing improve their management and
service delivery
Need for Study
Multiple stakeholders were demanding accountability
Lack of clarity among stakeholders about Board roles and responsibilities
Trend toward outcome measurement at state level
Purpose of the Study
Incorporating data from internal and external sources
Using benchmarking strategies Identifying best practices
Establishment of a Performance Measurement System
History
In 1998, the NCSBN Board of Directors appointed a project advisory group to provide oversight and guide development of a performance measurement system that incorporates data collection from internal and external sources, identification of best practices, and the use of benchmarking strategies.
History
Twelve dedicated member boards of nursing (Kentucky, Louisiana-RN, Maryland, Missouri, Nebraska, New Mexico, North Carolina, North Dakota, Ohio, Tennessee, Texas-RN and West Virginia-PN) contributed time and resources to the development of 10 instruments used to gather data in five areas: discipline, licensure, practice, education and governance.
Phase 1 of the Project
1. Validation of Board Roles
Study conducted by The Urban Institute
Methods to Gather Information
Interview– Focus Groups– Individuals
Review of Secondary Documents
Results of Phase 1:
Roles Identified– Establish scope of practice for nurses– Issue licenses to qualified nurses– Assure continued competence– Investigate complaints and impose
disciplinary sanctions as appropriate
Phase 2 of the Project
1. Validation of Board Roles
2. Identification of Performance Indicators
Results of Phase 2:
Technical Work Group developed– Performance indicators– Outcome Indicators– Output Indicators– Efficiency Indicators
Examples
Performance Indicator– Timeliness of complaint handling
Outcome Indicator– Average time for complaint resolution
Examples continued
Output Indicator– Number of complaints resolved in FY
Efficiency Indicator– Average cost per completed complaint
Phase 3 of the Project
1. Validation of Board Roles
2. Identification of Performance Indicators
3. Tool Development
Tool Development Process
Original tools developed and piloted– Because the science of regulation was not formed
enough to guide their efforts, they did not know which data would prove to be the crucial evidence and so they included all data.
Original tools revised and further tools developed– 6 data collection tools for boards of nursing– Surveys to collect data from 6 stakeholder groups
Board Surveys
1. Discipline
2. Licensure
3. Education Program Approval
4. Practice
5. Governance (Executive Staff)
6. Governance (Board President)
Board Surveys Included
Processes used, e.g.:– Investigator caseloads– Use of site visits or self-reports for
education programs
Timeliness issues, e.g.:– Days needed to processes license request
Board Surveys Included
Outcomes achieved, e.g.:– Number of discipline cases closed
Opinions, e.g.:– From Executive Staff and Board President
Stakeholder Groups
1. Licensed nurses
2. Health care employers
3. Nurses who had been the subjects of complaints
4. Persons who had lodged complaints
5. Nursing associations
6. Nursing education programs
Stakeholder Surveys Included
Perceptions of board’s– Timeliness,– Fairness,– Adequacy of regulation, etc.
Satisfaction with board’s– Communication with stakeholder group,– Nursing program approval process, etc.
Phase 4 of the Project
1. Validation of Board Roles
2. Identification of Performance Indicators
3. Tool Development
4. Data Collection
Data Collection
Stakeholder contact information submitted by boards– Random samples selected from those
submitted
6 data collection tools sent to boards of nursing
Data Collection Wisdom
Not all boards routinely collected the data asked for– Many boards used this as an opportunity to
improve/modify amount and types of data collected
Language/definitions (i.e., financial data, board processes) differed among boards
Phase 5 of the Project
1. Validation of Board Roles
2. Identification of Performance Indicators
3. Tool development
4. Data Collections
5. Reports of Findings
Report Format
I. Aggregate findingsA. Data results
1. From board surveys
2. From stakeholder surveys
B. Relationships among variables
II. State-specific findingsA. Comparison of state with all states
B. Comparison of state with “like” boards
Example of Comparison with Aggregate Data, i.e., Ed. Program Perceptions
Approval Process State Rating
Aggregate Rating
Interval betweenboard visits
2.66 1.40
Preparation time for board visits
2.45 1.41
Feedback/evaluation provided by board
2.16 1.38
Comparisons with Similar Boards
Boards evidenced a wide variety of resources, structures & processes
Boards were compared to other boards similar in a number a variables
Comparison Variables
Size of staff Staff assigned to specific functions Numbers of investigators Whether or not state mandated
reporting of errors Processes related to complaint review
Comparison Variables continued
Board structure Standard of proof Staff autonomy Number of board meeting per year Timeliness of discipline processes Timeliness of licensure processes
Comparisons of Sample Board with Other Boards with Similar Numbers of Staff
Sample Board
Similar Boards
Number of nursing programs 78 60.53
Rate of complaints resolved 0.78 0.66
Complaints per investigator 116.9 95.78
Nurses' perceptions of courteousness 1.23 1.23
Number of staff involved with investigations 12 7.92Number of board meetings/year 10 6.4
Phase 6 of the Project
1. Validation of Board Roles
2. Identification of Performance Indicators
3. Tool development
4. Data Collections
5. Reports of Findings
6. Search for “best practices”
The Search for Best Practices
Data were used to identify boards with consistently high ratings in– Outputs– Effectiveness
Ratings were explored in 5 functional areas– Discipline, licensure, education program
approval, practice and governance
The Search for Best Practices
Selected boards were interviewed to discover– Practices common among boards with
consistently high ratings– Differences from boards with lower ratings
Discipline Best Practices
Boards with the highest ratings on discipline outcomes– Delegated authority to board staff– Communicated well with stakeholders– Hired investigators and attorneys &
actively managed discipline process– Trained and mentored investigative staff– Applied discipline sanctions consistently
Licensure Best Practices
Boards with the highest ratings on licensure outcomes– Secured essential human and other
resources– Made an aggressive commitment to
customer service
Education Best Practices
Boards with the highest ratings on education outcomes– Provided consultative, as well as
evaluative services to education programs– Took a leadership role in establishing
congruence between education and regulation
Practice Best Practices
Boards with the highest ratings on practice outcomes– Facilitated understanding of legal scope of
practice– Made an aggressive commitment to customer
service– Established a high level of involvement with the
statewide nursing community– Delegated authority to board staff
Governance Best Practices
Boards with the highest ratings on governance outcomes– Promoted an understanding of the respective
roles of staff and board members– Built an effective working relationship and a high
level of trust between board and staff– Facilitated an effective working relationship
among board members– Demonstrated a commitment to board member
development
Phase 7 of the Project
1. Validation of Board Roles2. Identification of Performance Indicators3. Tool development4. Data Collections5. Reports of Findings6. Search for “best practices”7. Development of Ongoing System of
Performance Measurement
CORE
Nursing Boards educated on CORE– Manuals prepared, distributed and
explained– Ongoing presentations and publications
Best Practice “Tool Kit”– Submissions by boards of systems and
processes that have facilitated best practice
CORE
Ongoing improvement of data collection system– All survey items linked to outcome and/or
best practice– Data collection streamlined– Additional tools created and piloted
• Information Technology• Finance• Board Member
CORE Timeline
2006
Advisory Committee
Formed
2000
Stakeholder Surveys
Board Surveys
20021998
Urban InstituteStudy
Stakeholder Surveys
1999 2001 2003 2004 2005
RevisedQuestionnaires
Schedule
Jan Feb March April May June July Aug Sept Oct Nov Dec
Jan Feb March April May June July Aug Sept Oct Nov Dec
Conduct Stakeholder Surveys
Analyze Data
Report Results in Aggregate & by State
Conduct Board Surveys
Analyze Data
Report Results in Aggregate & by State
X
X
X
X
X
X X
X X
X
Board Participation2006 Stakeholder Surveys
Nurses
Nurses Subject of a Complaint
Persons Who Made
a Complaint About a Nurse Associations Employers
Education Programs
30 24 17 20 25 26
Total = 33 different boards participatedin 2006 Stakeholder Surveys
Utilization of Data by Boards of Nursing
Data has been used to: Support decision-making Develop mandated reports Provide information to legislators Change data management processes Improve stakeholder satisfaction Streamline processes Determine priorities
Benchmarking
"How are we doing?" "Are we tracking the right measures?" "How do we compare with others?" "Are we making progress fast enough?" "Are we using the best practices?"
CORE
PerformanceBenchmarking
Process Benchmarking
Identifying best practices
Adopting best practices
Performance benchmarking
The collection of (generally numerical) performance information and making comparisons with other compatible organizations.
Benchmarking of Board Performance
Quality Efficiency (cost effectiveness & timeliness)
Stakeholder survey Stakeholder surveyBoard survey (?) Board survey
Criteria of Item Selection
Measurable Comparable between states Objective, if possible
Licensure– Cost per licensee– Cost per license processed– Timeliness to process licensure
• by initial examination
• by endorsement
• by renewal
• by reinstatement
Benchmarking Board Efficiency
Practice– Cost per licensee– Number of practice questions handled per FTE
Benchmarking Board Efficiency
Education Program Approval– Rate of education site visits per staff
Benchmarking Board Efficiency
Discipline– Average number of investigations completed by each
investigator– Rate of investigations completed– Rate of complaints resolved– Rate of nurses disciplined– Timeliness of complaint handling – Cost per resolved complaint– Cost per investigation– Cost per licensee– Cost per discipline case
Benchmarking Board Efficiency
Mock Benchmarking Report
My board Similar boards All boards Complaints per investigator
117 95 112
Rate of complaints resolved
.78 .66 .66
Process Benchmarking
The comparison of practices, procedures and performance answering the question of What the best practice is in this topic and how to adopt the best practice in my board
Identification of Best Practice
How to define best practice How to identify best practice
Level I – Properly conducted RCT, systematic review or meta-analysis
Level II –Quasi-experimental, correlational, descriptive, survey, evaluation, and qualitative
Level III –Expert opinion or consensus statements
Levels of Evidence
From Nancy Spector, who adapted from Polit and Hungler (2004) & Gallagher (2003)
Identifying best practice
Choose a process to study: •One process at a time•If desire to study more than one process, it is best to set up separate studies for each process.
Identification of best practice
– With clear research questions– With clearly defined conceptual model
For example
What is the best practice for a discipline process?
A Conceptual Model
Board Practice Authority of staff Communication with PMCs
& CNAs Actively Managing
disciplinary process Mentoring investigative staff Consistent discipline
sanctions
Desired outcomes
Cost effectiveness
Perceived fairness and timeliness of the discipline process
Perceived quality of the board’s disciplinary process
Perceived effectiveness of board’s disciplinary process
Operationalize Variables
Authority of staff• Staff autonomy in resolving selected discipline cases
• Staff/Board relationship
Communication with PMCs and CNAs• Board communication to PMCs and CNAs before initiation
of investigative process and after the completion of the disciplinary process
• Board communication to PMCs and CNAs on rationale for the board action
• Availability of real person for questions
• Attitude of board toward CNAs
Operationalize Variables
Actively Managing disciplinary processa)Authority to hire or directly contract with the
investigators and attorneys involved with the disciplinary process.
b) Investigators being an RNc) Consistent process (including case
management and resolution)d) Investigators involved in the entire
disciplinary process
Operationalize Variables
Mentoring investigative staff Training for investigators that may or may
not include CLEAR training. Training materials available to novice
investigators. Mechanism of mentoring investigators
Operationalize Variables
Consistent discipline sanctions Enforcement processes are reviewed on a regular
basis to ensure consistency of discipline sanctions. The board develops policies and procedures to
ensure consistent handling of complaints. Formal mechanics are in place to review proposed
sanctions for consistency with previous action taken for similar cases.
Possible Areas of Interest
For example: What is the best time to renew a license? Are there any difference in outcomes
between using a RN as investigator vs. an non-RN investigator?
Performance Benchmarking
ChooseMeasuresChoose
MeasuresCollectData
CollectData
AnalyzeData
AnalyzeData
ProduceReport
ProduceReport
ReportAnnuallyReport
Annually
Process Benchmarking
ChooseProcessesChoose
ProcessesFormTeamFormTeam
DevelopBaselineDevelopBaseline
CompareProcessesCompare
ProcessesPlan forChangePlan forChange
Manage forChange
Manage forChange
Asked A Lot of Questions
? ? ????
????
? ?? ? ?
????
? ? ??? ?
??
?
?
Relationships
Timeliness of handling complaints # of FTEs or staff workload
Example:
More PrecisionQ: Were you treated fairly?
ORConsider the matters under review without any existing prejudice towards the people or organizations involved ( the decision-maker should act in a manner that is unbiased, fair and open-minded)
Communicating to the parties involved what is at issue, what decisions might be made adversely to the parties involved and what evidence and information the board may use in reaching its decision (the person should be informed of the matter)
Offering a procedure by which the parties involved may test, challenge or contradict the information and evidence used by the board in reaching its decision
Offering a procedure by which the evidence, information and submissions from the parties involved can be submitted to the board (opportunity to make oral or written submissions to the board)
The reasons for the board’s decisions must be based on the facts of the case under consideration The person should be told the reasons for the decision There is consistency between previous actions and proposed sanctions for similar cases
Next Steps
Identify and remove barriers to participation
Identify and remove barriers to participation
Performance Measures
Support member boards’ adaptations of best practices
Best Practices Identified by Focus
Groups
OR Committee Identified Best
Practice