Program Evaluation of In-Situ Simulation Team Training
-
Upload
melissa-powell -
Category
Education
-
view
318 -
download
0
description
Transcript of Program Evaluation of In-Situ Simulation Team Training
Program Evaluation of
In Situ Simulation-based Team Training
Melissa Powell BSN, MSHCA, RN
I’m a novice to conducting research I’ve made a few mistakes along the way I’m eventually going to get there
Just getting started:This is an early pilot and investigation
Late 2009 and early 2010, multiple anecdotal stories of confusion and communication problems
Post Code Quality Survey Out of ICU/RRT debrief Committee meets
weekly. Remediation and debrief process by
educator in place for failure to initiate early rescue and RRS activation. Qualitative data collected after each meeting.
Needs assessment
1. “He or she {physician} is right here at the bedside with me. And that feels very uncomfortable as though it says to the physician that I do not think they are capable.”
2. “He or she went to medical school. I didn’t. I’m just a nervous new nurse. They know what they are doing.”
3. “I feel like I’m tattling on them.” 4. “The doctor is responding to my concerns with orders.” 5. “There is nothing anyone can do for this patient anyway.” 6. “There is something not right about this patient but my vital signs are normal, I
just don’t want to call them over here and waste their time.” 7. “There aren’t any critical care beds right now. The ED is on diversion. I know
they are busy.” 8. “The {responder} nurses just roll their eyes at me and don’t do anything.” 9. “The charge nurse thinks everything is okay so I don’t want to call and override
her.” 10. “Multiple nurses at bedside: “I didn’t call because my charge nurse was there and
there were others with me and felt I had enough different people involved”. 11. “What else would RRT do, besides what we are already doing?” 12. If the patient wasn’t transferred they think that it wasn’t the right thing to do to
call. 13. If the patients code status is DNR/DNI they do not think they should call,
regardless of goals of care or team knowledge.
Qualitative data from nurses related to delay and failure to rescue
Authority gradient phenomenon Inability to express concern clearly Other staff discounting judgment Lack of organization support Issues of diminished culture of safety Responder communication poor Lack of knowledge System and process failure
Themes extracted from qualitative information
Ineffective communication is a root cause for nearly 66 percent of all sentinel events reported*• (The Joint Commission Root Causes and Percentages for
Sentinel Events (All Categories) January 1995−December 2005)
• Vanderbilt Innovation Pillar goal 2011: Innovate with new approaches to Interprofessional learning.
What is the problem? The driver?
Experiential learning theory -- (Kolb, 2000) Social Cognitive learning theory -- (Bandura,
1960) Interprofessional team training evidence --
Multiple studies (Salas, 2008) (Gaba, 2004) (Miller, 2008) (Small, 2008)
AHRQ TeamSTEPPS evidence (Salas, 2008) Diffusion through Innovations -- (Rogers,
1962)
Theoretical framework for developing the intervention
Will in situ team training improve team member perceptions of “chaos” during acute events?
Will in situ team training with TeamSTEPPS improve team member perceptions of “good” communication during acute events?
Will in situ team communication training with TeamSTEPPS curriculum decrease delay and failure to rescue?
Hypothesis Questions:
Program Evaluation using Moore’s Framework for determining Learning and Assessment
Level 1 • Participation
Level 2• Satisfaction - Learners
reactions to the training
Level 3
• 3a Declarative Knowledge - Modifications to knowledge, attitudes, perceptions
• 3b Procedural Knowledge - Acquisition of procedural knowledge and skills
Level 4• Competence - Demonstration
of skill in training
Level 5
• Performance - Changes in workplace behavior, competencies and attitudes
Level 6
• Patient Health – Changes in patient health or workplace systems as a result
Level 7 •Community Health – Changes in the health of a community
Unit determined by the following VUH standard:
Required Bi-annual AHA Training for all staff delivered by VRP. (Policy 30-1.24)
Education regarding adherence to Rapid Response System process (Policy 32.89.0) received in orientation.
Required Bi-annual “Mock Code” simulation in “Hands on Clinical Safety”.
All nurses debriefed about failure to rescue.
Description of control group
Convenience sample Required Bi-annual AHA Training for all staff.
(Policy 30-1.24) Education regarding adherence to Rapid
Response System process (Policy 32.89.0) Blended Learning Intervention:
◦ Online education regarding teamwork and communication
◦ On unit setting with interprofessional team ◦ Briefing before simulation◦ High fidelity Interprofessional simulation of patient
decline and cardiopulmonary arrest requiring early activation of rapid response system and BLS
◦ Debrief using Rudolph’s “non-judgmental” process
Description of Intervention group
Post code data & # failure to rescue events
Intervention group (100% unit) Level 1
Online education (Level 3a)
Simulation (Level 3b, Level 4)
Debrief (Level 4)
Survey (Level 2)
Real event (Level 5)
Post
code
quality
survey
(Level 5) &#
Failure to rescue events
(Level 6)
Program evaluation using Moore’s Method
Participation was mandatory for intervention unit. 100% completion.
Most staff was excited to have an opportunity to be a part of simulation training and signed up early for sim times.
Few staff had anxiety. Reassurance and encouragement was necessary for a few staff members.
Satisfaction: Initial paper survey response rate very highly rated simulation experience: 63 surveys received: 92% rated unit based training with simulation “very satisfactory” as a learning method.
Participation and Satisfaction:Level 1 and Level 2
Post training simulation survey
Strongly agree strongly disagree
Post training survey
Excellent poor
All complete AED Heartsaver or BLS course Performance of CPR Pass a test on AHA guidelines for rescue
All staff are communicated to about RRS All new hires are verified that they were informed
and read policy about RRS. Signs in all patient rooms and staff break room Cultural awareness No formal test of knowledge
Declarative and Procedural Knowledge Evaluation (Level 3a &b)
BLS skills (80%) Procedural knowledge (50%) Equipment knowledge (unable to assess during due
to SimMan3G barriers) Assessment and Intervention (99%) Recognizing and Initiating RRT immediately (55%) Communication skills
◦ SBAR (20%)◦ CUS (.1%)◦ Check Back (34%)◦ Two-Challenge Rule (0%)◦ Situation Monitoring including Shared Mental Model
knowledge (34%)
Simulation Checklist (Level 4)
***This data is based on video observation and very rough and unable to validate fully.
Post training survey for commitment to change practice
Strongly agree strongly disagree
Post RRT and Code event data obtained from team members obtained through convenience and ability to discover who would voluntarily complete online survey.
No systematic process in place currently to identify and contact all or some members of an acute event.
The sample frame or the population size is unknown and varies event to event.
Level 5 Performance
Pre July 2010
Post InterventionDec 2010
difference Post interventionConfidence Intervals
Intervention Unit
8% 8.3% 0.3% +/- 8%
Control Unit 9% 9.5% 0.5% +/- 8%
Post Acute Event Data Analysis % Team member perceptions of “good communication” “strongly agreed”
With 90% confidence 8% of the participants in the survey sample answered “strongly agreed," we could be sure that between 1% and 18% of the members of the entire target population would also say “strongly agreed" when asked the same question. The confidence interval, in this case,is +/- 8%.
My sample is not random so my ability to report is not accurate.
Pre July 2010
Post Dec 2010
difference Confidence Intervals
Intervention Unit
20% 0% 20% +/- 6%
Control Unit 19% 4.8% 14.2% +/- 6%
Post Acute Event Data Analysis % Team member perceptions of “chaotic” that “strongly agreed”
Interviewed 6 team leaders of STATS obtained through ability to identify and meet.
Questions asked: ◦ Tell me how the code went that occurred on (unit)?◦ How was the communication?◦ What could the team have done better?◦ Did you all debrief afterward?
Unable to record conversations completely. Some situations were random. Information regarding team work was consistently very satisfactory in regards to teamwork on intervention unit. Additionally, this approach yielded the highest quality feedback.
Attempted to triangulate through interview
Data obtained from Out of ICU meetings and review of events
There were fewer delay and failure to rescue.
The behavior is still occurring.
Level 6 Patient Health
Jan 2010 to July 2010
Jan 2011 to July 2011
Intervention Unit
23 6
Control Unit 18 12
Out of ICU Committee Data Analysis Number of delay and failure to rescue
Delay to rescue is determined by patient meeting a rapid response trigger criteria for more than 60 minutes.Failure to rescue is determined by patient declining over more than 6
hours before RRS activated.
Debriefing nurses post intervention yield stronger self reflective ability and that learning occurred
Using concepts such as authority gradient to communicate what they experienced
Staff now often identifies their own barriers and role in early activation and failure to rescue
Level 5 Performance
Did not continue qualitative data collection of debriefs
In Situ Simulation Team Training is effective. I recommend Vanderbilt leaders take a look at equipping unit educators with the skills.
Communication skills are a hard skill. Best methodology is not a power point but simulation.
Culture of safety is important and must take multiple angles to address.
To get to “zero” delay and failure to rescue, we must improve process, system, education and communication skills of providers.
Diffusion of Innovation is occurring
CONCLUSIONS
Researcher bias, confounded by known identity of intervention group
Training works versus no training. Team Training works with multiple levels of evidence for the last 10 years.
Teams that are trained in team communication rate themselves lower after training.
No triangulation of observed effects. No independent raters of performance.
Inability to train all educators to level of competence for intervention
Doesn’t inform beyond existing research literature. Must find measurable outcomes.
Limitations to generalization and publication of findings
Pilot idea, discussed in multiple meetings with leaders and experts in simulation team training.
Held initial simulations with team training experts and received training in event based scenario development, team observation, checklist development and rating, and debriefing skills.
Ran 68 simulations in a 10 month period. Videoed simulations and debriefs to improve and practice observation skills and debrief skills.
Evolution of investigation
July 2012, acquired TeamSTEPPS Master Trainer training and status through AHRQ.
Continue to refine process through improved coordination of departments and leadership buy in.
Created a temporary unit simulation space in a large equipment room.
Refined data collection methods and survey instrument. NO MORE SLIDER SCALES. LIKERT ONLY.
Begin collaboration with other units.
Evolution continues
Bandura. (2001). Social cognitive theory: An agentic perspective. Annual Review of Psychology, 52(1), 1. Clancy, C. M. (2007). TeamSTEPPS: assuring optimal teamwork in clinical settings. American journal of
medical quality, 22(3), 214. Gaba, D. M. (2004). The future vision of simulation in health care. Quality & Safety in Health Care, 13(suppl
1), i2. Holzman, R. S. (1995). Anesthesia crisis resource management: real-life simulation training in operating
room crises. Journal of clinical anesthesia, 7(8), 675. Kolb, D. A. (1983). Experiential Learning: Experience as the Source of Learning and Development (1st ed.).
Prentice Hall. Miller, K. K. (2008). In situ simulation: a method of experiential learning to promote safety and team
behavior. The Journal of perinatal & neonatal nursing, 22(2), 105. Moore, D. E., Green, J. S., & Gallis, H. A. (2009). Achieving desired results and improved outcomes:
Integrating planning and assessment throughout learning activities. Journal of Continuing Education in the Health Professions, 29(1), 1–15. doi:10.1002/chp.20001
Nunnink, L. (2009). In situ simulation-based team training for post-cardiac surgical emergency chest reopen in the intensive care unit. Anaesthesia and intensive care, 37(1), 74–8.
Peberdy, M. A., Callaway, C. W., Neumar, R. W., Geocadin, R. G., Zimmerman, J. L., Donnino, M., Gabrielli, A., et al. (2010). Part 9: Post–Cardiac Arrest Care 2010 American Heart Association Guidelines for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care. Circulation, 122(18 suppl 3), S768–S786. doi:10.1161/CIRCULATIONAHA.110.971002
Rogers, E. M. (1995). Diffusion of innovations. Free Press. Rudolph, Jenny, W. (2006). There’s No Such Thing as “Non-judgmental” Debriefing: A Theory and Method
for Debriefing with Good Judgement. Simulation in Healthcare, 49–55. Salas, E., DiazGranados, D., Klein, C., Burke, C. S., Stagl, K. C., Goodwin, G. F., & Halpin, S. M. (2008). Does
Team Training Improve Team Performance? A Meta-Analysis. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50(6), 903–933. doi:10.1518/001872008X375009
Small, S. D. (2008). Demonstration of high‐fidelity simulation team training for emergency medicine. Academic emergency medicine, 6(4), 312.
Vanderbilt Policy Rapid Response Team Activation Policy number CL 30-08.16 Vanderbilt Policy Cardiopulmonary Resuscitation Policy number CL 30-08.21.
References