Quantitative Analysis of the Effectiveness of a Newly ... › careers › nursing › upload ›...
Transcript of Quantitative Analysis of the Effectiveness of a Newly ... › careers › nursing › upload ›...
Quantitative Analyses of the Effectiveness of a Newly
Designed Preceptor Workshop
VA Palo Alto Health Care SystemLilly Liu BMed, RN, CCRN, RN-BC
Denise Fillipucci PhD, RN, GCNS
Satish Mahajan PhD, MStat, MEng, RN
VA Palo Alto Health Care System
Covers 3 campuses- Palo Alto, Menlo Park, and Livermore serving more than 85,000 veterans
Operates >800 beds, 3 extended care facilities, and 100-bed Homeless Domiciliary
Home of several Regional Treatment Centers- Polytrauma Rehabilitation/Brain Injury, Spinal Cord Injury, Western Blind Rehabilitation Center, Geriatric Research Education and Clinical Center, National Center for PTSDTeaching hospital serving 29 schools of nursing and Stanford University School of Medicine
Approximately 1800 nursing staff members
Background
1- day Preceptor Workshop first implemented in 2010
• Adult learning theory
• Communication
• Giving feedback
• Learner evaluation
Internal data suggested important gaps existed
• Knowledge in areas of preceptor roles/responsibilities
• Teaching in clinical settings
• Providing support to learners
• Structuring orientations
• Critical thinking & evaluation
• Giving corrective feedback
• Managing challenging behaviors
Literature Review
Precepting perceived as stressful internationally
Preceptors lack educational preparation
Role ambiguity
Role overload
Lack of teaching time
Sense of added responsibilities
Lack of leadership support
Lack of recognition
(Hautala, Saylor, & O’Leary-Kelly, 2007; Panzavecchia & Pearce, 2014)
Literature Review
Formal preceptor programs elements
Educational preparation
Ongoing education
Support for preceptors
Delineation of preceptor roles and responsibilities
Specification of preceptor selection criteria
(L’Ecuyer, Hyde, & Shatto (2018); Ward & McComb (2017))
Literature Review
Learning Needs of Preceptors
Foy, Carlson & White (2013) Top 5 needs - feedback, critical thinking, prioritization, organization, expectations of preceptor
Johnstone & Mohide (2009) Addressing diversity in clinical education & preceptors’ understanding of own beliefs/values of diversity
Benston & Carlson(2015); Chang, Lin, Chen, Kang & Chang (2015); L’Ecuyer et al (2018); Tsai et al (2018)
Communication, teaching, learning strategies, creating caring learning environment, drafting clinical education plans, evaluating preceptees, managing stress and conflict
Literature Review
Best Practices for Program Design
Senyk & Staffileno (2017) Online learning or blended online and live classes -expand accessibility
Wilson et al. (2013) Simulation as an adjunct method - more helpful than lecture alone
Cotter & Dienemann(2016)
Create opportunities for networking and discussion among preceptors
Literature Review
ProgramEvaluation
Quantitative and qualitative measures are used in program evaluation
Literature review showed a lack of interventional studies designed with valid and reliable assessment tools
None with control groups showing support of preceptor development
Most studies addressed learner satisfaction and self-efficacy rather than behavioral change
(Windey et al. (2015))
New Preceptor Workshop Design
Goal of the new workshop
To provide comprehensive evidence-based knowledge and skills to novice and experienced nurse preceptors
Sources of learning needs assessment
Prior 1-day preceptor workshop evaluation results
Preceptors’ anecdotal feedback
Preceptors’ learning needs assessment findings
Literature review findings on preceptor preparation and program evaluation
New Workshop Agenda
Day 1 Day 2
1. Welcome/Introduction 1. Reflection of Day 1 learning/Kahoot
2. Overview of preceptor roles, responsibilities, and qualifications 2. Competency validation and performance evaluation
3. Preceptorship planning and implementation 3. Time management and prioritization
4. Teaching adult clinicians 4. Using evidence-based practice in precepting
5. Critical thinking 5. Managing challenging behaviors in precepting
6. Professional relationship, communication, and feedback 6. Ethical and legal consideration in precepting
7. Conflict resolution 7. Reality shock and transition shock
8. Putting it all together/simulation
Teaching Strategies
Didactic presentation
Case studies
Group discussion
Role-playing
Simulation
Games
Video clip
Comprehensive workbook
Program Implementation
Officially implemented in March 2017
Seven workshops offered in 2017
Audience All new nurse preceptors (RNs and LVNs) Experienced preceptors
Nurse manager’s approval required Inpatient and outpatient settings
Respiratory therapists
Program Evaluation
Kirkpatrick Model of 4 Levels of Evaluation
Level 1 measures the learners’ reaction or satisfaction with the training
Level 2 measures the extent of learning
Level 3 measures the learners’ behavioral change after the training
Level 4 measures tangible long-term outcomes that can be directly related to the training
(Kirkpatrick & Kirkpatrick (2005))
Program Evaluation Questions
What was the effect of a 2-day preceptor workshop on preceptors’ confidence and comfort in various precepting roles?
Sandau et al.’s (2011) preceptor survey instrument
Is there an improvement in participants’ self-reported perception of precepting knowledge, skills, and attitudes before and after attending the workshop?
Self-Assessment tool
How satisfied were the participants in terms of the workshop adding to their knowledge, skills, and attitudes in precepting?
Course evaluation form
Instruments
Sandauet al.’s (2011) Preceptor surveys
Validated instrument: quantitative and qualitative
Baseline surveys and post-workshop surveys
To measure the preceptors’ knowledge, skills, and behavioral changes in precepting
Addressed levels 2-3 of Kirkpatrick model
Instruments
Quantitative 2 sets of questions
5-point Likert scale
To measure preceptor’s confidence and comfort in five preceptor roles
Ability to precept a new nurse
Active coaching in critical thinking
Working with an orientee with a different personality or learning style
Working with an orientee with a different ethnic background
Providing positive and constructive feedback to an orientee
To assess changes in preceptor after the workshop in actively coaching critical thinking and providing feedback to orientee
Sample Questions on Sandau’s Survey Instruments1. How satisfied are you with your previous education regarding precepting an orientee?
2. How confident are you with your ability to precept a new orientee?
3. How comfortable are you in actively coaching critical thinking with your orientee?
4. How comfortable are you in working with an orientee who has a different personality or learning style than yours?
5. How confident are you in working with an orientee of a different ethnic background than yours?
6. How confident are you in providing both positive and constructive feedback to an orientee?
Self-Assessment Tool
Developed by Nursing Professional Development practitioners based on the workshop learning outcomes
5-point Likert scale
To be completed by the participants before the workshop and immediately after the workshop
Eight items to measure self-reported change in knowledge, skills, and attitudes before and after the workshop
Addressed level 2 of Kirkpatrick model
Self-Assessment Questions 1. Describe preceptor roles and responsibilities
2. Explain how to plan and implement preceptorship (learning needs assessment, select proper instructional methods, etc.)
3. Apply adult learning principles in the clinical setting
4. Implement strategies for promoting critical thinking skills
5. Explain how to provide constructive feedback
6. Identify the strategies for conflict resolution
7. Identify the strategies for proper time management, delegation, and managing challenging behaviors
8. Recognize the importance of competency verification and performance evaluation
Course Evaluation Form
Developed by Nursing Professional Development practitioners following the organization’s recommended course evaluation standards
To assess overall value added for the preceptors by attending this workshop
5-point Likert scale addressed level 1 of Kirkpatrick Model
Sample Questions on Course Evaluation Form
1. Overall, I was satisfied with this learning activity
2. The learning activities and/or materials were effective in helping
me learn the content (handouts, simulation, assessment tools, etc.)
3. I learned new knowledge and skills from this learning activity
4. The scope of the learning activity was appropriate and relevant to my professional needs
5. I will be able to apply the knowledge and skills learned to improve my job performance
6. The training environment was effective for my learning
Quantitative Data Analysis: Setup
Workshop (2 days)
Pre-workshopSurvey Instrument (6 Qns.)
Pre-workshopSelf-assessment (8 Qns.)
Post-workshopSurvey Instrument (6 Qns.)
Post-workshopSelf-assessment (8 Qns.)
Post-workshopCourse evaluation (6 Qns.)
Time lapse (4 months)
Cluster sampling method – classroom environment
No identification of participants & anonymity of responses
Likert scale and continuity of measure assumed
Each question treated separately for testing and analysis
7 cohorts of preceptorship class ranging from 6 to 17 students
n=89 pre-workshop samplesn=16 post-workshop samples
Quantitative Data Analysis: Test Assumptions
Test assumptions for each of five datasets separately
1. Power Calculations
• Do we have sufficient data for the set error tolerance and effect size?
• Type I Error 1-5%
• (1-Type II Error) = Power of Test 80-99%
• Effect Size Cohen’s d (measure of effect size)
• Alternate Hypothesis (Ha) > or ≠
• If sufficiently powered small non-response rate is not a major concern
Quantitative Data Analysis: Test Assumptions
2. Normality of datasets
• Many tests are based on normality assumption – what’s that?
• Graphical test: QQ Plot
• Analytical test: Shapiro-Wilks test
Quantitative Data Analysis: ResultsObjective 1: To compare pre- & post-workshop group averages
If the data shows normal distribution use independent/paired t-test
Otherwise use Wilcoxon Rank Sum test
Used Wilcoxon Rank Sum test
Sandau’s Survey Instrument showed improvements in post- workshop ratings when
compared to pre-workshop ones*
Self-assessment tool showed improvements in post-workshop
ratings when compared to pre-workshop ones
Quantitative Data Analysis: ResultsObjective 2: To compare across cohorts
If the data shows normal distribution use one-way ANOVA
Otherwise use Kruskal-Wallis test
Used Kruskal-Wallis test
Pre-workshop Sandau’s Survey Instrument baseline assessment of preceptors
Pre- & post-workshop differences in process of learning for preceptors
Self-assessment tool
Post-workshop Course Evaluation Instructor’s contribution to process of
learning for preceptors
Quantitative Data Analysis: ResultsObjective 3: Notable characteristics of the analysis
How confident are you in working with an orientee of a different ethnic background than yours? Not statistically significant Consistent exposure to diversity in workplace
Behavioral Change: Have you given formal feedback, both positive and constructive? Statistically significant Improved attitude towards providing feedback
Behavioral Change: Have you used techniques to promote critical thinking? Not statistically significant Room for improvement in efforts to promote critical thinking
Quantitative Data Analysis: ResultsObjective 3: Notable characteristics of the analysis
Course Evaluation Lowest Score: “The learning activities and/or materials were effective in helping me learn the content”
Course Evaluation Highest Score: “I learned new knowledge and skills from this learning activity”
Instructors teaching this workshop provided the essential skills and knowledge to the workshop participants but could improve on teaching techniques and course materials
All datasets (both pre & post) compared across all 7 cohorts
Uniformly statistically significant results
Students were uniformly uninformed about preceptorship
Instructors were prepared equally for all the classes and consistently delivered effective workshops throughout the year
Uniform class environment, materials, and take home messages
Limitations
Kirkpatrick Level 4 not addressed
Which long-term outcomes should be considered for program evaluation
Qualitative data analysis remains unaddressed
Implications for Clinical Practice and Future Research
Lack of preceptor preparation
Strategies for teaching preceptors on how to improve critical thinking for orientee
Strategies to improve post-workshop survey response rate
Involve orientee in the evaluation process
Conclusion
Quantitative analyses have shown the effectiveness of our newly designed 2-day preceptor workshop
Integrated quality improvement process with evidence-based practice
Added additional evidence to the literature in enhancing preceptor education
Provided directions for future research
References Bengtsson, M., & Carlson, E. (2015). Knowledge and skills needed to improve as preceptor: Development of a
continuous professional development course—A qualitative study Part I. BMC Nursing, 14(1), 51. doi:10.1186/s12912-015-0103-9
Chang, C., Lin, L., Chen, I., Kang, C. Y., & Chang, W. Y. (2015). Perceptions and experiences of nurse preceptors regarding their training courses: A mixed method study. Nurse Education Today, 35(1), 220–226. doi:10.1016/j.nedt.2014.08.002
Cotter, E., & Dienemann, J. (2016). Professional development of preceptors improves nurse outcomes. Journal for Nurses in Professional Development, 32(4), 192–197. doi:10.1097/ nnd.0000000000000266
Foy, D., Carlson, M., &White, A. (2013). RN preceptor learning needs assessment. Journal for Nurses in Professional Development, 29(2), 64–69. doi:10.1097/nnd.0b013e318287aa12
Hautala, K. T., Saylor, C. R., & O’Leary-Kelley, C. (2007). Nurses’ perceptions of stress and support in the preceptor role. Journal for Nurses in Staff Development, 23(2), 64–70. doi:10.1097/01.nnd. 0000266622.39304.31
Johnston, C., & Mohide, E. A. (2009). Addressing diversity in clinical nursing education: Support for preceptors. Nurse Education in Practice, 9(5), 340–347. doi:10.1016/j.nepr. 2008.08.005
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2005). Transferring learning to behavior: Using the four levels to improve performance. San Francisco, CA: Berrett-Koehler.
References L’Ecuyer, K., Hyde, M., & Shatto, B. (2018). Preceptors’ perception of role competency. The Journal of Continuing Education in
Nursing, 49, 233–240.
Panzavecchia, L., & Pearce, R. (2014). Are preceptors adequately prepared for their role in supporting newly qualified staff? Nurse Education Today, 34(7), 1119–1124. doi:10.1016/j.nedt.2014.03.001
Sandau, K. E., Cheng, L. G., Pan, Z., Gaillard, P. R., & Hammer, L. (2011). Effect of a preceptor education workshop: Part 1. Quantitative results of a hospital-wide study. The Journal of Continuing Education in Nursing, 42(3), 117–126. doi:10.3928/00220124-20101101-01
Senyk, J., & Staffileno, B. A. (2017). Reframing nursing preceptor development. Journal for Nurses in Professional Development, 33(3), 131–137. doi:10.1097/nnd. 0000000000000343
Tsai, Y., Lee-Hsieh, J., Turton, M. A., Li, S., Tseng, H., Lin, H., & Lin, H. (2014). Nurse preceptor training needs assessment:Views of preceptors and new graduate nurses. The Journal of Continuing Education in Nursing, 45(11), 497–505. doi:10.3928/00220124-20141023-01
Ward, A., & McComb, S. (2017). Precepting: A literature review. Journal of Professional Nursing, 33(5), 314–325. doi:10.1016/j.profnurs. 2017.07.007
Wilson, R., Acuna, M., Ast, M., & Bodas, E. (2013). Evaluation of the effectiveness of simulation for preceptor preparation. Journal for Nurses in Professional Development, 29(4), 186–190. doi:10.1097/ nnd.0b013e31829aec2e
Windey, M., Lawrence, C., Guthrie, K.,Weeks, D., Sullo, E., & Chapa, D. W. (2015). A systematic review on interventions supporting preceptor development. Journal for Nurses in Professional Development, 31(6), 312–323. doi:10.1097/nnd.0000000000000213
Thank You! Questions?
VA Palo Alto Health Care SystemLilly Liu BMed, RN, CCRN, RN-BC
Denise Fillipucci PhD, RN, GCNS
Satish Mahajan PhD, MStat, MEng, RN