Research Paper Critical Research Paper Critical AnalysisAnalysis

10 ways to look at a research 10 ways to look at a research paper systematically for paper systematically for

critical analysiscritical analysis..

NUR 599T Human Factors In NUR 599T Human Factors In Healthcare EngineeringHealthcare Engineering

Dr. Vincent DuffyDr. Vincent DuffyResearch Paper AnalysisResearch Paper Analysis

Submitted by Submitted by RuthAnn Smolen, RN, MS Ed, HNP,RuthAnn Smolen, RN, MS Ed, HNP,

July 19, 2007July 19, 2007

STUDY CRITIQUESTUDY CRITIQUE

Title: Assessment of Clinical Title: Assessment of Clinical Performance during Simulated Crises Performance during Simulated Crises Using Both Technical and Behavioral Using Both Technical and Behavioral

RatingsRatings

Authors: Gaba, D., Howard, S., Flanagan B., Authors: Gaba, D., Howard, S., Flanagan B., Smith, B., Fish, K., & Botney, R.Smith, B., Fish, K., & Botney, R.

1.1. Research Ideas or Research Ideas or QuestionQuestion

(Study and Purpose)(Study and Purpose)

Study: “Assessment in clinical performance” Study: “Assessment in clinical performance” Gaba, et.al.)Gaba, et.al.)

The purpose is clearly identified in the introduction - The purpose is clearly identified in the introduction - to evaluate ratings of performance (technical and to evaluate ratings of performance (technical and

behavioral), and variability of ratings made by multiple behavioral), and variability of ratings made by multiple independent observers viewing videotapes of simulated independent observers viewing videotapes of simulated crisescrises

The use of human patient simulators (HPS) in health-care The use of human patient simulators (HPS) in health-care training is increasing to gain knowledge, teach technical training is increasing to gain knowledge, teach technical skills, complete drill training, and human factors training for skills, complete drill training, and human factors training for single or multidisciplinary groupssingle or multidisciplinary groups

2. Background2. Background

Anesthesiology is recognized as the leading medical Anesthesiology is recognized as the leading medical specialty in addressing patient safety with HPSspecialty in addressing patient safety with HPS

Pioneered development of computer screen and mannequin Pioneered development of computer screen and mannequin based interactive patient simulators and training plans based interactive patient simulators and training plans ((Schwid, 1987; Small, Wuetz, Simon, Shapiro, Conn, Setnik, 1999).Schwid, 1987; Small, Wuetz, Simon, Shapiro, Conn, Setnik, 1999).

A persistent initiative in graduate education is to find more A persistent initiative in graduate education is to find more effective methods to assess specialists’ skillseffective methods to assess specialists’ skills

““Techniques are needed to assess anesthesiologists’ Techniques are needed to assess anesthesiologists’ performance when responding to critical events” performance when responding to critical events” (Gaba, 1998)(Gaba, 1998)

Traditionally there has been concern about the potential Traditionally there has been concern about the potential subjectivity of expert raters. subjectivity of expert raters.

3. Theoretical 3. Theoretical Basis/HypothesisBasis/Hypothesis

Helmreich, RI Theory underlying CRM training: Psychological Helmreich, RI Theory underlying CRM training: Psychological issues in flight crew performance and crew coordination issues in flight crew performance and crew coordination (NASA, 1986) in teams(NASA, 1986) in teams

Behavioral psychologists and learning theories were used Behavioral psychologists and learning theories were used originallyoriginally

In anesthesia, patient safety depends on a complex In anesthesia, patient safety depends on a complex interaction between a skilled practitioner and technologyinteraction between a skilled practitioner and technology

A human patient simulator may reduce the risk of learning-A human patient simulator may reduce the risk of learning-

curve error in anesthesiologists technical and behavioral curve error in anesthesiologists technical and behavioral clinical performance without subjecting patients to riskclinical performance without subjecting patients to risk

4. Literature Search4. Literature Search

13 reports published between 1980 & 2000; 4 were 13 reports published between 1980 & 2000; 4 were designed to investigate validity or reliability of HPS designed to investigate validity or reliability of HPS systems; overview of existing evidence of the systems; overview of existing evidence of the problem is addressed in follow-up searches of todayproblem is addressed in follow-up searches of today

In this study the literature review is supportive of In this study the literature review is supportive of purpose and results, current for 1998, and thoroughpurpose and results, current for 1998, and thorough

Summaries of other comparative studies are Summaries of other comparative studies are discussed, relevant to the study and includes discussed, relevant to the study and includes strengths and weaknesses strengths and weaknesses

Chopra, et al. (pg. 13)Chopra, et al. (pg. 13) Devitt, et al. (pg. 13)Devitt, et al. (pg. 13) Law and Sherman (pg. 15)Law and Sherman (pg. 15)

5. Applicability5. Applicability

The aviation industry’s commitment to The aviation industry’s commitment to simulation and similarities between simulation and similarities between anesthesia and aviation was a key rationale anesthesia and aviation was a key rationale for developing courses in crisis for developing courses in crisis management and using simulation in management and using simulation in anesthesia educationanesthesia education

Generalization among various acute events Generalization among various acute events in simulation teaching and evaluation in simulation teaching and evaluation environments across healthcare disciplines environments across healthcare disciplines is a reality todayis a reality today

6. Methodologies6. Methodologies

Raters scored the videotapes of 14 Raters scored the videotapes of 14 different teams that were managing two different teams that were managing two scenarios: malignant hyperthermia (MH) scenarios: malignant hyperthermia (MH) and cardiac arrest. Technical and cardiac arrest. Technical performance and crisis management performance and crisis management behaviors were rated. behaviors were rated.

7. Statistical Analyses & 7. Statistical Analyses & AssumptionsAssumptions

Using three observers, the study shows Using three observers, the study shows good interrater reliability in scoring good interrater reliability in scoring technical technical achievement with less with less agreement between five raters scoring agreement between five raters scoring ACRM behaviors. ACRM behaviors.

The authors analyze the statistics of The authors analyze the statistics of interrater reliability in depth and note interrater reliability in depth and note that the scoring system requires further that the scoring system requires further development before it can be used for development before it can be used for assess clinical competence for assess clinical competence for residency/graduation, or board residency/graduation, or board certification certification

8. Presentation of Results8. Presentation of Results

What do the results really mean? What do the results really mean? Team performance varied and correlations performance varied and correlations among interraters were better for technical among interraters were better for technical markers than for behavioral markersmarkers than for behavioral markers

9. Conclusions 9. Conclusions (are they reasonable?)(are they reasonable?)

The rating system needs refinement before The rating system needs refinement before it can be used to assess clinical it can be used to assess clinical competence for residency/graduation, or competence for residency/graduation, or board certificationboard certification

This study suggests that simulation can be This study suggests that simulation can be used to measure more complex skills used to measure more complex skills expected in specialty trainingexpected in specialty training

Similar to other studies assessing complex Similar to other studies assessing complex technical and behavioral performance technical and behavioral performance multiple encounters are needed to multiple encounters are needed to estimate skill effectively and accuratelyestimate skill effectively and accurately

10. Future Work/Research 10. Future Work/Research DirectionDirection

Studies emphasize the cognitive and social aspects of designing Studies emphasize the cognitive and social aspects of designing and operating advanced technologies yet to be developedand operating advanced technologies yet to be developed

The behavioral rating system can be improved – difficulty The behavioral rating system can be improved – difficulty

aggregating a single rating that fluctuated with timeaggregating a single rating that fluctuated with time

Human Factors and ergonomics link quantitative skills to the Human Factors and ergonomics link quantitative skills to the integrated systems and make sure design of systems act in integrated systems and make sure design of systems act in accordance with requirement of users; includes interactions with accordance with requirement of users; includes interactions with technologiststechnologists

Questions: Questions: Is there a need for simulation in assessment of clinical Is there a need for simulation in assessment of clinical

performance?performance? Do current simulators match the fidelity of a real-live clinical Do current simulators match the fidelity of a real-live clinical

setting? Are they realistic?setting? Are they realistic? What are the potential uses of clinical simulation in What are the potential uses of clinical simulation in

medical/healthcare education?medical/healthcare education? What role exists for simulation in healthcare professional What role exists for simulation in healthcare professional

training and assessment? training and assessment? Do collaborative opportunities exist?Do collaborative opportunities exist?