Post on 03-Jan-2016
1905
Is The ACGME Resident/Fellow Is The ACGME Resident/Fellow Survey a Valid Tool to Assess Survey a Valid Tool to Assess General Surgery Residency General Surgery Residency
Programs Compliance with Work Programs Compliance with Work Hours Regulations?Hours Regulations?Robert P. Sticca, MD, FACS
Jay M. MacGregor, MD
Randolph E. Szlabick, MD, FACS
Department of Surgery
University of North Dakota SMHS
1905
BackgroundBackground• 2003 – ACGME work hours regulations
implemented• 2004 – ACGME Residents/Fellows Survey• 2008 – Special Message from CEO
ACGME (Thomas Nasca, MD)– Standard Approach to Programs Across
All Specialties with Potential Duty Hours Violations Identified in the Resident Survey
1905
ACGME Special MessageACGME Special Message• The Monitoring Committee has accepted the Resident Survey
as a reliable, sensitive and reasonably specific tool• The Monitoring Committee has set reasonable thresholds for
categorization of a program as having Potential Duty Hours Violations (PDHV) based on the Resident Survey
• The validity of the Resident Survey has been demonstrated though significant correlations with site visit findings to warrant added scrutiny for programs meeting the thresholds of potential noncompliance found in the group targeted for follow-up.
• There is a significant correlation between resident reported violations of duty hours with deficiencies in other important areas of the learning environment (correlation 0.51, p<0.0001)
Nasca, T. www.ACGME.org
1905
ACGME Algorithm for ACGME Algorithm for Noncompliant ProgramsNoncompliant Programs
Nasca, T. www.ACGME.org
1905
UND ExperienceUND Experience• 2005 – RRC site visit – additional categorical resident
requested– 5 year approval, no deficiencies– Additional resident denied, insufficient educational
justification• 2007 – RRC site visit – additional categorical resident
requested– 5 year approval, no deficiencies– Additional resident approved
• 2009 – RRC site visit – mandatory due to residents responses on survey– 5 year approval - commendation
1905
ACGME Survey - ProblemsACGME Survey - Problems
• Answer selections vague
• High threshold for compliance
• Small program effect
• Consequences of survey punitive
• Consequences based on survey results and not actual work hours data
1905
Current StudyCurrent Study• Purpose – obtain general surgery residents views
on work hours regulations and ACGME survey• Design – 20 question survey emailed to all Gen.
Surg. Program Directors – 5/09– 9 questions – work hours
– 9 questions – ACGME survey
– 1 both, 1 demographic, comments
• Study period – 5/09 – 10/09– Multiple reminders, pleas for participation
• Internet Survey – surveymonkey.com
1905
QuestionsQuestionsTABLE 1: Questions from Surgical Residents Views on the
Accreditation Council for Graduate Medical Education (ACGME) Resident Survey.
1. How many times have you taken the ACGME Resident survey?
2. Did you have any difficulty understanding the questions as written in the ACGME Survey?
3. Did you discuss the ACGME Resident Survey with your residency Program Director or any of your faculty prior to completing the survey?
4. Were you, in any way, instructed how to answer questions on the ACGME Resident Survey?
5. Did you answer all questions on the ACGME Resident Survey truthfully?
6. Are you aware how your answers to the ACGME Resident Survey can effect your surgery program's accreditation?
7. If you indicated an ACGME duty hours requirement is "SOMETIMES" met, which of the following is true?
8. If an ACGME duty hours requirement is "SOMETIMES" met, how often are you NONCOMPLIANT with this requirement?
9. When answering the ACGME Survey, what percentage of residents in a program need to indicate a rule violation for this to be deemed significant by the ACGME?
10. Do you feel the ACGME Resident Survey accurately evaluated the hours that you work in your Residency Program?
1905
ResultsResults
• Answers collected, analyzed - 11/09
• Aggregate
• Level of training– Junior – PGY 1, 2– Senior – PGY > 2
• Statistical analysis – openepi.com– P value < 0.05, statistically significant
1905
Results - ResponsesResults - Responses
Resident Year of Training by
Post Graduate Year (PGY)
Number of
Resident ResponsesPercent of Total
PGY 1 246 25.6
PGY 2 195 20.3
PGY 3 181 18.8
PGY 4 160 16.7
PGY 5 or Greater 179 18.6
Total 961 100
965 – 13.1 % of U.S. General Surgery Residents961 completed entire survey (99.6%)
Junior Residents – 441 Senior Residents – 520
1905
Results Results Number Times Survey Number Times Survey
Taken Taken
Question Junior Residents
(% of Junior Residents)
Senior Residents(% of Senior Residents)
Total (%) P values
How many times have you taken the
ACGME Resident/Fellow
Survey?
1 time = 254 (58) 2 times = 148 (34) 3 times = 25 (5) 4 times = 9 (2) 5 times = 1 (0.1)> 5 times = 4 (0.9)
1 time = 93 (17) 2 times = 95 (18) 3 times = 164 (31) 4 times = 90 (17) 5 times = 58 (10)> 5 times = 40 (7)
1 time = 347 (36)2 times = 243 (25)3 times = 189 (19)4 times = 99 (10)5 times = 59 (6)
>5 times = 44 (4)
<0.0000001
1905
Results Results Instruction on SurveyInstruction on Survey
Question
Junior Residents(% of Junior
Residents)
Senior Residents(% of Senior Residents)
Total (%) P values
Discuss survey with PD or faculty
Yes = 156 (35)No = 285 (65)
Yes = 185 (36)No = 335 (64)
Yes = 341 (35)No = 620 (65)
0.8881
Instructed on how to answer survey
Yes = 84 (19)No = 357 (81)
Yes = 82 (16)No = 438 (84)
Yes = 166 (17)No = 795 (83) 0.2242
1905
Results – Survey Results – Survey InterpretationInterpretation
Question Junior Residents
(% of Junior Residents)Senior Residents
(% of Senior Residents)Total (%) P values
Difficulty Understanding
Questions
Yes = 61 (14)No = 380 (86)
Yes = 118 (23)No = 402 (77)
Yes = 179 (19)No =782 (81)
0.0002662
When requirement
sometimes met, how often
noncompliant
In compliance = 121 (27)
In violation = 202 (46)
Neither compliant nor in violation = 118 (27)
In compliance = 156 (30)
In violation = 236 (45)
Neither compliant nor in violation = 128 (25)
In compliance = 280 (29)
In violation = 438 (45)
Neither compliant nor in violation = 246 (26)
0.4817
Meaning of sometimes
1 - 5 times/year = 212 (48)6 - 10 times/year = 117 (27)11 - 15 times/year = 41 (9)16 - 20 times/year = 27 (6)21 -25 times/year = 11 (3)> 25 times/year = 33 (7)
1 - 5 times/year = 288 (55)6 - 10 times/year = 109 (20)11 - 15 times/year = 49 (10)16 - 20 times/year = 25 (5)21 -25 times/year = 10 (2)> 25 times/year = 39 (8)
1 - 5 times/year = 500 (52) 6 - 10 times/year = 226 (24) 11 - 15 times/year =90 (10)
16 - 20 times/year = 52 (5) 21 -25 times/year = 21(2) > 25 times/year = 72 (7)
0.2852
1905
Results Results Effect on ProgramEffect on Program
QuestionJunior Residents
(% of Junior Residents)
Senior Residents(% of Senior
Residents)
Total (%) P values
Aware of effect on program
accreditation
Yes = 337 (76)No = 103 (24)
Yes = 399 (77)No = 121 (23)
Yes = 736 (77)No = 224 (23) 0.9081
What percent necessary to be
deemed significant by ACGME
5% = 131 (30)10% = 131 (30)15% = 37 (18)20% = 68 (15)25% = 52 (12)50% = 22 (5)
5% = 161 (31)10% = 117 (23)15% = 51 (10)20% = 97 (18)25% = 43 (8)
50% = 51 (10)
5% = 292 (30)10% = 248 (26)
15% = 88 (9)20% = 165 (17)25% = 95 (10)50% = 73 (8)
0.005127
1905
ResultsResultsHonesty/AccuracyHonesty/Accuracy
Question Junior Residents
(% of Junior Residents)
Senior Residents(% of Senior Residents)
Total (%) P values
Answer questions truthfully
Yes = 382 (87)
No = 59 (13)
Yes = 446 (86)
No = 74 (14)
Yes = 828 (86)No = 133 (14)
0.8856
ACGME Survey accurately
evaluated hours that you work
Yes = 287 (65)No = 154 (35)
Yes = 319 (61)No = 201 (39)
Yes = 606 (63)No =355 (37) 0.2231
1905
ConclusionsConclusions• Evaluation mechanism in which– 52% do not understand questions– 14% did not answer truthfully– 37% do not feel is accurate
• Not best tool to evaluate work hours compliance in residency programs
• Consequences significant• Open letter to GME community 10/28/09– www.acgme.org
Nasca, T. www.ACGME.org
1905
Thank you !!Thank you !!
Program Directors
Residents