Non NATO PC’s RISK ASSESMENT LEVEL ENVIRONMENTAL CONSIDERATIONS SAFETY CONSIDERATIONS EVALUATION.
Ethics Education: Considerations for Development and Evaluation
-
Upload
fatima-washington -
Category
Documents
-
view
43 -
download
3
description
Transcript of Ethics Education: Considerations for Development and Evaluation
Ethics Education: Considerations for Development and Evaluation
Presented at:University of Illinois - Champaign
May 2012
Presented by:Dr. Mike Mumford
Ethics Education: Needs Assessment
• New federal mandates highlight need for effective, validated ethics education programs
• Yet, validated instruments or methods for evaluating ethics programs are not applied
• Moreover, traditional approaches to ethics lack complexity and empirical evidence
Why Evaluate Ethics Education?
• Specific federal mandates to do so
• Quantify the real value of training program
• Ethics education is a loaded gun– It may encourage ethical behavior or it may actually
discourage it, and the consequences for either are great– Some (widely-used) programs haven’t worked……
Traditional Ethics Programs
• Kohlberg-Based Training– Help individuals move from one level of moral
development to another– Emphasize others’ rights or philosophical rights of
humanity in making ethical decisions – Deemphasize self-interest in ethical dilemmas
• CITI Training– Provide instruction on the major ethical guidelines
and rules – all declarative knowledge
Meta-analyses of Prior Ethics Course Results
Md
SD
k
N Overall Effectiveness .42 .27 26 3041
Criterion Type Moral Development .36 .26 17 2229 Ethical Analysis .61 .16 9 812
Program Quality Below Average .18 .25 5 674 Average .39 .02 8 1301 Above Average .72 .15 6 445
Study Quality Below Average .16 .23 10 817 Average .48 .20 12 1688 Above Average .65 .20 4 536
Investigator’s Field Health/Medicine .38 .17 9 1313 Philosophy .54 .21 4 648 Psychology .80 .00 6 441
Mandatory Course No .53 .25 10 909 Yes .32 .23 13 1818
Program Setting Integrated into Curriculum .37 .22 11 1832 Stand-alone Seminar .51 .30 15 1209
Md
SD
k
N Instructional Approach Decision-Making .52 .15 9 1234
Ethical Sensitivity .42 .11 7 780 Moral Development .17 .28 4 619
Type of Skills Real-World, Global .64 .11 9 744 Limited, Domain-Specific .35 .27 15 2257
Reasoning Errors No .33 .17 10 1845 Yes .57 .30 9 575
Strategies No .22 .25 7 1229 Yes .52 .20 13 1245
Ethical Domains No −.11 .14 4 460 Yes .48 .14 16 2014
Delivery Method Classroom-Based .36 .25 8 1091 Case-Based .53 .14 9 1214
Learning Activities 3 or more .12 .34 8 692 4 or more .48 .14 13 1995
Participant Interaction Low .05 .10 4 411 Moderate .37 .16 6 1198 High .63 .09 7 722
Ethics Education Evaluation: Federal RCR Mandates
• RCR training required in exchange for federal funds– Faculty, graduate students, some undergraduates– No online substitute
• Evidence of training effect required– Validated measurement
• Failure to comply will have implications for grant and contract awards– Institutions are requiring training for IRB
Ethics Education Evaluation: Federal Standards
• United States Code and the Code of Federal Regulations requires that all training programs be evaluated– Annual evaluation to demonstrate program effectiveness
towards organizational goals
• Executive Order 11348: Training for all government sponsored employees– Agency heads must plan, implement, evaluate, and
maintain records of training programs – Agency heads must monitor training effectiveness and
actively conduct research to improve program effectiveness
Ethics Education Evaluation: Quality Control
• Systematic evaluation provides insight into program success– What worked and why did it work
• Systematic evaluation provides insight into program failure– Is failing a structural problem or a conceptual one
• Increases public perception that program has real value– Increases stakeholder “buy-in”
Evaluation Methods
• Kirkpatrick’s four level model of training evaluation– Trainee reactions: how do trainees like elements of
the training• Affect-based; immediate and short-term
– Learning: Tests to determine acquired declarative knowledge of training material
Evaluation Methods
• Kirkpatrick’s four level model of training evaluation– Behavior: Transfer or application of trained knowledge
and skills (i.e. trainees’ post-training application of ethics education content)
– Results: Impact of training on organizational outcomes (e.g. actual increases or decreases in ethical misconduct, financial gains or losses as a result of training outcomes)
Sensemaking Training
• Ethics education programs should be grounded in realistic theory of ethical behavior
• Traditional models to ethics emphasized:– Moral development – Rational, black-and-white models
• Sensemaking training is grounded in EDM
Sensemaking Training Overview• Two-day seminar (16 hours of instruction)• 10 modules
– 2 out-of-class modules – 8 in-class modules
• Key Instructional Objectives– Develop understanding of the ambiguous, complex
nature of problems that may be encountered in professional work
– Learn strategies that facilitate identification, analysis, and acting within ethical situations
Sensemaking Training Content• Knowledge of guidelines• Awareness of own biases and common errors• Model of ethical decision-making (EDM)• Situational analysis and interpretation• Strategies, or tools, for decision-making• Field differences
Sensemaking Training Validation• Social & biological sciences graduate student
sample• Engineering graduate student and professional
sample• Undergraduate REU sample• Pre-post ethical decision-making test
– Outcomes examined for pre-post changes• Ethicality of decisions• Strategy use
Sensemaking Training Validation
Effect Size (Cohen’s d) Scientist Sample
Engineer Sample
Undergrad Sample
Decision-Making Ethicality Data Management .66** .53* 1.79** Study Conduct 1.46** .66* .34 Professional Practices .61** 1.06** 1.68** Business Practices .49* 1.82** 1.09** Strategies Recognizing one’s circumstances 1.24** .16 .35 Seeking help .84** −0.65* −0.01 Questioning one’s judgment 1.27** 1.92** 2.45** Dealing with emotions .90** 1.51** 2.56** Anticipating consequences .77** .33 .79** Analyzing personal motivations 1.36** .60* 1.05** Considering others’ perspectives .45** .08 .27
* Effects held over 6-month follow-up period
EDM Measure Development
• SMEs generate field-specific ethical examples– Real-world, ambiguous examples
• Equal representation of EDM items across ethical behavior taxonomy– Study Conduct– Data Management– Professional Practices– Business Practices
EDM Measure Development
• Parallel forms of measure for pre-post format– Test forms counterbalanced
• “Pick two” scoring scheme– Pick the “best” two out of possible eight
• Code response options for:– Ethicality of Response– Strategies underlying response– Social motives underlying response
Example EDM Measure Item(Social Science)
Dr. Cedar, a young developmental psychologist, obtained an Early Career Research Grant from the National Institute of Child Health and Human Development to study aggression in elementary school children. Cedar suspects that some children with a certain genetic makeup will be especially susceptible to the effects of television violence. Part of the project requires obtaining a cheek swab for DNA analysis, but interviewing and observing children in the classroom constitutes the major effort. Cedar is collaborating with a well-known senior social psychologist, Dr. Dawson. Given the geographic distance between their universities, the labs communicate primarily via email.
1. Considering the sensitive nature of the project, Cedar is concerned that parents will be reluctant to
allow their children to participate. He is writing the informed consent form and worries that too much detail might discourage participation. How should Cedar handle this issue? Choose two from the following:
a) Use wording like that of a past study involving similar procedures. b) Offer parents the option to contact participants from past data collection efforts to alleviate their concerns. c) Contact parents via phone to explain the importance of granting permission for their child’s participation. d) Send a follow up letter to non-consenting parents explaining that the risks are minimal. e) Mention the minor potential risks in the informed consent along with the benefits of the research. f) Describe all possible risks in the informed consent form, no matter how trivial. g) Send a handout along with the informed consent form addressing parents’ common questions and concerns. h) Since the risks of the study are trivial, deemphasize them in the informed consent form.
EDM Measure Validation
• Divergent validity evidence:– Narcissism & cynicism (r = -.01 to -.29*)– Exposure to unethical events (r = .02 to -.51*)
• Substantive validity evidence:– Meta-cognitive reasoning strategies
(r = .12 to .52)– Anti-social behaviors (r = -.07 to -.52)
EDM Measure Validation
• Criterion-related validity evidence:– Severity of violation (β = -.02 to .35*)– Perceived frequency of violation (β = -. 04 to .04)– Perceived importance of punishment (β = -.01 to .30*)– Severity of punishment recommended (β = -.01
to .26*)
EDM Process Strategy Studies• Purpose
1. Provide evidence for the validity of key EDM processes2. Identify strategies underlying those processes3. Identify interventions to promote strategy application
• 5 key process strategies1. Forecasting2. Emotion regulation3. Framing4. Self-reflection5. Ethical sensemaking
EDM Process Strategy Model (Sensemaking)
Framing Emotions
Forecasting
Self-reflection
Sensemaking
Decision
Prior Professional Experience
Prior Personal
Experience
Perceived Causes of the
Situation
Professional and Personal
Goals
Perceived Requirements for Attaining Goals
RCR Principles & Guidelines
EDM Process Strategy Studies
• Sample protocol (Self-Reflection)1. Recall past or predict future experiences that are
either positive or negative• Consider outcomes or the process of making ethical
decision2. Complete EDM measure
• Sample protocol (emotion regulation)1. Train on the use of ER strategies2. Induce anger via feedback (anger vs. fear)3. Complete EDM measure
EDM Process Strategy Studies• Key Findings: Best Strategies
1. Forecasting• Analysis of critical causes improved forecasting & EDM• Forecasting critical consequences improved EDM• Time pressure didn’t matter
2. Self Reflection • Reflection on positive events and process increased EDM• Using reflections to predict future ethical events
improved EDM3. Emotion Regulation
• Anger inhibited EDM, moderate fear facilitated EDM• Reappraisal buffered anger – was an effective technique
Case-Based Learning Studies
• Purpose:– Cases are frequently used in ethics education, yet
little is known about development of cases – Little is known about methods for case delivery– Three rounds of studies:
1. Case content - finished2. Process methods (strategies for delivering cases) – in
progress3. Content & process interactions - 2013
Case-Based Learning Studies
• Case Content:– Character affect & power dynamics– Codes of conduct & forecasting– Critical causes & outcomes– Social context & goals
• Process methods:– Elaboration & self-generation of cases– Sequential case presentation vs. case comparison– Alternative scenarios & outcome evaluation– Incremental case building & forecasting– Note-taking & review
Case-Based Learning Studies - Findings• Content Studies:
– Character emotions enrich cases increase in knowledge acquisition & transfer of ethical principles
– Highlight codes of conduct “ “– Include causal information and potential outcomes
• Process Studies:– Ask individuals to elaborate on cases– Have individuals compare multiple cases– Providing alternative case endings confuses readers
Alternative Methods of Evaluation
• Knowledge measures– Declarative Ethical Knowledge Measure (DuBois and
Colleagues (2012)– Knowledge of codes of conduct and regulations– Average Cronbach’s alpha = .71– Correlates highly with EDM measure (r = .40, p < .01)
Alternative Methods of Evaluation
• Trainee reactions– Trainees in sensemaking training have reported high
levels of satisfaction with training activities and content (M = 5 - 6 on a 7-point scale)
• Change in ethical conduct– Distal evaluation– Track ethical violations over time
A Cautionary Ending
• It is dangerous not to evaluate ethics training programs– We expect that ethics programs only help, but in
reality they can do more harm than good
• Success or failure (and the consequences) grounded in:– Effective models for understanding ethical behavior– Multi-faceted, comprehensive program evaluation