Post on 08-Jan-2016
description
Leading (and Assessing) a Learning Intervention
IMPACT Lunch and Learn Session
August 6, 2014
Facilitated By
Ozgur Ekmekci, EdD
Interim Chair, Department of Clinical Research and Leadership
School of Medicine and Health Sciences
The George Washington University
1
Overview
Leading a Learning Intervention
In today's complex social environment, anyone who has been assigned the task of leading a learning
initiative has to approach the process in a systematic way in order to succeed. The crucial success
factor is often creating alignment amongst components that constitute a well-articulated learning
intervention proposal. This session is offered for those who would like to better understand the four
key questions that allow educational leaders to effectively envision, design, implement, and assess
a meaningful learning intervention, whether this intervention takes place in the classroom or in the
boardroom.
Learning Objectives
•Discuss challenges associated with implementing a learning intervention
•Analyze major components of an effective learning intervention proposal
•Analyze four key questions that drive a learning intervention
•Discuss how assessment fits into the larger context of a learning intervention proposal
•Review six major assessment methods used in health sciences
2
• Learning Community Context (overview of the learners’ location, background, roles, and interests)
• The Problem Statement (current state of the learners – in measurable outcomes - and why this is a problem)
• The Statement of Significance (why it is important that the intervention be implemented and who suffers in what way, if not implemented)
• The Proposed Solution (envisioned learning objectives - including the statement of tangible, measurable objectives, which tie back to the problem statement - i.e., current state)
• Implementation Time Line (a graphical depiction of the implementation plan, with major milestones along the way)
• Evaluation Method(s) (how to measure progress against learning objectives and how it will be known when the learners finally get to the envisioned future state)
• Conclusion (summary of key messages with which to leave the audience)
ALIGNMENT
KEY SUCCESS FACTOR
Alignment of Major Components
3
Define Measurable Learning Objectives:
- Individual Level- Change in behavior- Change in perception- Change in performance
- Organizational Level- Change in outcomes
- Societal Level- Change in environment
What do I want to change, for whom, by how much, and by when? In other words, what will success look like?
Learning Intervention Time Line
Plan Learning Intervention:
- Identify learners - Select content- Develop materials- Design delivery method- Construct learning environment- Develop delivery schedule
How will I change what, for whom, by how much, and by when? In other words, how will I achieve success?
Design Learning Evaluation:
- Identify key variables to be measured at the appropriate level of analysis (i.e., individual, organizational, societal)
- Decide on the most appropriate method(s) for conducting formative and summative assessments that measure the key variables
- Determine frequency and timing for formative assessment and summative assessments
- Plan delivery format, medium, frequency, and timing of feedback to learners
- Select quantitative statistical methods that will help determine the effectiveness of intervention
How will I know what I am
changing, for whom, by how much, and by when? In other words, how will I monitor progress against set objectives that define success?
Implement Learning Intervention and Conduct Learning Evaluation:
- Deliver content- Assess learning - Provide feedback to learners- Analyze data using statistical methods- Determine effectiveness of learning intervention- Declare success!
Has what I wanted to change, for those whom I wanted to change it, actually changed by the amount I had wanted it to change?
Conduct Formative
Assessment
Conduct Summative Assessment
Review and Revise Content
Method or Schedule
Deliver Content
START
Provide Learner
Feedback
Provide Learner
Feedback
STOP
Delivery Complete
?
Conduct CAT
Determine Effectiveness of Intervention
YES
NO
Learning Intervention Timeline
4
PHASE I PHASE II PHASE III PHASE IV
Ekmekci, O. (2013). Being there: Establishing instructor presence in an online learning environment, Higher Education Studies. 3(1). 29-38.
Design Learning Initiative Evaluation
How will I know what I am changing, for whom, by how much, and by when? In other words, how will I monitor progress against set objectives that define success?
By asking this question, the leader will be able to identify key variables to be measured, decide on methods and frequency for formative and summative assessment, plan delivery format, medium, frequency, and timing of feedback to participants, and select statistical methods that will help determine the effectiveness of the initiative
Reflecting on Four Key Questions
5
Define Measurable Learning Objectives
What do I want to change, for whom, by how much, and by when? In other words, what will success look like?
By asking this question, the leader will be able to contemplate and quantify the envisioned type of change, its scope (in terms of breadth, depth, and timeline), and the level at which change is to take place (i.e., individual, organizational or societal) – which will help ability to monitor progress against objectives
PHASE IPHASE I
Plan Learning Initiative
How will I change what, for whom, by how much, and by when? In other words, how will I achieve success?
By asking this question, the leader will more easily identify participants, select content, develop materials, design delivery method, construct learning environment, and develop delivery schedule
PHASE IIPHASE II PHASE IIIPHASE III
Implement Learning Initiative & Conduct Evaluation
Has what I wanted to change, for those whom I wanted to change it, actually changed by the amount I had wanted it to change?
By asking this question, the leader will be able to effectively deliver content, assess progress, provide feedback to participants, and analyze data using statistical methods to ultimately determine how successful the initiative has been
PHASE IVPHASE IV
Ekmekci, O. (2013). Being there: Establishing instructor presence in an online learning environment, Higher Education Studies. 3(1). 29-38.
Assessment as a Process
6
Assessment is the ongoing process that allows one to:
•establish clear, measurable expected outcomes of learning;
•ensure learners have sufficient opportunities to achieve those outcomes;
•systematically gather, analyze, and interpret evidence to determine how well
learning matches expected outcomes of learning; and
•using the resulting information to understand and improve learning.
Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge. p. 4.
Threats to Validity
7
Construct Underrepresentation (CU)
Not measuring the right things: under-sampling of the achievement domain – such as tests that are too short to support legitimate inferences to the domain, trivial questions written at low-levels of the cognitive domain, or mismatch of sample to domain
Construct Irrelevant Variance (CIV)
Not measuring (the right) things in the right manner:erroneous inflation or deflation of test scores due to certain types of uncontrolled or systematic measurement error – such as poorly crafted test questions, rater bias, or environmental conditions
Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge.
Levels of Assessment and Matching Methods
8
Behavior
Cognition
DOESRequires methods
that provide an assessment of routine “real” performance for
the learner
SHOWS HOWRequires methods that allow learner to demonstrate
the integration of knowledge and skills that will translateinto successful routine performance
KNOWS HOWRequires methods that provide an assessment of how well learner uses
knowledge in the acquisition, analysis, and interpretation of data and the development of an approach or plan
KNOWSRequires methods that provide an assessment of how well learner has acquired the fundamental knowledge
on which to build higher levels of knowledge and behavior
Written TestOral Examination
Observational Assessment
Performance Test
Simulation
Adapted from: Downing, S.M. & Yudkowsky, R. eds. (2009). Assessment in health professions education. New York: Routledge.
Portfolio
Wrap Up
9
Questions?