Possibilities of IT security evaluations based on Common Criteria in Hungary (in Hungarian)
Outcome Mapping for Planning Evaluations in American K-12 Urban Education: Potent Possibilities
Transcript of Outcome Mapping for Planning Evaluations in American K-12 Urban Education: Potent Possibilities
Outcome Mapping for Planning Evaluations in American K–12 Urban
Education: Potent Possibilities
American Evaluation Association 2015 Conference
Author: Tabia Lee, Ed.D. © 2015, e-mail for permissions
[email protected] +1 916-588-7776
Schools as Complex Adaptive Social-Ecological Systems The
Evaluation Process
Teachers as Curriculum Evaluators
Outcome Mapping & Internal Self-Evaluation Planning
Potent Possibilities
Presentation Roadmap
I.
II.
IV.III.
V.
Locating Evaluation Studies & Action Research
Usefulness in facilitating
decision making & transformations
Discovery or validation of generalizations
Hypothesis-driven & product-focused
Decision-driven &
process-focused
Goal
s & A
ims
Drivers & Focus
EVALUATION STUDIES & ACTION RESEARCH
GENERALEDUCATIONAL
RESEARCH
Ill-Defined Problems
& Ill-Definable
Solutions
Continuous Resolutions
Waves of Consequences
Every Trial Counts
Significantly
Ideology
Matters
Butterfly-Effect
Decisions Affect Lives
Schools and Wicked Problems
ContextSpecific
Adapted from Rittel & Webber, 1973
Goodness of Fit
Continuous, Recursive, &
Divergent
Unpredictable Outcomes
Sociopolitical
CollaborativeTeaching
& Learning
EmergentCreates Reality
Source: Guba & Lincoln, 1989
TheEvaluation
Process
support p
ressure
development
accountability
inside outside
internal evaluation
external evaluation
botto
m-u
p
top
-dow
n
self-evaluation
others
The Evaluation Cube
Source: Schratz, 1999
Senge’s 5 Disciplines & School Change
Personal Mastery
Systems Thinking
Mental Models
Team Learning
Shared Vision
Continuous Learning
Improvement Innovation
Source: Cropper, 2003
Linear Perspective Nonlinear PerspectiveMetaphor M a c h i n e L i v i n g Sy s t e m
Part & Whole Relationship
Hierarchy Sum of the Parts
Fractal Geometry Emergence
Understanding of Schools
Fo r m a l O rga n i za ti o n s
L e a r n i n g O rga n i za ti o n s /C o m m u n i ti e s
Cause & EffectOne-Way Causality (Open-Loop)Seeking Stability
Webs of Reciprocal Causality (Closed-Loop) Dynamic Equilibrium
Locus of Change E x t e r n a l E x t e r n a l & I nt e r n a l
Schools as Social Systems
The Ecology of Educational Environments
Macro-system
Exo-system
Meso-system
Micro-system
Source: Bronfenbrenner, 1976
Knowledge Cultures as a Nested System Individual
Knowledge
Local CommunityKnowledge
SpecializedKnowledge
Organizational Knowledge
Holistic Knowledge
KEY
Collective Thinking, Learning, & Inquiry
isAll the decision-making knowledges generating synergy
HABITUAL PRACTICES
PERCEIVED REALITY
CORE BELIEFS
ASSUMPTIONS(examined and unexamined)
HABITUAL PRACTICES
REALITY
CORE BELIEFS
SYSTEMSTHINKING
VersusIncompetent
SystemsCompetent
Systems
Schools as Complex Adaptive Systems
GenerateSystem-wide
Patterns
AgentsInteract Source: Patterson,
Holladay, & Eoyang, 2012
Schools as Complex Adaptive Social-Ecological Systems (CASS)
Schools as
CASS
Diverse & Dynamic Agents
Non-Linear &Unpredictable
Change & Co-Evolution Rules
Flow of Information
Feedback Loops
Nested Systems Structure
EmergenceInteraction
Dependent but
Autonomous
Source:Keshavarz, Nutbeam,
Rowling, & Khavarpour,
2010
Teachers as Curriculum Evaluators
Improvement of Teaching
Improvement of Learning
Political & Social
Accountability
Professional & Organizational Development
Significant and Lasting Change is Teacher-Led
Inquiry Design
Framework
Collective Learning Spiral
(P4D4)
Participatory Action Research Outcome
Mapping
Deliberative DemocraticCurriculum Evaluation
Participatory Action
Research
ParticipatoryPractical
& Collaborative
Critical
Transforms
Theory & Practice
Reflexive
Emancipatory
Social Process
Documents the Change Process
Outcome Mapping
Principles
Actor-Centered Development & Behavior Change
Non-Linearity &
Contribution
Participation &
Accountability
Continuous Learning & Flexibility
Source: Jones & Hearn, 2009
Democratic Evaluation Orientations
DiscursiveSampwith the
peoplele text
Elitis
tfo
r the
pe
ople
Participatory
by the people
1
3
2
Dialogue
Deliberation
Inclusion
1
2
3
1
1
2
2 3
3
Adapted from: Hanberger, 2004
Less More
0. Situation Analysis
Design
Do
Develop
Desc
ribe 1. Intentional
Design
Design
Do
Develop
Describe
2. Evaluation
Planning
Design
Do
Develop
Desc
ribe
START Reports
Reports
Reports
3. Outcome &
Performance Monitoring
GO TO START
CollectiveInquiry
Flow
Cycle 0: Situation Analysis
Literature & Document Review
Work Teams
Surveys &
ClassroomObservations
Boundary Partner, Influence, & SWOT Analyses
Interviews & conversations
Observethe Landscape
Analyzethe Patterns
Schools & Systems Thinking
Student-Centered Learning
Constructivist & Inquiry-
Based
Varied Measures of Achievement
Commitment to Life-Long
Learning
A Whole- Community Approach
Culture of Teaming
Indicators of
Readiness
Source: Benson, LaVigne, Marlin, & Yates, 2010
Influence Analysis
High
LowLow High
Leve
l of P
ower
Level of Interest
Mixed Blessing Supportive
Marginal Non-Supportive
Monitor Defend Against
InvolveCollaborate With
KEY
Teacher
Admin
Support Staff
Solid
Flexible
Not Movable
T
A
SS
A 1
A 3
A 2
A 4
SS1T
6T 5 T
3T 2
T 4
T 1
T 7
T 8
T 9
T 10
SS2
T 11
Cycle 1: Intentional Design
Outcome Challenges Progress
Markers
Organizational
Practices
Strategy Maps
Vision & Mission
Collective Visioning
Collective Action
Monitoring Priorities
Vision & Mission
Teachers critically examine the civic curriculum and collaborate to support the development of students’ civic competence.
Increased Civic Competency
Whole- School Effort
Support Networks
& Processes
Civic Education Resources
Informed Teachers
Transdisciplinary teacher teams promote
local and global civic competence using authentic learning activities.
Mapping the StrategyIn
divi
dual
/Col
lecti
veST
RATE
GIES
Environmental STRATEGIES
Establish Newsletter and Good Citizen of the Month Billboard (I-2)
Conduct Student and Teacher Surveys and Interviews (I-1)
Organize School-wide Civic Activities (I-2)
Support Students’ Civic Competence
Report Civic
Findings& News
to Faculty
(E-2)
Implement Observ- ation Teams (E-3)
Establish Critical Friends
Groups & Processes
(E-3)
Develop Observ-
ation Criteria
(E-1)
Facilitate Mock Election Workshops and Training (I-2)
Distribute Civic
Lesson Tips,
Plans, & Materials
(E-2)
Cycle 2: Evaluation Planning
Determine Evaluation Questions
Establish Timelines
Select
Evaluation
MethodsTheory of Change
Chart the Pathways
Plan the Route
IdentifyInformation
Sources
Theory of Change Excerpt The Project intends to see Social StudiesTeachers (SST) who recognize the importanceof increasing students’ civic competency
Teachers commit to action for increased student civic competency
Surveys InterviewsReport results in workshops
80% SST participate
SST reflect & engage ingenerative dialogue
SST collab-orate on civic curri-culum action plans
Assumption
Goal
Outcome Challenge
Strategy
Expect to See
Like to See
Love to See
KEY
“Hard” and “soft” data inform about civic competence levels, curricular strengths, and challenges
Increased ownership & responsibility
Cycle 3: Monitoring & Evaluation
Set Monitoring Priorities
Performance Journal
Strategy Journal
Analyze Cycle 0, 1, & 2 Data
Focus the Monitoring
Document the Changes
Outcome Journals
Documenting Teacher Change
20% regularly facilitated learning opportunities related to civic competency
61% reported students never researched local community problems in class
44% reported they never discussed civic responsibility with students
72% regularly facilitated learning opportunities related to civic competency
30% reported students never researched local community problems in class
32% reported they never discussed civic responsibility with students
Prior to Project Two Years Into Project
Documenting Student Change
11% of students reported that civic competency was regularly addressed in Social Studies class
62% reported they never talked about the importance of voting in Social Studies class
20% were able to correctly identify the current mayor
Prior to Project Two Years Into Project55% of students reported that civic competency was regularly addressed in Social Studies class
16% reported they never talked about the importance of voting in Social Studies class
66% were able to correctly identify the current mayor
Lessons Learned
More extensively targeting all subject-matter teachers
More deeply engaging Administrators
Building networks with other area schools
Intensified whole-school focus on civic competence
A network of educators committed to civic education that is less susceptible to being dismantled
ReferencesAslin, H. J. & Brown, V. A. (2004). Towards whole of community engagement: A practical toolkit (MDBC Publication No. 14/04). Canberra City,
Australian Capital Territory: Murray-Darling Basin Commission.Benson, T., LaVigne, A., Marlin, S., & Yates, J. (2010). Indicators of site readiness for innovation: Systems thinking in schools. Retrieved from
http://watersfoundation.org/wp-content/uploads/2013/08/Indicators-of-site-readiness.pdfBrodnick, R. J., Jr. (2000). Conceptualizing social systems: A critical argument for the nonlinear perspective (Doctoral dissertation). Available from
ProQuest Dissertations and Theses database. (UMI No. 9965960)Bronfenbrenner, U. (1976). The experimental ecology of education. Paper presented at American Educational Research Association Annual
Meeting, San Francisco, CA. Brown, V. A. (2001). Planners and the planet. Australian Planner, 38(2), 67–73. doi:10.1080/07293682.2001.9657941Brown, V. A. (2004). The more we are together... Collaborative decision making, social planning, and sustainability. Australian Planner, 41(3), 42–
45. doi:10.1080/07293682.2004.9982371Brown, V. A., & Lambert, J. (2013). Collective learning for transformational change: A guide to collaborative action. New York, NY: Routledge.Clarke, A. E. (2005). Situational analysis: Grounded theory after the postmodern turn. Thousand Oaks, CA: Sage Publications.Cropper, B. (2003). Fact-file 2: Unpacking the 5 disciplines... Becoming a learning organisation. Retrieved from
http://www.thechangeforum.com/factfiles/FactFile_2-The_5_Disciplines.pdfCrosby, D. A. (2014). In Imago Naturae: The ultimacy of nature as a closing of the gaps. In B. Donaldson (Ed.), Beyond the bifurcation of nature: A
common world for animals and the environment (pp. 180–191). Newcastle upon Tyne, UK: Cambridge Scholars Publishing. de Guerre, D. W. (2001). Doing action research in one's own organization: An ongoing conversation over time. Systemic Practice and Action
Research, 15(4), 331–349. doi:10.1023/A:1016348421584Dooley, K. J. (1996). A nominal definition of complex adaptive systems. The Chaos Network, 8(1), 2–3.Dooley, K. J. (1997). A complex adaptive systems model of organizational change. Nonlinear Dynamics, Psychology, and Life Sciences, (1)1, 69–97.
doi:10.1023/A:1022375910940Earl, S., Carden, F., & Smutylo, T. (2001). Outcome mapping: Building learning and reflection into development programs. Ottawa, Ontario,
Canada: International Development Research Centre.Eguren, I. R. (2011). Theory of change: A thinking and action approach to navigate in the complexity of social change processes . The Hague, The
Netherlands: Humanist Institute for Development Cooperation (Hivos).Eoyang, G. H. (2001). Conditions for self-organizing in human systems (Doctoral dissertation). Available from ProQuest Dissertations and Theses
database. (UMI No. 3040770)
ReferencesForrester, J. W. (1995). Counterintuitive behaviors of social systems. [based on October 7, 1970 testimony for the Subcommittee on Urban
Growth of the Committee on Banking and Currency, U.S. House of Representatives]. Retrieved from http://static.clexchange.org/ftp/documents/roadmaps/RM1/D-4468-2.pdf (Original work published 1971)
Giancola, J. M., & Hutchison, J. K. (2005). Transforming the culture of school leadership: Humanizing our practice. Thousand Oaks, CA: Corwin Press.
Greenwood, D. J., Whyte, W. F., Harkavy, I. (1993). Participatory action research as a process and a goal. Human Relations, 46(2), 175–192. doi:10.1177/001872679304600203
Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Thousand Oaks, CA: Sage Publications.Hanberger, A. Democratic governance and evaluation. Paper presented at the Sixth European Evaluation Society Conference, Berlin, Germany.
Retrieved from http://www.edusci.umu.se/digitalAssets/66/66094_hanbergergovernance04.pdfHenderson, J. G., & Gornick, R. (2007). Transformative curriculum leadership (3rd ed.). Upper Saddle River, NJ: Pearson Education.Jones, H., & Hearn, S. (2009, October). Outcome mapping: A realistic alternative for planning monitoring, and evaluation. Overseas
Development Institute Background Note. Retrieved from http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/5058.pdf
Kemmis, S., & McTaggart, R. (2005). Participatory action research: Communicative action and the public sphere. In N. K. Denzin & Y. S. Lincoln (Eds.), The SAGE handbook of qualitative research (3rd ed., 559–604). Thousand Oaks, CA: Sage Publications.
Keshavarz, N., Nutbeam, D., Rowling, L., & Khavarpour, F. (2010). Schools as social complex adaptive systems: A new way to understand the challenges of introducing the health promoting schools concept. Social Science and Medicine, 70(10), 1467–1474. doi:10.1016/j.socscimed.2010.01.034
Krathwohl, D. R. (1998). Methods of educational and social science research: An integrated approach (2nd ed.). Long Grove, IL: Waveland Press. Kurtz, C. F., & Snowden, D. J. (2003). The new dynamics of strategy: Sense-making in a complex and complicated world. IBM Systems Journal,
42(3), 462–483. doi:10.1147/sj.423.0462MacBeath, J. (1999). Schools must speak for themselves: The case for school self-evaluation. New York, NY: Routledge.MacDonald, B. (1976). Evaluation and the control of education. In D. Tawney (Ed.), Curriculum evaluation today: Trends and implications (pp.
125–136). London, UK: Macmillan.Mendelow, A. L. (1981). Environmental scanning—The impact of the stakeholder concept. Proceedings of the International Conference on
Information Systems, USA, 2, 407–418. Retrieved from http://aisel.aisnet.org/icis1981/20Merry, U. (1995). Coping with uncertainty: Insights from the new sciences of chaos, self-organization, and complexity . Westport, CT: Praeger.
Mertens, D.M. (2005). Research and evaluation in education and psychology: Integrating diversity with quantitative, qualitative, and mixed methods (2nd ed.). Thousand Oaks, CA: Sage Publications.
Morgan, G. (2006). Images of organization (Updated ed.). Thousand Oaks, CA: Sage Publications.Organizational Research Services. (2004). Theory of change: A practical tool for action, results and learning. Retrieved from
http://www.aecf.org/m/resourcedoc/aecf-theoryofchange-2004.pdf Patterson, L., Holladay, R., & Eoyang, G. (2012). Radical rules for schools: Adaptive action for complex change. Circle Pines, MN: Human Systems
Dynamics Institute.Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage Publications.Richmond , B. (2004). An introduction to systems thinking. Lebanon, NH: isee systems.Rittel, H. W. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4(2), 155–169. doi:10.1007/BF01405730Roduner, D., & Hartmann, O. (2009, October). Module on step-0: How to prepare for OM Intentional Design. Outcome Mapping Ideas
(Paper No. 3). Retrieved from http://www.outcomemapping.ca/download/simonhearn_en_OMidea3.pdfSavage, G. T., Nix, T. W., Whitehead, C. J., & Blair, J. D. (1991). Strategies for assessing and managing organizational stakeholders. Academy of
Management Executive, 5(2), 61–75. doi:10.5465/ame.1991.4274682 Schön, D. A. (1995). Knowing-in-action: The new scholarship requires a new epistemology. Change, 27(6), 26–34. Senge, P. M. (2006). The fifth discipline: The art and practice of the learning organization (Revised ed.). New York, NY: Doubleday.Sergiovanni, T. J. (1994). Building community in schools. San Francisco, CA: Jossey-Bass.Shapiro, I. (2005). Theories of change. Beyond Intractibility (G. Burgess & H. Burgess, Eds.). Conflict Information Consortium, University of
Colorado, Boulder. Retrieved from http://www.beyondintractability.org/essay/theories-of-changeThemessl-Huber, M. T., & Grutsch, M. A. (2003). The shifting locus of control in participatory evaluations. Evaluation, 9(1), 92–111. doi:
10.1177/1356389003009001006Wadsworth, Y. (1998). What is Participatory Action Research? Action Research International, Paper 2. Retrieved from:
http://www.scu.edu.au/schools/gcm/ar/ari/p-ywadsworth98.htmlWatkins, A., & Wilber, K. (2015). Wicked and wise: How to solve the world’s toughest problems. Croydon, UK: Urbane Publications.Zmuda, A., Kuklis, R., & Kline, E. (2004). Transforming schools: Creating a culture of continuous improvement. Alexandria, VA: Association for
Supervision and Curriculum Development.
References
Visual SourcesSlide 1: Cartoon evaluation image from Scheers & Wilson-Grau, 2008, GAN-Net Impact Community of Practice Meeting PresentationSlide 3: Project Information graphic template from www.powerframeworks.com Slide 4: Schools and Wicked Problems adapted from Rittel & Webber (1973); graphic template from www.SlideHunter.comSlide 5: Locating Evaluation and Action Research adapted from Krathwohl (1998 )using positional mapping as described by Clarke (2005)Slide 6: The Evaluation Process was adapted from Guba & Lincoln (1989) graphic template remixed from www.fppt.com Slide 7: The Evaluation Cube was adapted from Schratz (1999) as cited in Themessl-Huber & Grutsch (2003) and MacBeath (1999)Slide 9: Schools as Social Systems adapted from Brodnick, 2000, p. 120Slide 11: Knowledge Cultures as a Nested System visual representation remixed from: solvingforpattern.org; Material from Valerie A. BrownSlide 12: Cynefin Framework adapted from Kurtz & Snowden, 2004Slide 13: Competent System Vs. Incompetent System from Zmuda, Kuklis, & Kline, 2004, p. 43Slide 14: Complex Adaptive Systems quasicrystal graphic from http://wallpapers24k.blogspot.com/2012_05_01_archive.html networked agents
image from http://necsi.edu/research/networks/man53/man53.png Slide 15: Schools as CASS graphic template remixed from www.fppt.com Slide 16: Teachers as Curriculum Evaluators is adapted from from MacBeath (1999); graphic template from www.showeet.comSlide 17: Inquiry Design Framework graphic template from www.SlideHunter.comSlide 18: Participatory Action Research graphic template from www.showeet.com Slide 20: Collective Learning Spiral graphic template remixed from www.slideteam.net; material from Aslin & Brown, 2004; Brown (2001, 2004);
Brown & Lambert (2013). Slide 22: Democratic Evaluation Orientations graphic template remixed from www.showeet.com. Information adapted from Hanberger (2004)Slide 23: Collective Inquiry Flow circular flow of process graphic template from www.slideteam.net Slide 24: Cycle infinity ribbon graphic templates from www.powerframeworks.com Slide 25: Schools and Systems Thinking graphic template from www.fppt.com Slide 26: The Influence analysis representation fuses the insights of Mendelow (1981) and Savage, Nix, Whitehead, & Blair (1991), as a visual
representation of the positioning of the Project’s boundary partners.Slide 28: Vision and Mission graphic template from www.fppt.comSlide 29: Mapping the Strategy graphic template from www.SlideHunter.comSlide 33: Documenting Teacher Change graphic template from www.slidehunter.comSlide 35: Lessons Learned graphic template from www.powerframeworks.com Slide 36: Teachers as curriculum evaluators handshake remixed from https://commons.wikimedia.org/wiki/File:Handshake2.svg ; Vertical Spiral
image from http://i2.wp.com/www.makeyourbestself.com/wp-content/uploads/2015/04/upwardspiral.jpg?resize=699%2C649
Outcome Mapping for Planning Evaluations in American K–12 Urban Education: Potent Possibilities by Tabia Lee is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Please contact for permissions:Tabia Lee, Ed.D.
916-588-7776, [email protected]