Michael Quinn Patton May, 2008
Evaluating the Complex:Getting to Maybe
Michael Quinn Patton
Oslo, Norway
29 May, 2008
Michael Quinn Patton May, 2008
Evolving Understandings
I keep changing what I said. Any person who is intellectually alive changes his ideas. If anyone at a university is teaching the same thing they were teaching five years ago, either the field is dead, or they haven’t been thinking.
Noam Chomsky
“The Professor Provaocateur,” The New York Times Magazine, Nov. 2, 2003: 13.
Michael Quinn Patton May, 2008
Interpretive Frameworks
• May 2003 Harvard Business Review "The High Cost of Accuracy." Kathleen Sutcliffe and Klaus Weber.
They concluded that "the way senior executives interpret their business environment is more important for performance than how accurately they know their environment."
Michael Quinn Patton May, 2008
They further concluded that it is a waste of resources to spend a lot of money increasing the marginal accuracy of data available to senior executives compared to the value of enhancing their capacity to interpret whatever data they have. Executives were more limited by a lack of capacity to make sense of data than by inadequate or inaccurate data. In essence, they found that interpretive capacity, or "mind-sets," distinguish high-performance more than data quality and accuracy.
Michael Quinn Patton May, 2008
Original Primary Options
Formative
and
Summative
Evaluation(Mid-term and End-of-Project Reviews)
Michael Quinn Patton May, 2008
Evidence-based Practice
Evaluation grew up in the “projects” testing models under a theory of change that pilot testing would lead to proven models that could be disseminated and taken to scale:
The search for best practices
and evidenced-based practices
Michael Quinn Patton May, 2008
Fundamental Issue:How the World Is Changed
Top-down dissemination of
“proven models”
versus
Bottoms-up adaptive management
Michael Quinn Patton May, 2008
Models vs. Principles
Identifying proven principles for adaptive management
(bottoms-up approach)
versus
Identifying and disseminating
proven models
(top down approach)
Michael Quinn Patton May, 2008
Conditions that challenge traditional model-testing evaluation
• High innovation
• Development
• High uncertainty
• Dynamic
• Emergent
• Systems Change
AdaptiveManagement
Michael Quinn Patton May, 2008
Mintzberg on StrategyTwo types of strategy: Intended & Emergent
Unrealized
Strategy
Intended
Strategy Deliberate Strategy
Realized
EmergentStrategy
Strategy
Michael Quinn Patton May, 2008
Re-conceptualizing Use
• Use is a process not a event
• Use involves an interaction not just a report
• Use involves training for use not just delivery of results
Michael Quinn Patton May, 2008
Some premises:• Evaluation is part of initial program design, including
conceptualizing the theory of change• Evaluator’s role is to help users clarify their purpose, hoped-
for results, and change model.• Evaluators can/should offer conceptual and methodological
options.• Evaluators can help by questioning assumptions.• Evaluators can play a key role in facilitating evaluative
thinking all along the way.• Interpretative dialogue is critical. • Designs can be emergent and flexible.
Michael Quinn Patton May, 2008
Three ways of conceptualizingand mapping theories of change
Linear Newtonian causality Interdependent systems
relationships Complex nonlinear dynamics
Michael Quinn Patton May, 2008
Linear Logic ModelINPUTS (people, materials)ACTIVITIES (processes)
OUTPUTS OUTCOMES
CHANGES IN PEOPLES LIVES IMPACTS
CHANGES IN COMMUNITIES
Michael Quinn Patton May, 2008
Pushing Force
(Non-directional)
Dissatisfaction with the
Status Quo
(Inertia)
a.k.a. “Cost of Change”
Resistance to Change
Pulling Force (Directional)
a.k.a. “Desirability of the end state”
Compelling Vision
Michael Quinn Patton May, 2008
Pushing Force
(Non-directional)
Dissatisfaction with the
Status Quo
(Inertia)
a.k.a. “Cost of Change”
Resistance to Change
Pulling Force (Directional)
a.k.a. “Desirability of the end state”
Compelling Vision
First StepsBelievability
Causal Diagram of Beckhard’sChange Formula
Michael Quinn Patton May, 2008
Output / ProductEssential Attributes
Attributes required to meet of exceedcustomer needs:
"Do the Right Thing"EfficacyAppropriate
Characteristics to meet or exceedcustomer wants and expectationsof excellence
"Do the Right Thing Well":EfficiencyDignity and RespectEffectivenessTimelinessReduce WasteSafetyContinuityAvailability
What inputs need to go into theprocess to make the productthat produces the desiredresult?
What steps need to be taken tocreate the product that achievesthe desired result?
What features / characteristics should the producthave?
Systems Logic Model
CustomerOutcomes
&Satisfaction
MeasureEffectiveness
MeasureSatisfaction
InformImprovementneeds
Effect
Inputs
Staff ResourcesFinancial resourcesInternal StandardsExternal Requirements
and InformationEquipment/Materials
Key Processes & Functions
Inputs organized and utilizedProceduresStepsKey processes
Measure VariabilityAssess Process ControlAssess fidelity to planned
proceduresAssess impact of variationEvaluate opportunity to raise the
bar
Cause
Feedback into process
What is the desired result?What should customerexperience?
Planning
Implementation
Structure Process OUTCOMES
Feedback
Michael Quinn Patton May, 2008
Division workplace that:· Offers a healthy
work environment· Recognizes
excellence· Provides quality
training and management
· Includes effective systems, procedures, and communication (Goal 5)
Increased adoption, reach, implementation, and sustainability of recommended public health strategies to achieve strategic plan goals:· Prevent risk factors for
heart disease and stroke (Goal 1)
· Increase detection and treatment of risk factors (Goal 2)
· Increase early identification and treatment of heart attacks and strokes (Goal 3)
· Prevent recurring cardiovascular events (Goal 4)
Division for Heart Disease and Stroke Prevention Evaluation Planning Logic Model
Internal
Workforce that is:· Diverse· Skilled
Resources that are:· Available· Timely
Division Leadership that provides sufficient:· Infrastructure· Policies· Strategic Planning
Leadership
Disparities
Surveillance
Research
Evaluation
Program
Translation and dissemination of the current knowledge base, and identification of ways to improve that knowledge base
Effective:· Management· Coordination· Staff
development
Enhanced competency of public health workforce
Enhanced ability of programs to apply findings to improve public health
Enhanced external application of Division goals and strategies
Increased advocacy and “activated constituency”
Engaged network of states and partners
Enhanced integration among chronic disease programs
Increased focus on heart disease and stroke prevention efforts by states and partners, especially with regard to disparities
Policy
Increased knowledge of signs and symptoms
Improved emergency response
Improved quality of care
Reduced risk factors
Reduced economic impact of heart disease and stroke
Eliminated preventable strokes and risks
Reduced levels of disparities in heart disease and stroke
Reduced morbidity and mortality of heart disease and stroke
External
Planning Activities Translation,
Dissemination Adoption, Practice, Sustainability Impact
WHAT WHYHOW
Communication
Collaboration
Michael Quinn Patton May, 2008
Step Two: THEN A MIRACLE OCCURS
“I think you should be more explicit here in step two.”
Michael Quinn Patton May, 2008
Systems
• Parts are interdependent such a change in one part changes all parts
• The whole is greater than the sum of the parts
• Focus on interconnected relationships
• Systems are made up of sub-systems and function within larger systems
Michael Quinn Patton May, 2008
Systems Concepts in Evaluation – An Expert Anthology. 2006.Bob Williams and Iraj Imam
AEA Monograph,
EdgePress/AEA Point Reyes CA.
Michael Quinn Patton May, 2008
The relationship between what goes in and what comes
out What conceptual framework informs front-end evaluation work?
Michael Quinn Patton May, 2008
Logic Model for Pregnant Teens Program1. Program reaches out to pregnant teens
2. Pregnant teens enter and attend the program (participation)
3. Teens learn prenatal nutrition and self-care (increased knowledge)
4. Teens develop commitment to take care of themselvesand their babies (attitude change)
5. Teens adopt healthy behaviors: no smoking, no drinking,attend prenatal clinic, eat properly (behavior change)
6. Teens have healthy babies (desired outcome)
Michael Quinn Patton May, 2008
Systems web showing possible influence linkages to a pregnant teenager
Teachers/ other adults
Youngpregnantwoman's
attitudes &behaviors
Herparents &
other familymembers
Child'sfather &
peers
Prenatal program
staff
Her peer group
Michael Quinn Patton May, 2008
Program systems web showing possible institutional influences affecting pregnant teenagers:
SCHOOL SYSTEM
Youngpregnantwomen's
attitudes &behaviors
PrenatalClinic andHospitalOutreach
Church
Prenatal program
Other community-based youth
programs
Other Systems-- welfare-- legal -- nutrition programs-- transportation-- child protection-- media messagesContext factors-- politics-- economic incentives-- social norms-- culture-- music
YouthCulture
Michael Quinn Patton May, 2008
Using Different System Lenses to Understand a “particular” System
Biologic System• Emergence• Coordination/synergy• Structure, Process, Pattern• Vitality
Sociologic System• Relationships• Conversations• Interdependence• Loose-tight coupling• Meaning/sense
Mechanical / Physical System• Flow• Temporal Sequencing• Spatial Proximities• Logistics• Information
Economic System Inputs/Outputs Cost/Waste/Value/Benefits Customers/Suppliers
Political System• Power• Governance• Citizenship• Equity
Anthropologic System• Values• Culture/Milieu
Information System•Access•Speed•Fidelity/utility•Privacy/security•Storage
Psychological System•Organizing•Forces Field•Ecological/Behaviour Settings
SYSTEMDIMENSIONS
Michael Quinn Patton May, 2008
5
Map Systems as Webs
Source: Digital Capital: Harnessing the Power of Business Webs,By Don Tapscott, David Ticoll and Alex Lowy
Michael Quinn Patton May, 2008
HIV/AIDS Example• Hits every system: health, family,
social, religious, economic, political, community, international
• Requires multiple interventions on multiple fronts in all subsystems simultaneously
• Resulting reactions, interactions, consequences dynamic, unpredictable, emergent, and ever changing
Michael Quinn Patton May, 2008
Contingency-basedEvaluation
• Situational analysis & responsiveness• Context sensitivity• Clarify and focus on intended users:
stakeholder analysis• Clarify and focus on intended uses• Methodological appropriateness• Criteria for evaluating the evaluation:
credibility, meaningfulness
Michael Quinn Patton May, 2008
Seeing Through A Complexity Lens
“You don't see something until you have the right metaphor to let you perceive it”. Thomas Kuhn
Michael Quinn Patton May, 2008
Complex Nonlinear Dynamics• Nonlinear: Small actions can have large
reactions. “The Butterfly Wings Metaphor”
• Emergent: Self-organizing, Attractors
• Dynamic: Interactions within, between, and among subsystems and parts within systems can volatile, changing
• Getting to Maybe: Uncertainty, unpredictable, uncontrollable
Michael Quinn Patton May, 2008
Michael Quinn Patton TEI 2008
SNOWDEN’s CYNAFIN FRAMEWORK
Simple, Complicated, Complex, Chaotic
and Disordered Behaviours
Michael Quinn Patton May, 2008
Michael Quinn Patton TEI 2008
Linear contextualisation: 1
Mostchaotic
Mostordered
orderunorder
complexchaotic simplecomplicated
Michael Quinn Patton May, 2008
Types of Community IssuesThe Stacey Matrix
Certainty
Ag
reem
en
t
Close to Far from
Far
from
Clo
se to
Michael Quinn Patton May, 2008
Simple
Certainty
Ag
reem
en
t
Close to Far from
Far
from
Clo
se to Simple
Plan, control
Michael Quinn Patton May, 2008
Michael Quinn Patton TEI 2008
Simple (Known)Inter-relationships between elements :
Tight, centralised connections. Anyone can see the things the way they are.
Very simple linear cause and effect.
Everyone knows the right answer within the current context (which of course may
not be self-evident or known to others – hence importance of context).
Michael Quinn Patton May, 2008
Following a Recipe A Rocket to the Moon Raising a Child
Complicated Complex
The recipe is essential
Recipes are tested to assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipes produce standard products
Certainty of same results every time
Simple
Michael Quinn Patton May, 2008
Michael Quinn Patton TEI 2008
Complicated (Knowable)Inter-relationships between elements:
Relationships are looser but still clustered around a central core. Cause and effect is non-linear.
Relationships able to be modelled and able to predicted.
An expert would know the right answer(s)
Michael Quinn Patton May, 2008
Technically Complicated
Certainty
Ag
reem
en
t
Close to Far from
Far
from
Clo
se to Simple
Plan, control
Technically Complicated Experiment, coordinate expertise
Michael Quinn Patton May, 2008
Following a Recipe A Rocket to the Moon Raising a Child• Formulae are
critical and necessary
• Sending one rocket increases assurance that next will be ok
• High level of expertise in many specialized fields + coordination
• Rockets similar in critical ways
• High degree of certainty of outcome
Complicated Complex
The recipe is essential
Recipes are tested to assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipes produce standard products
Certainty of same results every time
Simple
Michael Quinn Patton May, 2008
Socially Complicated
Certainty
Ag
reem
en
t
Close to Far from
Far
from
Clo
se to Simple
Plan, control
Technically Complicated Experiment, coordinate expertise
SociallyComplicated Build relationships, create common ground
Michael Quinn Patton May, 2008
Socially complicated
Implementing human rights agreements, like gender equity or outlawing child labor
Environmental Initiatives Many different and competing
stakeholders Diverse vested interests High stakes
Michael Quinn Patton May, 2008
Socially complicatedsituations
pose the challenge
of coordinating and
integrating
many players
Michael Quinn Patton May, 2008
Stakeholder Mapping
High Interest/
Low Power
THE INVOLVED
High Interest/
High Power
THE PLAYERS
THE CROWD
Low interest/
Low Power
CONTEXT SETTERS
Low Interest/
High Power
Michael Quinn Patton May, 2008
Michael Quinn Patton TEI 2008
ComplexInter-relationships between elements:
Centre is loosely connected to network. Cause effect difficult to understand
in current setting. Situation understandable only in retrospect.
Not predictable.
Michael Quinn Patton May, 2008
Know When Your Challenges Are In the Zone of Complexity
Certainty
Ag
reem
en
t
Close to Far from
Far
from
Clo
se to Simple
Plan, control
Zone of Complexit
y
Technically Complicated Experiment, coordinate expertise
SociallyComplicated Build relationships, create common ground
Systems ThinkingRelationship BuildingCollaborationGood Enough VisionChunking Around Drivers Minimum SpecificationsMultiple Actions Adaptability & Organic
Michael Quinn Patton May, 2008
Following a Recipe A Rocket to the Moon Raising a Child
Sending one rocket increases assurance that next will be ok
High level of expertise in many specialized fields + coordination
Rockets similar in critical ways
High degree of certainty of outcome
• Formulae have only a limited application
• Raising one child gives no assurance of success with the next
• Expertise can help but is not sufficient; relationships are key
• Every child is unique
• Uncertainty of outcome remains
Complicated Complex
The recipe is essential
Recipes are tested to assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipes produce standard products
Certainty of same results every time
Simple
Michael Quinn Patton May, 2008
Michael Quinn Patton TEI 2008
Chaotic (Unordered)Cause and effect unknowable,
unattributable even in retrospect.No right answer(s)
Michael Quinn Patton May, 2008
Know When Your Challenges Are In the Zone of Complexity
Certainty
Ag
reem
en
t
Close to Far from
Far
from
Clo
se to Simple
Plan, control
Zone of Complexity
Technically Complicated Experiment, coordinate expertise
SociallyComplicated Build relationships, create common ground
ChaosMassive
Avoidance
Michael Quinn Patton May, 2008
Following a Recipe A Rocket to the Moon Raising a Child• Formulae are
critical and necessary
• Sending one rocket increases assurance that next will be ok
• High level of expertise in many specialized fields + coordination
• Separate into parts and then coordinate
• Rockets similar in critical ways
• High degree of certainty of outcome
• Formulae have only a limited application
• Raising one child gives no assurance of success with the next
• Expertise can help but is not sufficient; relationships are key
• Can’t separate parts from the whole
• Every child is unique
• Uncertainty of outcome remains
Complicated
Complex
The recipe is essential
Recipes are tested to assure replicability of later efforts
No particular expertise; knowing how to cook increases success
Recipe notes the quantity and nature of “parts” needed
Recipes produce standard products
Certainty of same results every time
Simple
Michael Quinn Patton May, 2008
Global Economic Complexity
Arthur Greenspan, Final speech to world’s Central Bankers, Jackson Hole, Wyoming
August 26, 2005
Michael Quinn Patton May, 2008
• “In the absence of a single variable, or at most a few, that can serve as a reliable guide, policymakers have been forced to fall back on an approach that entails the interpretation of the full range of economic and financial data.”
Michael Quinn Patton May, 2008
• “Despite extensive efforts to capture and quantify what we perceive as the key macroeconomic relationships, our knowledge about many critical linkages is far from complete and, in all likelihood, will remain so. Every model, no matter how detailed or how well conceived, designed, and implemented, is a vastly simplified representation of the world, with all of the intricacies we experience on a day-to-day basis.”
Michael Quinn Patton May, 2008
• “We all temper the outputs of our models and test their results against the ongoing evaluations of a whole array of observations that we do not capture in either the data input or the structure of our models. We are particularly sensitive to observations that appear inconsistent with the causal relationships of our formal models.”
Michael Quinn Patton May, 2008
Oct 8, 2007 on The Daily Show“I was telling my colleagues the other day...I’d been dealing with these big mathematical models for forecasting the economy, and I’m looking at what’s going on the last few weeks and I say, “Y’know, if I could figure out a way to determine whether or not people are more fearful, or changing to euphoric... I don’t need any of this other stuff. I could forecast the economy better than any way I know. The trouble is, we can’t figure that out. I’ve been in the forecasting business for 50 years, and I’m no better than I ever was, and nobody else is either.” Alan Greenspan.
Michael Quinn Patton May, 2008
The Age of Turbulence: Adventures in a New World, 2007, by Alan Greenspan
Two evaluation locations
for
Evaluating the Complex:
Prospective
and
Retrospective
Michael Quinn Patton May, 2008
Michael Quinn Patton May, 2008
Prospective Example
The McGill-McConnell Leadership Program Example
Simple elements
Complicated elements
Complex elements
Simple outcomes
• Increase knowledge and skills of participants
Evaluation: Pre-post data and documentation of learning
Michael Quinn Patton May, 2008
Complicated Impacts
• Change participants’ organizations
Evaluation:
Case studies
of
organizational change
Michael Quinn Patton May, 2008
Complex Vision
• Infuse energy into the moribund not-for-profit (voluntary) sector
• Make the sector more dynamic
• Create network of leaders who actively engage in change
Michael Quinn Patton May, 2008
Evaluating the Complex
• Real time follow-up of network connections and actions
• Follow-up is an intervention
• Rapid feedback of findings permits infusion of resources in support of emergent outcomes
Michael Quinn Patton May, 2008
Michael Quinn Patton May, 2008
Process Use
Infusing evaluative thinking as a primary type of process use.
Capacity-building as an evaluation focus of
process use.
Michael Quinn Patton May, 2008
Retrospective Example
Advocacy Evaluation
Final Push Campaign
to overthrow the
Juvenile Death Penalty
In late 2003 several petitions on behalf of juvenile offenders facing the death penalty were filed with the U.S. Supreme Court. On January 26, 2004, the Court granted certiorari in Roper v. Simmons and the case was argued before the Court on October 13, 2004. The decision was announced March 1, 2005. The Court ruled 5-4 that capital punishment for juveniles was unconstitutional.
Michael Quinn Patton
May, 2008
In the brief window of time between when the Court agreed to hear the case and the case was argued, roughly nine months, a coordinated campaign was organized and funded aimed at overturning the juvenile death penalty. Organizing, public education, networking & communications continued through to the Court's ruling in March, 2005.
Michael Quinn Patton May, 2008
Retrospective Evaluation
To what extent, if at all, was the Court’s decision influenced by the campaign?
Modus Operandi
or
General Elimination Method
Michael Quinn Patton May, 2008
Michael Quinn Patton May, 2008
1. Strong high capacity coalitions. Working through coalitions is a common centerpiece of advocacy strategy.
2. Strong national-state-grassroots coordination. Effective policy change coalitions in the United States have to be able to work bottoms-up and top-down, with national campaigns supporting and coordinating state and grassroots efforts, while state efforts infuse national campaigns with local knowledge and grassroots energy. Strengthening strong national-state coordination is part of coalition development and field building.
3. Disciplined and focused messages with effective communications. Effective communications must occur within movements (message discipline) and to target audiences (focused messaging). Strengthening communications has been a key a key component of advocacy coalition building.
4. Solid research and knowledge base. The content of effective messages must be based on solid research and timely knowledge. In the knowledge age, policy coalitions must be able to marry their values with relevant research and real time data about dynamic policy environment.
5. Timely, opportunistic lobbying and judicial engagement. The evaluation findings emphasize that effective lobbying requires connections, skill, flexibility, coordination, and strategy.
6. Collaborating funders engaged in strategic funding. Effective funding involves not only financial support, but infusion of expertise and strategy as part of field building.
Michael Quinn Patton May, 2008
Targeted StateCampaigns & GrassrootsOrganizing
OmnibusCoordinatedIntegrated
Strategy and Implementation
Effective Litigation Credible, Useful and Amicus Up-to-date Research Briefs
United Focused Coalition Communications Partners Campaign
Knowledgeable Funders
Michael Quinn Patton May, 2008
Overall Lesson Learned for Effective Advocacy
In essence, strong national/state/grassroots coordination depends on having a high capacity coalition. A solid knowledge and research base contributes to a focused message and effective communications. Message discipline depends on a strong coalition and national-state coordination, as does timely and opportunistic lobbying and judicial engagement. To build and sustain a high capacity coalition, funders must use their resources and knowledge to collaborate around shared strategies. These factors in combination and mutual reinforcement strengthen advocacy efforts. In classic systems framing, the whole is greater than the sum of parts, and the optimal functioning of each part is dependent on the optimal integration and integrated functioning of the whole.
Michael Quinn Patton May, 2008
Strong
Timely,National/
GrassrootsCoordination
OpportunisticLobbying &
JudicialEngagement
StrongHigh Capacity
Coalitions
EFFECTIVEADVOCACY
DisciplinedFocusedMessage/Effective
Communications Collaborating Funders/ Strategic Funding
SIX INTERCONNECTED FACTORS,DYNAMICALLY INTERACTING,
THAT STRENGTHEN ADVOCACY
SolidKnowledge & Research Base
Michael Quinn Patton May, 2008
Strong, high capacity coalition
Collaborating funders/ strategic funding
Targeted timely
lobbying
National state
grassroots coordinatio
n
Disciplinedfocused message
Relevantresearch
The interdependent system of factors that contribute to effective advocacy and change
Michael Quinn Patton May, 2008
Creative Challenge
Situational adaptability: Contingency-based evaluation Appropriateness
--Using standard forms of evaluation and
-- Going beyond standard forms when appropriate and useful
Michael Quinn Patton May, 2008
“The range of what we think and do is limited by what we fail to notice. And because we fail to notice that we fail to notice, there is little we can do to change, until we notice how failing to notice shapes our thoughts and deeds.”
Scottish psychiatrist, R. Laing
Paradigms and Lenses
• The importance of interpretive frameworks
• Complexity as an interpretive framework
Michael Quinn Patton May, 2008
Michael Quinn Patton May, 2008
Getting to Maybe: How the World Is Changed? 2006Frances Westley, Brenda Zimmerman, Michael Q. Patton
Random House Canada,
Michael Quinn Patton May, 2008
Complex Situations• Highly emergent (difficult to
plan and predict)
• Highly dynamic, rapidly changing
• Relationships are interdependent and non-linear rather than simple and linear (cause-effect)
Michael Quinn Patton May, 2008
Certainty
Ag
reem
en
t
Close to Far from
Far
from
Clo
se to
?Zone of Complexity
Michael Quinn Patton May, 2008
Certainty
Ag
reem
en
t
Close to Far from
Far
from
Clo
se to
Search for root cause
Michael Quinn Patton May, 2008
Certainty
Ag
reem
en
t
Close to Far from
Far
from
Clo
se to
Search fo
r syste
mic feat
ures
DEVELOPMENTAL EVALUATION DEFINED
Evaluation processes, including asking evaluative
questions and applying evaluation logic, to support program, product, staff and/or organizational development. The evaluator is part of a team whose members collaborate to conceptualize, design and test new approaches in a long-term, on-going process of continuous improvement, adaptation and intentional change. The evaluator's primary function in the team is to elucidate team discussions with evaluative questions, data and logic, and facilitate data-based decision-making in the developmental process.
Michael Quinn Patton May, 2008
Michael Quinn Patton May, 2008
Other names
Real time evaluation Emergent evaluation Action evaluation Adaptive evaluation
Michael Quinn Patton May, 2008
CONTRASTS
Traditional evaluations…
• Testing models
Complexity-based, Developmental
Evaluation…
• Supporting innovation and adaptation
Michael Quinn Patton May, 2008
Traditional
Evaluation…
• Render definitive judgments of success or failure
Developmental Evaluation…
• Provide feedback, generate learnings, support direction or affirm changes in direction in real time
Michael Quinn Patton May, 2008
Traditional
Evaluation…• Render definitive judgments of
success or failure
• Measure success against predetermined goals
Developmental
Evaluation…• Provide feedback, generate
learnings, support direction or affirm changes in direction
• Develop new measures and monitoring mechanisms as goals emerge & evolve
Michael Quinn Patton May, 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Evaluator external, independent, objective
• Evaluator part of a team, a facilitator and learning coach bringing evaluative thinking to the table, supportive of the organization’s goals
Michael Quinn Patton May, 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Evaluator determines the design based on the evaluator’s perspective about what is important. The evaluator controls the evaluation.
• Evaluator collaborates with those engaged in the change effort to design an evaluation process that matches philosophically and organizationally.
Michael Quinn Patton May, 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Design the evaluation based on linear cause-effect logic models
• Design the evaluation to capture system dynamics, interdependencies,
and emergent interconnections
Michael Quinn Patton May, 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Aim to produce generalizable findings across time & space
.
• Aim to produce context-specific understandings that inform ongoing
innovation
Michael Quinn Patton May, 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Accountability focused on and directed to external authorities and funders.
• Accountability centered on the innovators’ deep sense of fundamental values and commitments –
and learning.
Michael Quinn Patton May, 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Accountability to control and locate blame for failures
• Learning to respond to lack of control and stay in touch with what’s unfolding
• And thereby respond strategically
Michael Quinn Patton May, 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Evaluation often a compliance function delegated down in the organization
• Evaluation a leadership function:
Reality-testing, results-focused, learning-oriented
leadership
Michael Quinn Patton May, 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Evaluation engenders
fear of failure.
• Evaluation supports hunger for learning.
Michael Quinn Patton May, 2008
Conditions
• High innovation
• Development
• High uncertainty
• Dynamic
• Emergent
Michael Quinn Patton May, 2008
SenseMaker software • Dave Snowden, Founder of Cognitive
Edge, former Director of Knowledge Management at IBM
• SenseMaker can code and map 95,000 stories in 24 hours
• See the world as others see it; anti-terror applications.
• See the quantitative patterns in the meta-data with qualitative context and meaning
Michael Quinn Patton May, 2008
Adaptive Cycle
Fore
loop
Growth r
ConservationK
Release
BackloopReorganization
Stored
Leadership/initiative
Exploration phase
System Capacity Building
Mature product and scaling up
Phases of Technological and Social Innovation
The birth, growth, destruction and renewal of a forest
Weak CONNECTEDNESS StrongLit
tle C
AP
ITA
L S
TO
RED
M
uch
Creative Destruction
1
The birth, growth, destruction and renewal of a forest
Weak CONNECTEDNESS StrongLit
tle C
AP
ITA
L S
TO
RED
M
uch
Creative Destruction
1
2
Renewal/Exploration
Renewal/Exploration
The birth, growth, destruction and renewal of a forest
Weak CONNECTEDNESS Strong
Exploitation
3
Lit
tle C
AP
ITA
L S
TO
RED
M
uch
The birth, growth, destruction and renewal of a forest
Weak CONNECTEDNESS Strong
Exploitation
3
Lit
tle C
AP
ITA
L S
TO
RED
M
uch
Conservation
4
Stored
HARVESTING LESSONS
DEVEOPMENTAL EVALUATION
FORMATIVE
SUMMATIVE
Phases of Technological & Social Innovation
Michael Quinn Patton May, 2008
Taking Emergence Seriously
• Beyond “unanticipated consequences” to genuine openness
Michael Quinn Patton May, 2008
SURPRISES
The psychologist Baruch Fischhoff wrote:
"The occurrence of an event increases its reconstructed probability"—in other words, surprises are psychologically untenable in some ways, and we reshape our memories and expectations until we believe that the surprising event was, in fact, likely.
Michael Quinn Patton May, 2008
The Epistemology of Surprise
MALCOLM GLADWELL
We hate surprises. We try to erase them from our memory. This is part of what keeps us sane. If, after all, we were always fully aware of the possibility of completely unpredictable events, would we be able to walk out the front door in the morning? Would we ever invest in the stock market? Would we have children? Generally speaking, people who have an accurate mental picture of why and how things happen tend to occupy mental hospitals—or, at the very least, a psychiatrist's office….
Michael Quinn Patton May, 2008
Information is like a stream teeming with fish, and if you stick out a net you'll collect something—but to decide what information is consequential. How does one do that?
Michael Quinn Patton May, 2008
At the heart is the process of distinguishing signal from noise.
How is that done? I have no idea, nor does anyone, I think, who isn't a seasoned analyst. Pattern recognition is something that comes only with experience. It's a matter of intuition, as much as anything.
People always want to reduce this sort of thing to a formula, or a system, and I'm not sure you can do that. I suspect that there are some artificial-intelligence systems that can help to sort through certain kinds of data. But that could only be a first cut, and eventually human judgment has to be involved.
Michael Quinn Patton May, 2008
Challenge:Matching the evaluation process and design to the nature of the situation:
Contingency-based
Evaluation
Top Related