Post on 14-Jul-2018
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Getting to Maybe:Evaluation,
Systems Thinking,and
Complexity Science
Tampa February 25, 2008
Michael Quinn PattonTampa 2008
Evaluation History
Setting the Context:Context Matters
So, In the beginning…
Michael Quinn PattonDecember 2007
Michael Quinn PattonDecember 2007
Utilization-FocusedEvaluation,
4th edition, May, 2008
•1st edition,1978•2nd edition, 1986
•3rd edition, 1997
Michael Quinn PattonDecember 2007
New Direction # 1
International and cross-cultural expansion ofevaluation:globalization and diversity
Michael Quinn PattonDecember 2007
New Direction # 2
From Studies to Streams
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Evaluation Trends:20 years ago
• One study for one user• Modest databases• Long time frames for studies• Presumption of direct use• Long reports• Generally single method• Dissemination was the written word
Michael Quinn PattonTampa 2008
10 years ago• One study for multiple users.
The age of stakeholders• Larger databases with computer support• More client focus• Not just decision use, but also conceptual use• Multiple teams producing information• Quantitative/qualitative wars come to an end:
valuing multiple methods
Michael Quinn PattonTampa 2008
And now…• We are moving from
discrete studies toinformation streams
• Systems not individual evaluatorsproduce evaluative knowledge
• Evaluative streams are multiple –integrating information from differentsources
Michael Quinn PattonTampa 2008
And now…• Data collection & reporting at multiple
levels by multiple-stakeholders• Databases are continuous and virtual• Time frames are immediate• Analysis is continuous• Virtual analysis of trends and conditions• Visual displays instead of narratives
Michael Quinn PattonTampa 2008
And now• Partnerships are dominant in collecting,
analyzing and sharing evaluativeknowledge
• Internet is the new information glue• Increased transparency of evaluative
knowledge• Emphasis on continuous organizational
adaptation and improvement
Michael Quinn PattonTampa 2008
New Direction # 3
Proliferation of evaluation models,theories, options, and methods,and approaches
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Original Primary Options
Formativeand
SummativeEvaluation
(Mid-term and End-of-Project Reviews)
Blandin CommunityLeadership Program
DevelopmentalEvaluation
Evidence-based Practice
Evaluation grew up in the “projects”testing models under a theory ofchange that pilot testing would lead toproven models that could bedisseminated and taken to scale:
The search for best practicesand evidenced-based practices
Fundamental Issue:How the World Is Changed
Top-down dissemination of“proven models”
versusBottoms-up adaptive management
Michael Quinn PattonTampa 2008
Conditions that challengeevaluation
• High innovation• Development• High uncertainty• Dynamic• Emergent• Systems change
Michael Quinn PattonTampa 2008
Challenge:Matching the evaluationprocess and design to thenature of the situation:
Contingency-basedEvaluation
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
New Direction # 4
Broader understandingand
conceptualization ofevaluation use
Michael Quinn PattonTampa 2008
Conceptualizing Use• Utilization-focused evaluation
now includes knowledgemanagement, organizationallearning, and facilitating change.The focus is as much oninstitutional uses of knowledgeas on individual users
Michael Quinn PattonTampa 2008
Re-conceptualizing Use
• Use is a process not a event• Use involves an interaction not just
a report• Use involves training for use not
just delivery of results• Use begins at the beginning not at
the endMichael Quinn Patton
Tampa 2008
New Direction # 5
Increased up-front role forevaluation & evaluatorsin intervention design:
Logic modeling &Theory of change work
Michael Quinn PattonTampa 2008
Some premises:• Evaluation is part of initial program design,
including conceptualizing the theory of change• Evaluator’s role is to help users clarify their
purpose, hoped-for results, and change model.• Evaluators can/should offer conceptual and
methodological options.• Evaluators can help by questioning assumptions.• Evaluators can play a key role in facilitating
evaluative thinking all along the way..• Designs can be emergent and flexible.
Michael Quinn PattonTampa 2008
New Direction # 6
Beyond linear logic models:
Systems Thinkingand
Complexity Science
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Three ways of conceptualizingand mapping theories of change
Linear Newtonian causality Interdependent systems
relationships Complex nonlinear dynamics
Michael Quinn PattonTampa 2008
Linear Logic ModelINPUTS (people, materials)ACTIVITIES (processes)
OUTPUTS OUTCOMES
CHANGES IN PEOPLES LIVES IMPACTS
CHANGES IN COMMUNITIES
Michael Quinn PattonTampa 2008
Pushing Force(Non-directional)
Dissatisfaction withthe
Status Quo
(Inertia)a.k.a. “Cost of
Change”
Resistance toChange
Pulling Force(Directional)
a.k.a. “Desirability ofthe end state”
Compelling Vision
Michael Quinn PattonTampa 2008
Pushing Force(Non-directional)
Dissatisfaction withthe
Status Quo
(Inertia)a.k.a. “Cost of
Change”
Resistance toChange
Pulling Force(Directional)
a.k.a. “Desirability ofthe end state”
Compelling Vision
First StepsBelievability
Causal Diagram of Beckhard’sChange Formula
Michael Quinn PattonTampa 2008
Output / Product
Essential Attributes
Attributes required to meet of exceed
customer needs:
"Do the Right Thing"
Efficacy
Appropriate
Characteristics to meet or exceed
customer wants and expectations
of excellence
"Do the Right Thing Well":
Efficiency
Dignity and Respect
Effectiveness
Timeliness
Reduce Waste
Safety
Continuity
Availability
What inputs need to go into the
process to make the product
that produces the desired
result?
What steps need to be taken to
create the product that achieves
the desired result?
What features / characteristics should the product
have?
Systems Logic Model
Customer
Outcomes
&
Satisfaction
! Measure
Effectiveness
! Measure
Satisfaction
! Inform
Improvement
needs
Effect
Inputs
Staff Resources
Financial resources
Internal Standards
External Requirements
and Information
Equipment/Materials
Key Processes & Functions
Inputs organized and utilized
Procedures
Steps
Key processes
Measure Variability
Assess Process Control
Assess fidelity to planned
procedures
Assess impact of variation
Evaluate opportunity to raise the
bar
Cause
Feedback into process
What is the desired result?
What should customer
experience?
Planning
Implementation
Structure Process OUTCOMES
Feedback
Michael Quinn PattonTampa 2008
Division workplace
that :
• Offers a healthy
work environment
• Recognizes
excellence
• Provides quality
training and
management
• Includes effective
systems ,
procedures , and
communication
( Goal 5 )
Increased adoption ,
reach , implementation ,
and sustainability of
recommended public
health strategies to
achieve strategic plan goals :
• Prevent risk factors for
heart disease and stroke (Goal 1 )
• Increase detection
and treatment of risk
factors (Goal 2)
• Increase early
identification and
treatment of heart attacks and strokes
( Goal 3 )
• Prevent recurring cardiovascular events
( Goal 4 )
Division for Heart Disease and Stroke Prevention Evaluation Planning Logic Model
Internal
Workforce that is :
• Diverse
• Skilled
Resources that are :
• Available
• Timely
Division Leadership
that provides
sufficient :
• Infrastructure
• Policies
• Strategic Planning
Leadership
Disparities
Surveillance
Research
Evaluation
Program
Translation
and dissemination
of the current
knowledge base ,
and identification
of ways to improve
that knowledge
base
Effective :
• Management
• Coordination
• Staff
development
Enhanced
competency of public
health workforce
Enhanced ability
of programs to apply
findings to improve
public health
Enhanced external
application of Division
goals and strategies
Increased advocacy
and “activated
constituency”
Engaged network of
states and partners
Enhanced integration
among chronic
disease programs
Increased focus
on heart disease
and stroke
prevention efforts
by states and
partners ,
especially with
regard to
disparities
Policy
Increased
knowledge of
signs and
symptoms
Improved
emergency
response
Improved quality
of care
Reduced risk
factors
Reduced
economic impact
of heart disease
and stroke
Eliminated
preventable
strokes and risks
Reduced levels of
disparities in heart
disease and stroke
Reduced
morbidity and
mortality of heart
disease and stroke
External
Planning ActivitiesTranslation ,
DisseminationAdoption , Practice , Sustainability Impact
WHAT WHYHOW
Communication
Collaboration
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
©TOM Jochen Enterprises, Möckernstr.78 10965 Berlin
Michael Quinn PattonTampa 2008
Step Two: THEN A MIRACLE OCCURS“I think you should be more explicit here in step two.”
Michael Quinn PattonTampa 2008
Systems
• Parts are interdependent such that achange in one part changes all parts
• The whole is greater than the sum ofthe parts
• Focus on interconnected relationships• Systems are made up of sub-systems
and function within larger systems
Michael Quinn PattonTampa 2008
Systems Concepts in Evaluation –An Expert Anthology. 2006.Bob Williams and Iraj Imam
AEA Monograph,EdgePress/AEA Point Reyes CA.
Michael Quinn PattonTampa 2008
Michael Quinn PattonTampa 2008
Understanding theElephant
froma Systems Perspective
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
The relationship between whatgoes in and what comes out
Whatconceptualframeworkinformsfront-endevaluationwork?
Michael Quinn PattonTampa 2008
Teen Pregnancy ProgramExample
Michael Quinn PattonTampa 2008
Logic Model for Pregnant Teens Program1. Program reaches out to pregnant teens
2. Pregnant teens enter and attend the program (participation)
3. Teens learn prenatal nutrition and self-care (increased knowledge)
4. Teens develop commitment to take care of themselvesand their babies (attitude change)
5. Teens adopt healthy behaviors: no smoking, no drinking,attend prenatal clinic, eat properly (behavior change)
6. Teens have healthy babies (desired outcome)
Michael Quinn PattonTampa 2008
Systems web showing possible influencelinkages to a pregnant teenager
Teachers/otheradults
Youngpregnantwoman's
attitudes &behaviors
Herparents &
other familymembers
Child'sfather &
peers
Prenatal program
staff
Her peer group
Michael Quinn PattonTampa 2008
Program systems web showing possible institutionalinfluences affecting pregnant teenagers:
SCHOOL SYSTEM
Youngpregnantwomen's
attitudes &behaviors
PrenatalClinic andHospitalOutreach
Church
Prenatal program
Othercommunity-based youth
programs
Other Systems-- welfare-- legal-- nutrition programs-- transportation-- child protection-- media messagesContext factors-- politics-- economic incentives-- social norms-- culture-- music
YouthCulture
Michael Quinn PattonTampa 2008
Sustainable change: Systems dynamic reinforcing feedback loops
Teenagerfeelsbetter
Teenagertells clinicnurseabout herchangedbehaviors
Teen tellsherfriendsand familyabout herchangedbehaviors
Teenagergetspositivefeedbackfrom clinicnurses
Teen getssupportfromfriendsand family
Teenagereatsproperly &takes careof herself
Pregnant teenagerlearns theimportance of notsmoking anddrinking -- andstops doing both.
ASustainableReinforcing
System
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Using Different System Lenses toUnderstand a “particular” System
Biologic System• Emergence• Coordination/synergy• Structure, Process, Pattern• Vitality
Sociologic System• Relationships• Conversations• Interdependence• Loose-tight coupling• Meaning/sense
Mechanical / Physical System• Flow• Temporal Sequencing• Spatial Proximities• Logistics• Information
Economic System Inputs/Outputs Cost/Waste/Value/Benefits Customers/Suppliers
Political System• Power• Governance• Citizenship• Equity
AnthropologicSystem• Values• Culture/Milieu
Information System•Access•Speed•Fidelity/utility•Privacy/security•Storage
Psychological System•Organizing•Forces Field•Ecological/BehaviourSettings
SYSTEMDIMENSIONS
Michael Quinn PattonTampa 2008
5
Map Systems as Webs
Source: Digital Capital: Harnessing the Power of Business W ebs,By Don Tapscott , David Ticoll and Alex Lowy
Michael Quinn PattonTampa 2008
Complex Dynamic SystemsConfiguration
Michael Quinn PattonTampa 2008
HIV/AIDS Example• Hits every system: health, family,
social, religious, economic, political,community, international
• Requires multiple interventions onmultiple fronts in all subsystemssimultaneously
• Resulting reactions, interactions,consequences dynamic, unpredictable,emergent, and ever changing
Michael Quinn PattonTampa 2008
Some system premises
Systems neutrality:An observed system isfunctioning as observed forsome reasons, fulfilling somefunctions.In whose interests is a systemfunctioning? Who benefits?
Michael Quinn PattonTampa 2008
Healthy systemIn a well-functioningsystem, no subsystem isoperating at its maximum.
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Systems changeDuring transitions
from one system toanother, things will getworse before they getbetter.
Michael Quinn PattonTampa 2008
Systems DynamicsDynamic system
interrelationships increase thelikelihood of unintendedconsequences as systemschange. Expect theunexpected.
Michael Quinn PattonTampa 2008
Challenges:
SituationRecognition
andAppropriate
EvaluationDesigns
Michael Quinn PattonTampa 2008
The nature ofEXPERTISE:
Situation Recognition
Michael Quinn PattonTampa 2008
Contingency-basedEvaluation
• Situational analysis & responsiveness• Context sensitivity• Clarify and focus on intended users:
stakeholder analysis• Clarify and focus on intended uses• Methodological appropriateness• Criteria for evaluating the evaluation:
credibility, meaningfulness
Michael Quinn PattonTampa 2008
Conceptual Options
•Simple
•Complicated
•Complex
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Types of Community IssuesThe Stacey Matrix
Certainty
Agr
eem
ent
Close to Far from
Far f
rom
Clo
se to
Michael Quinn PattonTampa 2008
Simple
Certainty
Agr
eem
ent
Close to Far from
Far f
rom
Clo
se to Simple
Plan, control
Michael Quinn PattonTampa 2008
Following a Recipe A Rocket to the Moon Raising a Child
Complicated Complex The recipe is essential Recipes are tested toassure replicability oflater efforts No particular expertise;knowing how to cookincreases success Recipes producestandard products Certainty of same resultsevery time
Simple
Michael Quinn PattonTampa 2008
Technically Complicated
Certainty
Agr
eem
ent
Close to Far from
Far f
rom
Clo
se to Simple
Plan, controlTechnically Complicated
Experiment, coordinate expertise
Michael Quinn PattonTampa 2008
Following a Recipe A Rocket to the Moon Raising a Child• Formulae are
critical andnecessary
• Sending one rocketincreasesassurance that nextwill be ok
• High level ofexpertise in manyspecialized fields +coordination
• Rockets similar incritical ways
• High degree ofcertainty ofoutcome
Complicated Complex
The recipe is essential
Recipes are tested toassure replicability oflater efforts
No particularexpertise; knowing howto cook increasessuccess
Recipes producestandard products
Certainty of sameresults every time
Simple
Michael Quinn PattonTampa 2008
Socially Complicated
Certainty
Agr
eem
ent
Close to Far from
Far f
rom
Clo
se to Simple
Plan, controlTechnically Complicated Experiment, coordinate expertise
SociallyComplicated Build relationships,create commonground
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Socially complicated
Implementing human rightsagreements, like gender equity oroutlawing child labor
Environmental Initiatives Many different and competing
stakeholders Diverse vested interests High stakes
Michael Quinn PattonTampa 2008
Socially complicatedsituations
pose the challengeof coordinating and
integratingmany players
Michael Quinn PattonTampa 2008
Stakeholder MappingHigh Interest/ Low Power
THE INVOLVED
High Interest/ High Power
THE PLAYERS
THE CROWDLow interest/ Low Power
CONTEXTSETTERSLow Interest/ High Power
Michael Quinn PattonTampa 2008
Know When Your Challenges Are In theZone of Complexity
Certainty
Agr
eem
ent
Close to Far from
Far f
rom
Clo
se to Simple
Plan, control
Zone ofComplexity
Technically Complicated Experiment, coordinate expertise
SociallyComplicated Buildrelationships,create commonground
Systems ThinkingRelationship BuildingCollaborationGood Enough VisionChunking Around DriversMinimum SpecificationsMultiple ActionsAdaptability & Organic
Michael Quinn PattonTampa 2008
Following a Recipe A Rocket to the Moon Raising a Child
Sending one rocketincreasesassurance that nextwill be ok
High level of expertisein many specializedfields +coordination
Rockets similar incritical ways
High degree ofcertainty ofoutcome
• Formulae have onlya limited application
• Raising one childgives no assuranceof success with thenext
• Expertise can helpbut is not sufficient;relationships arekey
• Every child isunique
• Uncertainty ofoutcome remains
Complicated Complex
The recipe is essential
Recipes are tested toassure replicability oflater efforts
No particularexpertise; knowing howto cook increasessuccess
Recipes producestandard products
Certainty of sameresults every time
Simple
Michael Quinn PattonTampa 2008
Know When Your Challenges Are In theZone of Complexity
Certainty
Agr
eem
ent
Close to Far from
Far f
rom
Clo
se to Simple
Plan, control
Zone ofComplexity
Technically Complicated Experiment, coordinate expertise
SociallyComplicated Buildrelationships,create commonground
ChaosMassive Avoidance
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Following a Recipe A Rocket to the Moon Raising a Child• Formulae are critical
and necessary
• Sending one rocketincreases assurancethat next will be ok
• High level ofexpertise in manyspecialized fields +coordination
• Separate into partsand then coordinate
• Rockets similar incritical ways
• High degree ofcertainty of outcome
• Formulae have only alimited application
• Raising one childgives no assurance ofsuccess with the next
• Expertise can helpbut is not sufficient;relationships arekey
• Can’t separate partsfrom the whole
• Every child is unique
• Uncertainty ofoutcome remains
Complicated
Complex
The recipe is essential
Recipes are tested toassure replicability oflater efforts
No particularexpertise; knowing howto cook increasessuccess
Recipe notes thequantity and nature of“parts” needed
Recipes producestandard products
Certainty of sameresults every time
Simple
Michael Quinn PattonTampa 2008
The Frogtown NeighborhoodChildren’s Community Initiative
in Saint Paul, Minnesota
Michael Quinn PattonTampa 2008
Seeing Through A Complexity Lens
“You don't see something until you have the right metaphor to let you perceive it”. Thomas Kuhn Michael Quinn Patton
Tampa 2008
Complex Nonlinear Dynamics• Nonlinear: Small actions can have large
reactions. “The Butterfly WingsMetaphor”
• Emergent: Self-organizing, Attractors• Dynamic: Interactions within, between,
and among subsystems and partswithin systems can volatile, changing
• Getting to Maybe: Uncertainty,unpredictable, uncontrollable
Major Sources ofUncertainty
• Human irrationality:Behavioral Economics
• Different contexts• Change in all its splendid
manifestationsMichael Quinn Patton
Tampa 2008
New Direction # 7
Methodological Flexibility &Creativityversus
Methodological Rigidity
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
The Debate AboutRandomized Controls in
Evaluation:
The Gold StandardQuestion
Michael Quinn PattonTampa 2008
Design Tension
Single StandardHierarchy
vsSituational Variation and
Appropriateness
Michael Quinn PattonTampa 2008
GOLD STANDARD:
METHODOLOGICALAPPROPRIATENESS
not
Methodologicalorthodoxy or rigidity
Michael Quinn PattonTampa 2008
Creative Design Thinking
David P. Billington:*
“The goal of good design is tointegrate efficiency, economyand elegance in a single design.”
* August 18, 2007, NY Times, A13
Michael Quinn PattonTampa 2008
Photo byLynsey Gornick
Michael Quinn PattonTampa 2008
“One Bridge Doesn’t Fit All”
“As many have pointed out, the deadlybridge failure in Minneapolis wassymptomatic of a system of bridges thatwill continue to corrode, crack and crumbleif not maintained. But maintenance is notthe only problem. We also need to designand build better bridges.”
Metaphor for evaluation design
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
“The Minneapolis collapse ishauntingly similar to the collapse in1983 of another interstate highwaybridge over the Mianus River inConnecticut. That disaster led toinspections of similar bridges, whichfound dangerous cracks fromdeferred maintenance….”
Michael Quinn PattonTampa 2008
What the Mianus and Minneapolisbridges had in common was not justneglect. Both were the products of adesign mentality in which engineerssimply used a standard form, andoften the same detailed features.Public bridges are all too oftendesigned by anonymous teams, andthe results can be seen on ourhighways.
Michael Quinn PattonTampa 2008
Creative Challenge
Situational adaptability: Contingency-based evaluation Appropriateness
--Using standard forms of evaluationand
-- Going beyond standard forms whenappropriate and useful
Michael Quinn PattonTampa 2008
Standard Evaluation Forms1. Inadequate upfront utilization focus2. Program/project as the unit of analysis3. Linear logic models4. Focus on findings use vs whole process5. Individual outcomes focus vs systems
change6. Preference for quantitative data & RCTs
as the methodological Gold Standard7. Static designs
Michael Quinn PattonTampa 2008
Getting to Maybe:How the World IsChanged? 2006Frances Westley, BrendaZimmerman, Michael Q. PattonRandom House Canada,
Michael Quinn PattonTampa 2008
Complex Situations• Highly emergent (difficult to
plan and predict)• Highly dynamic, rapidly
changing• Relationships are non-linear &
interdependent rather thansimple (linear cause-effect)
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Complex Interdependencies
Michael Quinn PattonTampa 2008
Insert action into thesystem
Michael Quinn PattonTampa 2008
EMERGENCE
Michael Quinn PattonTampa 2008
Contingency-basedDevelopmental
Evaluation
Michael Quinn PattonTampa 2008
Improvementversus
Development
Michael Quinn PattonTampa 2008
BeyondjustSummativeandFormative
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
BeyondStatic Accountability
Models
Michael Quinn PattonTampa 2008
Example of an emergent option:
DevelopmentalEvaluation
Michael Quinn PattonTampa 2008
DEVELOPMENTAL EVALUATIONDEFINED
Evaluation processes, including asking evaluativequestions and applying evaluation logic, to supportprogram, product, staff and/or organizationaldevelopment. The evaluator is part of a team whosemembers collaborate to conceptualize, design and test newapproaches in a long-term, on-going process of continuousimprovement, adaptation and intentional change. Theevaluator's primary function in the team is to elucidateteam discussions with evaluative questions, data and logic,and facilitate data-based decision-makingin the developmental process.
Michael Quinn PattonTampa 2008
CONTRASTS
Traditionalevaluations…Testing models
Complexity-based,DevelopmentalEvaluation…
• Supportinginnovation andadaptation
Michael Quinn PattonTampa 2008
TraditionalEvaluation…
• Renderdefinitivejudgments ofsuccess orfailure
DevelopmentalEvaluation…
• Provide feedback,generate learnings,support directionor affirm changesin direction in realtime
Michael Quinn PattonTampa 2008
TraditionalEvaluation…
• Measuresuccessagainstpredeterminedgoals
Developmental Evaluation…
• Develop newmeasures andmonitoringmechanisms asgoals emerge &evolve
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Evaluatorexternal,independent,objective
• Evaluator part of ateam, a facilitatorand learning coachbringing evaluativethinking to the table,supportive of theorganization’s goals
Michael Quinn PattonTampa 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Evaluatordetermines thedesign based on theevaluator’sperspective aboutwhat is important.The evaluatorcontrols theevaluation.
• Evaluatorcollaborates withthose engaged inthe change effort todesign anevaluation processthat matchesphilosophically andorganizationally.
Michael Quinn PattonTampa 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Design theevaluation based onlinear cause-effectlogic models
• Design theevaluation tocapture systemdynamics,interdependencies,
and emergentinterconnections
Michael Quinn PattonTampa 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Aim to producegeneralizablefindings acrosstime & space
.
• Aim to producecontext-specificunderstandingsthat informongoing
innovation
Michael Quinn PattonTampa 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Accountabilityfocused on anddirected to externalauthorities andfunders.
• Accountabilitycentered on theinnovators’ deepsense offundamental valuesand commitments –and learning.
Michael Quinn PattonTampa 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Accountability tocontrol and locateblame for failures
• Learning to respondto lack of controland stay in touchwith what’sunfolding
• And therebyrespondstrategically
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonTampa 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Evaluation often acompliance functiondelegated down inthe organization
• Evaluation aleadership function:
Reality-testing,results-focused,learning-oriented
leadership
Michael Quinn PattonTampa 2008
Traditional DevelopmentalEvaluation… Evaluation…
• Evaluationengendersfear of failure.
• Evaluation supportshunger for learning.
Michael Quinn PattonTampa 2008
Conditions
• High innovation• Development• High uncertainty• Dynamic• Emergent• Systems change
Michael Quinn PattonTampa 2008
SenseMaker software• Dave Snowden, Founder of Cognitive
Edge, former Director of KnowledgeManagement at IBM
• SenseMaker can code and map 95,000stories in 24 hours
• See the world as others see it; anti-terrorapplications.
• See the quantitative patterns in the meta-data with qualitative context and meaning
Michael Quinn PattonDecember 2007
New Direction # 8
Infusing evaluative thinking as aprimary type of evaluationprocess use.
Capacity-building as an evaluation focus.
Michael Quinn PattonTampa 2008
Thinking about:
The role can evaluation playwith complex dynamicinnovations….
21st Annual RTC ConferencePresented in Tampa, February 2008
Michael Quinn PattonDecember 2007
Michael Quinn PattonDecember 2007
And the beat goes on…
Evaluation as anever-evolving field
Michael Quinn PattonTampa 2008
References
• Getting to Maybe: How the World IsChanged? Frances Westley, BrendaZimmerman, Michael Q. PattonRandom House Canada, 2006.
• Utilization-Focused Evaluation, 4th ed.,Michael Quinn Patton, Sage Publications,2008