RSD4 1C Moore - Evaluation at the frontiers of systemic design
Transcript of RSD4 1C Moore - Evaluation at the frontiers of systemic design
![Page 1: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/1.jpg)
Evaluation at the Frontiers of Systemic DesignPresentation at RSD4 Banff, AB, August 1, 2015*Marah Moore, (Claire Nicklin, Keith Miller)
![Page 2: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/2.jpg)
This presentations . . .
1. Systems perspective for complex problems
2. R&D as a human centered design process
3. From product to outcomes
Paradigm Shifts
What can we—International R&D—learn from design practitioners?
What can design practitioners learn from evaluators?
![Page 3: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/3.jpg)
High Andean Cropping systems
W Africa: Millet- and sorghum-
based cropping systems
E/H of Africa: Crop Improvement
Southern Africa: Integrating
legumes in cereal-based systems
CCRPThe McKnight Foundation’s Collaborative Crop Research Program (CCRP): Quick Overview
![Page 4: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/4.jpg)
CCRP Program
4 Regions
Grantee Projects
![Page 5: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/5.jpg)
Grants & TA
R&D/Farmer Capacity
Relevant Results
Communities grow local nutritious food,
sustainably
![Page 6: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/6.jpg)
Research and Development as a Design Process
![Page 7: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/7.jpg)
Traditional R&D Design: Linear Problem Solving Approach “Product” Design
![Page 8: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/8.jpg)
Soil
Pests
Climate
![Page 9: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/9.jpg)
SocialContext
PoliticalContext
HistoricalContext
CulturalContext
InstitutionalContext
EnvironmentalContext
Soil
Pests
Climate
![Page 10: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/10.jpg)
![Page 11: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/11.jpg)
Rethinking R&D as HC Design: A Systems Approach (R&D as a “service” to design outcomes)
![Page 12: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/12.jpg)
Complex Adaptive Systems
. . . dynamic systems able to adapt in and evolve with a changing environment.
![Page 13: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/13.jpg)
I M E P(Integrated Monitoring Evaluation and Planning)
![Page 14: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/14.jpg)
The Role of Evaluation
Traditional evaluation mirrors traditional R&D in many ways:
• Top down• Narrow focus• Isolated results• Or, sometimes only process . . .
![Page 15: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/15.jpg)
In Michael Quinn Patton, Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, Guilford Press, June 2010
![Page 16: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/16.jpg)
Design Evaluation
Collect Data
Shape Practice
Make Meaning
from Data
Traditional Evaluation Framework
![Page 17: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/17.jpg)
Collect Data
Make Meaning from Data
Design Evaluation
SHAPE PRACTICE
Complex Systems-Oriented View of Phases of Evaluation
![Page 18: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/18.jpg)
An Alternative: Developmental Evaluation
Evaluation to support the development of innovation in complex situations
![Page 19: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/19.jpg)
CCRP DE Challenges:
1. Fostering Adaptive Capacity
2. Building Coherence: Articulating and Testing Assumptions About Change
3. Going to Scale in a Complex Environment
![Page 20: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/20.jpg)
Systems Thinking
Design Thinking
CCRP DE Challenge # 1: Fostering Adaptive Capacity
Para
digm
Shi
ft
LEARNING
![Page 21: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/21.jpg)
Adaptive Action: iterative cycles, design mindset
So What?
Now What?
What?Program
RegionsProjects
![Page 22: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/22.jpg)
Example: Soils, Northern Andes
What? So What?
Now What?
![Page 23: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/23.jpg)
CCRP DE Challenge #2: Building Coherence, Articulating and Testing Assumptions about Change
• Expanded vision
• Contextualized R&D
• Focus on outcomes
![Page 24: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/24.jpg)
1. Make
Hypotheses Explicit
• Theories of change at the project, region, and program levels
• Targeted research and evaluation questions
Project Level
![Page 25: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/25.jpg)
1. Make
Hypotheses Explicit
Adaptive Action in complex
systems
3. Collaborative interpretation,
planning, and design
2. Collect cross-cutting and deep data
![Page 26: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/26.jpg)
Going Deeper: Roles/jobs of women and men
Productive (10 tasks) (18 tasks)Agriculture:Place seed during sowingWork in sowing and harvesting
Animals:
Pasturing sheep and cows on the hillsBring cows to hillUse of local technologyWeaving and SpinningPut on the yokeBuild the house
Temporary migration to the cityGive money to the mothersLeave to earn money in the city
Services (14 tasks) (14 tasks)Outside the home:Purchase clothes and necessitiesTake sick children to the hospital or traditional healerBring firewood to the house
In the home:Prepare and serve food to the family
Toast quinoa for pito drinkClean the houseWash clothes
Help to make breadFeed children
Acculturation Raise childrenTeach our childrenTeach children how to work
Educate childrenReproductive Breastfeed childrenCarry children in the wombMake woman pregnant and form the familySlide: WNB
![Page 27: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/27.jpg)
• When men learned that malnutrition effects mental capacity not just growth, they became much more interested in improving it
• The support of men and mothers-in-law has been critical in increasing the frequency and quality of feedings
![Page 28: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/28.jpg)
CCRP DE Challenge #3: Going to scale in complex environments
Innovation??
??
![Page 29: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/29.jpg)
Replication “Scale”
![Page 30: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/30.jpg)
Scale: Farmers need to make and apply compost to their fields to increase fertility
![Page 31: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/31.jpg)
1. Make a bin: wire cylinder that is 3-4 feet in diameter
2. Add 6 inch layer of brown organic material to the bottom
3. Add 2-3 inch layer of green organic matter
4. Repeat layers until pile is 4-5 feet high
5. Mix the layers in 2 days
**From “Compost for dummies”
![Page 32: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/32.jpg)
UniversalistJim Hancock, FAO
Evaluation Tasks
Fidelity
Outcomes/Impact
![Page 33: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/33.jpg)
In complex systems …
(i.e. the “real world”)
this usually looks a bit different ...
![Page 34: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/34.jpg)
The real world . . .
![Page 35: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/35.jpg)
Scaling models
Universalist Contextualist
Jim H
anco
ck, F
AO
![Page 36: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/36.jpg)
Compost in the real world?
It depends…
![Page 37: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/37.jpg)
What does this look like on the ground?
UNIVERSALIST
CONTEXTUALIST
Case of Soil Fertility (complex system)
Policies and practices at scale
Innovation Networks• Inspiration• Adaptation• Influence
Options = success at small scale• Compost• Green Manure• Legumes• Fallows• Do nothing• Chemical
fertilizer
Farmer typologies
+ =
![Page 38: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/38.jpg)
Jim H
anco
ck, F
AO
Evaluation Tasks
Outcomes/Impact
Understanding Context Contextualist
![Page 39: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/39.jpg)
Jim H
anco
ck, F
AO
Examples of Evaluation Questions
Contextualist
Factors for success or failure?
Understanding drivers: what starts something, and helps it grow?
Constraints, bottlenecks?
Context, or ‘spaces’: understanding the enabling environment
External triggers? How can these be harnessed?
![Page 40: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/40.jpg)
Systems Thinking
Design Thinking
The deep learning that comes after the innovation is out in the community
Para
digm
Shi
ft
LEARNING
![Page 41: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/41.jpg)
Now What?
![Page 42: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/42.jpg)
What design can teach the development/research world• If people don’t use it, it’s not their fault, it’s our fault
• Participation: be creative about getting into people’s hearts and minds understanding what they need, want and will use—iterative
• Not just following a protocol—you have to get creative, observe, set up tests—you need good human capital
• Other?
![Page 43: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/43.jpg)
What the evaluation world can teach design
• There is use and then there is dis-use. Where are the incentives to track if it is useful over time, for who and why?
• To what depth are designers really understanding the systems, and using a contextual approach to scale?
• Other?
![Page 44: RSD4 1C Moore - Evaluation at the frontiers of systemic design](https://reader033.fdocuments.net/reader033/viewer/2022051503/588839021a28ab7a298b527b/html5/thumbnails/44.jpg)
Marah Moore, [email protected]