Using Developmental Evaluation to Support Prototyping:A Workshop

Leading Boldly Network:Calgary’s Network for Collaborative

Social Innovation

Workshop Agenda

• Getting Grounded: – The Collaborative Social Innovation Process– Your own experience with or questions about

prototyping

• What is Prototyping & How Is It Unique?

• What is Developmental Evaluation (DE)? – Using DE to Support Prototyping

Collaborative Social Innovation is…… the process of developing social innovations

that are both social in their ends and collaborative in their means. Specifically, collaborative social innovations include new ideas (services, models and resource flows) that simultaneously address complex social issues and create new social relationships or collaborations. In other words they are innovations that are both good for society and enhance society’s capacity to act.

Adapted from the Young Foundation

The LBN Process of Collaborative Social Innovation*

Secure resources

1. Co-Initiate – Set the right conditions and culture

2. Co-Sensing – Frame the issue or design question

Secure resources

3. Co-Presence – Generate ideas

4. Prepare for Prototyping – Planning

5. Co-create / Prototype – Experiment to test ideas

If prototype results prove to be promising, secure resources

6. Co-Evolve - Implement

7. Scale Out – Share innovations with a broader set of stakeholders

8. Scale Up – Systems Change

* This is a working model

What is Prototyping?

Prototype literally means first impression (proto = first, typos = impression)

• A series of small-scale experiments, in simulated or real environments, that test a ‘first impression’ or ‘hunch’ regarding how to positively influence change within a defined “problem space”

What is Prototyping? • A core activity within the

collaborative social innovation process

• Action– and learning–oriented

• Undertaken with the purpose of designing, testing and rapidly iterating new and unverified processes, services or programs in order to discover how they truly influence the complex problem you are trying to address

• A practice for situations that are non-routine and unpredictable, and therefore different than what is required for ongoing operations

• Best done collaboratively with diverse stakeholders

What is Prototyping & How is it Unique?

4 Prototyping Principles: 1. It is used at the beginning, or developmental

phase, of a new process, service or program, where the way to the desired outcome is unknown

2. It anticipates failure. Indeed a key objective of prototyping is to create the conditions to be able to ‘fail safe’, namely endure failure that is survivable (hence small scale) and provides learning to inform the next iteration.

What is Prototyping & How is it Unique?

4 Prototyping Principles3. It embraces rapid iteration informed by

feedback from the system (stakeholders, end-users, etc.) it is trying to change

4. It includes reflection-in-action, the intentional recording and documenting of what is being learned as projects are implemented

What is Prototyping & How is it Unique?

The focus is on learning:• Using evidence to move from “wild

guesses” to “reliable forecasts”–The focus is not on results, but on

the overall trajectory of the initiative

What is Prototyping & How is it Unique?

Moving from Wild Guesses to Reliable Forecasts

Wild guesses

Informed estimates

Reliable forecasts

Prediction

Learning

Time

How is Prototyping a Unique Practice?

This practice enables us to:• “Think with our hands” as a way to investigate the

merit of our proposed solution;• collect user feedback; and • observe how our actions influence (for better or

worse) change within the domain we are seeking to transform.

“The key to dealing with complexity is to focus on having good conversations about assumptions.”

Why Is Prototyping an Important Practice?

• In a time of tight budgets, prototyping can be used to test an unverified process, service or program before considerable resources are invested into a large-scale initiative. – It helps to expose flaws and limitations in an

approach before significant investment has been made

• Can help leaders/Boards manage the risks involved with trying something different or new

• Can help us develop better products and services

How to Prototype

1. Know when a prototype project is the right course of action

– Developmental phase / uncertain outcomes

2. Develop a plan – Write down what you plan

to do, what you expect to happen and why (Hypothesis of Record)

– “In an innovative initiative the plan is a hypothesis”

3. Develop a clear learning agenda (Developmental Evaluation)

– Compare results to explicit set of assumptions

4. Execute the Plan – Analyze differences

between what you thought would happen and what actually happened

5. Based on what you learned, revise the plan

How to Prototype

Small Group Conversation

What stood out for you?

What is becoming clearer?

What questions are you carrying?

Developmental evaluation is a small but demanding niche

in the world of evaluation.

Five Niches

DE: 7 Things You Need to Know

1. The Essence of DE

• Developmental evaluation isn’t some particular methods of recipe-like steps to follow. It doesn’t offer a template of standard questions. It’s a mindset of inquiry into how to bring data to bear on what’s unfolding so as to guide and develop the unfolding. What that means and the timing of the inquiry will depend on the situation, context, people involved, and the fundamental principle of doing what makes sense for program development (Patton 2010: pp.75-6).

Development is about creative

thinking.

Evaluationis about criticalthinking.

Developmental Evaluationholds theseIn balance.

The Essence of DE

2. Purpose

• Developmental evaluation focuses on developmental questions: What’s being developed? How is what’s being developed and what’s emerging to be judged? Given what’s been developed so far and what has emerged, what’s next?

TRADITIONAL EVALUATIONS DEVELOPMENTAL EVALUATIONS

Render definitive judgments of success or failure.

Provide feedback, generate learnings, support direction or affirm changes in direction.

Measure success against pre-determined goals. Develop new measures and monitoring mechanisms as goals emerge and evolve.

Position the evaluator outside to assure independence and objectivity.

Position evaluation as an internal, team function integrated into action and ongoing interpretive processes.

Design the evaluation based on linear cause-effect logic models.

Design the evaluation to capture system dynamics, interdependencies, and emergent interconnections.

Aim to produce generalizable findings across time and space.

Aim to produce context-specific understandings that inform ongoing innovation.

Accountability focused on and directed to eternal authorities and funders.

Accountability centered on the innovators’ deep sense of fundamental values and commitments.

Accountability to control and locate blame for failures.

Learning to respond to lack of control and stay in touch with what’s unfolding and thereby respond to strategically.

Evaluator controls the evaluation and determines the design based on the evaluator’s perspective on what is important.

Evaluator collaborates in the change effort to design a process that matches philosophically and organizationally.

Evaluation engenders fear of failure.

Evaluation supports hunger for learning.

3.1 The Evaluator

• The developmental evaluator inquires into developments, tracks developments, facilitates interpretation of developments and their significance, and engages with innovators, change agents, program staff, participants in the process, and funders around making judgments about what is being developed, how it is being developed, the consequences and impacts of what has been developed, and the next stages of development (Patton 2010, p. 227).

3.2 Evaluator Roles

3.2.1 Surfacing Issues

Raise critical questions, tensions, debates that emerge as part of the developmental process.

3.2.2 Framing Concepts

Flush out and flesh out problem definitions, lenses, frameworks and ‘working models’ underlying the conception and design of the intervention.

3.2.3 Testing Quick Iterations

Surface –and interpret, draw conclusions and judge -- real time data through techniques such as rapid reconnaissance, secondary evidence, modeling, outcome mapping, simulations, etc.

3.2.4 Tracking Developments

Documenting concrete developments – e.g. learning, new models, design features, forks in the road – in the evolution of a group’s work.

3.3 The Evaluator-User Relationship

Traditional Evaluation

• Positioned outside the action to assure independence and objectivity.

Developmental Evaluation

• Positioned as team member integrated into the action and ongoing process of gathering and interpreting data, framing issues, surfacing and testing model developments. May be situated externally or internally.

3.4 Competencies

Traditional Evaluator Competencies

Reflective Practice. Technical Practice Situational Practice. Management Practice. Interpersonal Practice.

Extra Competencies for DE

Comfort with adaptive processes

High tolerance for ambiguity Can react/adapt quickly Effective communication with

hyperactive, short-attention-span, action-oriented innovators

Larger tool kit of evaluative methods

4. Methods

• [I]t’s worth emphasizing that no definitive list of developmental evaluation inquiry approaches can or should be constructed. Developmental evaluation creatively adapts whatever approaches and methods fit the complexities of the situation and are responsive, appropriate, and credible to social innovators in opening up new understandings and guiding further development.

• In being creative, the developmental evaluator is also practical and pragmatic, doing the best job possible within available resources and other constraints. Constraints always exist and do what constraints do -- constrain. Our ability to think of alternatives is limited. Resources are always limited. Time is of the essence. We do what we can. Part of what we can do is adapt other inquiry traditions to the purposes of developmental evaluation.

Methods continued

Key Features

• Contingency

• Real Time

• Flexible

Adaptive/Evolutionary

• Evaluation methods need to be continually adapted to reflect the changes in the emerging innovation and the learnings and questions of the innovators.

5. Steps

6. Reports

• Dynamic complexities don’t slow down or wait for evaluators to write their reports, get them carefully edited, and then approved by higher authorities. Any method can be used but will have to be adapted to the necessities of speed, real-time reporting and just-in-time, in-the-moment decision-making.

• That is a major reason the developmental evaluator is part of the innovation team, to be present in real time as issues arise and decisions have to be made […] Contrary to the usual practice in evaluation of fixed designs that are implemented as planned, developmental evaluation designs can change as the innovation unfolds and changes (Patton 2010, p. 335-6).

Emerging Practices on Reports

Ongoing Real Time Feedback• A variety of formats that

capture key evaluation questions, results and learnings, and new developments (e.g. memos, PPT, visual diagrams, etc.) in a way that is fits the pattern of information flow and decision-makers of the evaluation users?

Periodic “Developmental Accounts”

• A record of the major developments that have emerged in a developmental process so far (interim accounts) and/or at the end of significant developmental period (e.g. a shift into formative evaluation).

The Tracking the EvolutionOf the Intervention

7. Accountability

To Whom

• People responsible for developing and/or adapting an intervention.

For What

• Developing a better understanding of the issue being addressed, leverage points for change, and developing an intervention.

• This may – or may not – develop into a workable, stable or ultimately successful intervention.

For How

• A rigorous, data-based, user-friendly and responsive, process of reality testing that balances creative and critical thinking.

• Guided by standard evaluation principles and ethics.

Tamarack Interview with Michael Quinn Patton on DEhttp://tamarackcommunity.ca/g3s61_CC4I5.html

Utilization-FocusedEvaluation: 4th Ed.Michael Quinn Patton

The DE Primer. Jamie Gamble.http://www.imprintinc.ca/

Getting to Maybe. Frances Westley Brenda ZimmermanMichael Quinn Patton

Resources

DE 201: Practitioners Guide Dozois, Langlois, Blanchet-CohenJW McConnell Site

Developmental Evaluation Michael Quinn Patton

Extras for Prototyping

types of prototyping

•exploratory prototyping – testing the demand, feasibility & viability of an idea, quickly and inexpensively.

•developmental prototyping – testing the demand, feasibility & viability of an idea more detail, to see if its works in practice.

•slow prototyping – a concurrent focus on developing an idea or practice AND testing the capacity of the group to carry out the idea.

•fast prototyping – an exclusive focus on testing an idea or practice.

fast prototyping

slow prototyping

exploratory prototyping

developmental prototyping

testing a collaborative fundingmodel (simulation)

exploring a new mission and design for employment program

(scenarios)

testing the efficacy of different campaign messages before roll out

(random controlled trials)

testing the buy-in, viabilityfeasibility of a fundingstrategy (role playing)

developing & adapting practices for inter-agency collaboration

(after action review)

Questions

• What questions – if any – emerge for you about DE?

• What is alive for you about DE might contribute to your work?

Key Resources

• Nesta (UK)• The Other Side of Innovation: Solving the

Execution Challenge • The Open Book of Social Innovation• The Austin Centre for Design• IDEO • Human Centred Design