blog: jabe.co

FAILINGprinciples and practices of

Vanity Validation a Paradox of Passionate Commitment

WELL

@cyetain

What was the last thing you failed at?

What did you learn?

Did you share your failure with anyone?

Pre-Talk Questions

blog: jabe.co

FAILINGprinciples and practices of

Vanity Validation a Paradox of Passionate Commitment

WELL

HELLLOOOO McFLYTLC LABS

blog http://jabe.co

Send Anonymous Feedback

http://sayat.me/jabebloom

Joshua (Jabe) BloomCTO : The Library Corporation

& Consulting Practioner TLC Labs

#agile2013

@cyetain

Fail Fast

@cyetain

@cyetain

Learning occurs when we detect and correct error. Error is

any mismatch between what we intend an action to produce

and what actually happens when we implement that action.

-Chris Argyris

@cyetain

How Do We Make Better Choices?

Why is it so hard to Fail?

Could we design a system to help?

@cyetain

One must treat his theory-in-use as both a psychological certainty and an intellectual

hypothesis.-Chris Argyris

@cyetain

How wonderful that we have met with a paradox. Now we have some hope of making

progress.-Niels Bohr

@cyetain

@cyetain

fake dictionary page from colbertnation.com

@cyetain

Although theory without experiment is empty,

experiment without theory is blind.

-Paul Thagard

@cyetain

3 Things To Leave With

• Failing Well Produces more Information than Failing Poorly

• “Passionate Beliefs, Loosely Held”

• Reducing Variability too, Soon risks suboptimal result, too Late increases Failure blindness

@cyetain

We simply cannot rely on randomness to correct

the problems that randomness creates.

-Don Reinertsen

@cyetain

undifferentiated streams of data

@cyetain

@cyetain

“Research is what I’m doing when I don’t know what

I’m doing.”-Wernher von Braun

@cyetain

@cyetain

The Principle of Optimum Failure Rate

0% 100%

Probability of Failure

50%

Pote

ntia

l Inf

orm

atio

n

@cyetain

0% 100%

Probability of Failure

50%

Pote

ntia

l Inf

orm

atio

n

Greater Asserted Information

Greater Asserted Information

@cyetain

0% 100%

Probability of Failure

50%

Pote

ntia

l Inf

orm

atio

n

Greater Asserted Information

Greater Asserted Information

Pretty Sure

theory is wrong

Pretty Sure

theory is right

Interesting Ideas

@cyetain

0% 100%

Probability of Failure

50%

Pote

ntia

l Inf

orm

atio

n

Greater Asserted Information

Greater Asserted Information

Pretty Sure

theory is wrong

Pretty Sure

theory is right

Interesting Ideas

Uncomfortable

Confident

@cyetain

Experience of Failure

Num

ber

of S

ampl

esThe Competency Trap

@cyetain

0% 100%

Probability of Failure

50%

Pote

ntia

l Inf

orm

atio

n

@cyetain

0% 100%

Probability of Failure

50%

Pote

ntia

l Inf

orm

atio

n

Pretty Sure

theory is wrong

Pretty Sure

theory is right

Interesting IdeasHidden

RiskHidden Value

The Line of SURPRISE!

@cyetain

0% 100%

Probability of Failure

50%

Pote

ntia

l Inf

orm

atio

n

Pretty Sure

theory is wrong

Pretty Sure

theory is right

Interesting IdeasHidden

RiskHidden Value

1

2

3

During Customer Development Focus on Interesting Ideas

Before Scaling Validate Your "We Know This Assumptions" to reduce risk of Failure Demand

After Customer Validation Run experiments to Validate Assumptions of Failure

2 3

1

@cyetain

“The typical sequence of coin tosses has high information content but little value; an

ephemeris, giving the positions of the moon and

planets every day for a hundred years, has no more

information than the equations of motion and

initial conditions from which it was calculated, but saves

it’s owner the effort of recalculating these positions.”

-Charles H. Bennett

@cyetain

Based on what we know right now, what problems

do we have the least amount of information

about that we can reasonably expect to

understand?

@cyetain

@cyetain

Risk vs Uncertainty

@cyetain

Alteaory vs Epistemic

Uncertainties

@cyetain

Gamble Invest

@cyetain

Justified MVP

Value of Information

Cost of Acquisition

Cost of MVP

UnjustifiedMVP

Over JustifiedMVP

JustifiedMVP

@cyetain

The first principle is that you must not fool

yourself--and you are the easiest person to fool.

-Richard Feynman

@cyetain

“Most people don’t know how to learn. What’s more, those

members of the organization that many assume to be the best at learning are, in fact, not very

good at it. I am talking about the well-educated, high-powered, high

commitment professionals” -Chris Argyris

@cyetain

“Expertise … breeds an inability to accept new

views.” -Laski

@cyetain

@cyetain

Vanity Validation

@cyetain

I need to be right even if I'm wrong.

@cyetain

Defensive Reasoning

@cyetain

Remain in unilateral control

@cyetain

Maximize "winning"

Minimize "losing"

@cyetain

Suppress negative feelings

@cyetain

Be as "rational" as possible -- by which people mean

defining clear objectives and evaluating their behavior in

terms of whether or not they have achieved them

@cyetain

Mindset Actions

Results Match

Results Mismatch

Single-loop

Double-loop

@cyetain

Valid Public Information

@cyetain

@cyetain

whenever we propose a solution to a problem, we ought to try as hard as we

can to overthrow our solution, rather than defend it.

-Karl Popper

@cyetain

• Identify Your Assumptions and Conclusions CLEARLY AS POSSIBLE PUBLICLY

• Question Your Assumptions and Conclusions

• Seek Contrary Data

• Learn when to correct your Actions and when to correct your Mindset

@cyetain

http://xkcd.com/

@cyetain

@cyetain

Abductionnot just for

Aliens

@cyetain

@cyetain

[Abduction] goes upon the hope that there is sufficient affinity between the reasoner's mind and nature's to render

guessing not altogether hopeless, provided each guess is checked by comparison with observation... The effort should therefore be to make

each hypothesis... as near an even bet as possible.

-Charles Peirce

@cyetain

ABDUCE

DEDUCE

INDUCE

Predictive

Probable

Plausible

The Way

Computers "Think"

The Way

Humans Think

Binary

Probability

Analogue

Justifiable

@cyetain

ABDUCE

DEDUCE

INDUCE

Experiences Hypothesises

ExpectedOutcomes

If Coherent

If Expected Outcomes Match Reality

EffectiveMatch

@cyetain

ABDUCE

DEDUCEINDUCE

@cyetain

ABDUCE

DEDUCEINDUCE

SURPRISE!!!

@cyetain

Multi-Hypothesis Research

!=

@cyetain

BRAINSTORM

@cyetain

TheoriesOpinions

Hypothesizes

The Facts and Just the Facts

@cyetain

TheoriesOpinions

Hypothesizes

ConstraintsCriteria

@cyetain

TheoriesOpinions

Hypothesizes

QuestionFacts

@cyetain

TheoriesOpinions

Hypothesizes Request More Information

@cyetain

NO TALKING!

@cyetain

How Would I Validate my

understanding of this

problem?How Would I

solve this

Problem?

•Based on your experiences, what would you do to solve this problem? This is your Hypothesis.•Identify What Needs to Be True if your Hypothesis is true.•Assert, Presume, Assume Truth•Imagine Experiments that would justify the Assumptions

@cyetain

I Assert that this I know this

0% 100%

Probability of Failure

50%

Pote

ntia

l Inf

orm

atio

n

Pretty Sure

theory is wrong

Pretty Sure

theory is right

Interesting Ideas

I Presume Somebody knows this

I am going to Assume

this is true for my

Hypothesis to be true

0% 100%

Probability of Failure

50%

Pote

ntia

l Inf

orm

atio

n

Pretty Sure

theory is wrong

Pretty Sure

theory is right

Interesting Ideas

0% 100%

Probability of Failure

50%

Pote

ntia

l Inf

orm

atio

n

Pretty Sure

theory is wrong

Pretty Sure

theory is right

Interesting Ideas

@cyetain

This is my Hypothesis,

Assumptions and

Experiments

ChallengeAssumptions & Experiments

Rotate Pairs 2-3 TimesAllow Time for Revision Between Rounds

@cyetain

This is my Hypothesis,

Assumptions and

Experiments

@cyetain

Multiple Smaller Experimentsagainst

Multiple Abductive Hypotheses

instead of

Single Large Experimentagainst

Single Hypotheses

@cyetain

Failing Well Produces more Information than Failing

Poorly

@cyetain

What are You Doing w All that Information?

Incremental: Confirm. Disconfirm.

Iteratively: Select Next Step. Generate More Options

@cyetain

Having “Passionate Beliefs, Loosely Held”

FAILURE

MUST

BE AN OPTION

@cyetain

Reducing Variability too soon risks suboptimal

result, too late increases failure demand

@cyetain

@cyetain

Influences &

Sources of More Information

@cyetain

Joshua (Jabe) BloomCTO : The Library Corporation

& TLC Labs

blog http://jabe.coSend Anonymous Feedback

http://sayat.me/jabebloom