MITSMR Making Better Decisions Collection

SPECIAL COLLECTIONWINTER 2015

sloanreview.mit.edu

Learn how asking the right questions can help you make smarter decisions, how to make decisions with data using simula-tions and how new strategies can improve your decision making.

Making BetterDecisions

A SPECIAL COLLECTION OF MIT SMR ARTICLES

Brought to you by:Brought to you by:

SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW i

CONTENTSSPECIALCOLLECTIONWINTER 2015

Making Better Decisions

1 The Art of Asking Pivotal Questions By Paul J.H. Schoemaker and Steven Krupp

`10 Using Simulated Experience to Make Sense of Big Data By Robin M. Hogarth and Emre Soyer

16 Why You Decide the Way You Do By Bruce Posner

Please note that gray areas reflect artwork that has been intentionally removed. The substantive content of the article appears as originally published.

PLEASE NOTE THAT GRAY AREAS REFLECT ARTWORK THAT HAS BEEN INTENTIONALLY REMOVED. THE SUBSTANTIVE CONTENT OF THE ARTICLE APPEARS AS ORIGINALLY PUBLISHED.

WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 39

The Power of Asking Pivotal QuestionsIn a rapidly changing business landscape, executives need the ability to quickly spot both new opportunities and hidden risks. Asking the right questions can help you broaden your perspective and make smarter decisions.BY PAUL J.H. SCHOEMAKER AND STEVEN KRUPP

GOOD STRATEGIC THINKING and decision making often require a shift in perspective particularly in environments charac-

terized by significant uncertainty and change. What worked in the

past simply may not apply in the future. Asking what if questions

about the future may create discomfort, since answers are often not

obvious. But asking such questions also forces you to step back and

challenge current assumptions that prevent you from seeing break-

through solutions. This article builds on our new book, Winning the

Long Game: How Strategic Leaders Shape the Future,1 by focusing on

the art of asking pivotal questions to improve strategic decision

making. (See About the Research, p. 41.) By presenting six ques-

tions that challenge executives to incorporate broader perspectives,

our aim is to stimulate out-of-the-box dialogues that help leaders

make better choices and find innovative solutions sooner.

Are You Solving The Right Problem?Back in the 1960s, IBM Corp. had the opportunity to buy or license

Xerox Corp.s new reprographic photo process. IBM hired the con-

sulting firm Arthur D. Little to answer a key question: If a more

reliable, cheaper and faster process for photocopying were available,

how many more copies would people make in a given year? Since

copies in those days could only be made from an original specimen,

ADL set out to estimate the number. Both companies framed the

problem too narrowly as copies from originals, ignoring a new

segment of the market that turned out to be many times larger

D E C I S I O N M A K I N G : T H E R I G H T Q U E S T I O N S

THE LEADING QUESTIONHow can managers make better decisions?

FINDINGSExamine both broad market trends and less visible undercurrents.

Seek out diverse views to see multiple sides of complex issues.

Push back when consensus forms too quickly.

SPECIAL COLLECTION MAKING BETTER DECISIONS MIT SLOAN MANAGEMENT REVIEW 1

40 MIT SLOAN MANAGEMENT REVIEW WINTER 2015 SLOANREVIEW.MIT.EDU


(namely, copies of copies of copies).2 This huge,

overlooked opportunity could only have been fore-

seen if different questions had been posed. IBM

might have owned this new revolutionary technol-

ogy if the key question had been framed as how

might the new Xerox process change when and how

people make copies, and what might this grow to in

total number of copies made in future years?

The questions leaders pose sometimes get in the

way of solving the right problem or seeing more

innovative solutions. They are often too narrow,

overly protective of the current business, or assume

that the old habits, business models and regula-

tions will remain largely intact. At Google Inc.,

CEO Larry Page challenges leaders to anticipate the

future better by not just asking what is or likely will

be true, but what might be true, even if unex-

pected.3 The matter of what is the right question

should be much more central when leaders tackle

complex and important decisions, especially in an

era of profound change.

Think Outside InQuestion One: How well do you understand the

implications of broad market trends and less vis-

ible undercurrents for your business and for

upcoming strategic choices? Entrepreneurs like

Elon Musk from Tesla Motors, Steve Jobs from

Apple and Jeff Bezos from Amazon became known

for spotting unmet market needs and figuring out

how to serve them profitably. The best entrepre-

neurs excel at peeking around the corner and seeing

the future sooner.4 Weve found that leaders can

learn to anticipate better by simply being more

curious, looking for superior information, con-

ducting smarter analyses and developing broader

touch points with those in the know.

In an interview on CNN, Musk was asked where

his forward-thinking, innovative ideas come from.

He replied, Just trying really hard the first order

of business is to try. You must try until your brain

hurts.5 Ever since he was in college in the early

1990s, Musk had a vision of commercializing elec-

tric vehicles for the mass market and was

questioning how this could be achieved, given the

historic pushback against this idea. He mused that

getting into the electric car business was probably

one of the stupidest things you could do.6 (Even

Toyota Motor Corp. chairman Takeshi Uchiya-

mada, known as the father of the Prius, had

reservations about electric cars: Because of its

shortcomings driving range, cost and recharg-

ing time the electric vehicle is not a viable

replacement for most conventional cars.7) Musk

saw electric vehicles as the future, but if their devel-

opment was left to traditional car companies, he

thought it would take a long time.

In Musks view, the industry was operating

under two false premises: One, that you could not

create a compelling electric car; and two, that no

one would buy it.8 The challenge was to demon-

strate that electric cars can be a mainstream

product and to reassure consumers that infrastruc-

ture can be developed to give them the freedom and

reliability of a regular car.9 Well before others,

Musk saw the possibilities and asked different ques-

tions. Although this story is far from over, Musks

vision has struck a chord with consumers and Wall

Street. He expanded his enterprise to include global

distribution and battery manufacturing shortly

after the Tesla Model S was rated the number one

car ever tested by Consumer Reports in 2013.

The Challenge Strategic leaders are focused on the future and are

masters at asking discerning questions and exploring

ideas and options that are outside the mainstream.

They are wary of status quo views and prefer honest,

transparent questions that focus on how much, or

how little, is really known about the issue at hand.

Many studies emphasize the importance of strategic

thinking and anticipation, while also lamenting the

shortage of leaders who do this well.10 Those who

miss the early signals often come late to the party

when customer tastes are changing or when nontra-

ditional competitors are preparing to disrupt or

blindside them.

To protect themselves, companies must keep an

eye on innovations from both existing companies

and startups. Some of the ideas could become game

changers, and you may have to team up with the in-

novators, as a number of big pharmaceutical

companies have done with biotech companies.

Tips and Pointers 1. Learn from startups. What are they doing and


SLOANREVIEW.MIT.EDU WINTER 2015 MIT SLOAN MANAGEMENT REVIEW 41

why? What do they see that you dont? Examine

their moves to detect market shifts and emerging

opportunities from the outside in.

2. Go to conferences outside your function or in-

dustry. In its Connect + Develop innovation

program, Procter & Gamble Co. reaches out to

companies outside the consumer products indus-

try to share lessons and explore joint challenges.11

Follow events in other regions and sectors, even if

they seem unrelated to your business at first.

3. Leverage current networks and join new ones.

How might you engage your existing networks

more systematically to stay on top of new develop-

ments? Join interest groups in adjacent businesses

or areas to expand your worldview and examine

questions you dont typically consider.

Explore Future ScenariosQuestion Two: How thoroughly have you ana-

lyzed major external uncertainties and future

scenarios that could significantly impact your

business decisions? Leaders must not only under-

stand the deeper trends but also the key uncertainties

that can rock their world. One way to do this is

through scenario planning and war gaming.12 For

example, a pediatric hospital in the U.S. Midwest

was grappling with rapid consolidation in its mar-

ket. Larger hospitals focused mostly on adult

patients and were actively looking to merge or to

form strategic alliances. In anticipation, the CEO of

the pediatric hospital engaged his board in a simu-

lation, presenting them with a hypothetical

scenario: a merger between two particular adult-

patient hospitals. He asked board members to

identify potential alliance partners, decide on an ac-

tion relative to competitors and assess their

hospitals readiness to execute the plan.

Then, the CEO introduced a second scenario: a

disruptive technology coupled with onerous new leg-

islation. The exercise spurred new questions and

helped the CEO crystallize a plan. The CEO deter-

mined that, if certain adult hospitals merged, the

competing pediatric hospital would likely want to

merge as well. Shortly thereafter, when two adult hos-

pitals in the region announced a major consolidation,

the CEO and his board were prepared to act. They

proposed a partnering arrangement to the other pedi-

atric hospital and were able to stay ahead of the curve.

The Challenge Developing different views of how the external en-

vironment may change allows leaders to better

determine whether the organization has sufficient

strategic flexibility to succeed. Scenarios can pick up

early indicators about how emerging technologies

or social trends might disrupt your current business

model, how customers preferences may change or

why new regulations could alter your industry.13

Asking what could happen in the future involves

imagination and curiosity. It pays, for example, to

ponder how and where a well-armed rival could at-

tack your business.

Even though good tools exist to raise important

questions about future uncertainties, time-pressured

executives occupied with putting out fires or

exploiting short-term gains arent always receptive

to them. For example, for several years leading up

to the U.S. subprime mortgage crisis that began in

2007, the investment community overlooked or

largely ignored the possibility that the subprime

mortgage boom might go bust. In a congressional

hearing in the fall of 2008, Standard & Poors presi-

dent Deven Sharma claimed, Virtually no one

be they homeowners, financial institutions, rating

agencies, regulators, or investors anticipated

what is occurring.14 Yet leading economists, in-

cluding Paul Krugman and Robert Shiller, and

savvy investors, such as Steve Eisman and John

Paulson, had been sounding the alarm.15 The in-

triguing question is not why top executives at large

rating agencies failed to acknowledge the elephant

in the room but why some investors and analysts

spotted the elephant sooner than others.

ABOUT THE RESEARCHThis article draws on multiple sources, including our ongoing research at the Wharton Schools Mack Institute for Innovation Management and our consulting work at Decision Strategies International. The discussion of the six questions we examine here draws on our book Winning the Long Game: How Strategic Lead-ers Shape the Future (PublicAffairs, 2014). We pretested the conceptual model underlying the book with many executives and then collected data from more than 30,000 managers representing diverse companies, functions and back-grounds around the world. Using factor analysis and other standard tests of validity, we refined the survey questions and identified remedies. The full-length version of our survey contains 39 items and has been used for self-assessment, peer-assessment and 360-degree feedback (sample size of 278). A shorter ver-sion of the self-assessment was published at Inc.com in March 2012, with a link enabling readers to complete a 12-item assessment online at www.decisionstrat .com (sample size of more than 30,000 at present). These two data sets were used to test the statistical reliability and validity of the instrument.




Tips and Pointers1. Identify weak signals at the boundaries of your

business. Strategic leaders ask questions about

the external business environment that have far-

reaching implications and then ask team

members to scout the periphery for emerging

trends.

2. Conduct war games to assess the perspectives of

competitors and stakeholders. Gauge their likely

reactions to novel opportunities or threats.

3. Analyze rivals, especially nontraditional ones,

and examine which of their moves puzzle

you and why.

Be a ContrarianQuestion Three: Do you regularly seek out diverse

views to see multiple sides of complex issues, and

do you purposely explore important problems

from several angles? A persistent problem for many

teams is promoting diverse thinking and creative

friction. Leaders must always ask if the team has

sought sufficient contrarian input and been ex-

posed to all sides of an issue before reaching a

decision. This can counter the tendency of many

team members to go along to get along. Offering

contrarian views is particularly essential when tack-

ling major strategic decisions in an uncertain

environment.

To promote diverse thought, Hala Moddelmog,

former president of Atlanta, Georgia-based Arbys

Restaurant Group Inc., a fast-food chain with

about 3,400 locations, surrounded herself with

colleagues of different races, geographies, socioeco-

nomic classes and personality styles. You really

dont need another you, she said. Staying open to

different viewpoints helps ensure leaders are not

unduly hindered by decision traps and can instead

open their eyes to information or solutions that

they may not have previously considered.16

Research shows that creative tension promotes

better idea generation and group problem solv-

ing.17 Constructive dissent and debate encourages

people to reexamine current assumptions to make

room for creative thinking. John Lasseter, chief cre-

ative officer at Pixar, Walt Disney Animation

Studios and DisneyToon Studios, has practiced a

powerful form of team challenge. Each morning at

Pixar, the team working on a movie would review

their previous days output and explore how to im-

prove. They were asked to provide tough questions,

offer honest critique and put alternatives on the

table.18 This practice was based on the belief that

team decisions were superior to any individuals,

but only if you pushed people out of their comfort

zones. Some team members had to get used to

being challenged and critiqued, but most came to

see how the product and decision improved.

Author Malcolm Gladwell has noted that the

best entrepreneurs and innovators are usually quite

disagreeable they love debate. He has gone so far

as to argue recently that an important role of senior

management in creating an atmosphere of inno-

vation is allowing people to be disagreeable.19 This

echoes an idea philosopher John Dewey presented

in 1922: Conflict is the gadfly of thought. It stirs us

to observation and memory. It instigates to inven-

tion. It shocks us out of sheep-like passivity, and

sets us at noting and contriving. Conflict is a

sine qua non of reflection and ingenuity.20

The Challenge The opposite of using questions to promote diver-

gent thinking is to coalesce around shared viewpoints

or succumb to groupthink. Amazons Jeff Bezos

decries social cohesion as the cloying tendency of

people who like to agree with each other and find

consensus comfortable.21 In response, he says he

tries to create a culture at Amazon where leaders

challenge decisions they disagree with, even when

doing so is uncomfortable or exhausting.

Bezos isnt the first business leader to value dis-

sent. As chairman of General Motors Corp., Alfred

P. Sloan Jr. told senior executives at the end of a

board meeting, I take it we are all in complete

agreement on the decision here. Then I propose

we postpone further discussion of this matter until

our next meeting to give ourselves time to develop

disagreement and perhaps gain some understand-

ing of what the decision is all about.22 Of course,

how conflict is handled differs strongly by culture.

Finding the right balance between encouraging

people to express diverse views and not offending

others requires cultural sensitivity, especially in

multinational settings. The benefits of frank debate

can dissipate quickly if they trigger resentment or

backstabbing.



Tips and Pointers 1. Foster constructive debate in meetings. Help

leaders and team members to get used to a more

candid dialogue with creative friction about ideas.

2. Keep teams small. Amazon forms task forces of

just five to seven people, which makes it easier to

test ideas and guard against groupthink.

3. Push back when consensus forms too quickly.

Insist on alternatives. Like GMs Sloan, challenge

teams if they agree too fast on a complex issue,

and ask them to reflect more deeply and develop

constructive disagreement.

4. Use devils advocates. Before meetings, ask

someone to prepare the case against the prevail-

ing view, and rotate this role. Train people to

question the status quo and get them to appreci-

ate the benefits of such questioning.

Look for PatternsQuestion Four: Do you deploy multiple lenses to

connect dots from diverse sources and stakehold-

ers, and do you delve deep to see important

connections that others miss? As the then-CEO of

DuPont, Charles O. Holliday Jr. picked up several

weak signals in the fall of 2008 that helped him pre-

pare his company for the deep recession that

followed. While visiting a major Japanese customer,

Holliday learned that the CEO had instructed his

staff to conserve cash, an indication that the

company was seeing or expecting a decline in prof-

itability. That got Hollidays attention, both in

terms of the potential for weaker economic condi-

tions and specific fears about DuPonts own cash

position. Upon his return, Holliday sought to get a

fix on DuPonts financial resilience. The leadership

team found that the initial signs of weakness were

spreading to the broader economy and beginning

to affect DuPonts business across the board.23

But how big a problem would it be? Holliday

learned that reservations at the Hotel du Pont,

located near the companys Wilmington, Delaware,

headquarters, had dropped 30% in 10 days, which

was highly unusual for the end of the year. He also

discovered that many corporate lawyers were set-

tling disputes rather than exposing clients to the

financial uncertainty of a trial. And several U.S.

automakers, huge DuPont paint customers, were

scaling back on production schedules. Holliday

wanted to know why. The answer wasnt compli-

cated: Orders for new cars were dropping as the

number of U.S. mortgage foreclosures increased,

and the economy was going downhill.

The Challenge What was impressive about Holliday was his ability

to amplify discrete data points, connect them and

take decisive action. Combining seasoned intuition

with vigilant questions, Holliday figured out that his

company was about to hit a wall. To test his fears, he

engaged his team and asked for candid feedback. His

team put a plan in place so DuPont would be ready if

financial markets hit rock bottom.

Leaders are often limited by selective percep-

tion and seek information that confirms what

they wish to believe. Unlike Holliday, most dont

ask tough questions because they filter out weak

signals that dont fit their mental models. When

faced with complex issues and conflicting infor-

mation, it is easy to fool yourself: If you torture

the data hard enough, it will confess to almost

anything! At Eastman Kodak Co., for example,

leaders failed to ask the right questions soon

enough to fully understand and act effectively on

the signs that photography was rapidly moving to

digital. This misperception reflected middle man-

agements belief that digital technology was

inferior to film and top executives belief that the

demands of Kodaks shareholders mattered more

than those of its consumers and engineers.24 These

flawed assumptions allowed Kodak to continue

When people feel pressed for time, they become less flexible and much prefer certainty to ambiguity. Ambiguity aversion is typically heightened in crisis situations and can lead to cognitive myopia, a narrow focus that can be counterproductive.




deluding itself about the urgency for change for

much too long.

Tips and Pointers1. Look for competing explanations to challenge

your observations. Engage a wide range of stake-

holders, customers and strategic partners to

weigh in.

2. When stuck trying to recognize patterns or

interpret complex data, step away, get some

distance and then try again. Sleep on the data,

since the mind continues to process information

when resting. Each time DuPonts Holliday took

a break, and then reengaged, he got a deeper

understanding and asked better questions.

3. Use visual graphs or flowcharts to juxtapose

the larger picture with the individual puzzle

pieces. Pattern recognition is easier when all the

information is clearly laid out and presented in

different ways. Try to leverage the power of visual

thinking.25

Create New Options Question Five: Do you generate and evaluate

multiple options when making a strategic deci-

sion, and do you consider the risks of each,

including unintended consequences? It may seem

obvious that leaders should examine multiple

options before making a big decision. Yet in the

heat of the battle, few leaders actually engage in

creative options thinking. A common refrain is:

We dont have time, weve got to move. Research

shows that when people feel pressed for time, they

become less flexible and will much prefer certainty

to ambiguity. Ambiguity aversion is typically

heightened in crisis situations and can lead to

cognitive myopia, a narrow focus that can be coun-

terproductive.26 Weathering storms, real or

metaphorical, requires strategic leaders to counter

this ambiguity aversion. Asking good questions

about alternatives or unintended consequences,

even if done quickly in a crunch, will provide a

wider-angle lens to include the less obvious and

potentially more strategic course of action.

When a devastating storm hit the annual Sydney

to Hobart Yacht Race in Australia in 1998, nearly all

of the more than100 yachts that started the race

were either trying to outrun the storm or heading

directly for the shore. A notable exception was the

crew of AFR Midnight Rambler. They asked a criti-

cal question in the midst of the life-threatening

storm: Are there other options? Rather than get-

ting ahead of the storm or racing to shore, the

Midnight Rambler saw a third possibility: sailing

directly into the storm. Although it was a noncon-

ventional choice, the Midnight Rambler crew

concluded that it would be the safest and the fastest

option. They also believed they had the skill to exe-

cute this bold plan. The Midnight Rambler not

only survived traumatic moments; it won the race.

Many boats were capsized and destroyed, few

finished and six sailors tragically died.27 The Mid-

night Rambler had the smallest boat and fewest

resources. But its crew was the only one to ask a

crucial question in the face of the storm: Are there

creative options?

The Challenge Major disruptions, such as the appearance of new or

unexpected competitors, often lead to quick

action with little reflection akin to the fight-or-

flight response of animals. When we are under the

gun, we frequently cut corners. This makes us prone

to the traps of narrow focus and inside-out thinking

that limit choices. We rely too much on ourselves or

on an inner circle. This can blind us to possibilities

that reflect outside perspectives and potential conse-

quences for customers or external stakeholders.

In 2011, Netflix Inc., which had been very suc-

cessful with its DVD rental-by-mail model, added a

Leaders are often limited by selective perception and seek information that confirms what they wish to believe. Most dont ask tough questions because they filter out weak signals that dont fit their mental models.



second delivery system based on Web download-

ing. To be competitive, CEO Reed Hastings decided

to unbundle the streaming service from the tradi-

tional model and offer it at a lower price. However,

the combined fee for both subscriptions ended up

being 60% higher than the original service. This in-

furiated consumers. In the following year, Netflix

lost 800,000 customers, and its stock price fell 65%.

By not asking the right questions, Netflix failed to

fully explore options that might be more flexible

and user-friendly. Although Hastings quickly

owned up to the mistake and publicly apologized,

the episode caused a lot of grief for both customers

and the company.

Tips and Pointers1. Rather than presenting binary go/no-go deci-

sions, reframe a situation to always examine

several more options. Always ask, What else

might we do?

2. Use impromptu meetings when time is limited

to generate more options, including unconven-

tional choices. The Midnight Rambler crew did

this during a major crisis.

3. Review alternatives based on clear criteria and

rank options accordingly. Clearly define deci-

sion criteria, make them explicit, weigh them and

then score each option against the criteria to

identify the best choice. Be disciplined when it

comes to making tough trade-offs.

Learn From FailureQuestion Six: Do you encourage experiments

and failing fast as a source of innovation and

quick learning? David Ogilvy, the advertising ge-

nius, purposely ran ads that he and his team did

not believe would work as a way to test their own

theories about advertising.28 One of the experi-

ments they tried was the famous Hathaway shirt

advertisement featuring a man with an eye patch.

This version of the ad (there were 17 others) was

an impromptu experiment whose success took

Ogilvy by surprise.29 The ad, in fact, was a brilliant

success, ran for a long time and received several

industry prizes.

Biologist Max Delbrck, who received a Nobel

Prize in 1969, believed in the principle of limited

sloppiness. He advised his students to be sloppy

enough in their lab experiments to allow for the

unexpected, but not so sloppy that they could not

identify the reasons for their anomalous results.30

Case in point: the eccentric Scottish scientist, Sir

Alexander Fleming, who received a Nobel Prize in

1945. His peers considered him brilliant but some-

what sloppy. In 1928, after a long summer holiday,

Fleming returned to his lab and began gathering up

the contaminated petri dishes for a good scrub-

bing. Suddenly, he noticed something different

about one of them: There was a halo where a blue-

green mold appeared to have dissolved the bacteria.

Many biologists might have missed the small irreg-

ularities, but Fleming knew bacterial growths as an

artist knows the color spectrum; in fact, he had

occasionally shaped colonies of Staphylococcus

into portraits of his coworkers. His keen perception

and curiosity led to the first breakthrough in what

later became the wonder drug penicillin.31

Talking about and even celebrating failure has

become central to the folklore about entrepreneurs.

Sara Blakely, who founded Spanx Inc., a fast-grow-

ing maker of slimming undergarments and other

apparel based in Atlanta, Georgia, got the idea for

the companys products from her own dissatisfac-

tion with the undergarment options available on

the market. She credits her success in part to a ques-

tion her father used to pose at the dinner table

when she was growing up: What have you failed at

this week? The message she got was that failure

wasnt about making mistakes it was about not

trying new things.32

Learning from mistakes has much to do with a leaders mind-set and the questions that he or she asks both before and after an unexpected event occurs. Strategic decision makers abandon the pursuit of perfection, allow some room for well-intentioned mistakes, and examine what went wrong and why.




The Challenge Learning from mistakes has much to do with a

leaders mind-set and the questions that he or she

asks both before and after an unexpected event

occurs. Strategic decision makers abandon the

pursuit of perfection, allow some room for well-

intentioned mistakes, and examine what went

wrong and why. What matters is how well a team

learns from setbacks and what mode of inquiry it

allows. The best teams try to fail fast, often and

cheaply in search of innovation.33

Few leaders are willing to give more than lip

service to failure; most corporate cultures view

missteps as crippling rather than as sources of

innovation. Only in Silicon Valley, perhaps, do

people wear their failures with pride.34 The

blame culture that permeates most organiza-

tions paralyzes decision makers so much that they

dont take chances and they sweep missteps under

the rug. Many companies dont learn and adapt

fast enough from missteps, as we saw with Black-

berry, Nokia and Microsoft in their early responses

to the iPhone.

To help a team learn faster, leaders must (1) frame

mistakes as valuable learning opportunities; (2) re-

spond to failure as temporary, isolated and not

personal;35 and (3) emphasize that learning from a

decision is a goal in itself.

In the U.S. and Israeli militaries, after-action

debriefs have become the norm.36 Debriefing in the

Israeli military is so highly valued that everyone is

graded on this skill, including their ability to create

a climate that accepts mistakes as natural and a

source for learning. This background training has

spread beyond the military to Israels venture com-

munity. Nearly all entrepreneurs in Israel have

served in the military, since such service is manda-

tory for both men and women. These shared

experiences, especially tolerance for failure and

after-action reviews, have helped Israel become a

technological innovation hub.37

Tips and Pointers 1. Shine a light on mistakes as sources of new

learning. Blakely of Spanx grew up in a home

where her parents admired her for trying and

failing. She incorporates this view into her lead-

ership philosophy.

2. Conduct after-action reviews to extract in-

sights. Define mistakes and successes in terms of

process rather than outcomes. Teach team mem-

bers to ask questions that elicit learning rather

than defensiveness.

3. Publicize stories about failed projects that led

to innovative solutions. Praise those who

learned from their errors and try to extract learn-

ing from near misses.

Start With Questions Not AnswersTypically, we dont judge leaders on the quality of

their questions, nor do we design our educational

systems or corporate training to develop this cru-

cial skill. If anything, we do the opposite. Television

game shows reward contestants who know answers

to preset questions and usually very trivial ques-

tions at that. Having encyclopedic knowledge may

win you a million dollars on a TV game show or

yield good grades in school, but it wont necessarily

make you successful in todays complex business

world. In changing environments, the big prizes go

to those who ask better questions and learn faster.

In organizations, this comes down to leaders teach-

ing and coaching others to think more strategically

and ask deeper questions. If you think like everyone

else, you are likely to be average. The best strategic

thinkers, leaders and entrepreneurs distinguish

themselves by how they frame decisions, the kinds

of questions they ask and their mode of inquiry.38

Paul J.H. Schoemaker is research director of the William and Phyllis Mack Institute for Innovation Management at the Wharton School of the University of Pennsylvania in Philadelphia and the founder and executive chairman of Decision Strategies International, a consulting firm with offices in Conshohocken, Pennsylvania, and London. Steven Krupp is the CEO of DSI. This article builds on the authors new book, Winning the Long Game: How Strategic Leaders Shape the Future (PublicAffairs, 2014). Comment on this article at http://sloanreview.mit.edu/x/56214, or contact the authors at [email protected].

REFERENCES

1. S. Krupp and P.J.H. Schoemaker, Winning the Long Game: How Strategic Leaders Shape the Future (New York: PublicAffairs, 2014).

2. V.P. Barabba, Meeting of the Minds: Creating the



Market-Based Enterprise (Boston: Harvard Business Press, 1995).

3. E. Schmidt and J. Rosenberg, How Google Works (New York: Grand Central Publishing, 2014).

4. G.S. Day and P.J.H. Schoemaker, Are You a Vigilant Leader? MIT Sloan Management Review 49, no. 3 (spring 2008: 43-51).

5. Transcript of Fareed Zakaria Global Public Square, aired Jan. 5, 2014, http://transcripts.cnn.com.

6. Making a Mark With Rockets and Roadsters, NPR, Aug. 9, 2007, www.npr.org.

7. W. Hall, Father of the Prius Declares Electric Cars Not Viable, Feb. 4, 2013, www.breitbart.com.

8. R. Lawler, Tesla CEO Elon Musk Says He Got Into the Electric Car Business Because No One Else Would, May 29, 2013, http://techcrunch.com.

9. E. Musk, http://videos.huffingtonpost.com.

10. R. Kabacoff, Develop Strategic Thinkers Throughout Your Organization, Harvard Business Review, Feb. 7, 2014, https://hbr.org.

11. L. Huston and N. Sakkab, Connect and Develop: In-side Procter & Gambles New Model for Innovation, Harvard Business Review 84, no. 3 (March 2006): 58-66.

12. For scenario planning books, see K. van der Heijden, Scenarios: The Art of Strategic Conversation (New York: John Wiley, 1996); P.J.H. Schoemaker, Profiting From Uncertainty: Strategies for Succeeding No Matter What the Future Brings (New York: Free Press, 2002); and L. Fahey and R.M. Randall, eds., Learning from the Future: Competitive Foresight Scenarios (New York: John Wiley, 1998).

13. P.J.H. Schoemaker, Scenario Planning: A Tool for Strategic Thinking, Sloan Management Review 36, no. 2 (winter 1995): 25-40.

14. Testimony of Deven Sharma, president of Standard & Poors, before the Committee on Oversight and Government Reform, United States House of Representatives (Oct. 22, 2008), http://oversight-archive.waxman.house.gov.

15. P. Krugman, That Hissing Sound, The New York Times, Aug. 8, 2005.

16. J.E. Russo and P.J.H. Schoemaker, Winning Deci-sions: Getting It Right the First Time (New York: Doubleday, 2001).

17. C.J. Nemeth, B. Personnaz, M. Personnaz, and J.A. Goncalo, The Liberating Role of Conflict in Group Cre-ativity: A Study in Two Countries, European Journal of Social Psychology 34, no. 4 (July/August 2004): 365-374.

18. J. Lehrer, Imagine: How Creativity Works (Boston: Houghton Mifflin Harcourt, 2012).

19. R. King, Malcolm Gladwell: Disruptive Innovators Are Usually Disagreeable, CIO Journal, Sept. 17, 2013, http://blogs.wsj.com.

20. J. Dewey, Human Nature and Conduct: An Introduc-tion to Social Psychology (New York: Henry Holt, 1922).

21. D. Baer, 5 Brilliant Strategies Jeff Bezos Used to Build the Amazon Empire, March 17, 2014, www.businessinsider.com.

22. Quoted in P.F. Drucker, The Effective Executive(New York: Harper & Row, 1967), 148.

23. R. Charan. DuPonts Swift Response to the Financial Crisis, Bloomberg BusinessWeek, Jan. 7, 2009, www.businessweek.com.

24. A. Hill, Snapshot of a Humbled Giant, Financial Times, Apr. 2, 2012. Also see G. Gavetti, Kodak: Inter-view with Dr. George Fisher, Oct. 1, 2005, DVD (Boston: Harvard Business School Publishing, 2006), http://hbr.org.

25. E.R. Tufte, Visual and Statistical Thinking: Displays of Evidence for Making Decisions (Cheshire, Connecticut: Graphics Press, 1997).

26. See H.J. Einhorn and R.M. Hogarth, Decision Mak-ing Under Ambiguity, The Journal of Business 59, no. 4, pt. 2 (October 1986): S225-S250, or the original study by Daniel Ellsberg, which became a classic in the field: D. Ellsberg, Risk, Ambiguity, and the Savage Axioms, Quarterly Journal of Economics 75, no. 4 (November 1961): 643-669.

27. R. Mundle, Fatal Storm: The Inside Story of the Tragic Sydney-Hobart Race (Camden, Maine: Interna-tional Marine/McGraw-Hill, 1999).

28. P.J.H. Schoemaker and R.E. Gunther, The Wisdom of Deliberate Mistakes, Harvard Business Review 84, no. 6 (June 2006): 108-115.

29. D. Ogilvy, Confessions of an Advertising Man (London: Southbank Publishing, 2004). First published 1963 by Holiday House.

30. R.S. Root-Bernstein, How Scientists Really Think, Perspectives in Biology and Medicine 32, no. 4 (summer 1989): 472-488.

31. W.H. Hughes, Alexander Fleming and Penicillin (London: Priory, 1974).

32. R. Frank, Billionaire Sara Blakely Says Secret to Suc-cess Is Failure, October 16, 2013, www.cnbc.com.

33. S. Kirsner, Campbell Soup CEO Denise Morrison Talks Corporate Innovation in Boston, Boston Globe, May 8, 2013.

34. C. Martin, Wearing Your Failures On Your Sleeve, The New York Times, Nov. 8, 2014, www.nytimes.com.

35. M.E.P. Seligman, Authentic Happiness: Using the New Positive Psychology to Realize Your Potential for Lasting Fulfillment (New York: Free Press, 2002).

36. After-Action Review: http://en.wikipedia.org.

37. D. Senor and S. Singer, Start-Up Nation: The Story of Israels Economic Miracle (New York: Twelve, 2009).

38. The power of critical inquiry is highlighted in P. Thiel, Zero to One: Notes on Startups, or How to Build the Future (New York: Crown Publishing Group, 2014).

Reprint 56214. Copyright Massachusetts Institute of Technology, 2015. All rights reserved.



Using Simulated Experience to Make Sense of Big DataAs data analyses get more complex, how can companies best communicate results to ensure that decision makers have a proper grasp of the datas implications?BY ROBIN M. HOGARTH AND EMRE SOYER

Many behavioral experiments have shown that when the same statistical in-formation is conveyed in different ways, people make very different decisions.

IN AN INCREASINGLY complex economic and social envi-ronment, access to vast amounts of data and information can help

organizations and governments make better policies, predictions

and decisions. Indeed, more and more decision makers rely on

statistical findings and data-based decision models when tackling

problems and forming strategies. Scientists, researchers, technol-

ogists and journalists have all been monitoring this tendency,

trying to understand when and how this approach is most useful

and effective.1

So far, discussions have centered mainly on analysis: data col-

lection, technological infrastructures and statistical methods. Yet

another vital issue receives far less scrutiny: how analytical results

are communicated to decision makers. As the amount of data gets

bigger and analyses grow more complex, how can analysts best

communicate results to ensure that decision makers have a proper

understanding of their implications?

Communicating Statistical InformationHowever well executed, the usefulness of an analysis depends on

how the results are understood by the intended audience. Con-

sider a patient visiting a doctor about an illness. Arguably, the

most important task is the diagnosis of the disease, as this can lead

to choosing an appropriate treatment. Yet even if the final deci-

sion lies with the patient, the chosen treatment may depend on

how the doctor communicates different options to the patient.

The same is true when an investor consults a financial expert or a

manager seeks the services of a consulting firm.

THE LEADING QUESTIONHow can com-panies best communicate analytical results to executives?

FINDINGSThere is often a large gap between conclusions reached by analysts and what decision makers understand.

Descriptions of complex statistical information can be misleading.

Interacting with a simulation model can help executives make better decisions.

D E C I S I O N M A K I N G : S I M U L AT I O N S




Data science, like medical diagnostics or scien-

tific research, lies in the hands of expert analysts

who must explain their findings to executive deci-

sion makers who are often less knowledgeable

about formal, statistical reasoning. Yet many

behavioral experiments have shown that when the

same statistical information is conveyed in differ-

ent ways, people make drastically different

decisions.2 Consequently, there is often a large gap

between conclusions reached by analysts and what

decision makers understand. Here, we address this

issue by first identifying strengths and weaknesses

of the two most common modes used for commu-

nicating results: description and illustration. We

then present a third method simulated experi-

ence that enables intuitive interpretation of

statistical information, thereby communicating

analytical results even to decision makers who are

nave about statistics.

Description Description is the default mode of presenting statistical information. This typically

involves a verbal statement or a written report,

which might feature one or more tables summariz-

ing the findings. The strength of this approach lies

in its speed in providing the decision maker with

the most essential and salient aspects of a given

analysis. But as problems get more complex, this

very strength turns into a major flaw. While

highlighting one issue, descriptions can end up

hiding details that have important decision-making

consequences. (See Describing an Investment

Problem.) The question then becomes: When it

comes to making decisions, are we able to differen-

tiate between good and bad descriptions?

Our own research suggests that descriptions can

mislead even the most knowledgeable decision

makers. In a recent experiment, we asked 257 econom-

ics scholars to make judgments and predictions

based on a simple regression analysis. (See About

the Research.) This is the most prevalent type of

analysis employed in applied economics in order to

identify and quantify a causal relationship between

two or more variables. To our surprise, most of

these experts had a hard time accurately decipher-

ing and acting on the results of the kind of analysis

they themselves frequently conduct. In particular,

we found that our description of the findings,

which mimicked the industry standard, led to an il-

lusion of predictability an erroneous belief that

the analyzed outcomes were more predictable than

they actually were.

The description obscured some sources of

uncertainty, and the decision makers became over-

confident about their prospects. Ultimately, we

managed to avert this illusion by substituting the

description with an illustration. This time, judg-

ments and decisions were accurate, suggesting that

the description of the results was indeed to blame

for the misperceptions.3

Illustration Illustrations in the form of a graph, figure, diagram or chart are also used regularly to

communicate statistical information. Unlike de-

scription, the primary objective is to give an

overview of the analysis and provide a bigger (al-

beit less precise) picture about the findings.

Consequently, decision makers are better able to

acknowledge the trends, effects and risks of their

prospective decisions.4 Using illustrations, it is

more difficult for crucial parts of the results to re-

main hidden. Hence, one benefit of visualizing data

is in making uncertainties more transparent. (See

Illustrating an Investment Problem, p. 52.) In

fact, a 2011 Science article that evaluates human

proficiency in visualizing data is aptly entitled

Visualizing Uncertainty About the Future.5

DESCRIBING AN INVESTMENT PROBLEMConsider a scenario where Y is a desirable variable such as wealth or health, and X is a valuable and scarce resource like money or time. You want to end up with more Y by making an investment in X. Analysis of past investments shows that, on average, increasing X by one unit leads to a one-unit increase in Y. Your current investment is X = 0, and you are considering increasing it to X = 5. With this amount of investment, what are the chances that you will actually end up with a negative Y? Alternatively, how many units of X would you need so that you can be 80% sure that you will end up with a positive Y?

The salient aspect of this description is the estimated 1-to-1 average effect. While highlighting this aspect, however, the description completely hides the uncertainties inherent in Y. In particular, the information provided gives no clues as to how random events, beyond the control of the decision maker, might affect the outcome. Such uncertainty could, in fact, result in someone with a large investment ending up worse off than someone with less or no investment at all. We could, for instance, agree that smoking is bad for health, but being a nonsmoker does not guarantee an individual a longer and healthier life relative to a heavy smoker.

Hence, armed only with the information about the average effect, it is impossible to answer accurately the particular questions posed in the problem or perceive the potential risks associated with the decision.



Yet for all of its benefits, illustration is not always

an ideal way to communicate complexities. The

more variables, structural changes, connections and

patterns there are in the data, the harder it becomes

to condense them into one display. Moreover, like

description, illustration is typically static and not

interactive. Hans Rosling, a professor of interna-

tional health at Karolinska Institute in Solna,

Sweden, has attempted to counteract these short-

comings with the aid of visual technology.

Specifically, he creates innovative visualizations of

trends in global development using dynamic graphs

that show changes across time.6 In a similar fashion,

Betterment LLC, an investing service based in New

York, has recently created a tool that lets users visu-

alize the evolution of an investment of $100 in the

S&P 500 over a given holding period.7

Nonetheless, the shortcomings of description

and illustration remain a pervasive problem. Pro-

jecting a multifaceted, convoluted and complicated

process into static words, tables and graphs inexora-

bly means omitting crucial parts of the results.

Relevant information is inevitably lost in transla-

tion. Would it be possible, therefore, to develop a

dynamic alternative to these available approaches

one that would let decision makers more readily

grasp the complexities and uncertainties inherent in

the analyses?

In a series of studies, we designed and tested the

effectiveness of an alternate method of communi-

cation. Instead of describing or illustrating the

analysis, we let decision makers gain experience on

the outcomes of different possible actions by inter-

acting with simulations based on the same analysis.

Our findings showed that regardless of their level

of statistical sophistication, people relate well to

such an approach. Moreover, as analyses become

more complicated, decision makers tend to trust

experience more than their analytical intuitions.

Most importantly, their judgments and decisions

improve in the face of increasing uncertainty and

complexity. We call this approach simulated

experience.8

Simulated ExperienceFor tens of thousands of years, humans formed

judgments and made decisions exclusively through

experience. Formal statistical reasoning and tools

are comparatively recent innovations. In particular,

probability theory, which constitutes the founda-

tion of our current methods of analysis, was only

conceived in the 17th century.9 The problem we

face today is that our ability to communicate and

understand statistical outcomes has not advanced

as rapidly as our proficiency in handling data. In

fact, all nonhuman animals still depend solely on

experience to make choices and solve their prob-

lems. In which locations are sources of food

available and with what regularity? Where are

predators present? Which meteorological patterns

and trends exist in a particular environment? These

are vital issues for survival. It is therefore unsur-

prising that evolution has endowed both animals

and humans with remarkable capacities to encode

information about past occurrences.10 When it

comes to understanding and communicating sta-

tistical information, experience is a powerful yet

often underappreciated tool.

Simulated experience exploits our natural abil-

ity to transform complicated information into

actionable knowledge. Essentially, it lets the

ABOUT THE RESEARCHWe recently published a series of papers in experimental psychology on the effectiveness of simulated experience as a communication tool for statistical information. To research these papers, we conducted several experiments. For example, in one of the experiments, we asked 257 economics scholars to make judgments and predictions based on a simple regression analysis a type of analysis with which they are extremely familiar. Yet most of these experts had a hard time accurately deciphering and acting on the results. In fact, our descrip-tion of the findings, while mimicking the industry standard, obscured some sources of uncertainty. Ultimately, as detailed in this article, we managed to avert this illusion by substituting the description with an illustration.

In another series of studies, we designed and tested the effectiveness of an alternate method of communication. Instead of describing or illustrating the analysis, we let decision makers whom we grouped into different pools based on their levels of statistical sophistication gain experience about the outcomes of different possible actions by interacting with simulations based on the same analysis. Our findings showed that regardless of their level of statisti-cal sophistication, people relate well to such an approach. Specifically, their judgments and decisions improve in the face of increasing uncertainty and complexity. Other insights we reached include: Descriptions are easy to construct, but tend to hide uncertainties by focusing attention on average effects. Illustrations make uncertainties more visible. However, they do not cope well with complex analyses involving multiple variables. As the uncertainties and complexities of decision situations increase, people tend to trust their experiences more than their analytic abilities. Regardless of their level of statistical knowledge, simulated experience helps decision makers form an accurate understanding about possible outcomes of the underlying statistical analysis.




decision maker live through the problem many

times with the aid of a simulation. (See Under-

standing an Investment Problem Through

Simulated Experience.) The implementation of

simulated experience involves four components:

Analysis Conduct an analysis using available data, from which an analyst might normally craft a de-

scription or illustration.

Simulation Instead of creating some type of descrip-tion or illustration, the analyst constructs a simulation

model based on the findings of the analysis.

Interaction Executive decision makers interact with the simulation model and can input potential

actions and observe the subsequent outcomes.

Experience Decision makers can experiment with changing their inputs. That way, they can experience

as many outcomes as they wish, given those inputs.

Executives gain experiential evidence about potential

consequences of their actions, based on the statistical

analysis.

Research attests to the effectiveness of this ap-

proach when the problem is both hard to handle

and rife with uncertainties. For example, one article

in the Journal of Consumer Research found that in-

dividuals create better retirement plans when they

interactively observe their potential future bene-

fits.11 Researchers at the University of Zurich have

found that simulated experience helps investors

perceive accurately the underlying risk-return pro-

file of their investments.12 A recent Management

Science article suggests that banks could employ

such a communication method to help their clients

accurately perceive the risks associated with differ-

ent investment products.13 John Sterman, the Jay

W. Forrester Professor of Management at the MIT

Sloan School of Management and director of MITs

System Dynamics Group, argues that climate

change debates involve crucial misperceptions

and he effectively removes them by simulating

viable scenarios.14

In our own research, we find that people have

difficulty assessing their chances of success in compe-

titions and market-entry decisions, but simulating

such situations leads to improved assessments and

decisions.15 Moreover, simulated experience has de-

monstrably helped to correct judgmental biases, such

as neglecting base rates in probabilistic statements.16

Accordingly, initiatives such as Probability Manage-

ment Inc. a nonprofit organization that aims at

improving communication of uncertainty through

open-source decision-support tools seek to put

such evidence to use, applying simulation-based

communication to improve actual managerial

decisions and public policies.17 (See Designing Sim-

ulations, p. 54.)

There are several software packages that provide

the necessary tools to create a wide range of

ILLUSTRATING AN INVESTMENT PROBLEMConsider the investment problem described in the previous exhibit. Here is the illustration of the X-Y relationship based on 250 individual observations, on which the analysis is conducted:

Seeing a graph of the past investments instead of a description that summarizes them helps a decision maker acknowledge the uncertainties inherent in the out-comes. The description featured previously only mentions the fitted line and disregards completely the cloud of data that surrounds it. Now it becomes clear that despite a 1-to-1 average effect, someone with a positive investment might end up with a negative outcome (see the data points below the horizontal line). Moreover, a larger investment would not always guarantee a larger return than that of someone who made a smaller investment.

Illustrating the relationship between two variables is easy. What if, however, there were multiple investment options instead of just the one? This is almost always the case in real-life analyses and situations. Illustrating many interrelated variables in one figure is unwieldy, if not unfeasible. In the face of such complexities, illustrations are bound to be less meaningful.

X

Y

0 5 10 15 20 25 30 35

-10

0

10

20

30

40



simulations for diverse decision-making situations.

Oracle Crystal Ball, Frontline Systems Risk Solver

Platform and Luminas Analytica are among the

most prominent. With a little bit of knowledge in

programming, one could also create simulations

from scratch using platforms such as MATLAB or

C++. Of course, Microsofts Excel is always handy if

the decision problem is relatively simple.

Simulated experiences are not without their

weaknesses and blind spots. For instance, if decision

makers simulate only a small set of experiences, the

resulting intuitions might be driven by sampling

variability. The remedy is to generate a large num-

ber of observations, given the same input, before

forming an idea about the uncertainties. Finally, like

all other communication methods, simulated expe-

rience is useful as long as the underlying analysis is

unbiased and accurate. If the analysis has blind

spots, so will the descriptions, illustrations and sim-

ulated experiences that stem from it.

Precisely Wrong Versus Approximately CorrectAdvances in computing technology allow us to

build simulations for virtually any scenario. How-

ever, this does not mean that we should employ this

method for all problems. For simple probabilistic

situations, a description or illustration may be a

wiser choice. But as complexity grows and uncer-

tainties arise, simulations can help managers better

understand statistical information and thus enable

them to make better decisions, regardless of their

levels of statistical expertise. We do not call for the

abandonment of descriptions or illustrations. In-

stead, we argue that these should sometimes be

augmented with add-on simulations.

Why is it, then, that we rarely encounter simula-

tions for critical decisions about medical treatments,

investment options, pension plans, insurance pro-

grams and so on? After all, simulation technology is

not new. In fact, it has always been essential to any en-

gineering process. The issue is that simulations are

still primarily seen as sophisticated tools for statistical

analysis as opposed to a means for communicat-

ing results.

There are two main reasons for this. The first is

technical: There is a cost to building simulations.

So it is easier for analysts to only craft descriptions

or build illustrations using the features in statistics

software. Second, and more importantly, simula-

tions are vague. An interface that lets you provide

your inputs, simulate the analysis, and sequentially

observe the related outcomes does not steer you

toward a precise answer. You have to make up your

own mind as you experience the simulated out-

comes. We typically do not appreciate such a fuzzy

approach, especially if the decision is important.

We seek perfect solutions exact maneuvers that

will lead to desired outcomes. As managers, politi-

cians and individual decision makers, we prefer

learning correct answers right away.

However, the very presence of uncertainty suggests

its wise to refrain from seeking fast solutions. No deci-

sion has a completely foreseeable set of outcomes. In

both business and life, chance has its say. Leaning too

heavily on likelihoods will inevitably lead to the belief

that we can predict outcomes more accurately than

we can. Such misperceptions can lead to precise but

wrong answers to important questions.

UNDERSTANDING AN INVESTMENT PROBLEM THROUGH SIMULATED EXPERIENCEThis time, consider a slightly more complex investment problem, where there are three possible investment options (X1, X2 and X3) affecting the outcome Y, instead of just one. The interface below lets decision makers enter their choices of X1, X2 and X3. When a user clicks the simulate button, the model simulates a corresponding Y based on the analysis conducted on the available data. Users are free to enter as many input sets and simulate as many outputs for each as they wish.

The simulation presented here records the previous entries and outcomes. This allows users to select a subsample of outcomes (the highlighted part of the simulated Y column) and obtain the average of that selected subsample. (The reset button clears all previously simulated data.) Hence, users not only gain insights by making decisions and experiencing the consequences, but they also can gather information about the average effects of their strategies. One could also construct a histogram based on the selected subsample an illus-tration of the frequencies of simulated outputs to visually display how potential outcomes are distributed.

Choice of X1

Simulate

Reset

X1 X2 X3 Simulated Y Selected YCount Average

101010

555555

333555555

000555555

5

Choice of X2

5

Choice of X3

5

0.5-0.33.24.7

10.84.24.4

-1.07.5

6 5.1




Simulated experience aims to counter this ten-

dency by letting decision makers feel the most likely

answers. In the words of a famous American math-

ematician, the late John W. Tukey, Far better an

approximate answer to the right question, which is

often vague, than an exact answer to the wrong

question, which can always be made precise.

Robin M. Hogarth is an emeritus professor in the department of economics and business at Universi-tat Pompeu Fabra in Barcelona, Spain. Emre Soyeris an assistant professor of judgment and decision making on the business faculty at zyegin Univer-sity in Istanbul, Turkey. Comment on this article at http://sloanreview.mit.edu/56215, or contact the authors at [email protected].

REFERENCES

1. P. Simon, Too Big to Ignore: The Business Case for Big Data (Hoboken, New Jersey: John Wiley & Sons, 2013) offers an overview of business applications of data science. See also R. Fildes and P. Goodwin, Against Your Better Judgment? How Organizations Can Improve Their Use of Management Judgment in Forecasting, Interfaces 37, no. 6 (November-December 2007): 570-576.

2. A. Tversky and D. Kahneman, The Framing of Deci-sions and the Psychology of Choice, Science 211, no. 4481 (January 30, 1981): 453-458; A. Tversky, P. Slovic and D. Kahneman, The Causes of Preference Rever-sal, American Economic Review 80, no. 1 (March 1990): 204-217; C.K. Hsee, G.F. Loewenstein, S. Blount and M.H. Bazerman, Preference Reversals Between Joint and Separate Evaluations of Options: A Review and Theoretical Analysis, Psychological Bulletin 125, no. 5 (September 1999): 576-590; A. Tversky and R.H. Thaler, Anomalies: Preference Reversals, Journal of Economic Perspectives 4, no. 2 (spring 1990): 201-211; and for a specific case study, see J. Koehler, Psychol-ogy of Numbers in the Courtroom: How to Make DNA-Match Statistics Seem Impressive or Insufficient, Southern California Law Review 74 (2001): 1275-1306.

3. E. Soyer and R.M. Hogarth, The Illusion of Predict-ability: How Regression Statistics Mislead Experts, International Journal of Forecasting 28, no. 3 (July-September 2012): 695-711.

4. R.M. Hogarth and E. Soyer, A Pictures Worth a Thousand Numbers, Harvard Business Review 91, no. 6 (June 2013): 26.

5. D. Spiegelhalter, M. Pearson and I. Short, Visualizing Uncertainty About the Future, Science 333, no. 6048 (September 9, 2011): 1393-1400.

6. H. Rosling, The Best Stats Youve Ever Seen, TED talk filmed February 2006, www.ted.com.

7. An interface shows the relationship between time and returns based on daily data. See: D. Egan, Its About Time in the Market, Not Market Timing. October 14, 2014, www.betterment.com.

8. R.M. Hogarth and E. Soyer, Sequentially Simulated Outcomes: Kind Experience Versus Nontransparent Description, Journal of Experimental Psychology: General 140, no. 3 (August 2011): 434-463; R.M. Hogarth and E. Soyer, Providing Information for Decision Making: Contrasting Description and Simula-tion, Journal of Applied Research in Memory and Cognition, in press, published online January 29, 2014; R.M. Hogarth and E. Soyer, Communicating Forecasts: The Simplicity of Simulated Experience, Journal of Business Research, in press.

9. G. Shafer, The Early Development of Mathematical Probability, in Companion Encyclopedia of the History and Philosophy of the Mathematical Sciences, Volume 2 ed. I. Grattan-Guinness (London and New York: Rout-ledge, 1993): 1293-1302.

10. L. Hasher and R.T. Zacks, Automatic and Effortful Processes in Memory, Journal of Experimental Psy-chology: General 108, no. 3 (September 1979): 356-388; L. Hasher and R.T. Zacks, Automatic Processing of Fundamental Information: The Case of Frequency of Occurrence, American Psychologist 39, no. 12 (De-cember 1984): 1372-1388; P. Sedlmeier and T. Betsch, Etc. Frequency Processing and Cognition (New York: Oxford University Press, 2002).

11. D.G. Goldstein, E.J. Johnson and W.F. Sharpe, Choosing Outcomes Versus Choosing Products: Consumer- Focused Retirement Investment Advice, Journal of Consumer Research 35, no. 3 (October 2008): 440-456.

12. M.A. Bradbury, T. Hens and S. Zeisberger, Improv-ing Investment Decisions With Simulated Experience, Review of Finance, published online June 6, 2014.

13. C. Kaufmann, M. Weber and E. Haisley, The Role of Experience Sampling and Graphical Displays on Ones Investment Risk Appetite, Management Science 59, no.2 (February 2013): 323-340.

14. J.D. Sterman, Communicating Climate Change Risks in a Skeptical World, Climatic Change 108, no. 4 (October 2011): 811-826.

15. R.M. Hogarth, K. Mukherjee and E. Soyer, Assess-ing the Chances of Success: Nave Statistics Versus Kind Experience, Journal of Experimental Psychology: Learning, Memory, and Cognition 39, no. 1 (January 2013): 14-32.

16. B.K. Hayes, B.R. Newell and G.E. Hawkins. Causal Model and Sampling Approaches to Reducing Base Rate Neglect, in Proceedings of the 35th Annual Conference of the Cognitive Science Society, eds. M. Knauff, M. Pauen, N. Sebanz and I. Wachsmuth (Austin, Texas: Cognitive Science Society, 2013.)

17. Probability Management is an organization that aims to improve communication of uncertainty through open-source decision support tools. More information can be found at www.probabilitymanagement.org.

Reprint 56215. Copyright Massachusetts Institute of Technology, 2015. All rights reserved.

DESIGNING SIMULATIONSHere are a few recommen-dations on how to design and use simulated experience:

The interface should be user-friendly. Interaction should not be costly.

Decision makers should be informed that simu-lated experience is a communication tool; it helps people understand the results of an analysis, but it does not prove the reliability of the analysis.

Decision makers should be advised about how the simulation works and how it calculates the outcomes given their inputs.

Decision makers should be given time to interact with the simulation and make up their minds at their own pace.

The simulation should incorporate uncertainties. For example, it should allow decision makers to experience different results given the same inputs when there is randomness in the underlying process.



Why You Decide the Way You DoFor executives, making good decisions is essential. New research offers insights into factors that can affect the decision-making process.BY BRUCE POSNER

HOW DO PEOPLE process different inputs and make complicated decisions? Variations on this question have engaged researchers for many years, with broad implications for a variety of individu-

als. But the topic is of particular interest to business executives, who must frequently make decisions.

Researchers have long sought to shed light on the inner workings of the human brain and the way

people make decisions. In recent years, curiosity about the decision-making process has heated up,

attracting academics from fields as diverse as neuroscience, management, behavioral economics and

psychology. Here are highlights of a handful

of recent scholarly articles that offer intrigu-

ing insights into decision making from

several disciplines.

1. The Advantage of Psychological DistanceInformation overload is a fact of modern

life, making many common decisions

(such as choosing a cellphone plan) un-

bearably confusing. Although choice offers

options to consumers, too many choices or

too many features per choice can cause

people to delay decisions or make less-

than-optimal choices. Recent research into

how individuals process information of-

fers some promising suggestions for

dealing with information overload. The

key may involve psychological distancing

removing oneself from the morass of

details surrounding a decision and consid-

ering the choices on a more abstract level.

As authors Jun Fukukura, Melissa

J. Ferguson and Kentaro Fujita explain

in their article in the Journal of Experimen-

tal Psychology: General, such distancing

THE LEADING QUESTIONWhat strategies can improve decision making?

FINDINGSIf you face informa-tion overload, psychological distancing can be helpful.

A willingness to ask for advice on diffi-cult problems can increase your per-ceived competence.

For minor, day-to-day choices, you may not need to de-liberate carefully to achieve a satisfac-tory choice.

D E C I S I O N M A K I N G : N E W R E S E A R C H

Too many choices or too many features per choice can cause people to delay decisions or make less-than-optimal choices.




(which can be either temporal or physical) can help

people to filter out the less-vital details and enable

them to focus on the gist of the matter. The authors

tested several aspects of how psychological distance

influences decision

making. In one study,

they asked some par-

ticipants, who were

students from Cor-

nell University in

Ithaca, New York, to

write about a car they

would buy next year,

and others to write

about a car they

would buy tomor-

row. (A control group

was not given a writing task.) Participants were then

given information to read about 48 individual fea-

tures (such as mileage, handling, year and trunk

capacity) of four different cars twelve features per

car and had only seven seconds to absorb each

piece of information before the next piece appeared

on a computer screen. Participants were then asked

to choose the car they thought was best. Those who

has written about the future before receiving infor-

mation chose the best car (the one whose features

were considered most important to people in an ear-

lier pilot) significantly more often than participants

who had written about a near-term purchase (69%

vs. 40%) or those in the control group (39%).

In another test of psychological distancing, the

researchers randomly assigned one group of indi-

viduals to write for three minutes about the

previous day and another group to write for three

minutes about a day about a year earlier. Then they

presented participants with sets of information

about the features of the four different cars; a com-

puter screen displayed information about the

features of one car at a time, and the participants

learned about the cars at their own pace. When the

participants were done reading, they were asked to

select the car they would buy and to characterize

the memory strategy they had used. Those who had

written about the past selected the best car at a

much higher rate than those who had written about

recent occurrences (59% versus 34%) or members

of the control group (29%), who had not done a

writing task. Whats more, those participants who

had written about the past reported relying on gist

memory in other words, memory about the gist

of a matter significantly more often than the

others. The researchers found that mind-sets

involving psychological distance enabled partici-

pants to organize related product features better.

To be sure, psychological distancing isnt appro-

priate for every situation. In instances where people

are expected to recall and piece together specific de-

tails (for example, jury trials or investigations), it may

be harmful. But in many circumstances involving in-

formation overload, it can result in better decisions.

2. Balancing Exploration and ExploitationScholars have argued that companies can develop

greater ambidexterity as they search for better ways to

balance practices supporting optimal exploitation

of existing opportunities and those promoting

exploration of new ones. Although much of the re-

search on corporate

ambidexterity has

been focused on how

companies can best

achieve ambidexter-

ity, less attention has

been paid to how the

cognitive processes of

individual managers

can shape perfor-

mance on a broader

level. New research by

Daniella Laureiro-

Martnez, Stefano

Brusoni, Nicola Can-

essa and Maurizio

Zollo shifts the discussion. In an article published in

Strategic Management Journal, the authors describe

how different regions of the brain control different

cognitive activities.

Exploitation, the authors explain, is behavior that

optimizes performance in current tasks, and explo-

ration is behavior leading to disengagement from

current tasks to search for alternatives. Exploitative

decisions take place in areas of the brain associated

with reward seeking and involve learning by doing.

Exploration choices, by contrast, activate the brains

REFERENCEJ. Fukukura, M.J. Ferguson and K. Fujita, Psycho-logical Distance Can Improve Decision Making Under Information Overload via Gist Memory, Journal of Experimental Psychology: General 142, no. 3 (August 2013): 658-665.

REFERENCED. Laureiro- Martnez, S. Brusoni, N. Canessa, and M. Zollo, Understanding the Exploration- Exploitation Dilemma: An MRI Study of Attention Control and Decision-Making Performance, Strategic Manage-ment Journal, in press, published electronically February 28, 2014.



attention control and executive functioning regions,

which are tasked with managing new situations.

The researchers studied the decision-making be-

haviors of 63 people who had at least four years of

experience making managerial decisions. Partici-

pants were asked to sit at computers and play a game,

the purpose of which was to accumulate points that

could be traded for cash. Following a brief warm-up,

they played the game while lying inside a functional

MRI scanner that took images of their brains. The

game featured four slot machines that awarded

points according to rules that changed from trial to

trial; each participant played a total of 300 trials.

However, the changing rules were never spelled out;

participants were expected to learn about them

through experimentation. Participants could choose

to pursue an option they were familiar with (exploi-

tation) or explore a new one (exploration).

The researchers compared the choices of study

participants (the number of exploration and exploi-

tation choices, and the number of times they

switched between the two) and their decision-

making performance. The authors found significant

links between greater activation of regions of the

brain associated with attention control and better

performance in the game, which supported their

hypothesis that increased attentional control is

linked to better decision-making performance. In

this study, participants who did less exploration gen-

erally performed better, but, more broadly, the

authors concluded that superior decision-making

performance relies on the ability to sequence exploi-

tation and exploration appropriately and to

recognize when to switch to exploration.

3. How to Tee Up ChoicesWhen does it make sense to let people make active

choices on their own, and when is it preferable to

design default rules that nudge people in a certain

direction (for example, to become an organ donor

or to use energy generated by wind)? In modern

societies, individuals face a barrage of complicated

choices: how to set up retirement accounts; how

much to save; whether to waive collision coverage

on rental car agreements, and so on. Decisions take

time and attention, and people are busy. Default

rules determine what happens if people choose to

do nothing.

Depending on what you are trying to achieve,

changing default rules can be a particularly power-

ful tool that institutions have, argues Harvard Law

School professor Cass R. Sunstein perhaps

more effective than significant economic incen-

tives. Writing in the University of Pennsylvania Law

Review, Sunstein ex-

amines the rationale

for default rules and

why and when orga-

nizations would use

blanket rules instead

of allowing individ-

uals to make their

own choices or establishing personalized rules

based on a persons individual profile (for example,

using demographic data). Default rules, he ex-

plains, dont impose mandates or bans. Rather, they

steer people in a particular direction (while offer-

ing opportunities to opt out), producing outcomes

that institutions want at costs that are lower than

economic incentives. By contrast, requiring indi-

viduals to make their own choices can impose high

costs in terms of the time it takes to learn about the

options. The job of choice architects, according to

Sunstein, is to understand decision costs (including

how confusing the decision is and how heteroge-

neous the pool of decision makers is) and the costs

of errors (what happens when people decide in a

way thats detrimental to them or to other members

of a group).

In Sunsteins view, the most desirable default

rules are informed chooser defaults, which align

with what most well-informed people would

The job of choice architects is to understand decision costs (including how confusing the decision is) and the costs of errors (what happens when people decide in a way thats detrimental to them or to other members of a group).

REFERENCEC.R. Sunstein, Deciding by De-fault, University of Pennsylvania Law Review 162, no. 1 (December 2013): 1-57.




choose. Such defaults appeal to those interested in

efficiency, welfare, autonomy or fairness. (On the

other end of the spectrum are default rules that are

either badly designed or intentionally misleading;

with so-called negative option marketing, for

example, companies offer people free products,

then enroll them in programs with a monthly fee

unless they make the effort to opt out.) Even when

its possible to develop default rules that are geared

to individuals personal needs and tastes (as in

algorithms that use your past choices in books or

music to make recommendations), Sunstein

argues that there may be an argument for preserv-

ing a system based on active choosing. Why? In

some areas, he believes, active choosing promotes

learning in ways the defaults do not, which may

generate long-term benefits.

4. Going With the FlowWhen you have a decision to make, you may

assume that you should focus rationally on the

choices and select the best one. Legal and economic

decision-making theory generally argues for care-

fully considering each option and then picking the

one that delivers the highest expected value. The

advantage of this approach is that the decision will

reflect your intentions, and you will be less likely to

have post-decision

remorse or so the

theory goes.

But new research

suggest that people

who make decisions

more spontaneously

by allowing their

thoughts to wander

until they arrive at a

choice that they feel drawn to can be as satisfied

with their decisions as those who choose more

deliberately. Writing in Frontiers in Psychology, re-

searchers Colleen E. Giblin, Carey K. Morewedge

and Michael I. Norton describe research they con-

ducted that included co

MITSMR Making Better Decisions Collection

Documents

Transcript of MITSMR Making Better Decisions Collection