Analyzing+Opportunities+ +AO 27July
Transcript of Analyzing+Opportunities+ +AO 27July
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
1/30
Analyzing Opportunities
In 2004, two Serbian men and a Hungarian woman won $2 million from the Ritz Casino in
London. Luck? Robbery? Or simply smart betting? How about a combination of all three. The
team had a small camera that scanned the roulette wheel and the position of the ball when it was
released by the croupier and sent this information to a remote location. Based on historical data
of that croupier and a simulation model of the mechanics underlying the movement of the ball, a
workstation recommended bets. Was the system able to perfectly forecast the outcome of the
roulette? No, but it was allegedly able to improve the gambling odds from the traditional 37 to 1
to 6 to 1. In other words, by making use of data analysis and a clever process, a losers gamble
was turned around into a profitable businessat least until the casino caught on.
Financial evaluations of innovation opportunities at times can feel like betting in a casino. After
all, many innovations fail, and most will not pay back their investment. But by following the
methods that we outline below, you can emulate that trio of London gamblers and begin to turn
the odds in your favor. By incorporating into your forecasts information about past innovations
and the innovation process, you can increase your likelihood of success.
version 080907
Horizon 1 Horizon 2 Horizon 3
Nature of
Uncertainty
Type of
Uncertainty
Example of a
Market
Uncertainty
Example of a
Technology
Uncertainty
Uncertainty about a
number, such as a sales
forecast
Uncertainty about a
scenario, such as an
approval or a cancellation
Many types; not even clear
which type matters
We think that sales volume
for the new product will be
between 120k and 150kper year.
We think there exists a 40%
probability that the results
of our pilot market warranta full launch of the new
product. Otherwise, the
product will be cancelled.
The market for transportation
devices smaller than cars seems to
have big potential. However, weneither understand the exact user
needs to peoples willingness to pay
We might be able to improve the
efficacy of drug x by customizing
the treatment to the patient.However, we dont know how to go
about this customization.
We think that the drug will
have a 20%-30% higher
efficacy than the drugcurrently on the market
We think that the drugs
toxicity might require the
termination of all clinicaltrials with a probability
of 20%.
Horizon 1
Horizon 2
Exhibit ESTIMATION FRAMEWORK: The three different types of opportunitiesdeal with different types of uncertainty requiring different analytical tools in theirevaluation.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
2/30
AO-2
As we discussed in the previous chapter, innovations can be classified into three groups, based
on their level of uncertainty and time horizon. For Horizon 1 opportunities, your main concern is
estimating the likely sales. For Horizon 2 opportunities, sales are also uncertain, but youre less
concerned with that than with a bigger riskwhether opportunity will reach the market. Early
cancellation is a real risk, and you have to factor that into your analysis. For Horizon 3
opportunities, the uncertainty is so great that a formal financial analysis is almost impossible.
You should instead seek basic insights about the quality of the opportunity.
Horizon 1 and Horizon 2 opportunities lend themselves to sophisticated analyses, such as
stochastic calculations that may seem intimidating at first sight. We call this rocket science
innovation management. Just as the trio at the Ritz benefited from modeling and analyzing
thousands of previous spins of the roulette wheel, so too can you benefit by applying rigorousanalysis to data from past innovations.
Evaluating Horizon 1 Opportunities
When evaluating a Horizon 1 opportunity you want to create an expected discounted cash flow
(DCF) model. You will also want to estimate the uncertainty of your DCF and can view the
variance of the DCF as a measure of risk. You create a DCF model in five steps, which Exhibit
HORIZON1 EVALUATION summarizes:
(1) forecast the expected sales of the opportunity;
(2) adjust the forecast based on the experience from prior opportunities;
(3) compute the variance of the forecast;
(4) use a typical product lifecycle pattern to estimate monthly (yearly) sales;
(5) build an economic model of costs and revenues.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
3/30
AO-3
version 081407
To Portfolio
Planning / Final
Selection
Define
opportunity
including target
segment and price
Concept test
with input from
potentialcustomers
Aggregate
opinions of
multiple experts
Computeexpected sales
and adjust for past
forecast errors
Spread sales
over the product
lifecycle. Build
a DCF modelComputevariance based
on disagreement
among experts.
Compute
variance
based on old
AF ratios
From Screening
and Scoring
Exhibit HORIZON1 EVALUATION: For each Horizon 1 opportunity coming fromscreening, five steps need to be taken to create an expected discounted cash flow.
Forming a sales forecast
Entire books have been written about forecasting sales for of new products, so we will focus on
the process more than the technical details. You can obtain a forecast using either of two basic
methods: surveying customers or relying on experts.
When doing a survey, identify customers in the market you are targeting with the opportunity.
Then try to discover their purchase intent with questions such as, If a web service reminding
you of important dates like your anniversary were available, how likely would you be to use it:
Very, Somewhat, Maybe/maybe not, somewhat unlikely, very unlikely. From this, you can
compute an average purchase probability and expected sales as shown in Exhibit CONCEPT
TEST. Note that in the exhibit, the customers expressed purchase intent is discounted (a 0.4
weight is applied to definite buy, and a 0.2 weight is applied to probably buy). Research has
shown that customers are much less likely to adopt an innovation compared with what they say
in surveys. These sorts of parameters as well as adjustments for product awarenessyou can
only purchase products you are aware ofdiffer widely by industry and should be obtained via
market research.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
4/30
AO-4
SpoilMy Spouse is an internet based service that will send an email or SMS to subscribers on a
weekly basis offering ideas on how to make their spouse happy (e.g. bring her flowers, come
home early, etc). These ideas are user-rated and recommendations are made based on the
experience of like-minded subscribers.
How likely are you to purchase subscription to this service?
Definitely
would buy
Probably
would buy
Might or might
not buyProbably
would not buy
Definitely
would not buySurv
eytoCustomers
InternalAnalysis
ofResults
Results
Proportion
Purchase
Probability
10 22 48 59 61
10/200 22/200 48/200 59/200 61/200
0.4 x + 0.2 x = 0.042
Expected sales = Population size x Awareness x Purchase Probability = 4M x 0.2 x 0.042 = 33,600version 080907
22200
10200
Exhibit CONCEPT TEST: To form a sales forecast, you estimate the averagepurchase probability for the population by learning the expressed purchase intent andthen adjusting based on historical data. You then multiply with the number ofpotential adopters aware of the product
1.
Alternatively, you can rely on experts to forecast sales. This method works best if your
innovation resembles products or services that have been launched in the past. The method is
similar to what you saw in the previous chapter on scoring and screening. The experts make
estimates independently. You then convene them a consensus-building discussion or simply
average their forecasts.
Dealing with forecast errors
Sales forecasts rarely are exactly right. If you forecast that 13 million people would see a movie
and 14 million did, you would pop open the champagne. But if you forecast 20 million people
would see it and only 5 million do, you would weep. The problem, of course, is that you only
know the accuracy of your forecast after the fact.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
5/30
AO-5
Project Forecast Actual AF ratio
7 5600 1000 0.178571
12 2500 500 0.2
5 6700 4000 0.597015
14 2100 1500 0.714286
1 11100 8000 0.720721
4 6700 5000 0.746269
2 9800 7500 0.765306
8 3879 3000 0.773395
18 1146 900 0.78534
10 3300 3300 1
9 3800 4000 1.052632
11 2500 3000 1.2
6 6300 8000 1.269841
20 1100 1400 1.272727
3 9000 11500 1.277778
16 1400 2000 1.428571
19 1100 2000 1.818182
17 1200 2400 2
Average 0.988924
St-Dev 0.483351
Product Forecast Actual
AF
Ratio Wine Year Color Forecast Actual
AF
Ratio
ZEN-ZIP 2MM FULL 470 116 0.246809MONTAGNE ST-EMILION 02 rouge 4000 468 0.117
ZEN 3/2 3190 1195 0.374608MONTAGNE ST-EMILION 02 rouge 6000 1236 0.206
ZEN 4/3 430 239 0.555814 CARTON PANACHE 1 n.a. mixed 1800 372 0.206667
WMS ELITE 3/2 650 364 0.56 BORDEAUX SUP 01 Rouge 1200 252 0.21
WMS HAMMER 3/2 FULL 6490 3673 0.565948 IROULEGUY 01 Rouge 3000 726 0.242JR HAMMER 3/2 1220 721 0.590984 CARTON PANACHE 2 n.a. mixed 900 297 0.33
HEATWAVE 4/3 430 274 0.637209 BORDEAUX 01 Rouge 1800 612 0.34
ZEN 2MM S/S FULL 680 453 0.666176 APREMONT 02 Blanc 1200 567 0.4725
EVO 4/3 440 623 1.415909 ROSE DE LOIRE 03 Ros 2300 2934 1.275652
WMS EPIC 4/3 1060 1552 1.464151 CTES DU RHNE (6) 01 Rouge 4000 5370 1.3425
JR EPIC 4/3 380 571 1.502632 CTES DU RHNE (6) 01 Rouge 3000 4146 1.382
EVO 3/2 380 587 1.544737 BORDEAUX 02 Rouge 2500 4057 1.6228
JR ZEN FL 3/2 90 140 1.555556CHTEAUNEUF DUPAPE 00 Rouge 300 703 2.343333
EPIC 3/2 2190 3504 1.6 CARTON PANACHE 6 na mixed 2160 5064 2.344444
Average 0.9966 Average 0.858337
St-Dev 0.369666 St-Dev 0.472292
Exhibit AF-RATIOS: Three cases of comparing forecasted numbers with the actualoutcomes in diverse settings including power and automation, surf apparel, andwines.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
6/30
AO-6
Even though forecasts are faulty, you should take time to examine your past forecasts and
contrast them with your actual performance. For each opportunity, compute the ratio between the
actual number and the forecasted number, providing anAF ratio for each opportunity. This
works for sales and financial forecasts. AF ratios can vary from figures that are significantly less
than onea ratio of 0.1 indicates that you over-forecasted by a factor of 10to figures of 3 or
highera ratio of 3 indicates that the real number was three times as high as the forecast.
Exhibit AF ratios shows examples from very different industries, including power and
automation, surf apparel, and French wines. At first sight, the analysis does not say much.
Sometimes, the forecasts are too high; sometimes, too low. But with a little more analysis and
graphical support similar to what is shown in Exhibit AF-GRAPH, you can obtain a couple of
powerful insights2.
By looking at the average AF ratio, you can see if a firm is forecasting correctly on average.
Most companies consistently forecast more than they actually sell; humans are optimistic
animals. As you can see in Exhibit AF-RATIO, the power and automation and surf apparel
forecasts are very accurate on average, with an average ratio close to 1. At least for the company
that provided the data underlying the exhibit, wines suffer from over-forecasting; on average,
sales are only 85 percent of the forecasts. (Maybe the forecasters spent too much time sampling
the product.) In the future, if you need to forecast the sales for a new wine, you should adjust the
forecast by multiplying it by 0.85 (and keep the forecasters away from the bottles).
The AF ratios also help you estimate your risk. A company that consistently produces forecasts
with AF ratios between 0.9 and 1.1 faces less risk (a lower forecast variance) than an outfit with
AF ratios ranging between 0.5 and 2. (Exhibit AF-RATIO (right) shows how you can use the
distribution of past AF ratios to estimate the standard deviation for the new product sales. For
every dollar forecasted, you really face a distribution of potential outcomes as is shown in the
distribution in Exhibit AF-GRAPH.)
Risk follows a common statistical pattern. To see this, compare the three distributions of AF
ratios in AF-Graph. The left and the right histograms resemble the normal distribution while the
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
7/30
AO-7
middle one seems closer to a uniform distribution. Once you understand the statistical behavior
of your old forecasts, you can build a forecast model that incorporates your tendency to err. Risk
can be modeled, in other words, whether the opportunity is a robot, surf shorts, or a Saint
Emilion wine.
version 082007
Frequencywithwhich
AFRatiowasObserved
0.0 0 1 .0 0 2 .000
1
2
3
4
5
6
Financial Returns for
Power and Automation
Products
AF Ratio
0.2 0 1 .0 0 1 .6 00
1
2
3
4
5
Product Sales of
Surf Apparel and
Wet-suits
AF Ratio
0 .00 1.0 0 2 .500
2
4
6
8
10
12
Product Sales of
French Wines
AF Ratio
Exhibit AF-GRAPH: Each of the three graphs shows the empirical distribution ofthe AF ratios shown in Exhibit AF-RATIOS.
Building a financial model
Once you have a sales forecast, you can create your overall financial model. For this, you first
distribute the total sales over the lifecycle of the product using a lifecycle similar to that of other
innovations of this type. Considering your development and launch costs as well as the major
fixed and variable costs, you can compute the discounted cash flow (DCF) for this innovation.
Exhibit ECONOMICS-EXAMPLE provides an example of this by plotting the cumulative cash-
flows (inflows and outflows) with the innovation. This graphic can easily be created in Excel and
provides a basis for the most common financial evaluations of Horizon 1 opportunities. (In
Chapter FI, we discuss several aspects related to how to appropriately choose discount factors
a topic that can lead to heated debates when evaluating innovation opportunities.)
Given your focus on value creation, your most important measure is the financial returns relative
to your investment. Yet with the discounted cash flow analysis, you can also compute other
commonly used metrics, including break-even time, time to your first commercial sale, and the
return on investment from the innovation.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
8/30
AO-8
Cumu
lative
Inflowor
Ou
tflow
($)
version 081507
Ramp-up Costs
Marketing and Support Costs
Production Costs
+$
-$
Sales Revenues
TimeDevelopment Costs
Exhibit ECONOMICS EXAMPLE: Most opportunities begin with a cash outflow,typically covering the costs of development and launch. Successful opportunities arethose in which the cumulative cash inflows (revenues net of production costs) justifythe initial investment3.
Now that you have computed the expected sales and the corresponding standard deviation, you
can also obtain the standard deviation of your financial measures. The easiest way to do this is to
perform a Monte-Carlo simulation in the Excel model, varying the sales number (or other
numbers that experience suggests are uncertain such as development time) according to the
histogram computed in Exhibit AF RATIOS. This can be simply done by randomly creating an
AF ratio between zero and two and then multiplying that AF ratio with the sales forecast.
As the opportunity moves through development, all forecasted numbers should be compared
with actual performance and updated. Because Horizon 1 opportunities are seldom terminated,
the development process should be executed with a focus on efficiency and timeliness.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
9/30
AO-9
Evaluating Horizon 2 opportunities
Earlier in this chapter we compared financial evaluations of innovations with bets at a casino.
Before you take your money to the roulette table, you want to make sure that you understand the
rules of the games and the odds. Imagine that you place a $100 bet on #14 when playing roulette.
You know that (unless you have secret camera) the odds of winning are 1 in 37. If you win, you
will receive $3,600. More likely, though, you will lose, and the casino will take your $100. From
simple statistics, you can value this bet using the expected payoff, which is computed as:
Expected payoff of putting $100 on #14 = 36/37 (-$100) + 1/37 ($3600-$100) = -$2.70
Thus your expected payoff is negative, and the casino makes more money than you do.
(Surprise, surprise.) Exhibit CASINO-EXAMPLE explains this bet. The concept of the expected
payoff works in cases where you make a lot of bets and what matters is the average payoff. If
you bet only once, you might also consider other measures, such as the probability of losing
money or the payoff in the worst (best) case scenario. On the left of the exhibit, you can see how
the uncertainty of the innovation can play out (also called an event tree). On the right, you can
see the various possible outcomes and associated probabilities.
version 081507
Probability
36/37
-100
1/37
+3500
Outcome is 0 (Probability 1/37) Pay-off is 0
Outcome is 1 (Probability 1/37) Pay-off is 0
Outcome is 2 (Probability 1/37) Pay-off is 0
Outcome is 14 (Probability 1/37) Pay-off is 3600
Outcome is 36 (Probability 1/37) Pay-off is 0
Exhibit CASINO-EXAMPLE: An event tree describes the odds of winning at the
roulette table (left) and a histogram shows the potential payoffs (right).
Expected payoff
Enough roulette. Now lets consider a real innovation project. In Exhibit PHARMA-EXAMPLE,
you have the event tree for a new chemical compound considered for investment. Based on
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
10/30
AO-10
Phase 1 clinical trials, you will learn how toxic the drug is for humans. If people cannot tolerate
the drug at all (10 percent probability), all trials stop. Otherwise, the drug moves into Phase 2
trials. A home run would be a drug with medium toxicity but high efficacy, in which case you
would make $500 million. If the toxicity is medium, you have a 20 percent chance of obtaining a
high efficacy score. For a more toxic drug (strong toxic activity, 40 percent probability), you
have a better chance of obtaining a high efficacy (80 percent). But because of the higher toxicity,
fewer patients could use the drug, and the payoff would be only $100 million.
version 081507
Probability
0.82
-30
0.1
+370
0.08
-130Pay-Off
Phase I shows mediumtoxic activity (p=0.5)
Phase Ishows strongtoxic activity(p=0.4)
Phase I shows substancecannot be tolerated byhumans (p=0.1)
Phase II showsmedium efficacy (p=0.8)
Pay-off: 100
Phase II showshigh efficacy
Pay-off: 500
Phase II showsmedium efficacy (p=0.2)
Pay-off: 0
Phase II showshigh efficacy (p=0.8)
Pay-off: 100
Investment Phase I: $30M Investment Phase II: $100M
Exhibit PHARMA-EXAMPLE: An event tree and histogram for a pharmaceuticalcompound.
To calculate the expected payoff for this drug, apply the same logic as before. The expected
payoff (not yet including the necessary investment) can be calculated as:
Expected payoff
= 0.5 Payoff if medium toxicity + 0.4 Payoff if strong toxicity + 0.1 Payoff if too toxic
= 0.5 (0.8 $100M + 0.2 $500M) + 0.4 (0.2 $0M + 0.8 $100M) + 0.1 $0M
= 0.5 $180M + 0.4 $80M = $90M + $32M = $122M
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
11/30
AO-11
If you factor in a development cost of $30 million for the Phase 1 trials and $100 million for the
Phase 2 trials, you end up with a net gain (loss) of: $122 million - $130 million = -$8 million.
And the drug seems as risky as a bet in roulette. But this analysis misses something critical.
Understanding option value
Despite the similarities with gambling, evaluating an innovation requires a more detailed
analysis of how the innovation will be created and how it will proceed through selection. In other
words, you can make an intermediate decision after obtaining information from Phase 1 but
before making the additional (large) investment in Phase 2. To see why a more detailed analysis
is needed, consider the three possible outcomes of the Phase 1 trials:
Case 1 (Medium toxicity): When you learn about the outcome of Phase 1, you have already
spent the associated $30 million. These costs are what economists call sunkthe money is gone
and you cant recoup it, so its no longer relevant to your analysis. Knowing a medium level of
toxicity, you can expect to earn a payoff of 0.8 $100 million + 0.2 $500 million = $180
million. For this, you would have to spend another $100 million. Would you do this? The answer
is an enthusiasticyes! This investment would create an expected $80 million in value.
Case 2 (Strong toxicity): When you learn about strong toxicity, you again should look forward,
not backward. What matters is that you have an expected payoff of $80 million (0.2 $0 + 0.8
$100M), not that you have spent $30 million. This is especially important when deciding on the
next $100 million investment. Investing $100 million to obtain an expected $80 million destroys
value. So, as painful as it might be having spent $30 million for nothing, you kill the project
(more on the organizational aspects of this later).
Case 3 (Too toxic to tolerate): Again, the $30 million is sunk and, because you have no hope of
making any money going forward, you should terminate the project after Phase 1.
Now, lets return to the initial decision concerning the initiation of Phase 1 trials. Is this project a
good investment or not? You have a 50 percent shot at $80 million and a 50 percent (40 percent
+10 percent) shot at nothing. For this, you have to spend $30 million for the Phase 1 trials.
Suddenly, what looked like a bad investment a moment ago looks much more attractivea $40
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
12/30
AO-12
million payoff for a $30 million investment. Note also that this investment is now less risky, as
you have eliminated the scenario in which you suffer a $130 million loss.
What can you learn from this example? First, most investments in innovation happen in phases,
and the early phases are usually less expensive than the later ones. While the first phases by
themselves typically do not lead to financial rewards, they provide something almost as valuable:
information. At the beginning of Phase 1, you had a compound about which you knew little.
Once you have spent the initial $30 million, you knew a lot more. The additional information
had the same effect as the techniques the gamblers used at Ritz: it improved your odds. You
could invest in promising compounds (those with medium toxicity) and abandon unpromising
ones.
Second, when investing into a new innovation, keep in mind that you have the right, even the
responsibility, to walk away if the information you gather points to failure. Failing is part of
innovation, and failing early makes more sense than failing late. Walking away from an
innovation is often referred to as an exit option, as it resembles the financial option of buying (or
not buying) a stock at a predefined price.
Estimating the probabilities of success
Once you have outlined all the ingredients for the calculations in Exhibit PHARMA-EXAMPLE,
valuing the compound (and making the necessary decisions) is a matter of calculus. The
calculation turns out to be simple, once you write them down. In contrast, the much harder
problem is to get the numbers required for the inputs. So how do you estimate success
probabilities? What does it mean to have a 50 percent probability of success? To see the
importance of these questions, consider again the payoffs described in Exhibit PHARMA-
EXAMPLE. By re-doing the above calculations with slightly lower (or higher) estimates for theprobability of obtaining a medium toxicity, you observe the following:
A 1 percentage point increase in probability (for example, a move the probabilityfrom 50 percent to 51 percent) is worth $0.8 million in value, which corresponds to
roughly 10 percent of the overall value of the innovation.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
13/30
AO-13
Moving the probability from 45 percent to 55 percent increases project value by 100percent.
Thus not only are the odds of innovation success important, their exact magnitude can have a
dramatic effect on your decision-making. For this reason, we recommend that you do the
computations outlined above with a range of plausible success probabilities, not just one. If the
opportunity is profitable only for a small range of probabilities, you probably should not pursue
it.
In general, you can choose among three estimation methods that yield probabilities of success
(POS). Given that even small changes in POS have large impact on the NPV of an opportunity,
consider using two or ideally all three of them.
Historical outcome data. You can use historical data about the proportion of similaropportunities that succeeded to estimate the POS for opportunities currently under
evaluation. Exhibit PHARMA-POS shows a set of probabilities for product approval
in the drug development process. Given that the POS data shows significant
differences across illnesses, you would want to group the data points of prior
opportunities according to illness. The main strength of forecasting POS using
historical data is that it provides a rigorous statistical approach. But it also assumes
that the future will resemble the past.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
14/30
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
15/30
AO-15
the opportunity under consideration. But they also can become mired in office or
professional politics.
Criteria. Many firms have clear criteria that define what POS level any givenopportunity deserves. Proctor & Gamble has found that by using explicit criteria of
what corresponds to, say, a 50 percent POS, expert discussion can focus more on
facts and less on opinions5. Experts, for example, are likely to agree on whether
theres a 90 percent probability that a new product can be made on existing machines
at high volume. This approach is objective, but defining the criteria consumes a lot of
time. And even more time gets spent translating the criteria to numeric probabilities.
Again, there is no one right way to forecast POS. You should use multiple approaches and
understand that you face an extra level of risk if these approaches arrive at very different
estimates. Any POS forecast should always be pressure tested: if a 5 percent reduction in POS
turns your return into a loss, the opportunity is probably not worth funding. As we discussed in
the previous chapter, accidentally killing a good opportunity might be painful, but is typically far
less expensive than further developing a poor one.
All three approaches benefit from feeding back the actual outcome into your forecasting for
future opportunities. We will elaborate on this point after discussing Horizon 3 opportunities.
Exhibit HORIZON2-EVALUATION summarizes the overall process.
version 081507
Forecast the pay-
off collected upon
launch (expected
value is enough)
Estimate the
probabilities that
the milestone is
completed (POS)
based on:
Historical data
Opinion
aggregation
Criteria
From Screening
and Scoring
Define all costs
required to get the
opportunity
launched.
Identify milestone(s)
at which the
opportunity could be
killed (and no furthercost incurred)
To Portfolio
Planning / Final
Selection
Build an integrated
model:
- Exit option
- Probability for
each pay-off
Exhibit HORIZON2-EVALUATION: Summary of steps that need to be takenwhen analyzing a Horizon 2 opportunity.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
16/30
AO-16
Evaluating Horizon 3 opportunities
Horizon 3 opportunities are all about uncertainty. But, in contrast to the tidy risks of Horizon 1
opportunities, the uncertainty for Horizon 3 opportunities is, simply put, one big mess. You dont
know many critical facts and you dont even know that you dont know even more things. So
how do you evaluate an opportunity like this?
Exhibit SEGWAY: The Segway scooter is a two-wheeled mobility device.
Consider the example of the Segway scooter, a personal-transportation device launched in 2001.
Given the radical nature of the innovation, the Segway team faced a number of uncertainties,
including:
Can you produce a self-balancing scooterthe Segway has two wheels on the sameaxleand sell it for less than $5,000?
Will local governments allow Segways on sidewalks? Are you able to produce more 100,000 units a year? Can you create a sufficient product awareness to sell more than 100,000 units a year? How will consumers respond to the Segway?
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
17/30
AO-17
Focus on hypothesis testing
With the Segway, youre not dealing with a Horizon 1 or 2 uncertainty. At inception, the Segway
represented a new radically technology is search of a market. The innovators did not know who
might adopt it. Commuters? People with disabilities and the elderly? Cops? Tour operators?When facing this much uncertainty, you focus on learning. You have only hypotheses about how
it might meet peoples needs but no solid evidence. For this reason, Horizon 3 evaluations focus
on the creation of knowledge, not potential sales.
Issue Current Understanding Type of Uncertainty
Technical Feasibility Technology has been used in a previous product. Detailed engineering not clearyet, but uncertainty primarily relates to component costs. Costs will be
between$3,000 and $5,000.
Horizon 1: Parameter uncertainty
Regulatory Approval Not clear if Segways will be allowed on sidewalks or if they will require driverslicense.
Horizon 2: Scenario uncertainty
Ability to Mass Produce No prior experience with mass production of a product of this type. Theproduct appears to be similar in its production process to golf carts and othersimple electric vehicles.
Horizon 1: Parameter uncertainty
Marketing Campaign Are you able to create sufficient product awareness? Currently, severalnewspapers and online chat rooms are hotly debating what product you willlaunch.
Horizon 1: Parameter uncertainty
Consumer Interest You are hoping to fundamentally change the way consumers think abouttransportation. You have no interest in niche markets such as golf carts. Giventhe risk that somebody could steal your idea, youve done no market research.
Horizon 3: Unknown unknowns
Exhibit HYPOTHESES1: To analyze a Horizon 3 opportunity, the various sourcesof uncertainty need to be identified and categorized into three horizons defined inChapter SO (Screening Opportunities).
When evaluating a Horizon 3 opportunity, you start by creating a table like the one shown in
Exhibit HYPOTHESES1. This table lists all aspects of this opportunity, such as its technical
feasibility, its fit with current regulations, and the customer response. For each one, you thenassess your current level of understanding:
Level 1 understanding (parameter uncertainty). You exactly understand the problembut have estimates for only some of its parameters (for example, the unit cost of the
Segway will between $3,000 and $5,000).
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
18/30
AO-18
Level 2 understanding (scenario uncertainty). You can envision all possibleoutcomes (and have some probability estimates) for your hypotheses (for example,
whether each state or major city will either approve the Segway for sidewalk use).
Level 3 understanding (unknown unknowns): You dont even understand all therelevant variables related to the hypothesis (for example, to support your sales
hypothesis, you need to first set a price point, identify market segments, learn the
purchase probabilities and appropriately adjust these probabilities)6.
version 081607
H1
: We can produce a self-balancingscooter for a unit cost below $5k
H2: Governments will allow people touse our scooters on side-walk
H3: We can produce at volumes>100k per year
H4: We can create product awarenesswithout having an established brand
H5: More than 100k consumers per yearwill want to purchase a scooter
Uncertainty
(phrased as hypothesis)
Full Understanding
(no uncertainty)
Build a protype ($2M)
Lobby for changes in regulation (>$5M)
Purchase intent test ($10M)
Public relations ($1M)
Exhibit HYPOTHESES2: Each uncertainty can be resolved at a cost. Your goal is
to find the uncertainty for which you can create a maximum of learning per dollar ofinvestment.
Once you have mapped out the uncertainties, you can answer two questions. What are the
biggest (and scariest) uncertainties? And where do you get the biggest amount of learning per
dollar of investment? As is illustrated by the leftward-pointing arrows in Exhibit
HYPOTHESES2, it is relatively expensive to reduce the uncertainty about regulatory approval.
True, regulatory approval is crucial, but before spending millions lobbying for Segways to beallowed on sidewalks, you might want to find out if consumers want to ride the machine at all.
Maybe, they want to use them only on golf courses or at airports. Creating a table such as
Exhibit HYPOTHESES is very helpful in pinpointing where small investments might create new
knowledge. In this case, a simple purchase-intent survey such as outlined in the section on
Horizon 1 evaluations would have revealed that mainstream consumers were not interested in the
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
19/30
AO-19
Segway as means of transportation at a price of $5,000. With this knowledge, the management
team probably would have waited to build a mass-production factory.
Learning vs. opportunity costs of time
Learning and testing hypotheses seem sensible steps. But they suffer from the motherhood and
apple-pie syndromethat is, who could disagree with them? Theyre too easy to pursue
mindlessly. Thus, before you plunge in, you need to consider what youre sacrificing when you
spend time on them. Your willingness to move ahead should be driven by two considerations.
The first is the opportunity cost of time, or, put differently, the urgency of a launch. This variable
is typically determined by the competitive landscape. The second is the chance of inexpensive
uncertainty resolution. This depends on how Exhibit HYPOTHESES looks for the opportunity at
hand, specifically if you have a key hypothesis that you can test for a small investment7.
Based on these two variables, you can create a matrix as described in Exhibit UNCERTAINTY
RESOLUTION. In some situations, you simply must place a big bet: if the opportunity cost of
time is high and you cant reduce uncertainty, there are no intermediate steps. You either
launch the rocket now or never. Granted, this scenario is rare. In many situations, you can simply
wait (and potentially become the second mover) or follow a staged investment approach as
discussed above.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
20/30
AO-20
version 080907
Rate at which Uncertainty Can be Reduced with
Incremental Investment.
OpportunityCostofTime
(e.g.,
immedia
cyofcompetitivethreat)
Uncertainty is high for
some time regardless
of our investment.
Uncertainty can be
reduced substantially
with modest investment.
HighOpportunity
CostofDelay
LowO
pportun
ity
CostofDelay
Go for it and translate
into Horizon 2
development
Adopt a dynamic,
contingent-action
approachstages
with decision points.
Wait and see
possibly
as second mover.
Take slow, deliberate,
and careful steps to
reduce uncertainty.
Exhibit UNCERTAINTY RESOLUTION: Depending on the possibility ofuncertainty resolution (that is, of learning via small investments) and on theopportunity cost of time, different strategies become optimal.
So how to you make a final decision? Again, recall that Horizon 3 opportunities need to be
evaluated based on the learning that they create. The investment in the opportunity needs to be
staged with the milestones corresponding to the hypotheses that you want to test. As you saw
with Segway, you generally want to start with the biggest uncertainty reduction per dollar
invested. Whether you make the initial investment, kill the opportunity or postpone requires a
group evaluation similar to what you have seen elsewhere in this chapter. If you cannot resolve
any of the uncertainty resolution quickly and you face time pressures (upper left box in Exhibit
UNCERTAINTY RESOLUTION), the potential payoff better be big and likely to happen under
a wide range of possible scenarios.
One venture capital firm that we studied uses the following approach to make the initial funding
decision for Horizon 3 opportunities. After hearing the creators pitch, each partner takes a turn
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
21/30
AO-21
and discusses (a) the key hypotheses that she would like to see tested and then (b) gives a
recommendation with respect to funding or not funding the opportunity. These discussions start
with the most junior partner and are wrapped up by the most senior partner. That way, opinions
of less experienced partners arent influenced by those of their more senior colleagues, who
should be more comfortable expressing opinions that dont line up with the groups consensus.
The final vote is typically pro forma. In most cases, the partners have already reached an
agreement.
Which Opportunities to Move Forward?
Our discussion so far has been on analyzing the three types of opportunities yet, we still have
not addressed the question of which opportunities to now move forward. Recall from our earlier
discussion of opportunity classification (see Chapter SO (Screening Opportunities)) that we
should not compare one type with another. The amount of resources that should be allocated to
each type of opportunity will be discussed at length in the following chapter (see Chapter SP
(Innovation Strategy and Opportunity Portfolio)).
So, how do we compare within each type? For Horizon 1 and Horizon 2, you select the setof
opportunities that creates the biggest value. In other words, you maximize the value of the
innovation portfolio. Too often, companies just rank opportunities according to expected returns
and then pursue the ones at the top of the list. But this approach ignores interdependencies
among the opportunities. The value of the portfolio is not just the sum of the individual values.
(See Chapter SP (Innovation Strategy and Opportunity Portfolio) for more details.)
For Horizon 3 opportunities, you should compare the required investments and the associated
learnings across opportunities and determine which ones would create knowledge most relevant
to your company as a whole. (Chapter SP (Innovation Strategy and Opportunity Portfolio) also
helps more with this question.)
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
22/30
AO-22
Human and Organizational Aspects of Opportunity Evaluation
All of the techniques we have discussed for evaluating opportunities are probabilistic in nature.
This reflects the uncertain reality of innovation. But in trying to implement these techniques, you
have to keep in mind that probabilistic thinking does not come naturally to many people.
version 081607
Forecast Probability of Rain
938
82159
147
203
172
257
589
575
2820
0.2
0.4
0.6
0.8
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
ProportionofDayswithRain
Exhibit POS-FORECASTING1: A comparison of the forecasted probability of rain(x-axis) plotted against with the proportion of days on which rain came (y-axis). Eachnumber along the plotted line corresponds to the number of times a specific forecast
was made. For example, a 0.3 probability of rain was forecasted 257 times8.
To get more comfortable with forecasting the probability of success of an innovation, consider
the example shown in Exhibit POS-FORECASTS-1. It shows data from another area in which
people often speak in terms of probabilities, namely weather forecasting. You might have asked
yourself in the past what it really means if you have a 70 percent probability of rain. After all, it
will either rain tomorrow or it will not. The graphic will help you ponder that question.
On the x-axis, you have different levels of rain probability that the National Weather Service has
announced for a particular region and period. Pick the point of 0.7, that is, a 70 percent
probability. By moving upwards in the graphic, you hit the point labeled 159, which means
that, for this region and period, the National Weather Service had announced a 0.7 rain
probability for 159 days. But what you want to know is how many times it actually rained the
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
23/30
AO-23
following day. To get to that number, move from this point to the left, towards the y-axis. If the
forecast had been perfect, you would expect that rain would have fallen in 70 percent of the
cases in which the National Weather Service forecast a 70 percent probability. You see that the
actual number was only slightly higher (about 72 percent). Overall, thats an impressive
performance.
version 081007
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Forecast Probability of Pneumonia
ProportionwithPne
umonia
,
asDeterminedby
X-ray
Exhibit POS-FORECASTING2: The graphic compares the forecasted probabilityof pneumonia as expressed by the doctors participating in the study (x-axis) with theproportion of times the patient indeed suffered from pneumonia. For example, only 5percent of the patients for whom doctors estimated a 25 percent probability ofpneumonia suffered from it
9.
Now consider a second example, created in the exact same way. Instead of weather forecasters,
our subjects are doctors who need to estimate the probability that patients have pneumonia.
Exhibit POS-FORECASTS2 shows the proportion of patients who turned out to have pneumonia
as a function of the probability forecast of the treating physician. For cases in which the doctor
forecast a 50 percent probability of pneumonia, x-rays indicated that only 10 percent of the
patients had the disease.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
24/30
AO-24
Unlike the weather forecasts, in which all data points tidily matched the predictions, you see that
their data points stray far from the good forecasting line Of course, just because doctors fail at
forecasting does not mean they mistreat their patients. From a forecasting perspective, you might
complain about their performance. From a medical perspective, the picture changes. What the
doctor really might have been saying when predicting 50 percent was, This patient is at risk for
pneumonia, and we should take a chest x-ray. Put differently, the doctor did not distinguish
clearly between the forecast (50 percent) and ordering an action (the chest x-ray).
Why does this distinction matter? Because the same problem exists in most innovation
processes. Both those proposing innovations to be fundedinventors, scientists, entrepreneurs
and those who control the requested resourcesmanagers and venture capitalistsoften confuse
the forecast (this innovation will or will not work) with the action (should we fund thisinnovation?). Of course, advocates will argue that the odds are good and will attempt to shift the
45 percent odds to 55 percent odds, while funders will argue the opposite. But to make a prudent
financial valuation, you must draw a clear line between your risk assessment and the associated
payoffs.
To improve your forecasting capability, the following three steps are useful:
(a) ensure that the evaluation of opportunities is done by independent reviewers (as opposed to
people who have a stake in the success of the opportunities);
(b) increase the independence of the review and improve its predictive power by involving a
larger population inside or outside your company;
(c) give regular feedback to all units involved in forecasting.
The need for an independent review board
The adage that You dont let the turkeys vote on Thanksgiving! has a direct relevance to the
evaluation of opportunities. You must separate those who select opportunities from those who
create and develop them, and this is where independent review boards come in. To understand
their usefulness, consider Exhibit INLICENSE-POS. The data summarizes a McKinsey study on
the innovation odds of compounds that drug companies created internally versus those that they
licensed from outsiders.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
25/30
AO-25
version 081407
Clinical Success Probabilities Among the Leading Pharmaceutical Companies
54
48
39
7068
80
14
27
Phase I Phase II Phase III
Cumulative SuccessClinical Trials
Internally developed
compounds
Licensed compounds
Exhibit INLICENSE-POS: Success probabilities for internally developed chemicalcompounds compared with licensed compounds. Internally developed compounds are
more likely to pass the Phase 1 review, yet their overall success rate is only half ofthat of licensed compounds
10.
Looking at the overall success ratesthat is, the probability of the innovation surviving all three
phasesyou notice that licensed compounds are twice as likely to succeed. Why is this? When
licensing a compound, an organization does exactly what we described above. It has a set of
criteria and applies them to the licensing candidates. In this setting, the people involved in the
valuation have no personal stake in the innovation and can be dispassionate.
But when you are evaluating an internally developed innovation, people involved with its
creation probably have some influence on the review. This often leads to a company being
reluctant to terminate a project early on. Just as parents would not harm their own children, many
innovators do not want to kill their projects. For this reason, as you can see in INLICENSE-POS,
internally developed projects are more likely pass through the first phase but also more likely to
fail later. This tendency can cost you real money. As you have seen in the context of valuing
individual projects, the value of an innovation depends crucially on your ability to walk away
from it. A review board can be truly independent in making these tough decisions.
An effective board is not just independent. Its also diverse, including the perspectives of
disinterested innovators as well as from such corporate divisions as finance, marketing, and
operations. The link between innovation and finance is often weak in big corporations. As a
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
26/30
AO-26
consequence, finance people will attempt to develop models similar to the one described in
Exhibit PHARMA-EXAMPLE but fail to incorporate insights from staffers most familiar with
the innovation.
Using large populations to support the evaluation process
In addition to this traditional approach towards forecasting success probabilities, one
additional tool deserves discussion:prediction markets. Prediction markets aggregate peoples
opinions using a basic principle of economics. If I believe that an event will occur with a 70
percent probability, I should be willing to pay up to 70 cents for a lottery ticket that pays $1 if
the innovation succeeds and nothing otherwise. If you feel more confident about the success of
the innovation, you should be willing to pay more. Prediction markets let you (and many others)
trade your bets in ways similar to how stocks are traded at Wall Street11. Just as stock prices
mirror the markets aggregate opinion on a companys future cash flow, a prediction market
reveals the participants aggregate opinion on the odds of an innovations success.
Several successful implementations of prediction markets for innovation opportunities have been
reported (links available at www.MasteringInnovation.com). You can trade prediction shares in
nearly anything, ranging from the next pope to the outcomes of elections. But there is little
empirical evidence that these markets really work when it comes to forecasting innovation
success.
Prediction markets certainly represent a step forward for most firms, relative to the typical status
quo of having no system of forecasting. Whether they can replace three-step probability
forecasting discussed above will depend on the specific application. Chapter OI (Open
Innovation) will elaborate more on how one can benefit from the wisdom of the crowds.
Feedback and calibration
One of the reasons that weather men do so well in forecasting the probability of rain (see Exhibit
POS FORECAST1) is that they have many opportunities to practice. But there is more to this
than practice alone. Practice only leads to better forecasting if there exists somefeed-back.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
27/30
AO-27
Doing many weather forecasts alone will not help, doing the forecasts and then contrasting them
with the actual weather, however, can be very powerful.
Why cant we do the same exercise that every new weather-person needs to go through for those
in charge of forecasting the odds that an innovation turns out successful? The idea ofcalibrating
the innovation forecasting process is compelling. However, it requires that the organization
overcomes two obstacles:
Forecasts will never be perfect, and this reality has to be accepted by forecasters andthe managers who evaluate them. Nobody likes to be reminded of yesterdays errors,
so most innovation forecasters prefer to not go on record when making a prediction.
Many companies fail to keep track of their old forecasts. Old forecasts get over-written by new forecasts if they are stored electronically, or they are filed way, never
to be retrieved.
Overcome these obstacles, and your firm can create a graph like the one shown in Exhibit POS-
FORECASTING3, which tracks the forecasting performance of Ely Lilly. Notice that the
probability forecasts by the companys review board are remarkably close to the actual
innovation success rates. With enough discipline and the right inputs, forecasting innovation
odds is possible.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
28/30
AO-28
version 080907
0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
IRB Probability Assessment
ActualSuccessRate
76
73
21
296722
15
12
9
815
6
IRB Performance, All Raw Data
Most of the high sample size
observations lie within a 0.10 band
about the target line
Exhibit POS-FORECASTING3: The graph compares the forecasted probability ofsuccess as expressed by the independent review board (x-axis) with the proportion ofactual innovation successes at Eli Lilly. Each number along the plotted linecorresponds to the number of times a specific forecast was made. For example, a 0.3probability of success was forecasted 8 times
12.
Chapter Summary
The objective of opportunity evaluation is to assign financial values to opportunities and to
identify the types and sources of risk. Where possible, you should aim to create rocket-science
evaluation models.
Horizon 1 opportunities can be evaluated based on discounted cash flows. The biggest threats to
a good evaluation are overly optimistic forecasts of sales and cost overruns. Their development
process should be highly structured and data driven.
Horizon 2 opportunities require an options-based evaluation, as they typically allow the
organization to abandon the opportunity after some bad news has surfaced. The biggest threats to
a good evaluation are overly optimistic POS forecasts and the unwillingness to kill an
opportunity.
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
29/30
AO-29
Horizon 3 opportunities should not be valued financially. Their focus should be learning, and the
company should try to maximize the knowledge per dollar of investment. If possible,
investments should be staged. Staging investments enables you to cash in on the full upside of a
good opportunity, yet limits the downside of a bad one.
Evaluation of opportunities should be done by an independent review board. Explicit data-based
feedback to a board helps it overcome decision biases.
Diagnostics
What measures do you use to evaluate the financial attractiveness of opportunities?Do you distinguish between different horizons?
Do you use measures for the probability of success? How are these probabilitiescomputed?
Does your company keep data on old forecasts with respect to financial returns, salesvolume, and probability of success?
When analyzing the financials of a new opportunity, do you factor in how previousforecasts fared?
Are you able to spot weak opportunities early on or do your opportunities tend tolinger, regardless of their prospects, once they have started development?
Do you evaluate Horizon 3 opportunities with a focus on financial returns orlearning?
Are your review boards independent or might members have conflicts of interestswhen selecting opportunities?
Chapter Notes
1 See the chapter of concept testing in Ulrich and Eppinger. K.T. Ulrich and S. Eppinger, Product Design and
Development,2nd ed. (New York: McGraw-Hill, 2000).
-
7/28/2019 Analyzing+Opportunities+ +AO 27July
30/30
2 See Cachon and Terwiesch for a detailed discussion of how to interpret AF ratios and how to use this information
to make decisions under uncertainty. G. Cachon and C. Terwiesch, Matching Supply with Demand: An Introduction
to Operations Management(New York: McGraw-Hill, 2006).
3 See Ulrich and Eppinger concerning details on how to build a discounted cash flow model for Horizon 1
opportunities. K.T. Ulrich and S. Eppinger, Product Design and Development,2nd ed. (New York: McGraw-Hill,
2000).
4 Source: ADIS International, see Girotra et al for further details. Karan Girotra, Karl Ulrich, and Christian
Terwiesch, Risk Management in New Product Portfolios: A Study of Late Stage Drug Failures, forthcoming in
Management Science.
5 Panel discussion of the POMS college on Product Innovation and Technology Management, Boston 2006.
6 See Loch et al. for more details on managing unknown unknowns as well as on how to create a table similar to
Figure HYPOTHESES 1. Christian Terwiesch and Christoph H. Loch, Measuring the Effectiveness of
Overlapping Development Activities,Management Science Vol. 45, Number 4 (1999): 455-465.
7 Terwiesch and Loch discuss the concept of uncertainty resolution. To measure uncertainty resolution, we have to
map out the amount of residual uncertainty over time. A steep initial decline in the resulting graph corresponds to a
fast uncertainty resolution. Christian Terwiesch and Christoph H. Loch, Measuring the Effectiveness of
Overlapping Development Activities,Management Science Vol. 45, Number 4 (1999): 455-465.
8 Example and data is taken from Murphy and Winkler 1977. A.H. Murphy and R.L. Winklder, Can weather
forecasters formulate reliable probability forecasts of precipitation and temperature?,National Weather Digest2
(1977): 2-9.
9 Example and data is taken from Christensen-Szalanski and Busyhead 1981. Christensen-Szalanski and
Bushyhead, Human Perception and Performance, Journal of Experimental Psychology Vol. 4 (August 7, 1981):928-35.
10 Example and data is taken from Booth et al 2004. B.L. Booth, D.J. Lennon, and E. J. McCafferty, Improving the
pharma research pipeline,McKinsey Quarterly (Web Exclusive 2004).
11 Wolfers and Zitzowitz discuss prediction markets including applications to new product sales forecasting. J.
Wolfers and E. Zitzewitz, Prediction Markets,Journal of Economic Perspectives Vol. 18 Issue 2 (Spring 2004).
12 John S. Andersen, Assessing Technical Feasibility of R&D Projects in Portfolio Management, presented at
INFORMS, Seattle, 1998, TA01.3.