Forecasting Revenues in Ancillary Markets - smawins.com New Market Forecasting 180114.pdftypes of...
Transcript of Forecasting Revenues in Ancillary Markets - smawins.com New Market Forecasting 180114.pdftypes of...
©2018 SMA, Inc. All Rights Reserved.
Forecasting Revenues
in Ancillary Markets
Ajay Patel and Eddie Solares
Technical Report
11 January 2018, V1
Forecasting Revenues in Ancillary Markets 1
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
Summary
Most companies use an expected value formula to forecast revenues from a sales or project opportunity
pipeline. The expected value is typically calculated as the aggregate of the estimated likelihood of the
opportunity being real P(go), times the likelihood of the company winning the opportunity P(win), times
the estimated annual project revenue. This approach is accurate when two conditions are met: 1) when
there are a large number of opportunities in the pipeline and 2) when timing of the project is well known.
However, when companies start to rely on new or ancillary markets for a larger share of the future rev-
enues, these two conditions of the traditional forecasting approach are rarely met for a variety of reasons.
Though new and ancillary markets may be a significant source of future revenues, companies typically
focus on few, large projects to optimize their investment (thus, any single outcome can significantly alter
the results from an expected value), and may not have the necessary insight to realistically estimate the
factors in the expected value formula, especially timing. This leads to the statistics behind the traditional
expected value formula breaking down and giving erroneous results. This could lead to dramatic sur-
prises in financial results from a win or loss of a single large project that was incorporated in the expected
value based forecast. Thus, traditional forecasting techniques are not adequate nor well suited to predict
future revenues for firms pursuing growth in new and ancillary markets. In this study, SMA’s analysis
shows that the traditional expected value approach systematically over-estimates forecasts by as much
as 60% in the near-term, and creates more than a 58% likelihood of a significant negative revenue
surprise.
The study shows that project timing is the largest source of forecast inaccuracy for a typical project pipe-
line. For the illustrative pipeline example used in the study, the small sample size of high value projects
typical of pursuing an ancillary market accounted for approximately 12.5% of the forecast inaccuracy,
whereas uncertainty in project timing is responsible for 50% (the remainder is from the inherent uncer-
tainty modeled by P(go) and P(win)). The study proposes a method to incorporate project timing
uncertainty as a fourth factor P(t) in the expected value formula. The proposed approach can be easily
implemented by the finance department, and as a component to monitoring business development
activities. If it were the case that a project decision and contract award was always as planned, then
within one standard deviation we would expect that the results of the traditional expected value formula
would be equal to the actual results that occur in real life. When you add the uncertainty that the contract
might not be awarded the year it is scheduled, a clearer picture emerges as to why the traditional
approach is not as robust as we would expect it to be. Using a Monte-Carlo method to simulate real life
scenarios, the study shows that by considering realistic delays we over-estimate near-term revenues by
20% to 60% using the traditional expected value formula. The more disconcerting result is that within
one standard deviation, the forecast can be as good as 10% or as bad as 150%. These are not the kind of
results the finance division of a company wants to be dealing with when planning company revenues.
So if timing is an issue how can we address it without completely scrapping the traditional approach?
In the study, we develop a unique approach to the traditional expected value approach where we
introduce a matrix that incorporates a timing delay on a year-by-year basis. The matrix is designed to
only take in one additional timing probability to simplify the approach. Running the same Monte-Carlo
simulation with this added matrix, we get a new expected value formula that improves accuracy by as
much as 57% and reduces the likelihood of any surprise by 30% within one standard deviation. This
added fix to the traditional approach can be modified and tailored to suit historical data and fit multiple
types of timing delays. It also is a relatively easy fix to the traditional approach and does not overly
complicate an already uncertain process that has been known to be difficult to pin down.
Forecasting Revenues in Ancillary Markets 2
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
Contents
1. Introduction ....................................................................................................................................................... 3
2. How Forecasts are Traditionally Calculated .................................................................................................. 4
3. The Challenge of Ancillary Markets ............................................................................................................... 5
4. Study Design Overview .................................................................................................................................... 7
5. Static Pipeline Design for International Opportunities in A&D .................................................................. 8
6. Creating Realistic Futures (i.e. Simulations) ................................................................................................ 13
7. Timing: The Hidden Third Parameter .......................................................................................................... 16
8. Obstacles in Forecasting ................................................................................................................................. 19
9. The New Expected Value Formula ............................................................................................................... 20
10. Expected Value vs. New Expected Value Analysis ................................................................................... 23
11. Conclusions and Recommendations ........................................................................................................... 25
Appendix A .......................................................................................................................................................... 26
Appendix B .......................................................................................................................................................... 27
About the Authors
Ajay Patel is President and CEO of SMA with over 30 years of strategy consulting, business development,
operations, program management, and systems engineering experience. He holds an MBA in Strategic
Planning and Finance from USC and a BS Physics from John Hopkins University. Eddie Solares is a
management consulting analyst at SMA with research experience focused on statistical analysis of large
datasets using programming languages. He holds a MS in Physics from UCLA and a BS in Astrophysics
from UCSC.
Forecasting Revenues in Ancillary Markets 3
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
1. Introduction
Most companies today rely on revenues from ancillary or adjacent markets as a part of their core growth
strategy. Revenues from these pursuits have been typically difficult to forecast, largely because of mis-
understood nuances of those markets especially customer buying behaviors and processes. To a new
entrant, and even established competitors, these markets can appear to have murky decision mecha-
nisms, informal influence networks, and significant uncertainties with customer budgets and needs.
This has been particularly true for project-driven industries pursuing international opportunities, such
as Aerospace and Defense, Engineering and Construction, and public sector consultancies. These pursuits
drive critical decisions on allocation of resources, especially for large projects that the company is relying
on to “make their numbers.”
Our experience working with clients across many industry sectors has been that revenue forecasting in
these markets has been persistently challenging, even for companies that have been in ancillary markets
for many years. The factors that are considered include estimates of the value and scope of the oppor-
tunity, the uncertainty of the project (or purchase) moving forward, the firm’s competitive position and
timing of the award. Traditional approaches to revenue forecasting clearly do not work for these types
of markets. The most widely used approach is an expected value calculation for each opportunity that
discounts the annual revenue estimate by a probability of the project moving forward and a probability
of winning the contract. The reason why this traditional approach fails in ancillary markets is that the
opportunities typically tend to be large in revenue and few in number, thus the uncertainties and out-
come of any individual opportunity can easily affect the overall revenue forecast and financial result.
To compensate for these uncertainties and the possibility of a surprise from a single project outcome
(win or loss), experienced firms typically use a rule-based or ad hoc decision as to whether to include
or not include a specific opportunity in the forecast. Though this approach compensates for overall
uncertainty, it also creates a systemic bias and relies heavily on individual judgement resulting in a
process where the forecasting accuracy is not predictable. See Appendix A for more details.
In this paper, we demonstrate through statistical simulations that the traditional forecasting approach
does not accurately predict realistic scenarios. The traditional approach systematically over-estimates
forecasts by 22–54% in the near-term, and creates more than a 58% likelihood of a significant negative
revenue surprise. We address the shortcomings of the traditional approach with an easily implemented
new expected value formula that improves accuracy by as much as 57% and reduces the likelihood of a
surprise by 30%. The new formula accounts for the following attributes: 1) the probability of the project
being real, 2) the probability of being awarded the project and 3) the probability of the contract being
awarded at the planned start year. We use an example of an aerospace and defense firm pursuing
international projects for illustration; but the approach can be implemented to any ancillary markets or
industry sector. Note that this paper applies only to revenue forecasting from new opportunities, and
does not address revenues from existing backlog.
Forecasting Revenues in Ancillary Markets 4
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
2. How Forecasts are Traditionally Calculated
The traditional approach to revenue forecasting is to use the expected value formula. The expected
value is an anticipated value for a given sales opportunity. The general expected value (EV) formula in
revenue forecasting is given by
EV𝑗[𝑡] = P𝑗 × R𝑗[𝑡] (1)
where P is the probability of opportunity j, and R is revenue of the opportunity at some time t.
Unfortunately, if either the opportunity itself is not certain or the confidence of winning is wrong, then
the forecast can be significantly misrepresented. For this reason, most companies uncouple these two
factors into individual probabilities: the probability of the opportunity occurring and the probability
that the opportunity is won. In this manner, the EV formula then becomes a realistic revenue
forecasting method. The modified formula uses these two probability variables as a compounded
probability:
P𝑗 = P𝑗(go) × P𝑗(win|go = 1)
The probability P𝑗(win|go = 1) is the conditional probability of winning given that the opportunity is real
(i.e. the project or purchase actually proceeds; whereas P𝑗(go) is the independent probability of the
project or purchase occurring. This enables an intuitive approach to estimating the confidence of each
opportunity in the revenue forecast. Taking all the opportunities in a pipeline, the sum of all the EV
revenues gives the total forecasted revenue. Therefore, the total revenue forecast F from an opportunity
j (of a total of n opportunities in the sales pipeline) for a given year i can be written as
F𝑖 = ∑ EV𝑗[𝑡𝑖] = ∑ 𝑛𝑗=1 P𝑗
𝑛𝑗=1 (go) × P𝑗(win|go = 1) × R𝑗[𝑡𝑖] (2)
where EV is the expected value, t is time, P are the probabilities and R is the revenue for the
opportunity. The traditional expected value method is a simple calculation, only requires two
probability estimates, and is intuitive for management and finance departments. However, it is a
statistical approach and requires a large number of opportunities for the statistics to give accurate
results which stems from the mathematical law of large numbers.
Forecasting Revenues in Ancillary Markets 5
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
3. The Challenge of Ancillary Markets
When pursuing sales opportunities in ancillary markets, there are sources of uncertainty that are hard
to manage. This is in comparison to core markets where the company has years of experience under-
standing buying processes and has experiential knowledge that can be applied to make reasonable
judgements on the viability of a project and the firms competitive position to win, namely P(go) and
P(win|go = 1). When pursing and competing in ancillary markets, as in Figure 1 for example, lack of
selling experience, lack of deep customer intimacy and lack of access to influence networks make it
difficult to ascertain project viability and competitive position.
Figure 1: Classification scheme tailored to revenue planning for new market opportunities. Although category E is
included for completeness, it is excluded in study once opportunities are classified.
Category E Dropped
No financial impact within the first five
years
Category BRevenue
Category D Revenue
Category ARevenue
Category C Revenue
Medium Outlook
Good P(go) probability
and decent P(win)
probability
4-5 Year Revenue
Meaningful financial
impact starting on the
fourth year
Good Outlook
High P(go) probability
and good P(win)
probability
High Outlook
High P(go) and
P(win), most likely
won opportunity
Forecasting Revenues in Ancillary Markets 6
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
Furthermore, some markets have inherent uncertainties with regard to customer needs, budgets, project
timing, and other factors that further complicate an already complex situation. This is particularly true
when pursuing international public-sector projects.
As an example, aerospace and defense firms have been pursing growth in international markets for the
past five years as a key complement to their core domestic business. These are typically large projects,
but few in number where winning or losing a handful of contracts can have an appreciable impact on
revenues. Companies in many industry sectors have faced similar obstacles on their path to globalization:
inability to make progress within the formal or imputed buying process, blind spots that arise from
pre-existing biases, influence levers that are obscured or concealed, and general difficulty discerning
“signal from noise” as an outsider looking in.
As these firms rely more on international sales, these obstacles make forecasting revenue with reasonable
certainty difficult. The probabilities P(go) and P(win), the program value and the start year of these
opportunities, are much less certain for international opportunities. On top of that, they are relatively
few in number but each with a much larger potential value creating the potential for significant revenue
surprises from a single loss or win. Therefore, in order to create a realistic pipeline we suggest that
companies include these factors explicitly as a way to categorize and prioritize opportunities in their
sales pipeline. An example screening process of a firm pursing international opportunities as an ancillary
market is illustrated in Figure 1. In this example, five categories to group each individual opportunity
are created to tackle the problem. The categories defined are generically referred to A, B, C, D, and E,
each has distinct attributes of timing, program viability and competitive position. We use this structure
to model traditional and alternate forecasting methodologies. Our models are limited to a five-year
horizon, so opportunities in category E are dropped from this study due to its lack of impact in the
revenue planning horizon.
Forecasting Revenues in Ancillary Markets 7
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
4. Study Design Overview
Our study systematically tests the traditional approaches of forecasting revenue at a detailed level to
gain insight into the sources of error and develop improved techniques. For our example in Figure 1 of
penetrating foreign markets, the following is an outline of our study design:
1. A static sample pipeline is created as a baseline to test the accuracy of different forecasting techniques.
The pipeline consists of 1,052 opportunities (1,000 domestic-market and 52 international-market
opportunities). This sample size choice was to simulate the small number of opportunities in an
ancillary market, compared to the large number of core-market opportunities. For each opportunity,
the P(go) and P(win) probabilities are assigned according to pipeline category shown in Figure 1.
2. The accuracy of the traditional forecasting method is tested by creating 1,000 future scenarios of the
pipeline using a Monte-Carlo technique. Each future scenario is a statistical simulation of each
opportunity going forward as a real program or purchase, and whether the company wins or loses
the competition for the opportunity. These binary outcomes are tested against the expected value
calculation. The difference between the sum of the expected values and the sum of the binary
revenues is a measure of the accuracy of the forecasting technique.
3. We analyze the sources of variability to better understand the accuracy of the traditional forecasting
technique. We demonstrate that most significant source of forecasting errors (and systemic revenue
misses) is with the estimate of the program or purchase planned start year versus actual start year.
4. Finally, an alternative forecasting method is developed to improve forecasting accuracy by consider-
ing a third probability parameter that takes into account the predicted program start year versus the
actual start year of the program.
Forecasting Revenues in Ancillary Markets 8
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
5. Static Pipeline Design for International Opportunities in A&D
The first step in our outline in Section 4 requires us to design the pipeline for our international market
example. A pipeline of new business opportunities is created in order to test the different forecasting
method approaches. A typical pipeline for a public sector contractor that is relying on international
opportunities as part of their core revenues is emulated.
To create a methodical way to categorize new market opportunities, a classification scheme based on
five categories that allows the differentiation and assignment of realistic variables for the probabilities
P(go) and P(win) is created. As stated Section 4, 1000 domestic opportunities (250 in each category
A/B/C/D) and 52 international opportunities (13 in each category A/B/C/D) are created. We randomly
assign a revenue value of $100 million, $500 million, or $750 million for all 1,000 domestic opportunities
giving a total domestic revenue of $212,500 million.
Similarly, revenue values of $500 million, $750 million, or $1,500 million for each of the 52 international
opportunities is assigned at random totaling $41,750 million in international revenues. In this model,
the international pipeline is 16% of the total. The distribution of these revenues can be seen in Figure 2.
These values are kept static throughout the study.
Figure 2: Revenue distribution across 1,000 domestic and 52 international opportunities.
Domestic
International
Opportunity Value, $ million
Number of Opportunities
Total Revenue, $ million
Opportunity Value, $ million
Number of Opportunities
Total Revenue, $ million
$100 750 $75,000
$500 25 $12,500
$500 200 $100,000
$750 15 $11,250
$750 50 $37,500
$1,500 12 $18,000
Totals 1000 $212,500
Totals 52 $41,750
From Figure 2, we can also see that the international opportunities are fewer quantity and larger in rev-
enue value. In general, the number of opportunities in ancillary markets is usually much less than core
markets. This comes with the territory of exploring and penetrating new markets. The larger revenue
reflects how companies prioritize opportunities; they generally seek new opportunities where the reve-
nue is worth the risk of entering ancillary markets.
Now that the revenue distribution is established, start years for each opportunity are assigned in order
to configure our pipeline. As Figure 3 shows, the 5-year pipeline is simulated with each opportunity
having a different start date. The opportunities in categories A, B, and C are apportioned to start in
different years based on a random seed such that 70% of the opportunities will start in 2018, 20% will
start in 2019 and 10% start in 2020.
Figure 3: Start year percentage distribution based on category for each opportunity.
Category Start Year Percentage Distribution
2018 2019 2020 2021 2022
A/B/C 70% 20% 10% 0% 0%
D 0% 0% 0% 70% 30%
For category D, based on our definition of the category in Figure 1, it is seen that this opportunity does
not have a meaningful impact in the first three years so the start year percentages do not begin until the
year 2021. 70% of the opportunities in category D are assigned to start in 2021 and 30% start in 2022. Note
that category E is excluded from the study since by design those opportunities have no significant
Forecasting Revenues in Ancillary Markets 9
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
revenue impact in the first five years. Once start years are assigned to each opportunity, for both domes-
tic and international, they are kept static throughout the course of the study.
The last step to finalize the baseline pipeline is to distribute the recognition of revenue for each oppor-
tunity over the years that the program is executed. We assume that each program is executed over a 5--
year period and has a simple increasing profile defined in Figure 4. The revenue recognized in year 1 is 2/20th of the total, increasing to 6/20th in year 5, resulting in recognizing 100% of the revenue over 5 years.
The increasing program revenue profile is not necessarily typical across all types of programs or pur-
chases, but compensates for the burn-off of backlog resulting in an overall growth forecast for the firm.
Figure 4: Revenue distribution for compensation of burn-off backlog of program opportunities.
Year Revenue Distribution
Year 1 2/20 x Revenue
Year 2 3/20 x Revenue
Year 3 4/20 x Revenue
Year 4 5/20 x Revenue
Year 5 6/20 x Revenue
For visualization purposes, Figure 5 shows a graph of the revenue distribution by category for domestic
and international. Note that the total international revenue is about a quarter of the total domestic
revenue.
Figure 5: Total revenue for both domestic and international category.
By design, an even distribution split among categories is noted. Recall that of the 1,000 domestic oppor-
tunities, 250 were assigned to each category and for the 52 international opportunities, 13 were assigned
to each category. To view the ramp up in revenue the graph of the revenue distribution by domestic and
international we graph it in Figure 6 and by the categories in Figure 7. Note, that in alignment to the
model design, category D does not begin until after 2020.
$0
$50,000
$100,000
$150,000
$200,000
$250,000
Domestic International
Reve
nue
($ m
illio
n)
Domestic and International
A B C D
Forecasting Revenues in Ancillary Markets 10
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
Figure 6: Revenue distribution per year for domestic and international opportunities.
Figure 7: Revenue distribution per year based on category distribution.
Now that the category, revenue, and start year distributed across all five years of our planning horizon
are created, probabilities are assigned for each opportunity. A systematic way of assigning the probability
of the opportunity occurring P(go) and the probability of winning the opportunity P(win) needs to be
created. Recall that in Figure 1, categories with qualitative definitions were classified among opportuni-
ties. In order to quantify the P(go) and P(win), a percentage range is assigned to each category for both
probabilities. The model must also take into account that ancillary markets have much smaller probabil-
ities due to the inherent risks a company takes in doing business in unknown markets.
$0
$10,000
$20,000
$30,000
$40,000
$50,000
$60,000
$70,000
$80,000
$90,000
2018 2019 2020 2021 2022
Reve
nue
($ m
illio
n)
Domestic International
$0
$10,000
$20,000
$30,000
$40,000
$50,000
$60,000
$70,000
$80,000
$90,000
2018 2019 2020 2021 2022
Reve
nue
($M
M)
A B C D
Forecasting Revenues in Ancillary Markets 11
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
To assign probabilities based on our category distribution, probability ranges are assigned according to
Figure 8. These probabilities are connected to the attributes of each pipeline category as defined in Figure
1. A random sampling based on these ranges is done and assigned a probability within the range. Con-
sistent with how the parameters are established, the probabilities of P(go) and P(win) are kept static
throughout the duration of the study.
Figure 8: Probability ranges for each category based on type of opportunity and category of opportunity. Category E
opportunities are removed from study but included for completeness.
Category Probability (Go) Probability (Win)
Do
mesti
c
A 90% – 100% 80% – 90%
B 80% – 100% 70% – 80%
C 70% – 100% 50% – 70%
D 50% – 100% 50% – 70%
E < 50% < 50%
Inte
rna
tio
nal A 70% – 100% 80% – 90%
B 60% – 100% 70% – 80%
C 50% – 100% 50% – 70%
D 30% – 100% 50% – 70%
E < 30% < 50%
Figure 9 column headings reflect all static variables that have been created so far in our pipeline. The
expected value is calculated in Figure 9 using the formula EV = P(go) × P(win) × Revenue. This is done
for all 1,000 domestic opportunities and all 52 international opportunities. These values are all kept static
for the remainder of the simulation in order to keep the relative comparisons free of any statistical bias.
Note that the expected value is less than the total pipeline, which makes sense since not all opportunities
will result in revenue. The reader is reminded that the aggregate sum of all expected values of the op-
portunities should equal the realistic revenue stream given a large enough sample size.
Figure 10 graphs the expected revenue from all opportunities over our 5-year planning horizon. As
described above, this is the aggregate of the expected revenue from each opportunity estimated by
multiplying the recognized revenue in each year by the two probabilities. Now that the static pipeline
is fixed, statistical simulations can be conducted and results tested to see whether or not the expected
value is an accurate measure for forecasting revenue. Forecasted revenue will be used interchangeably
with the expected value (EV).
Figure 9: Sample of static pipeline created showing opportunity number, whether domestic D or international I, cate-
gory, planned start year, two probabilities assigned to them, total opportunity revenue and expected value (EV) calcu-
lated from traditional formula.
Opportunity Dom/Int Category Year P(go) P(win) Revenue $ million EV, $ million
1001 D A 2018 90.52% 83.52% $100 $75.61
1002 D A 2018 99.19% 82.45% $500 $408.92
1003 D A 2018 94.45% 88.53% $500 $418.12
1004 D A 2018 91.32% 87.78% $500 $400.82
1005 D A 2018 95.72% 83.73% $100 $80.15
1006 D A 2019 91.86% 81.50% $500 $374.33
1007 D A 2019 94.44% 84.01% $100 $79.33
1008 D A 2020 99.84% 88.56% $100 $88.42
… … … … … … … …
Forecasting Revenues in Ancillary Markets 12
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
Figure 10: Sum of EV throughout five years for both international and domestic.
$0
$5,000
$10,000
$15,000
$20,000
$25,000
$30,000
$35,000
$40,000
$45,000
$50,000
2018 2019 2020 2021 2022
Reve
nue
($ m
illio
n)
Domestic International
Forecasting Revenues in Ancillary Markets 13
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
6. Creating Realistic Futures (i.e. Simulations)
Moving on to Step 2 of the study approach (see Section 4), testing of the traditional forecasting method-
ology is done to see how well the expectation value formula predicts future revenues. Future scenarios
are created by sampling from the probability distributions defined in the previous section; each scenario
represents the collection of outcomes of all opportunities determined from the sampling. For each op-
portunity, a random number is generated representing the outcome for the variable P(go) to determine
the outcome pertinent to whether the project will proceed or not. If the random number is equal to or is
less than the defined P(go), then the project will be awarded (i.e. it is assigned a value of 1) and if is
greater than P(go) then the project was not real (i.e. it is assigned a value of 0). We do the same for P(win).
To avoid any statistical correlations, different random samples for each probability are used. As an exam-
ple, if a random sample of s = 0.78 is generated and compared to a probability of P(go) = 0.92, it is seen
see that 0.78 < 0.92 and thus falls within the probability, so a binary value of B(go) = 1 is assigned. If,
however, a random sample of s = 0.96 is generated then it is greater that P(go) = 0.92 and thus assigned
the binary value of B(go) = 0. The actual scenario value (SV) is then calculated by SV = B(go) × B(win) ×
Revenue. This is done for all 1,000 domestic opportunities and all 52 opportunities.
From Figure 11, it is seen that if either or both opportunities get a binary value of 0, the entire scenario
value is $0.00. This scenario value simulates an actual real-life scenario where the opportunity is either
won or lost.
Figure 11: Pipeline with random sample drawn which assigns binary 0 or 1 to B(go) and B(win) to calculate Scenario
Value (SV).
Opportunity Dom/Int Category Year P(go) P(win) B(go) B(win) Scenario Value, $ million
1245 D A 2018 90.27% 89.63% 1 1 $500
1246 D A 2018 96.59% 87.38% 1 1 $100
1247 D A 2019 90.77% 86.47% 1 1 $500
1248 D A 2018 98.09% 88.52% 1 1 $750
1249 D A 2019 93.50% 80.41% 1 1 $100
1250 D A 2018 98.91% 84.69% 1 0 $0
1251 D B 2018 91.35% 74.05% 1 1 $100
1252 D B 2018 82.11% 74.91% 0 1 $0
1253 D B 2019 97.67% 77.04% 1 1 $100
1254 D B 2018 91.45% 79.65% 1 0 $0
1255 D B 2018 85.74% 70.23% 1 0 $0
1256 D B 2018 99.29% 77.34% 1 1 $100
… … … … … … … … …
From the law of large numbers, given a large enough sample size, it is expected that the sum of all the
expected values (EV) equals the sum of all scenario values (SV), that is SV – EV = 0. The sum of all of the
EV and the sum of all SV in our pipeline is taken and the difference between them for each of the five
years give us our percent difference. That is, the percent difference can be written as
% Diff = Σ(SV − EV) = ∑[B(go) × B(win) - P(go) × P(win)] × R (3)
where R is the revenue. This is done for both the domestic and the international opportunities. This pro-
cess counts as one simulation that produces one value for the percent difference. In order to generate a
standard deviation and determine the accuracy of our results 1,000 simulations are done.
Forecasting Revenues in Ancillary Markets 14
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
Once the 1,000 runs are completed, the average of these is taken along with the corresponding standard
deviation of each. The mean value is calculated and an error bar of one standard deviation is assigned
for each of the five. The process for running one of these simulations can be quite long so the Python
programming language is used to quickly run 1,000 of these simulations.1
Running and plotting the percent differences between the expected value (EV) and the scenario value
(SV) for all 1,000 runs and taking the average percent difference gives us the results of Figure 12. Fore-
cast for core domestic markets is spot on and there is very little mean error between the differences. This
means that SV = EV on average and there is very little variation within the error bars. The standard
deviation, or error bar, is less than 5% for the domestic market and is relatively small which reflects the
high number of opportunities.
Figure 12: Percent difference between the expected value (EV) and the scenario value (SV) with one standard deviation
as the error bar.
From a mathematical standpoint, this reflects the law of large numbers since there is a large sample size
of 1,000 opportunities the results tend to not deviate as much since the EV approaches SV as the sample
size nears infinity. The mean percent difference for the international ancillary market is also relatively
small. It is still slightly larger than for the domestic opportunities, but the difference is within 5% points
of the 0% difference. However, the error bar is considerably larger than the domestic one with values
greater than 15%. This is reflective of the sample size since there are fewer opportunities, 52 to be exact,
and each individual opportunity has a more significant impact on the revenue. A point should be made
to state that the size of the error bar is what indicates whether the traditional approach has flaws—
namely, if any single future scenario could result in a significant revenue forecast miss. This is mostly
1 Programming languages are useful to do routine calculations in a quick and efficient way. Since the simulation is
run 1,000 times, it is much easier for a computer to do this job as opposed to manually recalculating the results
one by one. High-level programming languages are useful for these types of calculations as they are designed for
general purpose programming. The programming language used in this study is Python, but there are other high-
level languages such as C++ and Java that can similarly be used.
-0.3
-0.25
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
2017 2018 2019 2020 2021 2022 2023
Perc
ent
Diff
ere
nce (1
-E
V/S
)
Domestic International Combined
Forecasting Revenues in Ancillary Markets 15
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
driven by the wide range of possible outcomes for smaller number but larger in value of the international
opportunities. However, from these results two things can be incorrectly assumed:
1. The traditional expected value represented in Equation 1 is a good way to forecast revenue;
2. Having a larger sample size reduces the standard deviation of your results.
We will next show that the first assumption is incorrect since there is a critical hidden parameter that is
assumed to be true that is not considered. We will also address the second assumption in our recommen-
dations for an improved forecasting approach. It will be seen that the true error bars of the traditional
forecasting method defined by Equation 1 will be large and the flaws of these results will be evident once
the hidden parameter is introduced.
Forecasting Revenues in Ancillary Markets 16
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
7. Timing: The Hidden Third Parameter
The results for the traditional expected value seem acceptable notwithstanding the possible wide range
of outcomes, but is missing an important aspect of forecasting. The timing of the opportunity is critical
to forecasting revenues and is the additional hidden parameter that failed to be taken into account. This
source of uncertainty will now be considered in the next step of the analysis (see Section 4). In the prior
analysis, it was assumed that the planned start year was the actual start year of the opportunity. From
experience, this is not realistic especially in public sector driven markets and in ancillary markets where
there is less budgetary and decision transparency. It is also known that forecasters have a significant
systemic bias for optimism. There is a tendency for companies to over-estimate their revenue in fore-
casting models for new markets and timing plays an important factor. Thus, it is crucial to estimate the
probability for the planned start year to be the actual start year. This introduces a new statistical param-
eter when running the future scenarios that was not considered before.
The question now becomes “how is uncertainty in timing simulated?” when running the future scenarios.
For each opportunity, when the dice is rolled for each future scenario value, not only is the probability
of whether or not the program happens and if it is won simulated but also when it occurs. To do this a
probability is associated with each start year and the remaining probabilities are pushed out to later
years. That is, there is a certain probability it starts in the planned year and if it does not fall under this
probability the project gets pushed out a year or two down the line. Using the timing probability matrix
in Figure 13, probabilities are assigned for each of the five years of forecasting revenues. From the
matrix, it is seen that for domestic opportunities with a planned year of 2021 there is a 65% chance it
actually starts in 2021, a 20% chance that it starts in 2022, and a 15% chance that it occurs in 2023 or
later. Since only five years revenue is considered in the forecasting horizon, this means that there is a
15% chance that this opportunity is not included in the model since it is pushed out past our five-year
range. This is similar to treatment of category E opportunities that do not have a financial impact within
the immediate five years that the revenue is forecasted.
Figure 13: Probability matrix of planned year versus actual start year.
Planned Year Probability Domestic Actual Year
2018 2019 2020 2021 2022 ≥ 2023
2018 80% 80% 15% 5%
2019 75% 75% 20% 5%
2020 70% 70% 20% 10%
2021 65% 65% 20% 15%
2022 60% 60% 40%
Planned Year Probability International Actual Year
2018 2019 2020 2021 2022 ≥ 2023
2018 80% 60% 25% 15%
2019 75% 55% 20% 10% 5% 1000%
2020 70% 50% 15% 5% 30%
2021 65% 45% 10% 45%
2022 60% 40% 60%
For this model, inferences were made based on reasonable assumptions from core and ancillary markets.
For core domestic markets, businesses are usually well informed on the timing and scope of the oppor-
tunity start dates and so high probabilities are assigned for the planned year. However, as the program
lies further out in the timeline, there tends to be less certainty as the opportunity may not be mature
Forecasting Revenues in Ancillary Markets 17
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
enough to have a well-planned start year. For international ancillary markets, there is less certainty in the
assumption that the planned start year is the actual start year. Whether that is an attribute of the market
or lack of intimacy and knowledge of the market, higher uncertainty is generally the case. Similar to the
domestic case, the probability for international opportunities also degrades as planned years moves
further out in the timeline. It is noted that since international opportunities planned start years are less
certain than domestic ones, more international opportunities than domestic opportunities will pushed
out beyond the five-year scope of our revenue forecast.
To provide some backup to these numbers, a study done on 18 major defense acquisition programs by
SMA looked into the delay for both the requests for proposals and the program awards in recent
contracts.2 For the Request for Proposal (RFP) delays, there was a 78% probability it was awarded the
same year it was planned, 17% within two years, and 6% within the three years.
Similarly, for the program award delays, there was a 72% probability it was awarded within same year
it was planned, 17% within the two years, 6% within three years, and 6% within four years. Taking the
average of these delays gives roughly 75% probability that there is a delay within the year it was planned,
17% chance of delay within two years, 6% chance of delay with three years, and a 3% chance of delay
within four years. These represent domestic program awards within the United States and the proba-
bilities assigned to the probability matrix in Figure 13 are roughly within the same order of magnitude
for the domestic opportunities. This is a good indicator that the numbers used are accurate representa-
tions of opportunity awards in core domestic markets. For international markets, it is well known these
probabilities are less certain, so probabilities assigned are smaller than their domestic counterparts.3
When running the scenario simulation, similar calculations as before are done. The reader is reminded
that the static values in the pipeline did not change. The only change is that this iteration of the 1,000
scenarios is run with the added probability that the planned year is the actual start year. The percent
difference formula is thus modified as follows
% Diff = ∑[B(𝑡) × B(go) × B(win) - P(go) × P(win)] × R (4)
where B(𝑡) is the binary value assigned depending on whether or not the planned year is the start year.
This value is assigned similarly to the other binary values using the probability values from Figure 13.
Using Equation 4, the results are graphed and the percent differences are shown in Figure 14. There are
some dramatic differences from this graph than that of Figure 12. The only similarity between them is
that the mean percentage difference is less for domestic opportunities than the international ones and
that as the years progress they get closer to 0% difference with smaller standard deviations.
2 The study done was a SMA funded project that analyzed the request for proposal (RFP) and award dates of
major defense acquisition programs (MDAP). 3 “Analysis of Major Defense Acquisition Programs (MDAP) Award Delays.” SMA,
www.smawins.com/Content/Files/SMA%20MDAP%20Award%20Delays%20171211.pdf.
Forecasting Revenues in Ancillary Markets 18
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
Figure 14: Calculation of percent difference between expected value (EV) and scenario value (SV) with additional prob-
ability of timing delays added to scenario calculations.
The percentage differences for both the domestic and international opportunities do not lie close to 0%
difference. This implies that there is a systemic bias (over-estimation) of approximately 20% to 60% in
the near-term forecast. In 2018 for the domestic case, the percentage difference is bigger than 20%. In the
latter years, one can see that the percentage difference starts to converge close to the 0% axis meaning
that the expected value results become closer to reality. The error bars on the domestic cases for the
standard deviation are relatively small so there is little variability as before.
As for the international case, there is a large percent difference of about 80% in 2018 with the years con-
verging to approximately 18% in the later years. The more disconcerting result is that within one stan-
dard deviation a large percentage difference of up to 150% may be possible. Putting this into context,
the revenue forecast can be as low as 10% or as high as 150%.
This is clearly a dramatically erratic result and tells us that timing is highly important when considering
forecasting revenues. Across the board, it is known that many companies have international revenue
forecasting models that give drastically inaccurate results. From doing these simulations, it is seen how
timing of the program start year is an attribute to factor into the expected value to develop a more
accurate forecast.
-1.8
-1.6
-1.4
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
2017 2018 2019 2020 2021 2022 2023
Perc
ent
Diff
ere
nce (1
-E
V/S
)
Domestic International Combined
Forecasting Revenues in Ancillary Markets 19
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
8. Obstacles in Forecasting
When entering new markets there are three underlying issues that create obstacles to forecast future
revenues accurately:
1. Small Sample Sizes: The relatively small number of opportunities in new or ancillary markets is inherent
with the pursuit of growth in these markets. Small sample sizes do not lend themselves to simple
statistical methods that can be easily and intuitively incorporated in revenue forecasting.
2. Large Revenues: When entering a new or ancillary market, firms typically focus on projects with
relatively large revenues to balance the pay-off against investment risk. These pursuits require
significant investment of resources especially since the firm is not as familiar with the market and
may be in a less advantageous competitive position. The large project revenue means that each
opportunity is critical to the total forecasted revenue, thus outcomes can vary significantly on a
single win or loss.
3. Timing: One of the most significant sources of uncertainty is timing. This is particularly true in public
sector markets where projects are not necessarily driven by an urgent competitive situation and sub-
ject to government budgets and lengthy approvals. As seen from the results of Section 7, including a
probability estimate for the planned start year in our testing creates drastically different results in
estimating forecast accuracy.
Each of these can have a significant impact on forecasting revenue techniques, specifically the expected
value formula. For the first case, there exist statistical methods to handle small sample sizes. However,
these statistics are difficult to implement as part of a routine business planning function and are non-
intuitive. Secondly, being more selective about which opportunities to include or exclude from the fore-
cast based on the static values assigned to the opportunity such as revenue and probabilities can be done.
However, eliminating too many of these opportunities limits the sample size and introduces the problem
inherent in the first case. See Appendix A for more details.
Also, selectively excluding opportunities introduces additional biases since it involves non-Bayesian
estimation and is largely done either ad-hoc or with arbitrary rules. The third issue can be effectively
addressed by modifying the traditional expected value formula with a probability estimate associated
with the planned start date. This approach also helps mitigate the first two issues by help push out the
revenues even more than the traditional expected value. Introducing an estimate of the probability that
the actual start date is the planned start date is consistent with the Bayesian approach of the traditional
forecasting methodology. The probability estimates can be informed and validated from historical data
in the individual markets. Although each approach can improve the predictive capability of the model,
our research indicates that addressing the timing issue with a new probability can significantly improve
forecasts and is a simple enhancement to revenue planning process being used today at most firms. Our
analysis also shows that small sample size statistics typically associated with pursuits in ancillary markets
only account for a small percentage of forecast inaccuracy when compared to addressing the timing issue.
See Appendix B for more details.
Forecasting Revenues in Ancillary Markets 20
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
9. The New Expected Value Formula
The introduction of an estimate of project timing uncertainty (i.e. the probability that the actual start year
is the planned year) will be added to improve the traditional expected value-based forecasting approach.
This requires the need to define a matrix that distributes the expected value based on probabilities of
planned start year versus actual start year. This can be simplified by using a single probability value
from which subsequent values in the matrix are derived.
Since the total revenue of the opportunity is already distributed throughout the five years, the probability
associated with timing to each of these five years is assigned and revenues are pushed with into the
later years based off the probabilities. This creates the matrix redistribution of revenue that is used to
create the New Expected Value (NEV) formula. Modifying the expected value formula from Equation 2,
the formula is written as
NEV[𝑡𝑖] = ∑ ( P(go) × P(win) × R[𝑡𝑖] × Mik[P(time), 𝑡𝑖])5𝑘=1 (5)
where 𝑖 is the planned year, 𝑘 is the actual year, P(time) is the probability that the planned year is the
start year and M[P(time), t ] is the matrix associated with the timing of the opportunity. This formula
redistributes out the revenue throughout the five years based on the individual timing probability.
Equation 5 is abstract so an example will be shown to illustrate the type of matrix used in the new ex-
pected value formula. Figure 15shows an example of how revenue for a project is distributed throughout
each year (for both the actual start year and the planned start year). This demonstrates the how the
expected value (EV) and the new expected value (NEV) differ. The yellow shaded section represents the
matrix M and the probability associated with it.
Figure 15: Example pipeline of how revenue is distributed based on expected start year 2018.
Opportunity (j) Actual Start Year (i) Total
Planned Year (k)* 2018 2019 2020 2021 2022 >2023
Rev Potential [$100M] [$150M] [$150M] [$150M] [$125M] [$0M] [$675M]
P(go) [90%] 90%
P(win|go = 1) [90%] 90%
2018 P(t)
[80%: $65M] (1 – P(t))*0.8 [16%: $13M]
(1 – P(t))*0.2 [4%: $3M]
[$81M]
2019
P(t) [80%: $97M]
(1 – P(t))*0.8 [16%: $19M]
(1 – P(t))*0.2 [4%: $5M]
[$123M]
2020
P(t) [80%: $97M]
(1 – P(t))*0.8 [16%: $19M]
(1 – P(t))*0.2 [4%: $5M]
[$123M]
2021
P(t) [80%: $97M]
(1 -P(t))*0.8 [16%: $19M]
(1 – P(t))*0.2 [4%: $5M]
[$119M]
2022
P(t) [80%: $81M]
(1 – P(t))*0.8 [20%: $20M]
[$101M]
[NEV] [$65M] [$110M] [$119M] [$121M] [$105M] [$25M] [$547M]
[EV] [$81M] [$122M] [$122M] [$122M] [$101M] [$0M] [$547M]
[X] example calculations
* For each planned start year, the matrix is displaced by one year down and one year to the right
This entire figure represents some opportunity 𝑗 that has a planned start year of 2018. The rows represent
the planned years and the columns represent the actual start year where P(𝑡) is the probability that the
planned year is the start year. Looking at the row for planned year 2018, it is seen that it has the values
P(𝑡), (1 − P(𝑡)) × 0.8, and (1 − P(𝑡)) × 0.2 for 2018, 2019, and 2020 respectively. When the terms are
added, the resulting total is 1 indicating that the total revenue assigned to planned year 2018 is distrib-
uted out three years.
Forecasting Revenues in Ancillary Markets 21
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
Similarly, this occurs for all the years and some of the revenue is distributed to 2023 and beyond which
is not included in our total five year forecasting range. Starting from the top, the revenue for the oppor-
tunity, the two probabilities, and the matrix columns of the revenue are pushed out from the timing ma-
trix. If the planned year is the expected year, that is P(𝑡) = 1, then only the diagonal entries of the matrix
would remain giving the traditional EV. Also, if the opportunity does not start for another n years the
entries would all be displaced n years down and n years to the right.
The matrix shows that the new expected value is lower in revenue in earlier years. This is expected since
a probability is introduced that delays the expected start year versus the actual start year resulting in
revenues being pushed out to later years. The last column in the table provides the total revenue for all
the years. Although the sums at the bottom right are equal, the NEV has $25 million allocated to 2023
and later years, which are outside the range of the five-year forecast. Another attribute to note is that
the matrix is designed to only assume one probability P(𝑡) and the probability was used to create a dis-
crete drop in revenue across the entire matrix. This makes it so that a company only needs to assign one
additional probability rather than design an entire matrix for the NEV. The delay in timing is inherently
embedded in this matrix and is easy to implement for companies developing revenue forecasting in
ancillary markets. The discrete drops in delay are simple, but more complicated and possibly continuous
model designs can fit actual historical data or known timing delays.
As done before, the 1,000 simulations are run once again and the percent difference is plotted using the
modified version of the percent difference formula given as
% Diff = ∑[B(t) × B(go) × B(win) − NEV] (6)
where NEV was given by Equation 5. Running the simulations using Equation 6, the results are shown
in Figure 16. Similar to Figure 14, the domestic percent difference is more accurate than the
international percent difference.
Figure 16: Percent difference between NEV and scenario value.
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
2017 2018 2019 2020 2021 2022 2023
Per
cen
t D
iffe
ren
ce (
1 -
NEV
/S)
Domestic International Combined
Forecasting Revenues in Ancillary Markets 22
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
The domestic percent difference also lies close to the zero percent difference line with much smaller error
bars than the international case. Again, this is because the international case has a smaller sample size
and larger revenue for the opportunities, and thus every opportunity has a significant impact on the
total forecasted revenue. A single opportunity—whether it proceeds, whether the firm wins it and when
the project starts—can drive the company’s financial results. Note that the revenue values for each year
are pushed out as before and one sees a convergence toward a value close to the 0% difference axis. As
the years pass, the forecasted revenue gets closer to the actual value. Another thing to note is the small
bump at the year 2021, which is because category D opportunities have no financial impact until year 4,
which is inherent, our study design. The start year 2018 for international opportunities has a percent
difference that is much smaller than the traditional EV results in Figure 14, suggesting the results for the
NEV formula revenue forecasting are more accurate.
Forecasting Revenues in Ancillary Markets 23
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
10. Expected Value vs. New Expected Value Analysis
The goal of any forecast model is to predict accurately future revenues, with accuracy measured by both
average error and the standard deviation of the error when compared to simulated outcomes. For this
study, this means that the percent difference mean is as close to zero as possible with a small standard
deviation represented by the error bar. Figure 17 shows the quantitative results of the EV and the NEV
percent differences when run against the scenario values. All five years are plotted with the percent
difference and plus/minus the standard deviation. Looking at the domestic values, for the EV the results
tend to converge to around the –4% percent difference meanwhile the new expected value it seems to
converge around the 0% difference.
Figure 17: Final Comparison for all five years for both domestic and international opportunities in both traditional
expected value (EV) and new expected value (NEV).
2018 2019 2020 2021 2022
EV Domestic –25 ± 7% –10 ± 4% –4 ± 4% –6 ± 4% –4 ± 3%
NEV Domestic –10 ± 6% –4 ± 4% –2 ± 4% –4 ± 4% 2 ± 3%
EV International –82 ± 71% –34 ± 31% –16 ± 20% –17 ± 19% –16 ± 18%
NEV International –36 ± 54% –17 ± 26% –10 ± 19% –13 ± 18% –8 ± 17%
EV Combined –54 ± 58% –22 ± 25% –10 ± 16% –11 ± 15% –10 ± 14%
NEV Combined –23 ± 41% –10 ± 20% –6 ± 14% –8 ± 14% –3 ± 13%
However, for the year 2018 the EV is around –25 ± 7% meanwhile the NEV sits at –10 ± 6%. This means
that the percent difference for the EV is less accurate than the percent difference of the NEV. This shows
that the NEV gives more accurate forecasts for the domestic case with the NEV method when compared
to the traditional EV. This is a good indication of a better formula but a check of the international case
shows a much clearer picture.
For planning international revenues, the traditional EV converges to about a difference of –15%. The
NEV converges to a difference of –8%. By 2022, the error bars are about equal but the NEV has about
half the percent difference than the EV indicating the NEV converges to a more accurate forecast
revenue. Furthermore, looking at the year 2018 the traditional EV has a difference of about –82 ± 71%
while the NEV has a difference of about –35 ± 54%. The forecasted revenue for the EV in the year 2018
is erratic with a range of –11% to –153% difference within one standard deviation. For the NEV there is
a range of +19% to –89% difference within one standard deviation. This range indicates that for the
NEV at worst there is an average percent difference of the EV, which is –82%, and at best there is an
underestimate of +19%. The result shows that there is a higher probability that the NEV will accurately
forecast the revenue and achieve a 0% difference within one standard deviation. The results of Figure
17 are graphed for visual comparison in Figure 18, Figure 19 and Figure 20. These results show that
using the NEV provides a significant increase in accuracy for simulated events. The NEV would only
require one additional probability estimate (associated with the project timing) and is a simple improve-
ment to the traditional EV. The model used employs a simple matrix for our percentage distribution of
timing. An examination of historical data on project delays for the firm’s specific markets can be used
to create a more precise and refined matrix to provide better results than the simulations conducted in
this study.
Forecasting Revenues in Ancillary Markets 24
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
Figure 18: Comparison of traditional expected values (EV) and the new expected value (NEV) for domestic.
Figure 19: Comparison of traditional expected values (EV) and the new expected value (NEV) for international.
Figure 20: Comparison of traditional expected values (EV) and the new expected value (NEV) for domestic and
international combined.
-0.35
-0.3
-0.25
-0.2
-0.15
-0.1
-0.05
0
0.05
0.1
2017 2018 2019 2020 2021 2022 2023
Perc
ent
Diff
ere
nce (1
-E
V/S
)
NEV EV
-1.8
-1.6
-1.4
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
2017 2018 2019 2020 2021 2022 2023
Perc
ent
Diff
ere
nce (1
-E
V/S
)
NEV EV
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
2017 2018 2019 2020 2021 2022 2023
Perc
ent
Diff
ere
nce (1
-E
V/S
)
NEV EV
Forecasting Revenues in Ancillary Markets 25
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
11. Conclusions and Recommendations
When running through the simulations it was discovered that the new expected value formula had a
significantly better forecasting accuracy than the traditional expected value. The new expected value
formula gives results closer to the simulated value and thus represents a better forecasting method.
This paper used an example of an aerospace and defense firm pursuing growth through international
projects. However, the recommended enhanced forecasting method is not necessarily limited to
aerospace and defense or similar public-sector firms, but extends to any business trying to penetrate
ancillary markets characterized by fewer but larger value projects. The proposed method described in
this paper can be further improved by incorporating actual project delay trends in specific markets. The
delays are typically as much a result of government processes and governance as the unique situation
of the individual project, thus enabling us to predict likely project delays. Unique knowledge can be
used to further inform the likelihood of a project delay, similar to the experiential estimates of the
probabilities P(go) and P(win) by the project leadership.
A company’s revenue forecasting approach should consider 1) the probability of the project being
awarded (i.e. is it real?), 2) the probability of winning and 3) the probability of the contract being
awarded at the planned start year. By considering this last source of uncertainty, companies can
improve their forecasts by 57% and avoid the possibility of a negative surprise by as much as 30%.
Tailoring the approach described in this paper to trends in specific markets can further improve
forecast accuracy and provide key market insights.
Forecasting Revenues in Ancillary Markets 26
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
Appendix A
A common approach to avoid strategic surprises and moderate the forecast is to exclude certain oppor-
tunities from the forecast such as those with very large potential values but low to medium probabilities
of win or occurrence. For example, a “billion dollar class” opportunity for a firm that is several hundred
million or less in annual revenues can easily skew the firm’s near term financial forecast. We use a
heuristic rule to test whether or not selectively eliminating specific opportunities from a forecast helps
increase forecasting accuracy. The procedures for defining the pipeline and conducting scenarios are
the same used in this paper following Section 7 with the timing parameter added. To test the forecast
accuracy of selectively excluding certain opportunities, we define a new probability called the total
probability as P(total) = P(win) x P(go). This probability is a representative proxy for an ad hoc rule that
is typically used by companies as a threshold. Setting a minimum total probability and excluding any
opportunity that falls below this threshold is a way of removing opportunities that the company has
little confidence of regardless of the value. The expectation is that by eliminating these opportunities,
we improve the forecast accuracy since low-confidence large-value projects will not swing the results.
The percentage difference defined by Equation 4 was determined for different thresholds and compared
to our enhanced approach in Figure 19. With a P(total) of 20% and a 30% it is seen that the difference in
results are negligible (within one standard deviation). However, when the threshold is increased to 40%
and above (that is only include medium and high-confidence opportunities in the revenue forecast, using
an ad-hoc selective threshold reduces the forecast accuracy significantly. This is because the international
opportunities have a small total sample size and eliminating more of these in the forecast increases the
susceptibility of the total revenue estimate to swing dramatically with a single project win or loss.
Figure 21: Percent difference calculation based on P(total) thresholds.
A threshold of P(total) of 40% removes nearly 50% of the total international opportunities thereby
reducing the of a Bayesian approach to forecasting (expected value statistics). This method of
systemically biasing forecasts can be done with a variety of heuristic rules including rules based on
P(win), P(go) and value or any combination of these. All the results lead to the same conclusion that
selectively removing opportunities to an already small sample of opportunities leads to poor statistics
and thus introduces additional inaccuracies in revenue forecasting.
-300.00%
-250.00%
-200.00%
-150.00%
-100.00%
-50.00%
0.00%
50.00%
2017 2018 2019 2020 2021 2022 2023
Per
cen
t D
iffe
ren
ce (
1 -
EV/S
V)
No Threshold P(total) = 20% P(total) = 30%
P(total) = 40% P(total) = 50% P(total) = 60%
Forecasting Revenues in Ancillary Markets 27
Technical Report 11 January 2018, V1
©2018 SMA, Inc. All Rights Reserved.
Appendix B
The relatively small number of opportunities in the pipeline for ancillary markets contributes to forecast
inaccuracy, and perhaps more significantly the likelihood of a surprise since these opportunities are
typically of large value. To better understand how much small size statistics contributes to forecast
inaccuracy compared to timing uncertainty, we conducted simulations on our illustrative pipeline with
twice the number of international opportunities (i.e. 104 opportunities instead of 52, with a total value
of $83.5B instead of $41.75B compared to the domestic pipeline of 1,000 opportunities valued at $212.5B).
Figure 20 compares the forecast accuracy of the traditional approach (EV) with the proposed new
approach (NEV) and the simulated pipeline with twice the number of international opportunities (EV
Double). We see that the larger sample size only slightly improves forecast accuracy by less than 10%
compared to the traditional expected value formula (EV), whereas considering timing uncertainty im-
proves forecast accuracy by a factor of two in the first year. The small sample size of high value projects
typical of pursuing an ancillary market accounted for approximately 12.5% of the forecast inaccuracy,
whereas uncertainty in project timing is responsible for 50%. This indicates that our proposed new
expected value formula addresses the most significant source of forecast errors.
Figure 22: Standard expected value (EV) compared to doubling of the sample size (EV double) to the new expected
value (NEV).
-1.8
-1.6
-1.4
-1.2
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
2017 2018 2019 2020 2021 2022 2023
Perc
ent
Diff
ere
nce (1
-E
V/S
)
NEV EV EV Double