NATO INDUSTRIAL ADVISORY GROUP (NIAG) LIFE CYCLE ...2018)0… · nato industrial advisory group...
161
NATO UNCLASSIFIED Releasable to PFP 22 May 2018 DOCUMENT NIAG-D(2018)0013 AC/327-D(2018)0005 NATO UNCLASSIFIED -1- NATO INDUSTRIAL ADVISORY GROUP (NIAG) LIFE CYCLE MANAGEMENT GROUP (LCMG) (AC/327) FINAL REPORT OF NIAG STUDY GROUP 218 - NIAG STUDY ON CONCEPTS AND RATIONALE FOR CONTRACTING FOR LOGISTICS CAPABILITY IN NATO ARMAMENTS SUPPORT PROGRAMMES Note by the NIAG Secretary 1. Enclosed is the Final Report on Concepts and Rationale for Contracting for Logistics Capability in NATO Armaments Support Programmes, as conducted by NIAG SG.218, which is now published to the sponsor. (signed) Nathalie Van Donghen 1 Enclosure Original: English NHQD99273
Transcript of NATO INDUSTRIAL ADVISORY GROUP (NIAG) LIFE CYCLE ...2018)0… · nato industrial advisory group...
NATO UNCLASSIFIED Releasable to PFP
22 May 2018 DOCUMENT NIAG-D(2018)0013
AC/327-D(2018)0005
NATO UNCLASSIFIED -1-
NATO INDUSTRIAL ADVISORY GROUP (NIAG)
LIFE CYCLE MANAGEMENT GROUP (LCMG) (AC/327)
FINAL REPORT OF NIAG STUDY GROUP 218 -
NIAG STUDY ON CONCEPTS AND RATIONALE FOR CONTRACTING FOR
LOGISTICS CAPABILITY IN NATO ARMAMENTS SUPPORT PROGRAMMES
Note by the NIAG Secretary
1. Enclosed is the Final Report on Concepts and Rationale for Contracting for Logistics Capability in NATO Armaments Support Programmes, as conducted by NIAG SG.218, which is now published to the sponsor.
(signed) Nathalie Van Donghen
1 Enclosure Original: English
NHQD99273
NATO UNCLASSIFIED
Releasable to PFP
ENCLOSURE TO
NIAG-D(2018)0013, MULTI REF
NATO UNCLASSIFIED
Releasable to PFP
NIAG SG218
ON
CONCEPTS AND RATIONALE FOR CONTRACTING FOR
LOGISTICS CAPABILITY ON NATO ARMAMENTS AND
SUPPORT PROGRAMS
(Main Document)
The work described in this report was carried out under the provisions of the NIAG
Study Order for Study Group 218.
Disclosure, utilization, publication or reproduction of this report by industry is subject
to pre-approval by NATO until such time as NATO may have released such work to the
public
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-2
1 EXECUTIVE SUMMARY
This report responds to the requirements of the NATO Study Order to “provide analysis, recommendations, and tools for NATO to effectively exploit outcome-based contracting”.
Commercial entities such as airlines and various Nations’ military organizations have employed outcome-based contracting with varying degrees of success. The beginnings of outcome-based sustainment have been attributed to the commercial aviation industry. The Rolls Royce company pioneered the concept of selling propulsion to aircraft, rather than selling engines, spare parts and repairs – Power by the Hour™ and has since expanded its concept to a number of Nation’s military forces. As well, other similar arrangements have been made between industry and Nations to provide logistics as an outcome-based service. Some examples include aerial refuelling in the U.K., and pilot training in Singapore. In both cases, the government has avoided the cost of investing in infrastructure, training and operations, by having industry provide the desired outcome (refuelled aircraft or trained pilots), at a cost less than the government could have done organically.
The evidence observed during the Study demonstrates that well-structured and defined OBCs will reduce total costs, increase system availability for the war-fighter, optimize industry performance (investments and costs) to deliver an improved outcome, i.e. a win-win for government and industry.
The most important findings of the Study are, that OBC processes have the same value for NATO as traditional contracting processes and the overarching recommendation is, that NATO fully adopts and implements OBC processes.
To implement OBC, it is recommended:
• NATO should develop a high level policy for contracting for systems and logistics services. This policy should cover all kinds of contracts (traditional contracting and outcome-based contracting e.g. for capability). This policy has to be in line with the NATO Policy for Systems Life Cycle Management (SLCM) approved by the North Atlantic Council and signed by the Secretary General in 2006.
• NATO should enter into OBC arrangements starting with two pilot programs, utilizing the Decision Support Tool developed as part of this Study. This trial should be performed by NSPA and should be limited to a defined period of time (e.g. 2 years).
• NATO should review and update the NATO publications on contracting including all aspects of contracting – traditional and OBC- based ones. Furthermore it is recommended to specify in the relevant NATO documentation at which point in the life-cycle of the system the responsible manager should evaluate if OBC contracting is preferred to traditional contracting
The Study provides 15 more recommendations e.g. for modification of NATO contracting regulations, doctrine, and sample contract language for NATO publications. An evaluation method and a decision support tool were developed to support new and legacy
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-3
NATO programmes and programme managers in determining the optimal combination of organic military and industry supported life cycle support for use at any stage of a programme.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-4
TABLE OF CONTENTS
1 EXECUTIVE SUMMARY ........................................................................................................ 2
2 LIST OF PARTICIPANTS ........................................................................................................ 6
3 ORGANIZATION .................................................................................................................. 9
3.1 Introduction ........................................................................................................................... 9
3.2 Background .......................................................................................................................... 10
3.3 Output of the Study ............................................................................................................. 11
3.4 Study Organization ............................................................................................................... 11
3.5 Programme .......................................................................................................................... 12
3.6 Sponsor Liaison .................................................................................................................... 13
3.7 Final Report structure .......................................................................................................... 13
4 FOCUS AREAS FINDINGS .................................................................................................... 14
4.1 Broad Spectrum Analysis of Programmes .............................................................................. 14 4.1.1 Introduction and Approach ....................................................................................................................... 14 4.1.2 Programmes Reviewed / Other Studies Leveraged .................................................................................. 14 4.1.3 Observations and Findings ........................................................................................................................ 15 4.1.4 Summary and Recommendations ............................................................................................................. 23
4.2 NATO policy and doctrine ..................................................................................................... 24 4.2.1 Introduction .............................................................................................................................................. 24 4.2.2 Findings & Conclusions ............................................................................................................................. 25 4.2.3 Recommendations .................................................................................................................................... 27
4.3 Decision support tool ........................................................................................................... 28 4.3.1 Description / rationale of the tool ............................................................................................................ 28 4.3.2 Recommendations .................................................................................................................................... 31
5 STUDY RECOMMENDATIONS............................................................................................. 33
6 APPENDICES ..................................................................................................................... 35
6.1 Appendix A – Programmes Reviewed / Other Studies Leveraged ........................................... 35
6.2 Appendix B – OBC Arrangement General Characteristics ....................................................... 37
6.3 Appendix C – Case Studies Supporting Results / Recommendations ....................................... 38
6.4 Appendix D – Sample language ............................................................................................. 42 6.4.1 Definitions ................................................................................................................................................. 42 6.4.2 Found Sources and Suggestions for Sample Language ............................................................................. 43
6.5 Appendix E – Decision Support Tool ...................................................................................... 62 6.5.1 Parameters List ......................................................................................................................................... 62 6.5.2 Definitions of the parameters ................................................................................................................... 63 6.5.3 Parameter list development ..................................................................................................................... 65
6.6 Appendix F – Reference Documents ...................................................................................... 67
Documents analysed separately ....................................................................................................... 67
ANNEX I GLOSSARY ............................................................................................................. 69
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-5
ANNEX II Development Tool and User Manual ...................................................................... 88
General logic for development of the tool ......................................................................................... 88
Considerations for implementation .................................................................................................. 89 Consideration for a Decision Tree ........................................................................................................................... 90 Consideration for a decision support tool ............................................................................................................... 90
Parameters ...................................................................................................................................... 96
Logic .............................................................................................................................................. 101
General Process when to identify parameter values ........................................................................ 103 Identification of the key players............................................................................................................................ 104 Identification of the Key documents ..................................................................................................................... 105 Identification of the Key set of parameters .......................................................................................................... 106 Key set of parameters ........................................................................................................................................... 107 Set of Parameters 1 (SoP1), ambition/objective ................................................................................................... 108 Set of Parameters 2 (SoP2), ambition/objective ................................................................................................... 109 Set of Parameters 3 (SoP3), ambition/objective ................................................................................................... 110 Set of Parameters 4 (SoP4), ambition/objective ................................................................................................... 111 General exclusions ................................................................................................................................................ 111
Tool Description / Handling of the tool ........................................................................................... 112 Register Main ........................................................................................................................................................ 113 Register Parameters .............................................................................................................................................. 115 Register Evaluation: .............................................................................................................................................. 116
Tests / Test cases and results .......................................................................................................... 118 Testcase UK Sentry Project: .................................................................................................................................. 118 Testcase UK Aerial Refuelling: ............................................................................................................................... 118
Description of single parameter values: .......................................................................................... 119
ANNEX III DECISION SUPPORT TOOL .................................................................................... 121
ANNEX IV FINAL PROOF POINT............................................................................................. 123
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-6
2 LIST OF PARTICIPANTS
Country Name Role Company
USA Mark Harnitchek Chairman
– till 29/06/2017
The Boeing Company
Germany Peter Janatschek Chairman
– from 29/06/2017
Peter Janatschek Projekte &
Logistik
Spain Juan Carlos Santos
Santos
Vice-Chairman AIRBUS DEFENCE & SPACE
Belgium Gustavo Scotti di Uccio Rapporteur AOS
Country Name Company
BEL Elena Manuela Tudosia PoleCM
BEL Nickola Carlomagno Vitrociset Belgium
CHZ David Hác STV Group a.s.
DEN Per Tony Frederiksen A/S Hydrema Export
DEN Christian Brochmann Jorgensen Nordic Camp Supply Fuel
DEN Jens Praestegaard Nordic Camp Supply Fuel
FR Jean-Charles Boulat Naval Group (ex DCNS)
FR Simon Payet Naval Group (ex DCNS)
FR Lionel Bourlard MBDA France
FR Louis Vergniolle de Chantal THALES International
FR Véronique Robineau THALES SA
GER Kurt Haeusler Aviation Consulting
GER Gabriel Jelodin ESG Elektroniksystem und Logistik GmbH
GER Wolfgang Mack MBDA Deutschland GmbH
GER Reinhold Ziegler MBDA Deutschland GmbH
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-7
GER Andreas Fenkner T-Systems International GmbH
GER Andreas Kirchhofer T-Systems International GmbH
ITA Giovanni Murettino Aerea SpA
ITA Luca Rancati Aerea SpA
ITA Gianluca Bove SIMAV
ITA Icilio Di Luzio SIMAV
ITA Attilio Perego UEE Italia S.r.l (EXPAL Systems Group)
ITA Luigi Ciolli Vitrociset
TUR Cengiz Oflaz ALTAY Information Tech. Def.Industrial Inc
TUR Alper Sueri TUBITAK SAGE
TUR Banu Tekşen TUBITAK SAGE
TUR Remzi Yüksekbulgu Turkish Aerospace Industries (TAI)
TUR Ahmet Kavcar Turkish Aerospace Industries (TAI)
UK Nigel Robert Hunter LEONARDO UK
UKR Ganna Pavlova SE KhMP "FED"
USA Joseph Spruill Lockheed Martin Corporation
LIST OF SG218 Quick Reaction Team (QRT)
Name Organization
Giuseppe Rampini ACO/SHAPE – LCMG-WG3
Steven Larcher Canadian Department of National Defence
Pierre Demers Canadian Department of National Defence
Andreas Keller Federal Office of Bundeswehr Equipment, Information
Technology and In-Service Support (BAAINBw)
Michail Bozoudis MOD GRE – LCMG-WG3
Alain Courtois NCIA
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-8
Stefan Limburg Federal Office of Bundeswehr Equipment, Information
Technology and In-Service Support (BAAINBw)
Oliver Brungs Federal Office of Bundeswehr Equipment, Information
Technology and In-Service Support (BAAINBw)
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-9
3 ORGANIZATION
3.1 Introduction
This report responds to the requirements of the NATO Study Order to “provide
analysis, recommendations and tools for NATO to effectively exploit outcome-based
contracting”.
Commercial entities such as airlines and various Nations’ military organizations
have employed outcome-based contracting with varying degrees of success. The
beginnings of outcome-based sustainment have been attributed to the commercial aviation
industry. Southwest Airlines champions outcome-based methods to reduce “turn-around
time” and increase profits. The Rolls Royce company pioneered the concept of selling
propulsion to aircraft, rather than selling engines, spare parts and repairs – Power by the
Hour™. Rolls Royce has since expanded its concept to a number of Nation’s military forces.
As well, other similar arrangements have been made between industry and Nations to
provide logistics as an outcome-based service. Some examples include aerial refuelling in
the U.K., and pilot training in Singapore. In both cases, the government has avoided the cost
of investing in infrastructure, training and operations, by having industry provide the desired
outcome (refuelled aircraft or trained pilots), at a cost less than the government could have
done organically.
This study explored documentation of many of these outcome-based arrangements
to determine commonality and opportunity for leveraging their successes in NATO.
The Study Order asserted: “Contracting for capability (readiness, availability, etc.)
has been proven in some situations to reduce cost and increase mission effectiveness. This
method of contracting is contrasted to traditional transactional contracting where payment
to a supplier is based on number of parts supplied or number of repairs.”
This Study provides NATO with the rationale and tools to support the determination
of whether, when, and to what extent outcome-based contracting best supports a specific
programme.
This Study addresses, describes and documents the various types of contracting
for capability implementations conducted in NATO and NATO Nations. Benefits, risks, and
lessons learned are addressed.
The Study provides recommendations for modification of NATO policy, doctrine, and
sample contract language for NATO publications. An evaluation method and a decision
support tool were developed to support new and legacy NATO programmes and programme
managers in determining the optimal combination of organic military and industry supported
life cycle support for use at any stage of a programme.
Many of the terms used in this report come from sources outside of NATO writing
convention. As well, there are many different phrases that describe outcome-based
arrangements, that are similar, but with nuanced differences (e.g. “outcome-based services
contracting”, and “performance-based logistics”). To assist the reader, a glossary of terms,
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-10
that is published as a separate document, has been prepared to provide context for those
terms used extensively in this Study. For consistency, in this document, we will use the term
“Outcome-Based Contracting” (OBC).
At the request of the sponsor organization, the scope of the Study is limited to
logistics support capabilities including any of the elements of logistics.
3.2 Background
Contracting for capability includes bringing outcome-based, commercially-provided
solutions to logistics and sustainment to increase user-defined availability and capability, for
the same or reduced costs. Examples maybe described as outcome-based services
contracting, performance-based logistics, fleet management, turn-key solutions, or total life
cycle management. These are methods of acquiring operational armament support that
acquires “performance” (outcomes) rather than materiel, maintenance and/or repair services
(outputs). OBC generally expresses requirements in terms of a desired end-state, rather
than providing for prescriptive or specification-based work statements. Simply put, OBC is
more about the “what”, than the “how” in delivering the desired product.
The approach is aimed at decreasing overall programme risk, improving reliability
and maintainability, and driving down life cycle costs.
Contracting for capability includes a wide spectrum of support solutions ranging from
minimum contractor involvement – the traditional model – where equipment is supported by
transactional repair contracts, to true contracting for capability where the contractor provides
a complete support package. Most solutions fall somewhere between the two extremes of
the continuum and include elements of both traditional and non-traditional support
arrangements. Identifying the options and the conditions under which each option optimizes
the support environment is a critical consideration during programme planning.
Increasingly, NATO Nation military organisations (e.g., UK, GER, NL, US) are
leveraging the defence industrial base to provide operational support, and are contracting
for that support, based on the delivery of performance outcomes.
The Study concentrates on three focus areas:
1. It contains a broad-spectrum analysis of programmes which have been conducted
using a Contracting for Capability model to achieve logistics support or sustainment.
The programme conditions, results, and lessons learned are documented. The
Quick Reaction Team (QRT) has given additional input on which programmes are
to be included in the study to ensure that a representative blend of conditions and
outcomes is addressed. The evaluation of the programmes includes a comparative
cost benefit analysis between the results of the programme and expected results
using traditional contracting methods.
2. Relevant NATO policy and doctrine are identified for updates, to include the
concepts of OBC. The Study provides sample language for consideration in NATO
publications.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-11
3. The Study provides a decision support tool that can be used by NATO / Nations to
determine whether, and to what extent, OBC might be appropriate, given the
programme parameters.
3.3 Output of the Study
The Study considers operational scenarios that require a component of life-cycle
product support. The operational scenarios include situations where it is necessary to
maintain the readiness and operational capability of a system or subsystem. The specific
conditions range from ensuring that parts are available for repair activities, to ensuring
operational availability of a weapon system. In general, the Study considers operational
scenarios where there is a mission availability or readiness requirement that can be fulfilled
by industry support, and where innovation can increase efficiency and reduce cost.
3.4 Study Organization
The Concepts and Rationale for Contracting Capability in NATO Support
Programmes (“Contracting for Capability Study”) has been conducted in two phases:
• 1st Phase: The Study Team has been organized in three Study Sub-Groups (SSG)
to:
o execute the analysis of programmes that have been conducted using a
Contracting for Capability model to achieve logistics or sustainment. This activity
included an analysis of the programme conditions, results, and lessons learned.
The QRT was consulted for the list of programmes analysed;
o identify relevant NATO policy and doctrine that may require updates to
implement contracting for capability; and
o develop a decision support tool that could be used by NATO programme
managers to determine whether, and to what extent, contracting for capability
might be appropriate given the programme parameters.
• 2nd Phase: the Study Team worked cooperatively to:
o perform a global analysis of the benefits, risks, and lessons learned from
National experience and extrapolate that into the multi-national environment;
o prepare recommendations on how NATO doctrine and policy might be
enhanced to enable to effective consideration of different support solutions; and
o set up a decision support tool to guide NATO programme development with best
practices of contracting for capabilities.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-12
3.5 Programme
Six Plenary Meetings of the Study Team took place, allowing the specialists to
develop this document (figure 1).
During such Plenary Meetings, the three SSGs collaborated on their respective
inputs. The proposals were shared with the QRT representatives, taking their feedback into
consideration.
Prior to Plenary Meeting #4, an initial draft of the Study Report was generated.
Several iterations of the draft document were circulated after each meeting for further
improvement and alignment.
All Plenary Meetings were conducted with the presence, and assistance of a QRT
Representative.
A final presentation to the QRT was performed to assure the consistency and
alignment with the Study Order.
The Final Report was presented to the Sponsor of the study in May 2018.
Figure 1 – Programme planning
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-13
3.6 Sponsor Liaison
The Study Team has been in continuous liaison with the QRT.
1st Name Last
Name
Organization Notes
Giuseppe Rampini ACO/SHAPE – LCMG-WG3 Met in Mons on Sept 26/2017
Steven Larcher Canadian Department of National
Defence (DND)
By Phone on the 4th and 5th Plenary
Meeting
Pierre Demers Canadian Department of National
Defence (DND)
Attended the Kick-Off (1st Plenary)
Meeting in Capellen
Oliver Brungs Federal Office of Bundeswehr
Equipment, Information Technology and
In-Service Support – LCMG-WG1
Attended the 2nd Plenary Meeting
Michail Bozoudis MOD GRE – LCMG-WG3 No show
Alain Courtois NCIA Interested to receive information, but
apologized for not attending.
Stefan Limburg Federal Office of Bundeswehr
Equipment, Information Technology and
In-Service Support
Attended the EG meeting and made the
initial sponsor presentation – plus
commenting on Final Report
Andreas Keller Federal Office of Bundeswehr
Equipment, Information Technology and
In-Service Support – LCMG-WG1
Attended the 3rd, 4th, 5th and the 6th
Plenary Meetings.
Stefan Limburg Federal Office of Bundeswehr
Equipment, Information Technology and
In-Service Support
On Apr 10/2018 meeting with the
Chairman discussing the findings and
recommendations.
3.7 Final Report structure
• Executive Summary (EXECUTIVE SUMMARY).
• Main Analysis (LIST OF PARTICIPANTS, ORGANIZATION, FOCUS AREAS
FINDINGS)
• Findings and Overall Study Recommendations (STUDY RECOMMENDATIONS).
• Appendices (APPENDICES)
• Annexes (separate documents are available on the DI/Web Sharepoint).
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-14
4 FOCUS AREAS FINDINGS
This chapter summarizes the work done within the 3 Study Subgroups:
• broad spectrum analysis of programmes;
• identification of relevant NATO Policy and Doctrine; and
• development of a decision support tool.
4.1 Broad Spectrum Analysis of Programmes
4.1.1 Introduction and Approach
Decision support tool of the Study Order directed the study group to “execute a
broad-spectrum analysis of programmes that have been conducted using a Contracting for
Capability model to achieve logistics or sustainment”.
The analysis was conducted to support the group’s overall recommendations, as
well as provide a basis for the other two tasks of the Study Order – recommend changes to
relevant NATO policy and doctrine (NATO policy and doctrine), and develop a decision
support tool for NATO / Nations decision makers to facilitate the selection of OBC to achieve
desired effects on cost and performance (Decision support tool).
Data sources included: available reports, personal knowledge of team members,
and findings from reports of previously conducted analysis.
The analysis considered the results from the perspectives of the end-user (i.e.
military forces), the contracting officials (i.e. NSPA/NCIA/MoDs), and the provider (i.e.
industry).
Characteristics considered included: capability, system supported, and the desired
outcome. It is important to point out that for many of the programmes reviewed, specific
performance and contractual data was withheld by Nation’s governing agencies for security
or business sensitivity reasons. In these cases, assessments were made using reasonable
estimates of similar situations, or anecdotal data.
This section addresses three areas:
1. discussion of programmes reviewed, or other studies leveraged for analysis;
2. observations of analysis to include characteristics and attributes of the contract,
results achieved, methods of delivery, and a summary of benefits;
3. summary of the analysis with recommendations; and
4. in addition to the glossary, annexes include examples of outcome-based
agreements and copies of source material reviewed.
4.1.2 Programmes Reviewed / Other Studies Leveraged
The Study Group assessed 23 programmes. These included 21 programmes
supporting NATO, and 2 commercial contracts supporting the private sector. The Group also
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-15
leveraged previous, similar analytical work, including 4 reports from academia and the U.S.
Government. A detailed list of programmes and previous studies is included in Appendix A.
4.1.3 Observations and Findings
4.1.3.1 General Observations
Many nations engage in outcome-based contracting for weapons systems support
and other logistics requirements. Many different terms are used to describe these
arrangements (e.g.: the U.S. uses the term “Performance-Based Logistics” (PBL)). This
Study collectively calls these arrangements “Outcome Based Contracts” (OBC).
In traditional methods, the government customer procures the required support and
service through transactional contractual arrangements. In OBC, the government procures
an outcome, where industry takes on the responsibility, and risk, to deliver those required
outcomes. Industry prime contractors include original equipment manufacturers (OEM),
non-OEM competitors, or joint ventures of interested parties.
OBC arrangements have common characteristics and generally seek to achieve
similar results: optimized performance and cost. By moving to OBC arrangements,
governments recognized that by shifting the responsibility and risk, contractors were now
incentivized to make improvements, thus providing better outcomes, at lower cost (see
figure 2).
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-16
Figure 2 – Traditional vs. OBC Support Strategies
Time
RELIABILITY
AVAILABILITY
COST
Implement a business model that inherently incentivises:
• Improving processes and products to improve reliability
and availability
• Resulting in O&S cost reduction & improved Performance
Co
st
an
d P
erf
orm
an
ce
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-17
4.1.3.2 Characteristics of Outcome Based Contracting
4.1.3.2.1 General
This section provides characteristics and attributes observed in the reviewed
programmes and other studies. There are universal characteristics of outcome-based
contracts found in practice. The list below is more fully described in Appendix B:
• specifies “what vs. how”;
• contains defined outcome metrics;
• explicit/implicit financial incentives tied to metrics;
• real financial risk/rewards;
• long-term arrangements; and
• authority to manage and optimize outcomes
4.1.3.2.2 Level and Complexity of the Supported System / Service Provided
Generally, the analysis showed that OBC arrangements could be grouped in three
categories: System Level, Sub-System Level and Component Level, Figure 3.
Figure 3 – OBC Levels of Complexity
System level refers to an arrangement that covers all the Integrated Logistics
Support (ILS) for an entire system (e.g. U.S. MH-60 helicopters) or provides system-level
support service (e.g. U.K. Aerial refuelling). These “Tip-to-Tail” programmes often include
support equipment, training, and infrastructure, providing a turnkey support solution.
System Level
Sub-System
Level
Component
Level
An entire Weapon System
or Major Service (e.g. UK
SENTRY, Aerial
Refueling)
A major subsystem
(e.g., Landing Gear)
A commodity item or service
(e.g., A/C tires, circuit cards,
garrison infrastructure
support)
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-18
Sub-system level refers to a group of components providing a major function within
a larger system. Examples include radars, sensors, communications systems, and aircraft
landing gear. The number of ILS elements engaged depends on the complexity and scope
of the subsystem.
Component level arrangements typically support smaller assemblies, subsystems
or systems (e.g. cockpit displays, H60 GE T700 engines), or provide supply chain services
for a single commodity, or class of commodities (e.g. tires, depot repair consumables). The
scope of ILS elements engaged in supporting component level requirements is generally
limited.
4.1.3.2.3 Metrics
Well defined, customer-focused, outcome metrics are the key to successful OBC
arrangements. Industry avoids unnecessary risks by accepting only those metrics that they
can control and deliver.
The use of a small number of outcome metrics (optimally 6 or less) empower
industry to take the necessary actions to provide requisite support. Contracts with too many
metrics risk becoming specification type contracts. In this case, the metrics measure
outputs, rather than outcomes.
Industry may employ these more specific output metrics to evaluate internal
processes, and control cost of delivery.
4.1.3.2.4 Contract Characteristics
Successful OBC contracts were generally observed to be long-term (5 years or
more), had stable funding, had incentives reflecting assumed risk, small numbers of top-
level outcome metrics, and based on a strategy of performance rather than product. See
figure 4.
The contract types used typically relate to the existence of system operation data.
Newer programmes may be supported with shorter, cost-type contracts, where programmes
with more operational experience use longer, fixed price-type contracts, with various
incentives tied to performance metrics.
While incentives are set to target performance, some OBC arrangements include
disincentives for both under and over-performance.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-19
Figure 4 – OBC Characteristics
4.1.3.2.5 Prime Contractors and Subcontractor/Partner Relationships
All OBC programmes had a single, prime contractor accountable for meeting overall
performance outcome metrics. These primes usually engage several subcontractors, or
partner companies, and in some cases specified government agencies, to deliver the
required performance. In this role they act as an integrator, performing trades between the
providers to optimize outcomes.
Depending on the complexity of the support provided, some integrators flow down
the outcome metric requirements, as well as the risk and rewards, to their
subcontractors/partners. Adequately gauging and managing subcontractor contributions to
overall performance are critical to overall system and financial performance.
4.1.3.3 Case Studies Supporting Results/Recommendations
To illustrate the observations and characteristics, this section highlights two
significant examples of the 23 reviewed programmes: the first is focused on a Life Cycle
Support Programme, while the second is focused on a Support Services Programme.
Details about all reviewed programmes are provided in Appendix C.
• UK SENTRY is a whole life-cycle support programme for the E-3D Airborne Warning
and Control System (AWACS) – including both the air vehicle and mission
packages, provided by a non-OEM prime contractor. The 20-year contract covers
support to seven mission aircraft. This arrangement has a single outcome metric of
OBC CONTRACTS CHARACTERISTICS
Contract Characteristics
Te
ne
ts o
f S
uc
ce
ss
Contract
TermLong Term; Five Years Plus
Funding Stable; Funding Floor
RiskProvider Delivers Top-Level
Performance-Support Outcomes
MetricsBased on a Few Top Level
Performance-Support Outcome
Metrics
Strategy
Focus
Aligned with Top-Level End Item
Performance
Results
High Incentive to Invest
ROI Business Case for Investment
Higher Risk Offset by Opportunity
for Higher Reward
Provider Has Broad Flexibility in
How to Deliver Outcomes
Clear System-Level Provider
Responsibility
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-20
aircraft availability (requiring 6 of 7 aircraft). The provider (Northrop Grumman
Corporation) is fully empowered to conduct requisite product support activities (e.g.
maintenance, supply support, etc.) to meet required outcomes.
• UK Aerial Refuelling is a contract between UK MOD and a joint venture (Air Tanker
Holdings, Ltd.) to procure and support tanker and transport aircraft. This 27-year
arrangement is a Private Finance Initiative (PFI), with all capabilities funded by
industry. The customer pays for on-demand air transportation of fuel and personnel,
as a service, on a flying hour basis. UK Aerial Refuelling has two outcome metrics
– aircraft availability and flying hours.
Although most of the outcome-based arrangements identified for review were US
DoD contracts, the two programmes reviewed as case studies were selected as they
provided a more established development of outcome-based principles and details for
analysis.
The analysis of programmes revealed a limited body of specific acquisition
regulations for outcome-based arrangements by the individual NATO nations. The
predominant body of national regulatory documents were seen with those nations who have
longer experience engaging in OBC (e.g. US DoD, UK MoD).
4.1.3.4 Other Studies Leveraged – US DoD Proof Point
This section highlights the findings of the document “Proof Point Project – A Study
to Determine the Impact of Performance Based Logistics (PBL) on Life Cycle Costs”, Office
of the Deputy Assistant Secretary of Defense (Materiel Readiness), Washington, DC,
November 2011. The executive summary states the purpose of this study: “The U.S.
Assistant Secretary of Defense (Logistics & Materiel Readiness) chartered this initiative to
provide conclusive evidence regarding the impact of Performance Based Logistics (PBL) on
the life cycle cost of sustaining Department of Defense (DoD) weapon systems, subsystems,
and major components when compared to non-PBL sustainment arrangements.”
The study reviewed 21 OBCs (PBL programmes), and concluded that PBL
arrangements, which adhere to generally recognized PBL tenets, reduce cost while
simultaneously improve system performance. Sample of findings are seen in Figure 5, and
key findings, stratified by level of evidence supporting the conclusions, are seen in Figure 6.
The entirety of the report is included in ANNEX IV FINAL PROOF POINT.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-21
Figure 5 – DoD Proof Point Summary Findings
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-22
Figure 6 – DoD Proof Point Key Findings
4.1.3.5 Benefit to doing Outcome Based Contracting
Key requirements for successful OBC are:
• Required services and responsibilities of all parties involved must be clearly defined.
• Industry party (prime contractor) must have full responsibility for delivery of the
service – Industry should set up and manage the arrangements between supplier
parties (sub-contractors)
• In systems with multiple providers (e.g. aircraft with different providers for the air
vehicle and the propulsion plant), the customer must designate a single integrator
to avoid ambiguity in responsibilities between parties for overall performance of the
system.
• Desired outcomes must be defined from the perspective of the end-user (e.g. the
operator).
• Specific authorities and responsibilities must be defined for changes in system
configuration (e.g. modifications, upgrades, service bulletin incorporation,
obsolescence management, etc.) prior to contract.
If these key requirements are met, OBCs will deliver the primary benefits of
improved system performance at the same, or reduced total cost (win-win). (Total cost is
defined as the cost incurred in a traditional support environment from both government
sources (organic support) and contracted services (industry)). Effects include:
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-23
• OBC recognizes the shifting of responsibility, and consequently risk, from the
government to industry in pricing and incentive structures;
• increases in revenue, cash flow and profit reflect the additional responsibilities and
risk assumed by industry;
• natural industry behaviour, with increased risk, is to affect continuous process and
product improvements to reduce costs;
• long term contracts are designed to give Industry a reasonable expectation to
achieve a return on invested capital. Profit levels at the end of a period of
performance may appear excessive, but customers must consider periods where
industry made investments to improve service delivery and performance. A suitable
long-term costing and performance model should be established;
• occasionally windfall profits are realized when results from improvements in process
and product exceed expectations. Well-structured OBC arrangements account for
this in the incentives structures, and through gain-sharing schemes, where a
predetermined percentage of profit above expected gains are returned to the
government;
• industry-standard performance levels may exceed what is desired by the
government (e.g. system availability at 90%, where government readiness
requirements or available funding support lesser needs). OBC arrangements should
reflect customer needs and related payment terms to incentivize industry to deliver
what the customer needs; and
• other benefits will be a reduction in total inventory of spares and repairs in the
system, as industry will optimise holdings against lead times, repair and overhaul
turn-around-times, against proven MTBFs to achieve the required system
availability metrics. Considerations, such as customer surge requirements and the
consequent emergency stock levels, need to be included within the contract.
4.1.3.6 Parameters to be Considered for Decision Support Tool
Based on the characteristics of successful OBC observed as part of the broad-
spectrum analysis, a list of parameters to be considered for the proposed Decision Support
Tool was developed. The table of parameters is described in Appendix E.
4.1.4 Summary and Recommendations
The evidence observed from the programmes and other studies reviewed,
demonstrates that well-structured and defined OBCs will reduce total costs, increase system
availability for the war-fighter, optimise industry performance (investments and costs) to
deliver an improved outcome, i.e. a win-win for government and industry, figure 7.
These results suggest that NATO, and the Nations supported by NATO contracting
agencies (e.g. NSPA, NCIA), would similarly benefit by employing OBC.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-24
The results also suggest that successful OBC requires proper planning, and due
diligence to understand the customer’s desired outcomes, set the appropriate performance
outcome metrics, and negotiate effective incentives.
Joint programmes, like the F-35 Lightning II Joint Strike Fighter, where many of the
participating nations are also NATO nations, present a unique opportunity to engage in OBC
and provide improved system availability with greater efficiencies – optimised outcomes.
The Study recommends NATO enter into OBC arrangements starting with two pilot
programmes, utilizing the Decision Support Tool developed as part of this Study, to identify
the target programmes.
Figure 7 – Win-Win: OBC Benefits
4.2 NATO policy and doctrine
4.2.1 Introduction
This section focuses on identifying “NATO policy and doctrine that would be relevant
to be updated to include the concepts of contracting for capability and provide sample
language for consideration by the document custodian”. Recommendations on how NATO
doctrine and policy might be enhanced to enable effective consideration of different support
solutions, and sample language for inclusion in NATO documents are included.
Contractor
Government
Other Programmes
Opportunity to:
• improve product reliability
• improve process efficiency
• decrease costs
• increase profit
Opportunity to:
• increase availability of system
• Increase reliability of system
• decrease programme costs
• achieve desired outcomes
Opportunity to:
• free up capacity
• gain lessons learned
WIN
WIN
WIN
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-25
The analysis also provided input to support the development of a decision support
tool.
A list of 52 documents was identified, and reviewed by the QRT. The documents
were selected according to levels of importance, and relevance with respect to the NATO
Policy for Systems Life Cycle Management
The 52 documents were selected to reflect:
1. The estimated impact on the object of the study;
2. the hierarchy of NATO documents, from high-level policy on systems life-cycle
management, to main standards and guidance on life-cycle and logistics support
(e.g. AAP-20; AAP-48; ALP-10), to more detailed standards and guidance related
to specific aspects (e.g. quality; risk management; life-cycle costs; acquisition
practices);
3. the majority of life-cycle stages;
4. the key procurement regulations (e.g. NSPA; NCIA); and
5. examples from NATO member nations.
Of the original 52, 20 documents were selected for in-depth analysis. These
documents are selected samples that well represent the points described above. The full list
of documents can be found in Appendix F – Reference Documents.
For the in-depth analysis of the 20 documents, reports have been produced which
include:
1. Name and version of the document;
2. short summary of document, followed by remarks concerning need for update and
recommendations for improvement, as well as connection with other NATO
documents;
3. sample language (when considered appropriate);
4. remarks concerning collaborative relationship (when applicable); and
5. identified need for new definitions or adaptation of old definitions.
These reports can be found in Appendix F – Reference Documents.
In order to further address the request for sample language, a separate section was
created where examples for such language are given in the form of “amendments” to the
current text. This document is to be found at Appendix D – Sample language.
4.2.2 Findings & Conclusions
No specific NATO policy, doctrine, or agreed definition on OBC was identified.
Therefore, the recommendations and sample language suggested are based on the Study
Group experience and analysis of existing OBC-type programmes. The group assumes that
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-26
the same characteristics would be considered acceptable for use by NATO and NATO
Nations.
Although certain elements, similar to OBC, could be found throughout the
documents, the concept of OBC itself does not appear to be integrated in a consistent and
coherent way in existing policies, doctrines and procurement regulations.
For example, guidelines like AACP-2 Vol. 2, suggest a preference, whenever
possible, for firm, or fixed price contracts. However, reference is not made to the possible
positive implications (e.g. on cost, incentives; risk premiums) that firm, or fixed price, has
when OBC is applied.
Other rules and procurement principles, for example those applied in the current
NSPO procurement regulations (e.g. “pre-award contact with suppliers”; “the most
economical price”; “balancing of production”) may not leave enough scope for applying OBC
if they are not adapted accordingly.
Implementation of OBC requires complete understanding of the requirement (“the
what”) by both the Contracting Officer and Industry. Close collaboration between Contracting
Officer and Industry must occur as early as possible in the definition of needs, to ensure
industry adequately addresses the delivery of the outcome (“the how”).
This applies equally for new capabilities, the improvement of an already existing
capacity / service, new contracts and the transition from traditional type of contracting to
OBC.
Some industry consultation instruments were found throughout the analysed
documents, which may be sufficient for the traditional transactional type of contracting but
not sufficient for OBC (e.g. “pre-award contact with suppliers”, “pre-award survey” within
NSPO Procurement Regulations).
Consultation of prospective contractors as early as possible (during the definition of
needs for services or capabilities) is a prerequisite for selecting the right type of contract (i.e.
traditional vs. OBC). If OBC is chosen, enough time is needed for defining the details (e.g.
exact scope of the contract, service level, performance metrics) and, most importantly, for
building trust between contractors and the customer. As such, if OBC is chosen, industry
needs to assess its capacity to properly deliver the requirements defined in contracts, its
readiness, its availability, as well as to estimate risks and benefits for the customer and the
prospective contractor.
Interdependence also exists between the implementation of OBC and the
Government Quality Assurance process. For example, the question can be raised regarding
the extent to which Quality Assurance should take into account “measurable outcomes” or
“performance metrics” identified as part of OBC.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-27
4.2.3 Recommendations
It is recommended:
• NATO should develop a policy on contracting for systems and logistics services
including “traditional contracting and OBC in order to secure common understanding
of the concept and of its underlying principles. Such policy and common
understanding are likely to facilitate decision-making in a multinational environment.
This policy has to be in line with the NATO Policy for Systems Life Cycle
Management (SLCM) approved by the North Atlantic Council and signed by the
Secretary General in 2006.
• The Study recommends NATO to enter into OBC arrangements starting with two
pilot programs, utilizing the Decision Support Tool developed as part of this Study.
This trial should be performed by NSPA and should be limited to a to a defined
period of time(e.g. 2 years.
• OBC training requires a Training Need Analysis (TNA), it is recommended to
consider, whether the aspect of “Contracting” can be included in the current Life
Cycle Management Training Analysis.
• The Study recommends to review and to update the NATO publications on
contracting including all aspects of contracting – traditional and OBS- based on
and in line with the NATO Policy for Systems Life-Cycle Management.
• Furthermore it is recommended to specify in the relevant NATO documentation at
which point in the life-cycle of the system in the decision-making process the
responsible manager should evaluate if OBS contracting is preferred to traditional
contracting.
• This work should be performed by a new AC/327 Working Group with members
both from Nations and the NIAG IIG and co-chaired by AC/327 and NIIG.
• The basis of this work are the results of the work of the former Group on
Acquisition Practices – AC/313 which was created to advise the CNAD on
government defence procurement practices and procedures relating to defence
trade, armaments cooperative programme arrangements and industrial
collaboration within the Alliance.
• In this context the Study recommends to modify several existing definitions
(Appendix D – Sample language) to recognize the concept of OBC. Additionally it
is recommended that NATO uses the term Outcome Based Contract (OBC)
exclusively.
• To specify in the highest-level NATO documentation (e.g. NATO Policy for Life
Cyle Management, AAP-20, AAP-48, ALP-10, and NSPA/NCIA procurement
regulations) at which point in the decision-making process the customer /
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-28
procurement officer should evaluate if OBC is preferred to traditional contracting.
This assessment should be done by applying the decision support tool;
• The choice between traditional contracting and OBC should be done as early as
possible in its life-cycle, both for the entire System of Interest (SoI) and its sub-
systems, via mutual consultation between industry and contracting authorities;
new instruments for industry consultation should be created to accommodate the
specificities of OBC.
• Any changes to high-level documents must also be reflected in subordinate
Implementing guidance and standards documents. In addition to general OBC
principles, guidance should include specific areas of OBC (e.g.: metrics/indicators,
requirements definition, risk vs. incentive determination);
• The inclusion of the metrics could be investigated in the framework of a follow-on
NIAG Study.
• The implementation of OBC within the NATO community should be promoted from
the highest to the lowest decision-making levels. To ensure its implementation,
specific instructions and the development of a training concept and dedicated
training modules are needed. The contracting officers throughout the NATO
organisations and agencies must be trained on the same basis to ensure
commonality;
• It is recommended to develop processes for Government Quality Assurance
Organisation to evaluate outcome metrics and determine if adequate OBC results
are delivered. The relation between Government Quality Assurance Organisation
and performance metrics specific to OBC implementation can be evaluated in the
frame of a follow-on NIAG study.
4.3 Decision support tool
4.3.1 Description / rationale of the tool
This part of the Study determined a methodology leading to a set of processes
gathered in a decision support tool to support the choice of the applicable contract type. The
result coming from the support tool is to be taken into consideration to select the type of
contract.
This activity was conducted through close discussions within the Study Group and
with some external contribution from relevant stakeholders. (e.g. NSPA/NCIA/MoDs).
The task performed at this stage was a preliminary reflection on how to best
determine a “Logical Choice”. Since it is difficult to consider all the relevant parameters for
a new project, the Study Group limited, for this preliminary thinking, the number of input
parameters for the model proposed. During the use of the tool, the input parameters should
be evaluated and weighted by contract specialists to adapt the tool for final use.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-29
The final decision for a support contract type could take into consideration further
applicable parameters.
The decision support tool has been benchmarked with two of the contracts analysed
in section 4.1 to check its efficacy.
The tool is expected to support the user in deciding a contract type (e.g. outcome-
based (“OBC preferred”) or activity based (“Non-OBC preferred”)).
The process used to develop the tool included:
• identification and refinement of a set of parameters from existing contracts
examined in Section 4.1;
• establishing a logic, based on the study members experience and taking into
account current NATO contracting regulations as investigated in Section 4.2;
• testing the tool with two test cases. Input for these test cases came from the
examined contracts in Section 4.1; and
• providing instructions on how to use the tool.
More details are described in Annex II
For the implementation of the tool, the Study Group completed the following:
• A sequence for a decision tree was developed. The challenge was to highlight a
logic that enables the user to take sound decisions, as values are applied for the
different parameters inside the decision tree. This logic is generic enough to cover
both OBC and Non-OBC.
• For the development of the decision support tool, the defined logic was turned into
a process, which was split into seven phases:
o Phase 1 – Preparation
o Phase 2 – Parameters valuation
o Phase 3 – Initial scoring
o Phase 4 – Selection of best suitable “Main contract type”
o Phase 5 – Pre-selection of best suitable “Sub contract type”
o Phase 6 – Verification of contracts types compatibility & optimization
o Phase 7 – Final selection of contracts
More details are described in ANNEX II Development Tool and User Manual.
The Tool itself is a protected Excel File and is posted together with a list of passwords on
the NATO DI Portal:
https://diweb.hq.nato.int/
(NIAG / Study Groups / SG-218 / Shared Documents)
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-30
The Study Group identified and refined different types of parameters, including:
1. Requirement
2. Cost Analysis
3. Metrics – Type
4. Metrics – Number used
5. Complexity of System Supported
6. Performance History / Experience
7. Contract starting point in life cycle
8. Availability of Data and Technology
9. Contractor Freedom of Solution Choice
10. Provision for Performance Improvement
11. Period of Performance: Return on Investment Opportunity
12. Scope – Extent of service provided
13. Contract Type
14. Obsolescence Management / Modernization and Upgrade
15. Number of Nations
16. Contractor OBC Experience
ANNEX II Development Tool and User Manual describes in detail the different
parameters used for the tool.
The Study Group reviewed NATO documents to identify policies and doctrines
relevant to OBC, further including regulations that could be considered for the decision
support tool. The findings included three suggested actions with nine specific
recommendations. The conclusions drawn from this are shown in ANNEX II
Development Tool and User Manual.
The group took into consideration the overall process of the OBC decision support
tool requirement in order to identify the stakeholders, and their expectations were provided
through discussions, review of documents and specifications, criteria, and “input
parameters”.
To ensure the requirement are fully identified, the following list stakeholders were
identified:
• End-Users (e.g. armed forces)
• Customer contracting organisation (e.g. national MoDs, …)
• NATO (e.g. NSPA, NCIA, …)
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-31
• Participating industrial partners
The requirement evolution and the values assigned to the parameters depend on
the level of comprehension obtained during the Study including the identification of the
parameters that will influence the decision making.
Annex II contains flowcharts, and it describes the process to reach a good level of
understanding and to be able to size the parameter level.
Annex II contains the description/handling (user manual) of the tool.
The tool itself is an Excel- file which consists of three worksheets:
• Main (user interface)
• Parameters (setting of parameters and values for the Main worksheet, setting of
digital values in order to enable comparisons)
• Evaluation (performing evaluations of selected parameter values. The result of the
evaluations is shown in the Main worksheet)
It contains the 16 parameters as listed above plus a provision for 3 additional
parameters to enable simple further development of the tool.
It is important to bear in mind that this tool was not planned to be the sole
discriminator for the user to make his decision on which contract type to use. The user must
take into account other circumstances, e.g. regulations and pre-conditions. It is a support
tool!
Being the first release of this tool, the Study Team strongly suggest following the
instructions of Annex II, in order to avoid misinterpretation.
Two test cases have been applied to the tool:
• The first for the UK Sentry contract, covers the Study Group order requirement to
run a test with data of a weapon system contract (result: OBC preferred strongly).
• The requirement for a test case with large/complex capability services is covered by
the test case UK Aerial Refuelling (result: OBC preferred strongly).
Detailed information about the test results is provided in ANNEX II
Development Tool and User Manual.
4.3.2 Recommendations
The task of the Study Group was not to develop a fully comprehensive tool based
on fully validated software, but to show that the processing of a broad set of key parameters
can be of great support for NATO in defining the best type of contract in answer to an RFP.
The Study has shown that the present process and decision support tool can be
adapted in different ways, which are suggested below:
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-32
At process level:
➔ Further develop the process, scope and capabilities
o extend to more contract types
o further test of cross parameters interaction to avoid unforeseen results
➔ Improve process through results from Lessons Learned (LL)
o an LL committee –including NATO personnel- should review the parameters
list. Detailed value changes should be suggested for the parameters based on
the related results
o potentially develop other applications to meet NATO’s personnel expectations
o improve the actual decision support tool, as the considered test cases are
somewhat subjective and have to be taken with caution. E.g. test cases for
Non-OBC have not been taken into account
At tool level:
➔ further develop the tool by continuing to consider the parameters
o pursue identifying and integrating all impacting parameters
o other parameters that could be considered for inclusion are:
▪ Budget
▪ Time frame
▪ Quantity of systems,
▪ Cyber security factors
▪ Intellectual property rights
▪ Detailed contract types
o the current prototype tool is based on Excel®. Consideration should be given
to implementation on dedicated COTS SW once mature.
➔ Improve the tool
o through additional functions
o through technology (e.g. parameter database, integration with Big Data and
Artificial Intelligence
o through innovation
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-33
5 STUDY RECOMMENDATIONS
The most important findings of the Study are that OBC processes have the same value for NATO as traditional contracting processes and the overarching recommendation is, that NATO fully adopts and implements OBC processes.
To implement OBC, it is recommended:
1. NATO should develop a high level policy for contracting for systems and logistics
services. This policy should cover all kinds of contracts (traditional contracting and
outcome-based contracting e.g. for capability).This policy has to be in line with the
NATO Policy for Systems Life Cycle Management (SLCM) approved by the North
Atlantic Council and signed by the Secretary General in 2006.
2. NATO should enter into OBC arrangements starting with two pilot programs,
utilizing the Decision Support Tool developed as part of this Study. This trial
should be performed by NSPA and should be limited to a defined period of time
(e.g. 2 years).
During this time it is recommended to perform a Training Needs Analysis (TNA) for contracting including the use of the Decision Support Tool to be included in the current LCMG TNA.
3. NATO should review and update the NATO publications on contracting including
all aspects of contracting – traditional and OBC- based ones. The updated
publications have to be in line with the NATO Policy for Systems Life-Cycle
Management.
Furthermore it is recommended to specify in the relevant NATO documentation at
which point in the life-cycle of the system the responsible manager should evaluate if OBC
contracting is preferred to traditional contracting.
The basis of this work are the results of the work of the former Group on Acquisition Practices (AC/313) which was created to advise the CNAD on government defense procurement practices and procedures relating to defense trade, armaments cooperative programme arrangements and-industrial collaboration within the Alliance. AC/313 was disbanded.
This work should be performed by a new AC/327 Working Group with members both from Nations and the NIAG IIG and co-chaired by AC/327 and NIIG.
The evidence observed during the Study demonstrates that well-structured and defined OBCs will reduce total costs, increase system availability for the war-fighter, optimize industry performance (investments and costs) to deliver an improved outcome, i.e. a win-win for government and industry.
The Study Group further recommends:
1. NATO should develop a policy on OBC in order to secure common understanding of the concept and of its underlying principles. Such policy and common understanding will facilitate decision-making in a multinational environment.
2. Responsibilities for all parties involved have to be defined clearly.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-34
3. Industry should have the full responsibility for delivery of the service, to include set
up and managing of the arrangements between supplier parties.
4. In systems with multiple providers, designate a single integrator to avoid ambiguity
in responsibilities.
5. Desired outcomes must be defined from the perspective of the end-user
6. Specific authorities and responsibilities must be defined for changes in system
configuration (e.g. modifications, upgrades, service bulletin incorporation,
obsolescence management, etc.) prior to contract.
7. Proper planning and due diligence are required to understand the customer’s
desired outcomes, set the appropriate performance outcome metrics, and
negotiate effective incentives.
8. NATO (e.g. NSPA/NCIA/) should start OBC with two pilot programmes using the Decision Support Tool developed as part of this Study.
9. The term Outcome Based Contracting (OBC) should be selected to cover all contracting functions that provide outcome-based or performance-based results.
10. The contracting method (OBC or traditional) should be selected as early as possible in the life-cycle of the system/subsystem, in consultation with industry starting with the Concept Stage of a programme.
11. The use of an analytical tool-set for contracting type selection at the highest-level NATO life-cycle policy and contracting regulations is required.
12. Conduct a follow-on study to determine the appropriate use of metrics in contracting. Define processes for Government Quality Assurance Organisations to evaluate outcome metrics and determine if adequate OBC results are delivered.
13. It is recommended to further develop the process, scope and capabilities of the decision support tool, to extend it to more contract types, and to enhance cross parameters interaction to better balance parameter ratings.
14. Initiate and conduct training in contracting processes, including the use of a decision support tool.
15. Improve the use of the tool through return of experience.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-35
6 APPENDICES
6.1 Appendix A – Programmes Reviewed / Other Studies Leveraged
Members of the study group represented companies of a wide range of industries
and Nations. They leveraged their individual backgrounds and experiences to analyse
programmes supported by outcome-based arrangements, and reviewed related studies
conducted by others. These arrangements, and studies, covered a full-spectrum of
scenarios – both in peace and wartime – providing services, material, and/or maintenance
support to specific end items, from components, to subsystems, to entire weapon systems.
The analysis group was aware of many programmes that are supported by outcome-
based arrangements. The table on page 36 provides a list of programmes reviewed.
Obtaining analysis data for some of these programmes proved difficult, for reasons
including national security, business sensitive, and proprietary nature. In cases where data
was unavailable or inadequate, the group elected to use anecdotal information or results of
previously completed studies. The table on page 37 provides a list of studies that were
leveraged to inform the analysis. The list of these reports is included in Appendix F –
Reference Documents.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-36
Programmes Reviewed
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-37
Study Title Source Date
Proof Point Project, A Study to Determine the Impact of
Performance Based Logistics (PBL) on Life Cycle Costs
US Department of
Defense
November 2011
Impact of Performance Based Contracting on Product Reliability:
An Empirical Analysis
Management
Science, Vol 58/No.
5
May 2012
Power by the Hour: Can Paying Only for Performance Redefine
How Products are Sold and Serviced?
University of
Pennsylvania,
Wharton School of
Business
January 2018
Performance Based Logistics, Conclusive Evidence Supporting
the Impact of PBLs on Life Cycle Costs, Public Presentation
US Department of
Defense
January 2013
Additional Studies Assessed
6.2 Appendix B – OBC Arrangement General Characteristics
This section provides characteristics and attributes observed in the reviewed
programmes and other studies. There are universal characteristics of outcome-based
arrangements found in practice. The list below is more fully described in Annex I:
• “What vs. How”: Requirements expressed as “what” is desired, not specifics on
“how” to achieve the results. This is the essence of OBC. “How” type contracts are
more typically defined as transactional. To achieve an outcome, the contractor is
given leeway to define how they will perform and deliver the desired outcome as
expressed by the government.
• Defined metrics: To determine if the desired outcome has been met, specific
metrics are defined to measure outcomes. Typically, the more successful
programmes had less than 6 outcome metrics for overall performance, e.g.: time-
on-wing, engineering response time, availability, and perfect order fulfilment.
• Financial incentives tied to metrics: These incentives include awards for
improving system performance or reducing overall cost to support the system. Some
incentives have come in the form of “award term”, providing additional contractual
period of performance without recompeting/renegotiating the contract. Incentive
awards, or conversely, withholding, are tightly associated with the contractor’s ability
to meet the defined metrics.
• Risk/Rewards and Long-term arrangements: Taking on the responsibility for
system performance increases contractor risk. Before, contractors sold spares and
repairs to the government on demand, with no incentive or capability to optimize
system performance and cost. With OBC arrangements, contractors are now
responsible for system performance and cost, raising risk to profit and reputation.
To mitigate this increased risk, contractors will typically make internal investments
to improve the system, or the reliability of spares, with the expectation of reduced
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-38
costs and improved performance. OBC arrangements are typically of a longer length
than transactional contracts – a period of performance long enough to allow
contractors a reasonable expectation for a return on invested capital. Most OBC
arrangements are between 3-5 years. Some arrangements exceed 25 years, made
up of 5-year contracts reviewed in the final year, and if metrics are met additional
periods of performance awarded.
• Authority to manage: To achieve optimal system performance and cost often
requires the integration of many product support, or logistics elements. The
successful OBC arrangements are those where the contractor was provided
authority to integrate, and make trade-offs, between all those supporting activities.
Some nations have legal requirements to control support activities and direct the
use of government-owned, or organic, support. In those cases, successful
programmes engaged in “public-private partnering” between the contractor and
other government support activities.
6.3 Appendix C – Case Studies Supporting Results /
Recommendations
UK Sentry Whole Life Support Program
Contract between the UK MoD Defence Equipment & Support (DE&S) and Northrop
Grumman for the support to the seven UK Airborne Warning and Control System (AWACS)
E-3D aircraft.
Aircraft are based at RAF Waddington, Lincolnshire. Aircraft were delivered in 1991
& 1992 with little consideration of support contracting at time of delivery. Initially there were
up to 12 supply contracts for repairs and spares for the total fleet of 7 aircraft, with the RAF
performing all front-line maintenance. Each aircraft required deep maintenance every 6
years, returned to industry for a period of time up to 9 months. This methodology of
sustainment resulted in only 5 of 7 aircraft being operational for significant periods.
The MOD Integrated Project Team (IPT) looked at creative and innovative methods
for contracting to achieve greater availability. It looked at combining:
• Aircraft maintenance,
• The provision of spares and repairs,
• Design Engineering Support Services (DESS),
• Technical Publications
• Maintenance training
The IPT adopted a Single Track Maintenance (STM) approach whereby only one
aircraft would be in maintenance at any time increasing the available aircraft from 5 to 6.
A competitive bid process was held between Boeing (aircraft OEM) and Northrop
Grumman (radar supplier – primary mission sensor), with an internal UK MOD maintenance
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-39
option as a comparator. Due to high complexity and the IPT looking for innovation and
creative solutions to the problem, a second round of bidding was necessary to refine
assumptions and reduce risk. Northrop Grumman was subsequently selected on a full
through life cost analysis basis.
To mitigate the additional time taken to choose a contractor, a risk reduction
preliminary contract was let to ensure work in critical areas. Requirements were tailored so
that if the full contract was not let as planned, then the work would still benefit reversion to
the UK MoD maintenance option.
The contract runs until 2025 and has firm prices agreed for the first five years. The
remaining period is subject to a Target Cost Incentive Fee (TCIF) arrangement under which
cost savings are split three ways; 50 per cent is retained by the contractor, 20 per cent is
retained by the Authority and 30 per cent is retained in a savings pool.
Money held in the savings pool is used to ‘reward’ the contractor if he meets certain
performance criteria. If there is no money in the saving pool then, irrespective of the
contractor’s performance, no ‘reward’ is paid. The incentive arrangements under the
contract are, therefore, ‘self-funding’.
The contract requires the contractor to make six of the seven Sentry aircraft
available at any time. The contractor is responsible for arranging maintenance, including
spares and repairs, to meet this requirement. Payments are made on the basis of aircraft
availability as set out in the contract.
The IPT over time plans to increase the scope of the contract to cover Whole Life
Support, with the ultimate goal of being to have all Sentry support activities under one
contract. Initially there were some elements of support that were still contracted separately,
leading to dependencies that fell to the Authority to manage under the project.
The contract length helps to mitigate the risks associated with the issue of
obsolescence. For example, smaller parts suppliers are more likely to keep their
manufacturing lines open if there is a long-term contract for aircraft maintenance.
The contract makes use of a partnering approach. This recognises that the IPT
cannot deliver the outputs that they require alone, and neither can industry. Both sides must
work together as partners to deliver the requirement. The difference with partnering is that
both parties are working closely together with greater visibility of the desired outcome. For
Sentry, a Partnering Principles document was generated after the final bidder was selected.
This was an aspirational code of behaviour rather than a legally binding contract.
The method used to draw up the Sentry contract was totally different from previous
methods. In the past, industry would be directed to the proposed solution, rather than
agreeing the most appropriate solution, as in this case. The Team took an innovative
approach to letting the contract, summarised as follows “There are no rules – only
authoritative guidance. So long as we don’t reduce the safety level of the aircraft, and we
are not doing anything illegally, then anything is possible.”
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-40
If the aircraft is required to go to war, the contractor will deploy key employees and
stock to the aircraft’s new operating base. Selected employees of the contractor are
sponsored reservists and, when required, will put on uniform and support the aircraft in
theatre.
UK Aerial Refuelling – RAF Voyager Force Contract
General:
Contract between UK MoD Defence Equipment & Support (DE&S) and Air Tanker
Holdings Ltd for the provision of 14 Voyager Aircraft and full Support for 27 years.
Aircraft are used for four roles:
• Aerial Refuelling (2 or 3 hose and drogue systems)
• Air Transport
• Medivac (40 stretchers and full medical support)
• VVIP transport
PFI Contract signed March 2008 with support to run until 2035 – includes
procurement and aircraft certification and set up of support infrastructure.
Air Tanker Holdings Ltd consortium is a joint venture between the following parties
and their responsibilities:
• Airbus – Basic aircraft Airbus A330-200 and modifications to tanker configuration
• Rolls Royce – Rolls Royce Trent 772 engines
• Thales – avionics supplier and simulator provision
• Cobham – Under-wing and Belly Flight Refuelling Units (FRU)
• Babcock – provision of labour
Infrastructure (provided under contract) located at RAF Brize Norton
• All maintenance facilities
• Training facility (aircrew & maintenance) including full flight simulator
• Aircrews and maintenance personnel all based at Brize Norton
• Aircraft and crews (air and ground) deployed to a number of operating locations in
accordance with MoD tasking demands
Crew (aircrew & maintenance) are a mix of Full-time RAF personnel, Sponsored
RAF Reserves and Civilians. Crew working on MAR aircraft are RAF personnel, crew on
CAR aircraft are generally civilian although military crew with civil certification can fly. The
contract also covers support aircrew for air transport duties (flight attendants).
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-41
Voyager fleet split into ‘Core’ and ‘Surge’ fleet as follows:
• Core fleet – 9 a/c (8 a/c on Military Air Register (MAR) providing Air Refuelling and
1 a/c on Civil Air Register (CAR) providing Transport duties)
• Surge fleet – 5 a/c, 2 a/c on MAR also used for reserves to Core fleet and 3 a/c on
CAR leased out to commercial operators, generates 3rd party revenue for Air
Tanker
• All aircraft have fixed fittings to enable Air Refuelling but when on CAR there are
significant role fit equipment removals and changes to the cabin required to
convert from MAR to CAR and vice-versa
• All aircraft can carry two underwing refuelling pods (hose & drogue) and a limited
number (≈6) also fitted to carry belly refuelling (hose & drogue)
Contract Key Features:
1. A contract planning baseline number of flight hours per annum used (scopes size
of support required) – actual hours over first few years approx. 10% below
planning baseline.
2. Guaranteed Minimum Usage (GMU) payment for flight hours on both MAR and
CAR fleet used to ensure guaranteed payment to Air Tanker (coverage of aircraft
loan repayment cost).
3. Guaranteed Minimum Availability (GMA) payment for 5 aircraft daily (core fleet) –
currently achieving very high availability of 9 aircraft (unscheduled maintenance is
only reflected in metric). Again to ensure guaranteed payment to Air Tanker.
4. Variable charges above GMU & GMA.
a. Available aircraft, usage hours, and number of sorties.
b. Special Equipment charge – dependent on role (e.g. AAR, VVIP transport).
5. Strategic Change in contract for Air Tanker to re-cost contract if usage varies by
±25% against planning baseline hours (see 1 above).
6. Air Tanker have exclusivity on all Air Transport tasks of 100+ people and Air
Refuelling tasks that can be accommodated in the annual planning baseline hours.
7. Contracted obligation for MoD to provide Government Furnished Equipment (GFX)
– Air Tanker receipts back to MoD for licenced RAF engineers.
8. MoD responsible for all fuel, contractor T&S, landing and handling costs.
9. Contract effectively incentivises Air Tanker to fly as many hours as possible (in all
roles) – variable fees for flying higher than variable costs at Air Tanker. Air Tanker
also incentivised to maximise core fleet availability – use of surge fleet to
supplement main fleet when necessary – but lower 3rd party lease revenue.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-42
10. Air Tanker responsibility to ensure appropriate insurance in place – aviation and
non-aviation insurance cover for assets in build and in-service phase. Indemnity
provisions assume assets are insured.
11. Air Tanker provides indemnity for MoD loses arising from design, manufacture or
operation of assets. MoD provides indemnity for other aspects including Air
Refuelling, environmental losses, operations.
12. Licences – Air Tanker required to achieve and sustain various licences.
13. Civil obsolescence fully covered (over 27 years), military obsolescence partially
covered by MoD.
14. Service Credits for MoD for any shortfall in delivering service
a. Late commencement of operations (after build up phase).
b. Failure to complete a sortie (e.g. because of an equipment failure) – very few
to date.
Contracting issues causing biggest impact on contract:
1. Aircraft are a mix of standard commercial aircraft with additional modifications (e.g.
refuelling provisions), supplemented by GFX (DAS, Secure radios, etc) when used
for military missions (MAR):
a. Obsolescence / Upgrade management – when, incentives – change is
difficult to manage.
b. Mission System Reliability – with mix of responsibilities for equipment and the
effect on system reliability – for example cryptos working with standard radios
and the aircraft integrated communication system including antennas.
2. Use of Military Pilots and Maintainers.
a. Difficulty if not being used in military service – MoD has to provide correct
numbers of staff under contract, risk of pilots and maintainers becoming civil
certified and migrating to Commercial fleet world.
6.4 Appendix D – Sample language
6.4.1 Definitions
Logistic Capability: A capability of supporting forces / supporting industry to
ensure / re-establish the performance / availability of an SOI.
• Remarks: Several Logistic Capabilities are to be defined.
• Reference: NATO et al
• Due to: both (Product / Service)
• Related Perspective: both (Customer / Contractor)
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-43
Performance metrics: A means of performance measurement which can be
defined / identified / assessed / evaluated / calculated / categorized. Also known as Key
Performance Indicator (KPI)
• Remarks: Several KPIs are to be defined in order to measure the performance of
the implementation of Logistic Capabilities.
• Reference: NATO et al
• Due to: both (Product / Service)
• Related Perspective: both (Customer / Contractor)
6.4.2 Found Sources and Suggestions for Sample Language
Reference Page Para Paragraph Text Suggestion – sample language
NATO Lifecycle
Management
Policy
1 2 "The aim is to promote the
acquisition of military
systems for NATO that
fulfil the full range of
through life requirements
in a cost effective
manner."
"The aim is to promote the
acquisition of military systems
and services for NATO that fulfil
the full range of through life
requirements in a cost effective
manner."
NATO Lifecycle
Management
Policy
1 3.1. "I recommend the NATO
Policy for SLCM as a
basis to enable the
contributors to the
fulfilment of NATO military
capabilities to work
together to achieve
efficient and timely
delivery of military
systems that meet the
military needs at
affordable cost."
"I recommend the NATO Policy
for SLCM as a basis to enable
the contributors to the fulfilment
of NATO military capabilities,
supported by logistic capabilities,
to work together to achieve
efficient and timely delivery of
military systems that meet the
military needs at affordable
cost."
NATO Lifecycle
Management
Policy
1-1 6 "This allows for traditional
acquisition as well as for
iterative developments
and procurement cycles
where necessary for the
realisation of required
capabilities."
"This allows for traditional
acquisition as well as for iterative
developments and procurement
cycles where necessary for the
realisation of required
capabilities and/or contracting
for logistics capability."
NATO Lifecycle
Management
Policy
1-2 6 "5.1.4 Collaboration with
Industry. SLCM needs a
close working relationship
with Industry, maximum
use of civil standards ( C-
M(2004)0009(INV) dated
25 February 2004. NATO
Framework for Civil
Standards) where
"5.1.4 Collaboration with
Industry. SLCM needs a close
working relationship with
Industry, maximum use of civil
standards ( C-
M(2004)0009(INV) dated 25
February 2004. NATO
Framework for Civil Standards)
where appropriate, full
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-44
appropriate, full
exploitation of new
technologies and shared
domain expertise in order
to benefit from
commercial best
practices."
exploitation of new technologies
and system life cycle
management processes, and
shared domain expertise in order
to benefit from commercial best
practices."
NATO AAP 20 10 1 "Only NATO, Nation,
groups of Nations have to
provide a certain military
capability, they are
accountable for the
Programme over its entire
life cycle of the system.
The industry takes the
responsibility for the
product life cycle as
integrated part of the SLC
on the basis of contracts
with the customer. The
figure 5 shows notional
this structured approach.
The real structure
depends on the contracts
in the specific
programme."
"Only NATO, Nation, groups of
Nations have to provide a certain
military capability, they are
accountable for the Programme
over its entire life cycle of the
system. The industry takes the
responsibility for the product life
cycle as integrated part of the
SLC on the basis of contracts
with the customer, where risks
and performance metrics are
defined depending on the
contract type and contract
duration. The figure 5 shows
notional this structured
approach. The real structure
depends on the contracts in the
specific programme."
NATO AAP 20 17 3 & 4 CONCEPT STAGE/ "The
main thrust of the Study
Phase is to conduct an
evaluation of alternative
technical concepts for
satisfying the identified
capability need and to
identify the most
promising technical
concepts for further
evaluation. The study
effort is an iterative
process and terminates
with the recommendation
of the preferred solution.
The conduct of market
surveys, including
evaluation of the
availability of off-the-shelf
solutions, is an essential
part of the effort. The
primary task of the study
efforts are to identify
possible system solutions
to the requirements, along
with performance
objectives that will guide
activities beyond this
CONCEPT STAGE/ "The main
thrust of the Study Phase is to
conduct an evaluation of
alternative technical concepts for
satisfying the identified capability
need and to identify the most
promising technical concepts for
further evaluation. The study
effort is an iterative process and
terminates with the
recommendation of the preferred
solution. The conduct of market
surveys, including evaluation of
the availability of off-the-shelf
solutions, is an essential part of
the effort. The primary task of
the study efforts are to identify
possible system solutions to the
requirements, along with
performance objectives that will
guide activities beyond this
phase, and to make appropriate
recommendations for the follow-
on phase. When industry
contracting is estimated
necessary for meeting the
identified capability need,
relevant industries will be
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-45
phase, and to make
appropriate
recommendations for the
follow-on phase."
informed about this need as
early as possible during the
concept stage. Follow-on
consultations with such
relevant industries will be
organised in order to better
evaluate the feasibility of
certain contract types and to
receive industrial feedback on
the capability to meet certain
performance objectives.
NATO AAP 20 20 2 The Concept Stage
should consider
architecture, integration,
and possibly certification
constraints. · Develop
initial Configuration
Management (CM) Plan ·
Develop initial Integrated
Logistics Support (ILS)
Plan · Develop initial
Obsolescence
Management
Strategy/Plan · Outline
design solutions in the
form of drawings, models,
prototypes, etc… ·
Prepare/Update
programme/project
management plan ·
Develop and refine life
cycle cost estimates and
human resource
requirements; · Develop
preliminary programme
schedule · Perform initial
risk management
activities · Estimate Life
Cycle Cost (LCC) using
Allied Life Cycle Cost
Publication (ALCCP-1) as
a guide · Conduct an
after action review to
capture lessons learned"
The Concept Stage should
consider architecture,
integration, and possibly
certification constraints. ·
Develop initial Configuration
Management (CM) Plan ·
Develop initial Integrated
Logistics Support (ILS) Plan ·
Develop initial Obsolescence
Management Strategy/Plan ·
Outline design solutions in the
form of drawings, models,
prototypes, etc… ·
Prepare/Update
programme/project management
plan · Develop and refine life
cycle cost estimates and human
resource requirements; ·
Develop preliminary programme
schedule. Based on prior
consultations with the
industry, the schedule should
also specify which portions of
the programme are expected
to be delivered through
outcome-based contracting,
the specificities of which will
be taken into account when
preparing the bidding
documentation. · Perform
initial risk management activities
· Estimate Life Cycle Cost (LCC)
using Allied Life Cycle Cost
Publication (ALCCP-1) as a
guide · Conduct an after action
review to capture lessons
learned"
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-46
NATO AAP 20 26 3 & 4 "Production Stage/ A
critical path analysis will
identify the essential
priorities and
synchronisation required
to realize the SOI and to
achieve efficiency. A full
risk assessment will
identify sub critical paths
and sensitive elements.
Production is entirely a
matter for the participating
nations and parties in the
programme. It is also
possible to include
nations who have not
participated in the
Development Stage. In
such cases the financial
and industrial implications
of such expansion will
require special
consideration and
agreement."
"Production Stage/ A critical path
analysis will identify the essential
priorities and synchronisation
required to realize the SOI and
to achieve efficiency. A full risk
assessment will identify sub
critical paths and sensitive
elements. Production is entirely
a matter for the participating
nations and parties in the
programme.
Based on prior information to,
and consultation with, relevant
industries during the concept
stage, the critical path
analysis will also take into
account the specificities of
outcome-based contracting
when this has been identified
as the preferred way to deliver
(parts of) the SOI. In this
context, risk sharing models
and incentives for the
industrial contractor(s) will
also be defined in detail. It is
also possible to include nations
who have not participated in the
Development Stage. In such
cases the financial and industrial
implications of such expansion
will require special consideration
and agreement."
NATO AAP 20 29 9 During the Utilisation
Stage, the following tasks
and activities should be
performed: · Obtain
enabling products and
services · Assign trained
and qualified operators ·
Activate the system in its
intended operational
environment · Monitor
operation to ensure that
the system is operated in
accordance with the
operations’ plans,
occupational safety and
environmental protection
regulations, and the
international humanitarian
law · Monitor the system
operation by collecting
data to confirm that
service performance is
within acceptable
During the Utilisation Stage, the
following tasks and activities
should be performed: · Obtain
enabling products and services ·
Assign trained and qualified
operators · Activate the system
in its intended operational
environment · Monitor operation
to ensure that the system is
operated in accordance with the
operations’ plans, occupational
safety and environmental
protection regulations, and the
international humanitarian law ·
Monitor the system operation by
collecting data to confirm that
service performance is within
acceptable parameters, to
include reliability, maintainability,
and availability. When
applicable, monitoring
activities should take into
account the specificities of
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-47
parameters, to include
reliability, maintainability,
and availability · Perform
failure identification
actions when non-
compliance has occurred
in the delivered services ·
Determine corrective
course of action, if
applicable · Update
operating procedures as
necessary · Solicit users’
feedback · Request for
corrective design change,
if applicable · Address
life-extension
considerations · Review
and implement
engineering changes
through a staged AAP-20
approach · Estimate Life
Cycle Cost (LCC) using
Allied Life Cycle Cost
Publication (ALCCP1) as
a guide · Conduct an
After Action Review to
capture lessons learned;
performance-based logistics
compared to other types of
contract. As such, a pre-
defined set of performance
metrics should be used that is
characteristic to PBL. ·
Perform failure identification
actions when non-compliance
has occurred in the delivered
services · Determine corrective
course of action, if applicable ·
Update operating procedures as
necessary · Solicit users’
feedback · Request for
corrective design change, if
applicable · Address life-
extension considerations ·
Review and implement
engineering changes through a
staged AAP-20 approach ·
Estimate Life Cycle Cost (LCC)
using Allied Life Cycle Cost
Publication (ALCCP1) as a guide
· Conduct an After Action
Review to capture lessons
learned;
NATO AAP 20 31 2 Support Stage/
Description/ "During the
support stage the
following tasks and
activities should be
performed: · Implement
the (...) · Monitor the
system’s capability to
deliver service and record
problems for analysis ·
Take corrective, adaptive,
(...)
Support Stage/ Description/
"During the support stage the
following tasks and activities
should be performed: ·
Implement the (...) · Monitor the
system’s capability to deliver
service and record problems for
analysis. When applicable,
monitoring activities should
take into account the
specificities of performance-
based logistics compared to
other types of contract. As
such, a pre-defined set of
performance metrics should
be used that is characteristic
to PBL. · Take corrective,
adaptive, (...)
NATO AAP 20 3-2 1 ANNEX 3: Modification
and upgrade procedure
within the utilisation and
support stages / Figure
20: Modification and
Upgrade Procedure
Overview / Inputs: Need/
stakeholder requirement
to change SOI; SOI and
ANNEX 3: Modification and
upgrade procedure within the
utilisation and support stages /
Figure 20: Modification and
Upgrade Procedure Overview /
Inputs: Need/ stakeholder
requirement to change SOI; SOI
and its configuration;
Obsolescence; Reliability,
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-48
its configuration;
Obsolescence; Reliability,
Availability,
Maintainability,
Supportability, Testability
(RAMST) Feedback
issues; Maintenance
Burden; Accidents,
Incidents and Defect
Investigation; Lessons
learned; Changing
environment; Changing
threats, scenarios;
Enhancements; Life
extension; Legal
Changes; In-Service data/
experience; Production
easement; Technology
insertion; Interoperability;
Programme/ Project
Plans; Etc....
Availability, Maintainability,
Supportability, Testability
(RAMST) Feedback issues;
Maintenance Burden; Accidents,
Incidents and Defect
Investigation; Lessons learned;
Changing environment;
Changing threats, scenarios;
enhancements; life extension;
Legal Changes; In-Service data/
experience; Production
easement; Technology insertion;
Interoperability; Programme/
Project Plans; Etc....
NATO AAP 20 3-3 3 & 4 "Study Phase/ Activities in
this phase will provide an
evaluation of alternative
technical solutions for
satisfying the
modification/upgrade
need and identify the
most promising technical
concepts for further
evaluation. The Study
Phase will terminate once
sufficient information has
been obtained to permit
the selection of the
preferred feasible
solution, thereby ensuring
that the entry criteria for
the Programme
Establishment Phase can
be met. Input
to this phase is a
modification/upgrade
need.
"Study Phase Activities in this
phase will provide an evaluation
of alternative technical solutions
for satisfying the
modification/upgrade need and
identify the most promising
technical concepts for further
evaluation. The Study Phase will
terminate once sufficient
information has been obtained to
permit the selection of the
preferred feasible solution,
thereby ensuring that the entry
criteria for the Programme
Establishment Phase can be
met. During the Study
Phase, an assessment should
also be performed of whether
and where outcome-based
contracting would be more
appropriate/ beneficial from a
cost and efficiency points of
view.
Input to this phase is a
modification/upgrade need (...)"
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-49
NATO AAP 20 3-5 3 & 4 4.2.1 Programme
Establishment Phase / At
the end of this phase,
confirm down selection of
the preferred option,
freeze the requirements,
agree on specifications,
and agree on
qualification, certification,
verification, and
acceptance criteria to be
included within the formal
proposal of Industry. The
overall output of the
Concept Stage is a full
Product Specification and
the Qualification/
Verification Plan for the
defined
Modification/Upgrade.
4.2.1 Programme Establishment
Phase / At the end of this phase,
confirm down selection of the
preferred option, freeze the
requirements, agree on
specifications, and agree on
qualification, certification,
verification, and acceptance
criteria to be included within the
formal proposal of Industry.
Prior consultations with
industry during the concept
phase (if any) will be taken
into account. The overall output
of the Concept Stage is a full
Product Specification and the
Qualification/ Verification Plan
for the defined
Modification/Upgrade.
AACP-02vol1 13 5 A contract clause on
pricing should consider
the following areas
commensurate with the
type
and scope of contract
involved:
Pricing arrangements
Cost determination
Price variation
Profit or fees
Taxes and duties
Cost sharing
Audit of costs
Sub-contracts
Currency.
A contract clause on pricing
should consider the following
areas commensurate with the
type
and scope of contract involved:
Pricing arrangements
Cost determination
Price variation
Profit or fees
Incentives
Taxes and duties
Cost sharing
Audit of costs
Sub-contracts
Currency.
AACP-02vol1 15 1 The method of payment
must be set forth clearly.
When establishing the
method of payment,
the type of contract
should b considered.
Normally, payment should
be made upon completion
of the
contract item. There are,
however, depending on
individual countries,
policies and procedures,
other
options such as milestone
payments, progress
payments and advance
payments, if any of the
The method of payment must be
set forth clearly. When
establishing the method of
payment,
the type of contract should b
considered. Normally, payment
should be made upon
completion of the
contract item. There are,
however, depending on
individual countries, policies and
procedures, other
options such as milestone
payments, progress payments
and advance payments, if any of
the latter
methods are chosen, it is
important that an adequate
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-50
latter
methods are chosen, it is
important that an
adequate amount of the
total payment be retained
in order
to settle outstanding
questions on contractual
completion. Moreover,
when advance payments
are
made, consideration
should be given to the
need for a bank
guarantee.
amount of the total payment be
retained in order
to settle outstanding questions
on contractual completion.
Moreover, when advance
payments are
made, consideration should be
given to the need for a bank
guarantee. Performance
payment concept should also
be taken into account. This
means payment might be done
according to the contractors
performance. So PBC should
be based: Specifying
performance in terms of
required outputs and/or
outcomes. Design incentives
to Designing incentives to link
supplier payment and/or non-
financial rewards to the
achievement of specified
outputs and/or outcome.
AACP-02vol1 16 1 The contract must make
provision for the
purchaser to be aware of
the status of the work on
the
contract. It is therefore
necessary to define the
mechanisms of keeping
the purchaser informed.
Such
mechanisms may include,
but are not limited to,
reports, progress review
meetings, site visits and
auditing of records. The
requirements associated
with necessary access to
premises should be
defined.
It is the purchaser's
prerogative to specify
these mechanisms.
The contract must make
provision for the purchaser to be
aware of the status of the work
on the
contract. It is therefore
necessary to define the
mechanisms of keeping the
purchaser informed. Such
mechanisms may include, but
are not limited to, reports,
progress review meetings, site
visits and
auditing of records. The
requirements associated with
necessary access to premises
should be defined.
It is the purchaser's prerogative
to specify these mechanisms.
In PBC, following assessments
should also be analysed:
Performance measurement and
reporting system.
Note that incentives impact on
supplier behaviour.
NSPA No. 4200 2 4.1 4.1 A principal objective of
the NSPA is to obtain,
through international
competitive bidding, the
most economical prices
for materiel and services.
The most economical
A principal objective of the
NSPA is to obtain, through
international competitive bidding,
the most economical prices for
materiel and services. The most
economical proposal meeting
the technical and contractual
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-51
proposal meeting the
technical and contractual
requirements stipulated in
the Request for Proposal
(RFP) shall normally be
accepted.”
requirements stipulated in the
Request for Proposal (RFP)
shall normally accepted.At the
request of the customer,
NSPA will evaluate if OBS
contracting would be overall
more economical and of better
quality than traditional
contracting.”
NSPA No. 4200 4 4.8 4.8 Non-Competitive
Solicitation “NSPA shall
promote and provide for
full and open international
competition in soliciting
offers and awarding
contracts. However,
under specific
circumstances, solicitation
procedures other than full
and open international
competitive bidding may
be utilized.
Circumstances under
which competitive bidding
rules need not be strictly
followed are: 4.8.1 Sole
Source (….); 4.8.2 Single
Source: (…)
To be added:
4.8.3 OBS contracting that
spans over a period of several
years, and subject to positive
assessment of its
performance, may justify a
decision to restrict the
solicitation process to the
contractor originally selected,
although other potential
sources may exist.
NSPA No. 4200 5 4.8 Page 5, Para 4.8.2.4.
Commonality of
Equipment
Page 5, Para 4.8.2.4.
Commonality of Equipment or
Services
NSPA No. 4200 8 5.8 5.8.3 Proposals received
shall be evaluated to
determine the most
economical technically
and contractually
compliant proposal.
5.8.3 Proposals received shall
be evaluated to determine the
most economical technically and
contractually compliant proposal
and / or best value
evaluations.
NSPA No. 4200 9 8 8.1 NSPA will normally
conclude contracts of
such duration as to
ensure completion of the
requirements. The
duration of these
contracts will normally not
exceed a 5-year period,
including option periods.”
COMMENT: 5 years is too short
to enable provision of
performance of
services/capabilities plus
renewal options.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-52
AQAP-2000 4 4 2.3.2 The acquirer
The acquirer’s
responsibility is to:
a. Ensure the capture of
the requirements to the
necessary extent by: (1)
Translation of the user's
needs into appropriate
functional and technical
requirements. (2)
Consideration of the
interests of all the other
interested parties in the
life cycle. (3)
Consideration of all
applicable national and
international regulations,
including environmental
regulations. (4) Reflection
of the resulting
requirements in the
project documents, e. g.
the contract, and in the
continuing project
management processes.
b. Ensure the preparation
of contracts that: (1) Allow
sufficient freedom and
incentives for the
suppliers, e.g. to use
commercial products. (2)
Allow the acquirer the
fullest view of the
suppliers’ activities in
order to gain confidence
in the resultant product.
(3) Contain appropriate
quality management
requirements. c. Ensure
compliance ...
2.3.2 The acquirer
The acquirer’s responsibility is
to:
a. Ensure the capture of the
requirements to the necessary
extent by: (1) Translation of the
user's needs into appropriate
functional and technical
requirements. (2) Consideration
of the interests of all the other
interested parties in the life
cycle. (3) Consideration of all
applicable national and
international regulations,
including environmental
regulations. (4) Reflection of the
resulting requirements in the
project documents, e. g. type of
contracting (traditional,
outcome-based), the contract,
and in the continuing project
management processes.
b. Ensure the preparation of
contracts that: (1) Allow
sufficient freedom and incentives
for the suppliers, e.g. to use
commercial products. (2) Allow
the acquirer the fullest view of
the suppliers activities in order
to gain confidence in the
resultant product. (3) Contain
appropriate quality management
requirements. (4) define
quantifiable targets and
performance metrics in case
of outcome-based contracting;
(5) define incentives for the
achievement of specific
targets within a specific
timeframe;
c. Ensure compliance ...
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-53
AQAP-2070 15 2 SECTION 6. RISK
IDENTIFICATION,
ASSESSMENT AND
COMMUNICATION
GUIDANCE (page
15,para 2)
“6.2. Unknown Risks It is
recognised that, in some
situations, risk information
may not be (…) especially
risks associated with the
Supplier's performance.
SECTION 6. RISK
IDENTIFICATION,
ASSESSMENT AND
COMMUNICATION GUIDANCE
(page 15, para 2)
“6.2. Unknown Risks. It is
recognised that, in some
situations, risk information may
not be (......) especially risks
associated with the Supplier's
performance.
If Firm Fixed Price contract
type is preferred in the
framework of Outcome-based
contracting, the Delegator may
delegate the assessment of
quality-related risks that may
occur from Contractor’s
necessity to cut/ control costs
despite acknowledged
uncertainty associated with
the development of new
technology or very complex
capability development. Such
risk assessment should be
undertaken in line with
potential risk premium
provisions (for the Contractor)
defined in the contract or the
in the intent to contract.”
AQAP-2070 19 6 SECTION 8. GQA
REQUEST GUIDANCE
(page 19, para 6)
“8.6. Associated
Documentation
The Delegator should
provide directly or through
the Supplier, the
documentation necessary
to plan and perform GQA
including the contract and
product specifications to
the Delegate. The
documentation should
detail, as applicable, the
following: a)
Legal/statutory
requirements that could
affect the contract and/or
the performance of
GQA,………….h)
Contract delivery
schedule requirements.
SECTION 8. GQA REQUEST
GUIDANCE (page 19, para 6)
“8.6. Associated Documentation
The Delegator should provide
directly or through the Supplier,
(,………….)h) Contract delivery
schedule requirements.; i) If
outcome-based contracting is
applied, the outcomes,
performance indicators and
associated Contractor
incentives as agreed between
the acquirer and the supplier.”
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-54
AQAP-2070 C2 –
C3
2 ANNEX C – GQA RISK
IDENTIFICATION,
ASSESSMENT AND
COMMUNICATION (page
C2 -C3,
C.3.3.1 Sources of Risk
Information
(….) Contract Review –
(...)
ANNEX C – GQA RISK
IDENTIFICATION,
ASSESSMENT AND
COMMUNICATION (page C2 -
C3, para 2)
C.3.3.1 Sources of Risk
Information
(….) Contract Review – (...)
Contract type and supplier
contractual constraints –
Contractual provisions (e.g.
fixed price contract vs.
contract length) that may
determine the supplier to
adjust costs and processes
while making sure that quality
is not affected.
AACP-2 – Vol. 2 G-15 1 AMENDMENTS (PD &
DD)
10.1 Rights and
responsibilities of the
contract parties to modify
the Contract must be
addressed in view of the
possible need for
changes. As part of this,
the manner in which
changes will be
accommodated as well as
the question of associated
costs should be defined. It
may ...
10. AMENDMENTS – Page G-
15
“10.1 Rights and responsibilities
of the contract parties to modify
the Contract must be addressed
in view of the possible need for
changes. As part of this, the
manner in which changes will be
accommodated as well as the
question of associated costs and
potential incentives to the
contractor should be defined. It
may be ...
AACP-2 – Vol. 2 G-28 2 PRICE
16.6 In firm or fixed price
situations, the provision
will normally require the
Contractor:
16.6.1 to make available
particulars regarding the
plans for the work, the
methods by which the
various tasks are intended
to be performed and the
basis of his estimated
costs; and
16.6.2 to permit the
examination of his
facilities and his
processes with a view to
facilitating the agreement
of a reasonable estimate
for the work.
PRICE
16.6 In firm or fixed price
situations, the provision will
normally require the Contractor:
(…..)"
In firm or fixed price situations
where outcome-based
contracting is involved, the
provision may require the
Contracting authority provide the
contractor with measurable
targets adapted to each
development stage to help the
Contractor to better estimate
costs and possible savings.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-55
AACP-2 – Vol. 3 G-12 2 2.5 Rights of contractors”
(Page G12)
2.5.2 Consultation,
preference, compensation
2.5.2.1 In considering
whether a contractor
should be given any right
to prior consultation,
priority treatment or
compensation when
awarding work to another
contractor, consideration
should be given, inter alia,
to the following:
- national practices;
- value for money
considerations;
- practicalities of
production (including
timescales) by another
contractor;
- work sharing
requirements.”
2.5 Rights of contractors” (Page
G12)
2.5.2 Consultation, preference,
compensation
2.5.2.1 In considering whether a
contractor should be given any
right to prior consultation, priority
treatment or compensation when
awarding work to another
contractor, consideration should
be given, inter alia, to the
following:
- national practices;
- value for money
considerations;
- contractor performance and
contract length provisions
negotiated in the framework of
outcome-based contracting.
- practicalities of production
(including timescales) by another
contractor;
- work sharing requirements.”
AACP-2 – Vol. 4 G-21 3 15. PRICE
15.3 Wherever possible,
Contracting Authorities
seek to proceed on the
basis of a firm (non
revisable) price or a fixed
(revisable) price. This
approach aims not only to
limit their financial liability
but also to provide an
incentive for the
contractor to perform
efficiently.
15. PRICE (page G-21)
15.3 Wherever possible,
Contracting Authorities seek to
proceed on the basis of a firm
(non revisable) price or a fixed
(revisable) price. This approach
aims not only to limit their
financial liability but also to
provide an incentive for the
contractor to perform efficiently.
When needed/ applicable, the
contract will include what
incentives should be given to
the Contractor for the latter’s
performance regarding price
(and other indicators, as
applicable e.g. quality).
AACP-2 – Vol. 10 COMMENT:
In performance based logistics;
performance, incentives and risk
issues are the key elements. As
the auditing process continues,
performance should be tracked.
The performing authority should
audit performance of the
contractor according to the
requirements which determined
by the requesting authority.
Afterwards the performing
authority can submit a
performance measurement
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-56
report to the requesting
authority.
NATO ALP 10 1 1.2. APPLICABILITY Decisions on support
requirements have the
greatest impact on system
performance, life cycle
cost and supportability
when taken early in the
period of the life cycle of a
programme and a system.
Decisions on support
requirements and contracting
method to provide logistics
capability have the greatest
impact on system performance,
life cycle cost and supportability
when taken early in the period of
the life cycle of a programme
and a system.
NATO ALP 10 2 3.1. General System operational
availability to improve the
military capabilities of the
alliance is a primary
objective of multinational
armament programmes. It
is NATO policy to ensure
that financial and other
resources required to
maintain operational
availability receive
the same emphasis as
those required to achieve
performance objectives
and timely delivery of the
system. These resources
should include those
necessary to design
desirable support
characteristics into the
system, implement them
through manufacturing or
integration, as well as
those to plan, develop,
acquire and evaluate
their support.
Performance-based
agreements, including
contracts, should include
appropriate ILS
performance
requirements.
System operational availability to
improve the military capabilities
of the alliance is a primary
objective of multinational
armament programmes and it is
dependent on logistics
capability level (i.e. material
availability). It is NATO policy to
ensure that financial and other
resources required to maintain
operational availability receive
the same emphasis as those
required to achieve performance
objectives and timely delivery of
the system. These resources
should include those necessary
to design desirable support
characteristics into the system,
implement them through
manufacturing or integration, as
well as those to plan, develop,
acquire and evaluate
their support. Performance-
based agreements, including
contracts, should include
appropriate ILS performance
requirements.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-57
NATO ALP 10 2 3.1. General The ILS process should
begin at the Concept
Stage of the life cycle of
the armament programme
and continue for the life of
the system. The primary
objective of the ILS
programme should be to
achieve the required
system operational
availability at minimum life
cycle cost. Early ILS
activitiesshould focus on
designing desirable
support characteristics
and on determining
support requirements.
Subsequent activity
focuses on equipment
support requirement
evaluation and
preparation of provision of
support resources. The
scope and level of detail
should be tailored to meet
specific programme
needs at each stage of
the system life cycle.
Annex C lists the ILS
considerations for
each stage of an
armaments programme.
The ILS process should begin at
the Concept Stage of the life
cycle of the armament
programme and continue for the
life of the system. The primary
objective of the ILS programme
should be to achieve the
required system operational
availability at minimum life cycle
cost. Early ILS activities should
focus on designing desirable
support characteristics and on
determining support
requirements. During
conceptual stage assessment
of contracting methodology
should be performed to
effectively satisfy required
logistics capability.
Subsequent activity focuses on
equipment support requirement
evaluation and preparation of
provision of support resources.
The scope and level of detail
should be tailored to meet
specific programme needs at
each stage of the system life
cycle. Annex C lists the ILS
considerations for
each stage of an armaments
programme.
NATO ALP 10 3 3.2. Systems
Engineering and
Utilisation/Support
Relationship
Information obtained from
systems engineering
sources is required to
ensure that all aspects of
utilisation support are
recognized and
considered during the
planning and acquisition
of the support elements
(i.e. support equipment,
repair parts, personnel
and training, facilities,
communication, security,
Information Technology
(IT) framework,
supply and maintenance
technical assistance,
equipment and software
publications).
Information obtained from
systems engineering sources is
required to ensure that all
aspects of utilisation support are
recognized and considered
during the planning and
acquisition of the support
elements (i.e. support
equipment, repair parts,
personnel and training, facilities,
communication, security,
Information Technology (IT)
framework,
supply and maintenance
technical assistance, equipment
and software publications) and
evaluating contract types for
logistics support.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-58
NATO ALP 10 3 3.3. ILS Elements The primary objective of
any new armament
programme isto provide a
military capability at
minimum life cycle cost.
Operational availability is
one of the principal
determinants of military
capability.
The primary objective of any
new armament programme isto
provide a military and logistics
capability at minimum life cycle
cost. Operational availability is
one of the principal determinants
of military capability and it is
tied to material availability.
NATO ALP 10 4 3.3.1. Maintenance
Planning
Maintenance Planning
includes, but is not limited
to the following:
- Levels of repair
– Repair times
- …
Maintenance Planning includes,
but is not limited to the following:
- Levels of Repair
- repair times
- ...
-Provide data for the
assessment of contracting for
logistics capability
NATO ALP 10 7 3.4. Life Cycle Cost
(LCC)
Early in the life cycle, the
LCC analysis
concentrates on
quantifying the cost
implications of selected
design alternatives, which
provide the desired level
of performance.
Early in the life cycle, the LCC
analysis concentrates on
quantifying the cost implications
of selected design alternatives
and contracting types for
logistics support, which
provide the desired level of
performance and logistics
capability.
NATO ALP 10 C-4 3. Development
Stage
Take steps to ensure that
ILS considerations are
given appropriate weight
in requests for criteria for
source contractor
selection and contract
provisions. Contract
requirements clearly
define a baseline
operational scenario,
baseline maintenance
concepts,NATO
peacetime availability and
wartime deployment
objectives and support
schedule objectives. ILS
programme and data
requirements should be
tailored to meet these
objectives.
Take steps to ensure that ILS
considerations are given
appropriate weight in requests
for criteria for evaluation of
contract types, source
contractor selection and contract
provisions. Contract
requirements clearly define a
baseline operational scenario,
baseline maintenance concepts,
NATO peacetime availability and
wartime deployment objectives
and support cost and schedule
objectives. ILS programme and
data requirements should be
tailored to meet these objectives.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-59
NATO ALP 10 C-4 4. Production Stage > Validate and deliver ILS
elements to meet
deployment needs.
Ensure that logistics
support arrangements will
be implemented prior to
the start of the Utilisation
Stage.
> Finalize and endorse a
formal document which
addresses the
multinationally organized
utilisation support and its
essential elements.
> Validate and deliver ILS
elements to meet deployment
needs. Ensure that logistics
support arrangements will be
implemented prior to the start of
the Utilisation Stage.
> Finalize and endorse a formal
document which addresses the
multinationally organized
utilisation support and its
essential elements.
> Should outcome-based
contracting be chosen as
preferred way to deliver (parts
of) the SOI (during the concept
phase), contract type, risk
sharing models and incentives
will also be defined in detail.
NATO ALP 10 E-3 2.2.1. 2.2.1 Support risks:
Identify risk associated
with system support
alternatives. As a
minimum, the following
areas should be
addressed:
§ What are the effects of
changing the level of
maintenance/repair
capability? § Are there
items or subsystems in
the inventory that can be
used to reduce
development
risk/requirements? § How
will the proposed materiel
system be integrated into
the Service’s structure at
maturation? (The system
must be designed to fit
into the appropriate
Service’s support
structure planned for the
fielding time frame to
reduce changes needed).
2.2.1 Support risks: Identify risk
associated with system support
alternatives. As a minimum, the
following areas should be
addressed:
§ What are the effects of
changing the level of
maintenance/repair
capability? § Are there items or
subsystems in the inventory that
can be used to reduce
development risk/requirements?
§ How will the proposed materiel
system be integrated into the
Service’s structure at
maturation? (The system must
be designed to fit into the
appropriate Service’s support
structure planned for the fielding
time frame to reduce changes
needed). What are the risks
for proposed logistic support
model?
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-60
NATO ALP 10 E-4 2.4 ILS/
Supportability
funding
§ Describe studies and
investigations to be
conducted and updated in
determining, by ILS
element, total life-cycle
cost estimates to include
an identification of the
scope and depth of
studies to be conducted.
Include plans for transition
of support to item
managers and the
respective sustainment
command.
§ State support models
and modifications to be
used in cost estimating
and limitations and
assumptions to be made
in modelling.
§Provide coordination
channels and reporting
schedules.
§ State results
(dollars/type funds) of
cost estimating, by ILS
element, major function,
operation and
maintenance. Include total
requirements.
§ Describe studies and
investigations to be conducted
and updated in
determining, by ILS element,
total life-cycle cost estimates to
include an identification of the
scope and depth of studies to be
conducted. Include plans for
transition of support to item
managers and the respective
sustainment command.
§ State support models and
modifications to be used in cost
estimating and limitations and
assumptions to be made in
modelling.
§Provide coordination channels
and reporting schedules.
§ State results (dollars/type
funds) of cost estimating, by ILS
element, major function,
operation and maintenance.
Include total requirements.
§ Define contracting types and
evaluate their impact on LCC.
NATO ALP 10 E-6 3.1 Maintenance
Planning
3.1.4.Indicate strengths
and weaknesses of each
support alternative and
the effect of the support
concept on the system
design, acquisition and
O&S costs, and on
affected ILS elements.
After 3.1.4 :
Indicate strengths and
weaknesses of each support
alternative and the effect of the
support concept on the system
design, acquisition and O&S
costs, and on affected ILS
elements.
Define benefits and risks of
possible contracting types
depending on proposed
support concepts.
NATO ALP 10 E-7 3.2 Supply Support 3.2.6.Address possible
need for inter-service
supply support
agreements
3.2.6 Address possible need for
inter-service supply support
agreements.
Determine the scope of supply
support that will be included
in outcome based logistics
support.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-61
NATO ALP 10 F-2 PROJECT
MANAGEMENT ILS
RESPONSIBILITIES
In the ILS process most
elements represent
functional areas which are
individually managed by
technical specialists. The
ILS Manager’s role should
be to coordinate and
interface these functional
areas to achieve
integration of all ILS
elements into an ILS Plan.
Thus, responsibilities of
the ILS Manager or staff
officer are intended to:
(a) Develop an ILS Plan
and ...
(b) ...
(i) Ensure an orderly,
timely and efficient
transfer of overall logistic
support responsibilities
and know-how to the
system user or in-service
support
organization.
In the ILS process most
elements represent functional
areas which are individually
managed by technical
specialists. The ILS Manager’s
role should be to coordinate and
interface these functional areas
to achieve integration of all ILS
elements into an ILS Plan. Thus,
responsibilities of the ILS
Manager or staff officer are
intended to:
(a) Develop an ILS Plan and ...
(b) ...
(i) Ensure an orderly, timely and
efficient transfer of overall
logistic support responsibilities
and know-how to the system or
in-service support organization
organization.
( ) Ensure ILS requirements
and outcomes are considered
when defining logistics
capability and evaluating
contract types of outcome
based logistics.
NSPA OI 4200-01 11 4 “3.2 Requests for Price
and Availability (P&A):
The general NSPA policy
is to solicit bids or
proposals in respect of
requirements only when
there is a definite intention
to award a contractual
instrument. If solicitation
is necessary for other
reasons, i.e. for
procurement planning or
information (exploratory)
purposes, the
Procurement Officer must
approve it. (…)
3.3 Letter of Interest.
3.3.1 It is frequently
necessary to contact
industry prior to
solicitation. (…) Contacts
with industry may be
made either through the
use of a Letter of Interest
or pre-solicitation
conferences. (…).”
COMMENT: In case of OBS
contracting, the provisions
related to “Solicitation” would
need to be adapted to allow
earlier integration of Contractors.
“Letters of Interest” and “Pre-
solicitation conferences” may not
be sufficient in this case. In
addition, some Definitions need
to be clarified.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-62
NSPA OI 4200-01 22 05-apr Evaluation of Proposals –
Proposals received shall
be evaluated to determine
the most economical
technically and
contractually compliant
proposal. (…)
COMMENT: Judgement of
Proposals of Capabilities
demands substantial, objective
competence to “evaluate to
determine the most economical,
technically and contractually
compliant proposal”. Would be
useful if NSPA OI 4200-01
described how that is done and
by whom. “Best Value evaluation
methodology” would also need
to be addressed.
6.5 Appendix E – Decision Support Tool
6.5.1 Parameters List
The following list describes the recommended parameters to consider when
engaging in outcome-based arrangements. These parameters were predominant factors
exhibited in the analysed programmes and studies. Such parameters were deemed useful
for inclusion in the decision-support tool described ANNEX II Development Tool and
User Manual.
Parameter Observations
1 Requirement Successful outcome-based contracts had clearly defined warfighter/end-user
requirements.
3 Metrics – Type
Use of top-level outcome metrics was more valuable than output metrics in
determining successful results (e.g. availability of a system vs. number of spares
repaired).
4 Metrics – Number used Successful outcome-based arrangements typically considered only 3-5 top-level
metrics. More than 5 metrics usually indicates a lack of clarity of desired output.
5 Complexity of System
Supported
The complexity of the system supported often determined the level of the
outcome-based arrangements (i.e. component, subsystem, or system level
support).
6
Life-Cycle: Contractor
Performance History /
Experience
The range of available and experienced suppliers/providers relates to the
maturity of the system supported; impacting the level of competition.
7 Life-Cycle: Contract type
System maturity (e.g prototype, development, production, support, or
sundowning) had a direct effect on type of contract/period of performance. Less
mature systems typically engaged OEMs on shorter, cost-type contracts, and
aged systems near end-of-life typically were not conducive to outcome-based
arrangements at all.
8 Life-cycle: Data and
Technology Available
Systems with significant performance data are more conducive to outcome-
based, fixed-price arrangements – less risk. New-start programmes, with little
performance data, were better managed with shorter-term and cost-type
arrangements.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-63
9 Contractor Freedom of
Solution Choice
Outcome-based arrangements totally managed by industry result in lower risk
and greater success. Imposed use of government agencies as part of the
solution resulted in less control or the options and higher risk.
10 Provision for Performance
Improvement
The greater the freedom to perform reliability improvements, and conduct
configuration changes, the greater the opportunity to improve performance and
cost.
11
Period of Performance:
Return on Investment
Opportunity
Longer periods of performance (minimum of 5 years) allow greater opportunity
for return on investments made by industry.
12 Scope – Extent of service
provided
The more control the contractor has over the integration of support elements, the
greater the opportunity for success.
13 Contract Type
More successful arrangements utilized fixed price contract structures with
incentives. Cost Plus fixed-fee is not considered an outcome-based contract
type.
14
Obsolescence
Management /
Modernization and
Upgrade
The broader the authorities for mitigating risk of obsolescence, improving the
capability, held by the contractor, the more successful the programme.
15 Number of Nations Greater numbers of nations covered by the same outcome requirements
simplifies contracting processes.
16 Contractor PBL Experience
Success leads to success. Those contractors who have previous successful
experience with PBL are more likely to leverage that experience in additional
PBL opportunities.
Selected Parameters
6.5.2 Definitions of the parameters
No Parameter Definition
1 Requirement
A formal statement of what is needed.
Example: service level requirement; project requirement; required deliverables for a
process. (NATO Standardisation Office, “TermOTAN”, approbation C3B)
An essential condition that a system has to satisfy. (ISO/IEC 2382-20: 1990, ADatP-
02 20.01.02 )
Requirement definition involves defining and describing what is needed and will be
procured, collecting information, identifying appropriate solutions and specifying these
in specifications for goods and equipment and/or statements of work (SOW) for works
and services. (NATO International Staff Procurement Manual,- EM(2010)0285-REV1)
2 Cost Analysis
Cost: The amount of cash or cash equivalents paid or the fair value of the other
consideration given to acquire an asset at the time of its acquisition or construction.
(AAP-43, NATO glossary Defense – budget et finances )
Cost analysis:
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-64
1 : the act of breaking down a cost summary into its constituents and studying and
reporting on each factor.
2 : the comparison of costs (as of standard with actual or for a given period with
another) for the purpose of disclosing and reporting on conditions subject to
improvement. (Webster dictionary)
3 Metrics – Type
Metric: A quantifiable entity that allows the measurement of the achievement of a
process goal. Something that is measured and reported to help manage a process,
service or activity. (COBIT – ITIL – C3 Policy Glossary)
4 Metrics – Number
used
Metric: A quantifiable entity that allows the measurement of the achievement of a
process goal. Something that is measured and reported to help manage a process,
service or activity. (COBIT – ITIL – C3 Policy Glossary)
5 Complexity of
System Supported
Consisting of many diverse and autonomous but interrelated and interdependent
components or parts linked through many (dense) interconnections. Complex systems
cannot be described by a single rule and their characteristics are not reducible to one
level of description. They exhibit properties that emerge from the interaction of their
parts and which cannot be predicted from the properties of the parts. (The Business
dictionary)
In our case, the complexity of the system supported derives series of contract clauses
that are necessary to address every and each of the desired system specificities.
(redactor)
6
Performance
History /
Experience
Performance: The accomplishment of a given task measured against preset known
standards of accuracy, completeness, cost, and speed. In a contract, performance is
deemed to be the fulfilment of an obligation, in a manner that releases the performer
from all liabilities under the contract (The Business dictionary)
Performance history: A record of an employee's history of achievement.(Oxford
dictionary)
Experience: Familiarity with a skill or field of knowledge acquired over months or
years of actual practice and which presumably, has resulted in superior understanding
or mastery. (the business dictionary)
7 Contract starting
point in life cycle
Life cycle: The period of time that starts with the assessment of the requirement for a
system, facility or product and ends with its final disposal. The life cycle may comprise
a number of sequential phases, including requirement assessment, planning,
feasibility studies, definition, design and development, production, installation and
testing, operation and maintenance, and final disposal (NC3B)
8 Availability of Data
and Technology
Point in time where data and technology will be mature enough and disseminated with
sufficient provision of due capacities to enable their incorporation into the project as a
part of the solution. (redactor)
9
Contractor
Freedom of
Solution Choice
Possibility granted to the provider by the contract manager to choose the type of
solution he wants, given the solution will fulfil all expressed requirements. (redactor)
10
Provision for
Performance
Improvement
Performance: the accomplishment of a given task measured against pre-set known
standards of accuracy, completeness, cost and speed. In a contract, performance is
deemed to be the fulfilment of an obligation, in a manner that releases the performer
from all liabilities under the contract. (the business dictionary)
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-65
11
Period of
Performance:
Return on
Investment
Opportunity
Return on investment: the earning power of assets measured as the ratio of the net
income (profit less depreciation) to the average capital employed in a company or
project. Expressed usually as a percentage, return on investment is a measure of
profitability that indicates whether or not a company is using its resources in an
efficient manner. (the business dictionary)
12 Scope – Extent of
service provided
Service provision: a task or service provided to a consumer in exchange for
acceptable compensation. (the business dictionary)
The extent of service provided is the quantified and qualified amount of service
defined by the contract to be provided. (redactor)
13 Contract Type
Contract: a voluntary, deliberate, and legally binding agreement between 2 or more
competent parties.
NATO defines several sorts of contract in adequate to the solution needed as well as
its desired completion. (the Business dictionary)
14
Obsolescence
Management /
Modernization and
Upgrade
Obsolescence: A lack of availability of an item or raw material resulting from statutory
and process changes, as well as new designs. (Defense Acquisition Glossary)
Upgrade: Raise (something) to a higher standard, in particular improve (equipment or
machinery) by adding or replacing components (Oxford Dictionary)
15 Number of Nations
The number of Nations creates difficulties of convergence towards a common target
ad roadmap, in task and funding sharing, as well as in the manner to achieve the
solution; (redactor)
16 Contractor PBL
Experience
Past experience of the contractor in fulfilling PBL contracts. (redactor)
6.5.3 Parameter list development
The initial parameter list considered the 3 point of view (ENDUSER – NATO -
INDUSTRY). This was further reflected against AACP-2, an “US assessment tool” and
reflected in the output of Section 4.1
6.5.3.1 Initial parameter list
The initial parameter list considered the 3 points of view (ENDUSER – NATO -
INDUSTRY)
1. Scope included in the contract- topic – What´s in /out – workshare – achievable?
2. Timeframe period of the contract
3. Total Cost of the contract
4. Commitment among the parties (KPIs and incentive /penalty models)
5. Availability of the Weapon System (Level of Service)
6. Risks
7. Control over the asset (Management of the Asset/Authority to Employ)
8. Surrounding Parameters (Legal, Interoperability & Regulatory Constraints)
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-66
9. Number of Costumers (Multinational, National, Pool)
6.5.3.2 Reflection of the parameters selected against parameters from other sources
The selected parameters have been reflected against parameters from other
sources: from a “US assessment tool” and against AACP-2.
Reflection against the Parameters (Factors) from “US assessment tool”
• Reduction in total cost of ownership, life cycle cost.
• Presence, determination of metrics, performances outcomes
• Improvement in performance
• Partnership, strategic alliance
• Length of contract
• Flexibility
• Better service, best value creation, best practices
• Successful implementation of BLK, PBL experience and training
• Innovation
• Data availability
• Candidates and competition
• Properties of system
• Up-front, early planning
Our parameters were also reflected against possible Parameters from AACP-2 (only
headlines shown here)
• Abstract & key aspects: most salient points of the Contract at a glance
• Order of precedence (requirements and parameters ranking)
• Scope of Work;
• Content (definition / selection of equipment)
• Content (Performance – level of service)
• Execution of the contracts (Operational management – i.e. premises, dates,
consignees);
• Contract administration (administrative management) and financial guarantees
• Language.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-67
6.6 Appendix F – Reference Documents
Documents analysed separately
• NATO Policy for Life Cycle Management
• NATO Standard AAP-48 (
• NATO Standard AAP-20 )
• NATO Guidance ALP-10 (ILS)
• AACP-2 (Acquisition Practices) –
• AACP-2 (Acquisition Practices) – Vol.2
• AACP-2 (Acquisition Practices) – Vol.3
• AACP-2 (Acquisition Practices) – Vol.4
• AACP-2 (Acquisition Practices) – Vol.5
• AACP-2 (Acquisition Practices) – Vol.6
• AACP-2 (Acquisition Practices) – Vol. 10
• AAP-06 (Terms & Definitions) 2016 AAP-15 (Abbreviations) 2016 )
• NATO Policy AQAP-2000 Quality Through Life Cycle NATO Standard AQAP-2070
(GQA)
• NSPO Procurement Reg. No. 4200
• NSPA Procurement Operating Instructions OI 4200-01
• ALP-10 Training Package
• STANAG 4597 (EDITION 2) – OBSOLESCENCE MANAGEMENT
• NATO STANDARD ACMP-2009 GUIDANCE ON CONFIGURATION
MANAGEMENT
Other NATO Documents Considered
• STANREC 4755 (Lifecycle Cost)
• NATO Standard ACMP-2000 (Config Mngt)
• STANAG 4728 (System Lifecycle Mngt)
• STANAG-4728-SRD.1 (Induction Training)
• NATO Guidance ALCCP-1 (Lifecycle Costs)
• NATO Standard ARAMP-1 (Risk Management)
• Investment Committee – Procedure Bidding Best Value
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-68
• NC3A – Best Value Evaluation Methodology
• Infrastructure Committee – Procedure Bidding Best Value
• Infrastructure Committee – Procedure Int. Compt. Bidding
• STANAG 4107 (Quality)
• STANAG 4427 (Configuration Management)
• STANAG 4739 (Risk Management)
• SRD-4739 (Training Package on Risk Management)
• STANREC 4755 (Lifecycle Cost)
• NATO Standard ACMP-2000 (Config Mngt)
• NATO Standard ACMP-2008 (Guidance)
• NATO Guidance AQAP-2009 (Use AQAP-2000)
• NATO Guidance AQAP-2105 (Quality Plans)
• NATO Quality Assurance AQAP-2110
• AQAP-2110-SRD.1 (Transition, Implementation)
• NATO Quality Assurance AQAP-2120 (Production)
• NATO Quality Assurance AQAP-2130 (Fin. Inspection)
• NATO Standard AQAP-2210 (Software Quality Assurance)
• NATO Standard AQAP-2310 (Aviation, Space, Def.)
• NATO Standard AQAP-4107 (Mut. Acceptance AQAP)
• AQAP-4107-SRD.1 (Nat. Authorities, Focal Points)
Non-NATO Documents
• U.S. PBL Guidebook, March 2016 Background
• U.S. IPS Element Guidebook Background
• U.S. DoD Directive – Defense Acquisition System Background
• U.S. DoD Instruction – Defense Acquisition System Background
• U.S. DoD Instruction – Defense Acquisition of Services Background
• U.S. GAO – 2000 CENSUS – Lessons Learned Background
• U.S. DAU – Introduction to Life-Cycle Logistics Mngt Background
• U.S. DAU – Overview 12 IPS Elements Background
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-69
ANNEX I GLOSSARY
The Study Team found sometime different definitions of terms used during the
performance of the Study. To assist the reader a glossary of terms used in this Study, was
developed.
Acquisition Strategy (AS)
Describes the Program Manager’s plan to achieve program execution and
programmatic goals across the entire program life cycle. Summarizes the overall approach
to acquiring the capability (to include the programs has been prepared to provide context
schedule, structure, risks, funding, and the business strategy). It contains sufficient details
to allow senior leadership and the Milestone Decision Authority (MDA) to assess whether
the strategy makes good business sense, effectively implements laws and policies, and
reflects management’s priorities. Once approved by the MDA, the Acquisition Strategy
provides a basis for more detailed planning. The strategy evolves over time and should
continuously reflect the current status and desired goals of the program. (Definition by US
DoD)
Availability of Data and Technology
Point in time where data and technology will be mature enough and disseminated
with sufficient provision of due capacities to enable their incorporation into the project as a
part of the solution.
Capability Package
A combination of national and NATO funded infrastructure, associated costs and
manpower which, together with the military forces and other essential requirements, enable
a NATO Commander to achieve a specific Military Required Capability. The Capability
Package directly links military requirements with established force goals by focusing on
those activities most essential to the new strategy, the resultant forces and command
structure and address overall resource implications, both national and international
identifying all elements necessary for the package to function. (NATO Logistics Handbook).
Capabilities-Based Assessment (CBA)
The CBA is an analytical method the US DoD uses to identify needed capabilities
and associated capability gaps that are eventually transformed into requirements. Source:
US CJCSI 3170.01I – Joint Capabilities Integration and Development System).
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-70
Complexity of System Supported
Consisting of many diverse and autonomous but interrelated and interdependent
components or parts linked through many (dense) interconnections. Complex systems
cannot be described by a single rule and their characteristics are not reducible to one level
of description. They exhibit properties that emerge from the interaction of their parts and
which cannot be predicted from the properties of the parts. (source: The Business
Dictionary)
Condition Based Maintenance Plus (CBM+)
In accordance with US Department of Defense (DoD) Instruction 4151.22 Condition
Based Maintenance Plus (CBM+), is the application and integration of appropriate
processes, technologies, and knowledge based capabilities to achieve the target availability,
reliability and operation and support costs of DoD systems and components across their life
cycle. At its core, CBM+ is maintenance performed on evidence of need, integrating
reliability centered maintenance (RCM) analysis with those enabling processes,
technologies and capabilities that enhance the readiness and maintenance effectiveness of
DoD systems and components. CBM+ uses a systems engineering approach to collect data,
enable analysis, and support the decision-making processes for system acquisition,
modernization, sustainment, and operations.
Configuration Management
A discipline applying technical and administrative direction and surveillance to the
following activities i.a.w. NATO ACMP-6:
• Configuration Identification and Documentation
• Configuration Control
• Configuration Status Accounting and
• Configuration Audits.
Contract starting point in life cycle
The period of time that starts with the assessment of the requirement for a system,
facility or product and ends with its final disposal. The life cycle may comprise a number of
sequential phases, including requirement assessment, planning, feasibility studies,
definition, design and development, production, installation and testing, operation and
maintenance, and final disposal (source: NC3B)
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-71
Cost Analysis
Cost analysis is the review and evaluation of the separate cost elements and profit
or fee in an offeror’s or contractor’s proposal to determine a fair and reasonable price or to
determine cost realism. Cost analysis includes the application of judgment to determine how
well the proposed costs represent what the cost of the contract should be, assuming
reasonable economy and efficiency.
Contractor Logistics Support (CLS)
Contracted weapon system sustainment that occurs over the life of the weapon
system. Can also be defined as the performance of maintenance and/or materiel
management functions for a DoD weapon system by a commercial activity or contractor
sustainment of a weapon system that is intended to cover the total life cycle of the weapon
system. CLS generally includes multiple integrated product support (IPS) elements, but does
not include interim contractor support (ICS), a temporary measure for a system's initial
period of operation before a permanent form of support is in place. CLS also excludes
contractor sustainment support for a specific sustainment task that a Service would
otherwise conduct itself; a typical example would be a weapon system's prime contractor
providing sustaining engineering. Also called Long-Term Contractor Logistics Support.
Source: US Logistics Assessment Guidebook.
Contract Types
The contract type defines the expectations, obligations, incentives, and rewards for
both Government and contractor during an acquisition. The Government contracting officer
selects the contract type based on analysis of the most effective way to satisfy mission
requirements. Many contract types are available to Government and contractors to provide
them with the flexibility they need to acquire/supply the large variety and volume of supplies
and services that the Government requires.
Cost Plus Award Fee Contracts
A cost-plus-award-fee contract is a cost-reimbursement contract that provides for a
fee consisting of an award amount that the contractor may earn for performance and that is
sufficient to provide motivation for excellence in such areas as cost, schedule and technical
performance. The amount of the award fee to be paid is determined by the Government’s
judgmental evaluation of the contractor’s performance in terms of the criteria stated in the
award fee plan (which the DFARS requires be made a part of the contract).
This determination and the methodology for determining the award fee are unilateral
decisions of the individual referred to as the Fee Determining Official (FDO) (FAR – Federal
Acquisition Regulation).
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-72
Cost-plus-incentive-fee contracts
The cost-plus-incentive-fee contract is a cost-reimbursement contract that provides
for the initially negotiated fee to be adjusted later by a formula based on the relationship of
total allowable costs to total target costs. This contract type specifies a target cost, a target
fee, minimum and maximum fees, and a fee adjustment formula. After contract performance,
the fee payable to the contractor is determined in accordance with the formula. The formula
provides, within limits, for increases in fee above target fee when total allowable costs are
less than target costs, and decreases in fee below target fee when total allowable costs
exceed target costs. This increase or decrease is intended to provide an incentive for the
contractor to manage the contract effectively. When total allowable cost is greater than or
less than the range of costs within which the fee-adjustment formula operates, the contractor
is paid total allowable costs, plus the minimum or maximum fee.
Cost-Plus-Fixed-Fee Contracts
A cost-plus-fixed-fee contract is a cost-reimbursement contract that provides for
payment to the contractor of a negotiated fee that is fixed at the inception of the contract.
The fixed fee does not vary with actual cost, but may be adjusted as a result of changes in
the work to be performed under the contract. This contract type permits contracting for efforts
that might otherwise present too great a risk to contractors, but it provides the contractor
only a minimum incentive to control costs.
Depot Level Maintenance
Depot-level maintenance and repair is intended as the material maintenance or
repair requiring the overhaul, upgrading, or rebuilding of parts, assemblies, or
subassemblies, and the testing and reclamation of equipment as necessary. It includes:
• All aspects of software maintenance classified by the Department of Defence as of
July 1, 1995, as depot-level maintenance and repair;
• Interim Contractor Support (depot maintenance performed by a contractor in the
intervening period between production of the first item and establishment of organic
maintenance capability);
• Contractor Logistics Support (depot maintenance performed by the contractor on a
long-term basis).
It does not include:
• The procurement of major modifications or upgrades of weapon systems that are
designed to improve program performance;
• The nuclear refuelling or defueling of an aircraft carrier and any concurrent complex
overhaul;
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-73
• The procurement of parts for safety modifications, although it does include the
installation of safety modifications.
Diminishing Manufacturing Sources and Material Shortages (DMSMS)
According to the January 2016 US DoD Diminishing Manufacturing Sources &
Material Shortages (DMSMS) Guidebook, DMSMS is “...the loss or impending loss of
manufacturers of items or suppliers of items or raw materials may cause material shortages
that endanger a weapon system's or equipment's development, production, or post-
production support capability.” DMSMS directly impacts system readiness, system
availability, potentially costing the Department of Defence hundreds of millions of dollars
annually.
Failure Modes & Effects Analysis (FMEA) and Failure Modes, Effects &
Criticality Analysis (FMECA)
The FMEA systematically identifies the likely modes of failure and the possible
effects of each failure on mission completion (MIL HDBK 502A, Product Support Analysis)
The FMECA addresses the same features as the FMEA; however, the FMECA additionally
addresses the criticality of each effect on mission completion, environmental impacts, health
hazards, and system safety (US MIL HDBK 502A, Product Support Analysis).
Fault Tree Analysis (FTA)
A fault tree analysis (FTA) analyzes high-level failures and identifies all lower-level
(sub-system) failures that cause it. Generally, the undesired event constitutes the highest
level (top) event in a fault tree diagram and represents a complete or catastrophic failure of
the system.
Firm-Fixed-Price Contracts
A firm-fixed-price contract provides for a price that is not subject to any adjustment
on the basis of the contractor’s cost experience in performing the contract. This contract
type places upon the contractor maximum risk and full responsibility for all costs and
resulting profit or loss. It provides maximum incentive for the contractor to control costs and
perform effectively and imposes a minimum administrative burden upon the contracting
parties. The contracting officer may use a firm-fixed-price contract in conjunction with an
award-fee incentive and performance or delivery incentives when the award fee or incentive
is based solely on factors other than cost. The contract type remains firm-fixed-price when
used with these incentives.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-74
Fixed-price incentive (firm target) contracts
A fixed-price incentive (firm target) contract specifies a target cost, a target profit, a
price ceiling (but not a profit ceiling or floor), and a profit adjustment formula. These elements
are all negotiated at the outset. The price ceiling is the maximum that may be paid to the
contractor, except for any adjustment under other contract clauses. When the contractor
completes performance, the parties negotiate the final cost, and the final price is established
by applying the formula. When the final cost is less than the target cost, application of the
formula results in a final profit greater than the target profit; conversely, when final cost is
more than target cost, application of the formula results in a final profit less than the target
profit, or even a net loss. If the final negotiated cost exceeds the price ceiling, the contractor
absorbs the difference as a loss. Because the profit varies inversely with the cost, this
contract type provides a positive, calculable profit incentive for the contractor to control
costs.
Fixed-Price Contracts with Award Fees
Award-fee provisions may be used in fixed-price contracts when the Government
wishes to motivate a contractor and other incentives cannot be used because contractor
performance cannot be measured objectively. Such contracts shall establish a fixed price
(including normal profit) for the effort. This price will be paid for satisfactory contract
performance. Award fee earned (if any) will be paid in addition to that fixed price.
Governmental organization
An organization controlled and financed by its national government (NATO AAP-06
Edition 2016).
Incentive Contracting – Incentives, Award Fee, Award Term
Incentive contracting describes contracts, from either the cost reimbursement or
fixed price families, that are structured to emphasize a specific acquisition objective(s) by
tying it to a financial reward1. A portion of profit, fee, or the term of the contract is dependent
upon criteria that further align Government and contractor interests. Award Fee and
Incentives are the two categories of incentive contracting described in FAR 16.4, DFARS
216.4, and PGI 216.4. Although not discussed in those regulations, Award Term has also
gained full acceptance as a type of incentive.
Infrastructure
A term generally applicable for all fixed and permanent installations, fabrications, or
facilities for the support and control of military forces. (AAP-6)
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-75
Integrated logistic support (ILS)
The management and technical process through which supportability and logistic
support considerations are integrated into the design and taken into account throughout the
life cycle of systems/equipment and by which all elements of logistic support are planned,
acquired, tested and provided in a timely and cost-effective manner (NATO AAP-06 Edition
2016)
Key Performance Parameters (KPPs)
Performance attributes of a system considered critical or essential to the
development of an effective military capability. Failure of a system to meet a validated KPP
threshold value triggers a review by the validation authority and evaluation of operational
risk and/or military utility of the associated system(s) if KPP threshold values are not met.
The review may result in validation of an updated KPP threshold value, modification of
production increments, or recommendation for program cancellation(Source: Manual for the
Operation of the Joint Capabilities Integration and Development System (JCIDS) – Updated
February 2015)
Level of Repair Analysis (LORA)
Level of Repair Analysis LORA is a prescribed procedure for defense logistics
planning. LORA is performed to determine the best, most efficient location where an item
can be repaired.
Life Cycle Cost (LCC)
LCC consists of all direct costs plus indirect variable costs associated with the Life
Cycle stages of the System of Interest (adapted from RTO TR-058 / SAS-028).
LCC comprises the marginal costs (both direct and indirect) of introducing a new
equipment or capability. LCC is used as a minimum for the analysis of alternatives; it does
not include notional allocation of costs, whereas TOC and WLC might do so. LCC is used
to compare options of alternatives, and often for economic analyses (NATO ALCCP-1).
Life Cycle Management
Also referred to as Total Life Cycle System Management (TLCSM), "life cycle
management is the implementation, management, and oversight by the program manager
of all activities associated with the acquisition, development, production, fielding, sustaining,
and disposal of a DOD system." In addition, "the tenets of life cycle management emphasize
sustainment planning early in the capability solution's life cycle, to include requirement
generation activities."
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-76
(Source: Manual for the Operation of the Joint Capabilities Integration and
Development System (JCIDS), (Appendix D to Enclosure D, Para 2))
Logistics Product Data
The portion or subset of Product Support Analysis (PSA) documentation consisting
of detailed data pertaining to the identification of Product Support resource requirements of
a product (US MIL-HDBK-502A, Product Support Analysis).
Logistics Response Time (LRT)
"The period of time from logistics demand signal sent to satisfaction of that logistics
demand required for weapon system logistics support." Source: Undersecretary of Defense
for Acquisition, Logistics and Technology Memorandum, “Performance Based Logistics:
Purchasing Using Performance Criteria,” 16 August 2004
"Logistics Response Time is the period of calendar time from when a
failure/malfunction is detected and validated by the maintainer to the time that the
failure/malfunction has been resolved. This includes: the time from when a need is identified
until the provider satisfies that need, all associated supply chain and maintenance time, and
delivery times of parts" Source: US Army PBL Guidebook
Measures the delivery times of parts to military customers for materiel requisitioned
through the DoD logistics systems or other delivery means to a predetermined location.
According to Benjamin S. Blanchard’s (2004), Logistics Engineering and
Management, 6th ed. Pearson Prentice Hall, Upper Saddle River, NJ, Logistics Response
Time (LRT), is “the time that is consumed from the point when a system support requirement
is first identified.
Logistic sustainment
The process and mechanism by which sustainability is achieved and which consists
of supplying a force with consumables and replacing combat losses and non-combat attrition
of equipment in order to maintain the force's combat power for the duration required to meet
its objectives (NATO AAP-06 Edition 2016)
Logistics
The science of planning and carrying out the movement and maintenance of forces.
In its most comprehensive sense, the aspects of military operations which deal with: a.
design and development, acquisition, storage, movement, distribution, maintenance,
evacuation, and disposal of materiel; b. transport of personnel; c. acquisition or construction,
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-77
maintenance, operation, and disposition of facilities; d. acquisition or furnishing of services;
and e. medical and health service support (NATO AAP-06 Edition 2016)
Maintenance
All action taken to retain materiel in or to restore it to a specified condition. It
includes: inspection, testing, servicing, classification as to serviceability, repair, rebuilding
and reclamation.
All supply and repair action taken to keep a force in condition to carry out its mission.
The routine recurring work required to keep a facility (plant, building, structure,
ground facility, utility system, or other real property) in such condition that it may be
continuously utilized, at its original or designed capacity and efficiency, for its intended
purpose (AAP-6)
Maintenance Task Analysis (MTA)
MTA is the identification of the steps, spares and materials, tools, support
equipment, personnel skill levels as well as any facility issues that must be considered for a
given repair task. Also included in the MTA are elapsed times required for the performance
of each task. MTAs cover both corrective and preventative maintenance tasks and, when
complete, identify all physical resources required to support a system.
Materiel Availability
Materiel Availability is a measure of the percentage of the total inventory of a system
operationally capable (ready for tasking) of performing an assigned mission at a given time,
based on materiel condition. This can be expressed mathematically as (the number of
operational end items divided by the total population). Determining the optimum value for
Materiel Availability requires comprehensive analysis of the system and its planned use,
including the planned operating environment, operating tempo, reliability alternatives,
maintenance approaches, and supply chain solutions. Materiel Availability is primarily
determined by system downtime, both planned and unplanned, requiring the early
examination and determination of critical factors, such as the total number of end items to
be fielded and the major categories and drivers of system downtime. The Materiel
Availability Key Performance Parameter must address the total population of end items
planned for operational use, including those temporarily in a non-operational status once
placed into service (such as for depot-level maintenance). The total life cycle time frame,
from placement into operational service through the planned end of service life, must be
included.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-78
Mean Time Between Failure (MTBF)
According to the Military Handbook on Maintainability Design Techniques Metric
(DOD-HDBK-791) 17 March, 1988, Mean Time Between Failure (MTBF) is “the total
functioning life of a population of an item divided by the total number of failures within the
population during the measurement interval.”
Military Capability
The ability to achieve a specified wartime objective (win a war or battle, destroy a
target set). It includes four major components: force structure, modernization, readiness,
and sustainability.
1. Force structure, numbers, size, and composition of the units that comprise defence
forces; e.g., divisions, ships, air wings.
2. Modernization e.g. technical sophistication of forces, units, weapon systems, and
equipment
3. Unit readiness- the ability to provide capabilities required by the combatant
commanders to execute their assigned missions. This is derived from the ability of
each unit to deliver the outputs for which it was designed.
4. Sustainability- the ability to maintain the necessary level and duration of operational
activity to achieve military objectives. Sustainability is a function of providing for and
maintaining those levels of ready forces, materiel, and consumables necessary to
support military effort.
Obsolescence Management
According to the US Defense Acquisition University’s Glossary of Defense
Acquisition Acronyms & Terms, 16th ed., obsolescence is a lack of availability of an item or
raw material resulting from statutory and process changes, as well as new designs.
Obsolescence deals with the process or condition by which a piece of equipment becomes
no longer useful, or a form and function no longer is currently available for production or
repair. Implementation of new technology causes older technology to become less
supportable because of the diminished availability of parts and suppliers. Mitigation
practices include reviewing proposed parts lists for obsolescence and being proactive in the
engineering design process prior to production.
Original Equipment Manufacturer (OEM)
An Original Equipment Manufacturer (OEM) is a company that produces parts and
equipment that may be marketed by another manufacturer.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-79
Organic
Forming an integral part of a military entity (NATO AAP-06 Edition 2016)
Outcome or result
Results are the same as outcomes. An outcome is a describable or measurable
change that is derived from an initiative's outputs or lower-level outcomes. Outcomes are
qualified as immediate, intermediate, or ultimate; outputs contribute to immediate outcomes;
immediate outcomes contribute to intermediate outcomes; and intermediate outcomes
contribute to ultimate outcomes. Outcomes are not entirely within the control of a single
organization, policy, program or project; instead, they are within the organization's area of
influence. In the context of development, these are also referred to as development results.
Outputs
Direct products or services stemming from the activities of an organization, policy,
program or project.
OBC – Outcome based contracting
Outcome-based contracting is a form of contracting that explicitly includes three
characteristics:
• Clear definition of a series of objectives and indicators by which to measure
contractor performance.
• Collection of data on the performance indicators to assess the extent to which the
contractors are successfully implementing the defined services.
• Performance leading to consequences for the contractors, such as provision of
rewards or imposition of performance adjustments.
OBS – Outcome Based Services
Outcome-based services is a generic term used to describe agreements, contracts
or arrangements between a customer and a supplier in which the supplier is contracted to
directly achieve business and/or service outcomes for and with the customer – rather than
being contracted in terms of delivery of the supplier’s inputs, outputs or deliverables.
Part
In logistics, an item of an assembly or sub-assembly, which is not normally further
broken down (NATO AAP-06 Edition 2016).
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-80
Parts Management
The US DoD Defense Standardization Program Office SD-19 document, Parts
Management Guide, defines ‘parts management’ as the practice of considering the
application, standardization, technology (new and aging), system reliability, maintainability,
supportability, and cost in designing or selecting parts and addressing availability, logistics
support, DMSMS, and legacy issues in supporting them throughout the life of the systems.
Parts are the building blocks from which systems are created and, as such, greatly
impact hardware dependability, readiness, and operating costs. Increasingly important in
today’s acquisition environment – characterized by rapidly changing designs and by
increased risk for US Department of Defense (DoD) weapons systems and equipment
acquisition contracts due to an increased emphasis on the use of commercial part types,
offshore manufacture of parts, Diminishing Manufacturing Sources and Material Shortages
(DMSMS), counterfeit parts, and the use of lead-free parts – the need for contractors to have
an effective Parts Management Program (PMP) is greater than ever before. The PMP is an
integral part of the acquisition process for design, development, modification, and support
of weapons systems and equipment.
Process
A series of time-based activities linked to complete a specific output.
Performance
Performance: the accomplishment of a given task measured against pre-set known
standards of accuracy, completeness, cost and speed. In a contract, performance is deemed
to be the fulfilment of an obligation, in a manner that releases the performer from all liabilities
under the contract. (source: the business dictionary)
Performance Based Logistics (PBL) Overview
Performance Based Logistics (also commonly referred to as Performance Based
Life Cycle Product Support and PBL) is an outcome-based product support strategy for the
development and implementation of an integrated, affordable, product support package
designed to optimize system readiness and meet operational requirements in terms of
performance outcomes for a weapon system through long-term product support
arrangements with clear lines of authority and responsibility.
According to the November 22, 2013 US ASD (L&MR) "Performance Based
Logistics Comprehensive Guidance" memo, PBL is synonymous with performance based
life cycle product support, where outcomes are acquired through performance based
arrangements that deliver operational requirements and incentivize product support
providers to reduce costs through innovation. These arrangements are contracts with
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-81
industry or inter-governmental agreements. Attributes of an effective PBL arrangement
include:
• Objective, measurable work description that acquires a product support outcome.
• Appropriate contract length, terms, and funding strategies that encourage delivery
of the required outcome.
• A manageable number of metrics linked to contract requirements that reflect desired
operational outcomes and cost reduction goals.
• Incentives to achieve required outcomes and cost reduction initiatives.
• Risks and rewards shared between government and commercial product support
integrators and providers.
• Synchronization of product support arrangements to satisfy operational
requirements.
Performance Based Logistics (PBL) Metrics – Overview
Identifying operational requirements is the first step toward establishing a
Performance Based Logistics (PBL) Product Support Arrangement (PSA). Per the ASD
L&MR “Performance-Based Logistics Comprehensive Guidance” Memorandum dated 22
Nov 13, PBL is synonymous with performance-based life cycle product support, where
outcomes are acquired through performance-based arrangements that deliver Warfighter
requirements and incentivize product support providers to reduce costs through innovation.
These arrangements are contracts with industry or intragovernmental agreements.
A key component of any PBL PSA is the establishment of metrics. The desired
performance outcome for the specific PBL arrangement(s) should be linked to the top-level
sustainment requirements of the system (such as Materiel Availability (Am) or Reliability
(R)).Additionally, the PBL arrangements must deliver the needed performance (R or Am) at
reduced total cost when compared to transactional arrangements.
Performance History / Experience
Performance: The accomplishment of a given task measured against preset known
standards of accuracy, completeness, cost, and speed. In a contract, performance is
deemed to be the fulfilment of an obligation, in a manner that releases the performer from
all liabilities under the contract (The Business dictionary)
Performance history: A record of an employee's history of achievement. (Oxford
dictionary)
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-82
Experience: Familiarity with a skill or field of knowledge acquired over months or
years of actual practice and which presumably, has resulted in superior understanding or
mastery. (the business dictionary)
Performance metric
A performance metric measures an organization's behaviour, activities, and
performance. It should support a range of stakeholder needs from customers, shareholders
to employees. While traditionally many metrics are finance based, inwardly focusing on the
performance of the organization, metrics may also focus on the performance against
customer requirements and value. In project management, performance metrics are used
to assess the health of the project and consist of the measuring of seven criteria:
1. safety,
2. time,
3. cost,
4. resources,
5. scope,
6. quality, and
7. actions.
Private Finance Initiative (PFI)
The private finance initiative (PFI) is a way of creating "public–private partnerships"
(PPPs) where private firms are contracted to complete and manage public projects.
Developed initially by the governments of Australia and the United Kingdom, and later used
extensively there and in many countries as part of the wider programme of privatisation.
The private finance initiative (PFI) is a procurement method which uses private
sector investment in order to deliver public sector infrastructure and/or services according
to a specification defined by the public sector.
Product Support Business Case Analysis (BCA)
The Product Support BCA is a structured methodology and document that aids
decision making by identifying and comparing alternatives by examining the mission and
business impacts (both financial and non-financial), risks, and sensitivities. Product Support
BCAs may be somewhat different from other decision support analyses through their
emphasis of the enterprise wide perspective of stakeholders and decision makers and
assessment of the holistic effects impacted by the decision. Other names for a BCA are
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-83
Economic Analysis, Cost-Benefit Analysis, and Benefit-Cost Analysis. Broadly speaking, a
BCA is any documented, objective, value analysis exploring costs, benefits, and risks.
The Product Support BCA concludes with a recommendation and associated
specific actions and an implementation plan to achieve stated organizational objectives and
desired outcomes. The Product Support BCA does not replace the judgment of a decision
maker. Rather, it provides an analytic, standardized, and objective foundation upon which
credible decisions can be made. The Product Support BCA should be a comprehensive,
fair, and accurate comparison when evaluating multiple alternatives. It should take into
account broad Department wide impacts and context throughout the analysis. The PSM
prepares a Product Support BCA for major product support decisions, especially those that
result in new or changed resource requirements. The Product Support BCA helps leadership
with significant investment and strategic decisions across all applications of Product
Support.
Project stakeholder
According to the US Project Management Institute (PMI), the term project
stakeholder refers to, "an individual, group, or organization, who may affect, be affected by,
or perceive itself to be affected by a decision, activity, or outcome of a project" (Project
Management Institute, 2013). ISO 21500 uses a similar definition.
Product Support Manager (PSM)
According to Para 1.3 of the US DoD Product Support Manager’s Guidebook, “the
program manager (PM) is charged with delivering warfighter – required capabilities while
the product support manager (PSM), working for the PM, is responsible for developing and
implementing a comprehensive product support strategy early in the acquisition cycle, and
during and after fielding, adjusting performance requirements and resource allocations
across Product Support Integrators (PSIs) and Product Support Providers (PSPs).
Furthermore, the PSM’s responsibility carries across the life-cycle of the weapon system,
requiring the revalidation of the business case prior to any change in the support strategy or
every five years, whichever occurs first. The PSM must be a properly qualified member of
the Armed Forces or a full-time employee of the DoD” (per Public Law (PL) 111-84 The
National Defense Authorization Act (NDAA) for Fiscal Year 2010, Section 805, and The John
Warner NDAA of 2007, Section 820a).
Product Support Integrator (PSI)
The term “product support integrator” means an entity within the Federal
Government or outside the Federal Government charged with integrating all sources of
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-84
product support, both private and public, defined within the scope of a product support
arrangement.
Product Support Provider (PSP)
The term “product support provider” means an entity that provides product support
functions. The term includes an entity within the Department of Defense, an entity within the
private sector, or a partnership between such entities.
Project
In contemporary business and science, a project is an individual or collaborative
enterprise, possibly involving research or design, that is carefully planned, usually by a
project team, to achieve a particular aim.
A project may also be a set of interrelated tasks to be executed over a fixed period
and within certain cost and other limitations.
It may be a temporary (rather than permanent) social systems as work systems that
is constituted by teams within or across organizations to accomplish particular tasks under
time constraints. A project may be a part of a wider programme management.
Provision for Performance Improvement
Performance: the accomplishment of a given task measured against pre-set known
standards of accuracy, completeness, cost and speed. In a contract, performance is deemed
to be the fulfilment of an obligation, in a manner that releases the performer from all liabilities
under the contract. (the business dictionary)
Period of Performance (PoP)
It is the timeframe allowed or required for performance to be completed.
Public-Private Partnerships (PPP)
According to DoD Instruction 4151.21, Public-Private Partnerships for Product
Support, “a public-private partnership for depot-level maintenance is a cooperative
arrangement between an organic depot-level maintenance activity and one or more private
sector entities to perform DoD or Defense-related work and/or to utilize DoD depot facilities
and equipment. Other government organizations, such as program offices, inventory control
points, and materiel/systems/logistics commands, may be parties to such agreements.”
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-85
The Public-Private Partnering for Product Support Guidebook clarifies that even
though the definition cited from this instruction is in a depot maintenance context, the term
actually applies to the broader range of Integrated Product Support (IPS) Elements and
activities.
Also, the terms 'public-private partnership' and 'public-private partnering' are
frequently used interchangeably and are both often abbreviated as “PPP.”
Requirement
A formal statement of what is needed.
Example: service level requirement; project requirement; required deliverables for a
process. (NATO Standardisation Office, “TermOTAN”, approbation C3B)
An essential condition that a system has to satisfy. (ISO/IEC 2382-20: 1990, ADatP-
02 20.01.02 ) Requirement definition involves defining and describing what is needed and
will be procured, collecting information, identifying appropriate solutions and specifying
these in specifications for goods and equipment and/or statements of work (SOW) for works
and services. (NATO International Staff Procurement Manual,- EM(2010)0285-REV1)
Reliability Centered Maintenance (RCM)
RCM is a systematic approach for identifying preventative or scheduled
maintenance tasks for an equipment end item and establishing necessary preventative (or
scheduled) maintenance task intervals.
Return On Invested Capital – ROIC
A calculation used to assess a company's efficiency at allocating the capital under
its control to profitable investments. Return on invested capital gives a sense of how well a
company is using its money to generate returns. Comparing a company's return on capital
(ROIC) with its weighted average cost of capital (WACC) reveals whether invested capital
is being used effectively.
One way to calculate ROIC is:
This measure is also known as "return on capital."
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-86
Risk
It is the potential of gaining or losing something of value. Values (such as physical
health, social status, emotional well-being, or financial wealth) can be gained or lost when
taking risk resulting from a given action or inaction, foreseen or unforeseen. Risk can also
be defined as the intentional interaction with uncertainty. Uncertainty is a potential,
unpredictable, and uncontrollable outcome; risk is a consequence of action taken in spite of
uncertainty.
Risk management
The process of identifying, assessing, and controlling risk arising from operational
factors, and making informed decisions that balance risk cost with mission benefits (NATO
AAP-06 Edition 2016)
Scope – Extent of service provided
Service provision: a task or service provided to a consumer in exchange for
acceptable compensation.
The extent of service provided is the quantified and qualified amount of service
defined by the contract to be provided.
Supply Chain Management (SCM)
The integration of the supplier, distributor, and customer logistics requirements into
one cohesive process to include demand planning, forecasting, materials requisition, order
processing, inventory allocation, order fulfilment, transportation services, receiving,
invoicing, and payment. Also, the management and control of all materials, funds, and
related information in the logistics process from the acquisition of raw materials to the
delivery of finished products to the end user. (“The Management of Business Logistics” by
John J. Coyle, Edward J. Bardi and C. John Langley (7th Edition dated 2003)
Sustainability
Sustainability’ and ‘sustainable’ mean to create and maintain conditions, under
which humans and nature can exist in productive harmony, that permit fulfilling the social,
economic, and other requirements of present and future generations of Americans.
Sustainability includes a range of initiatives, including solid waste reduction, reduction of
petroleum use for vehicles, sustainable buildings, greenhouse gas reduction, green
(sustainable) procurement, energy reduction, water reduction, and electronic
stewardship/hazardous chemical reduction.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-87
The ability of a force to maintain the necessary level of combat power for the
duration required to achieve its objectives. (AAP-6)
System
A system is a regularly interacting or interdependent group of items forming a unified
whole. Every system is delineated by its spatial and temporal boundaries, surrounded and
influenced by its environment, described by its structure and purpose and expressed in its
functioning.
Warfighter
A person, especially a member of one of the armed services deployed to an area of
conflict, who is responsible for making decisions involving the use of military force.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-88
ANNEX II Development Tool and User Manual
General logic for development of the tool
Study Sub Group 3 was tasked to develop a kind of a decision tool or a decision
tree.
In the beginning a very general approach was determined, which has been followed
through the completed study.
General approach for creating a decision tool / decision tree
Following rough steps have been foreseen:
• From Study Sub-Group 1, parameters were expected out of the contracts they
examined.
• From Study Sub-Group 2, regulations out of NATO documents were expected. This
regulations, could be used for implementation of some logic to run the tool
• The implementation should be tested with 2 test cases, input for this test cases
should come from the examined contracts from subgroup 1
• Finally we had to create some instructions for the user in order to work with the tool
During the discussion of different considerations for the development and layout of
the decision tool / decision tree, major questions came up:
➔ What is the outcome/result of the tool?
➔ How can a decision be taken with the tool?
The overall answer was: The user expects a suggestion from the tool, which
supports him in taking a decision for a contract type (e.g. performance based or activity
based).
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-89
Therefore the output of the tool was defined as follows:
• The user shall receive a suggestion if a performance based logistic (PBL) contract
or an activity based (Non-PBL) contract should be installed.
• For the parameters values must be assigned. For the parameters’ values, digital
values must be assigned for PBL and for Non-PBL, so that a mathematical
comparison is possible. The larger the difference between the digital values of a
selected parameters’ value, the stronger the preference for a contract type will be
shown
Limitation:
• There are many different contract types existing. For this tool only a suggestion for
PBL or Non-PBL contracts is implemented. A refinement of the contract types might
be subject to further development.
Regarding the data above, the proceeding for developing the tool was as follows:
• Determination of parameters.
• Check if values can be determined for the individual parameters.
• Consider if the information about the values is available, before a contract type is
chosen.
Values cannot be used, when they are not known at the point, when the contract
type is chosen.
• Assign digital values (10 is best, 1 is worst) to each individual value for PBL and for
Non-PBL, so that a comparison is possible.
• If the digital values are the same for each individual value of a parameter, the
parameter cannot be used for comparison.
• Taking recommendations from subgroup 2 into account – regulations of NATO
document which could be used for determination of the parameters, individual
values and assigned digital values for PBL and Non-PBL
Considerations for implementation
Initial considerations for a decision tree and a decision support tool have been
developed. Later on, the decision to develop a decision support tool has been taken. This
chapter describes the considerations leading to the design of a decision tree and the main
approach for a decision support tool.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-90
Consideration for a Decision Tree
A sequence for a decision tree was developed. The challenge was to highlight a
logic that enables the user to take sound decision in its tasks, as he applies values for the
different parameters inside the decision tree. This logic is due to be generic enough to cover
both PBL and non-PBL contracts, or underline foreseen incompatibility features or domains.
Consideration for a decision support tool
The defined logics was turned into a process, which was split into seven phases,
the completion of which leading to the development of the decision support tool.
Phase 1 – Preparation
Process phase 1 sets the scene in order to enable the cross cutting between types
of contracts and parameters (process for sorting out the due parameters is described
thereafter in Decision support tool). Indeed, not only must the tool support the selection of a
main contract type out of several possibilities, but also must it propose options for optimizing
this overall contract with embedded sub-contacts from different types, better suited to handle
included specificities.
At this stage, it was highlighted that each contract has a dedicated objective and
specific characters, and that the spirit of the contract as well as its major features have to
be truly represented. Consequently, all parameters have to be qualified and quantified with
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-91
values (variables) as to represent their relative importance. They were flagged with a
scoring, to be implemented in the tool. Scoring the parameters did allow to sum up a global
scoring highlighting a main type of contract that takes into account the specific focuses of a
contract, while differentiating potential sub-contracts types showing more adapted to support
specific tasks within the main contract (e.g. “run” services within an equipment).
Phase 2 – Parameters valorisation
Parameters were therefore flagged with coefficients showing their intrinsic value in
order to enable a truly representative scoring.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-92
Phase 3 – Initial scoring
Phase 3 cross-cuts parameters value with different types of contract, with the effect
of a relevant, dedicated scoring for each parameter.
At this stage, it is possible to compare the relative interest of each type of contract
for a specific parameter. The user has the ability to focus on the essential, core parameters
to deduce which type of contract seems best suited for supporting the driving parameters.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-93
Phase 4 – Selection of Best suitable « Main contract type »
At the end, the global, total scoring highlights the main type of contract that should
be used as the main contract. In the following figure, contract type “B” displays a rating of
21 points. As a side effect, it is possible to value the score difference between B type contract
and the next one (A type), which difference clearly shows no ambiguity in selecting “B” type
contract as the main.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-94
Phase 5 – Pre-selection of Best suitable « Sub contract type »
This phase considers further developments of the tool.
According to the level of complexity, the user may take the opportunity to handle
some parameters through embedded sub contracts.
Having pre-selected the main type of contract against which to rule the project, it
still remains to seek any necessary sub-contract types for handling sub-tasks in the main
contract. Highest scores are made obvious for each parameter, meaning tasks, and suggest
afferent sub contract types per parameter.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-95
Phase 6 – Verification of contracts types compatibility & optimization
Phase 6 considers the implementation of phase 5.
Adding several types of sub-contracts may increase management difficulties, and
there must be some decision at this point on whether it is useful to add as many sub
contracts.
Comparison of the respective parameters scoring values highlights the usefulness
of inserting a new type of contract. E.g. parameter #2 displays a difference of 1 point
between the selected major contract type “B” and a possible sub contract “C”. This difference
may not be worth the change, and it is up to the user to decide if he really wants to change
or not. In this case, it is supposed he will not.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-96
Phase 7 – Final selection of contracts
At the end of the selection process, it is obvious that all final decision lays in the
hand of the user, when the tool has proposed some options based on logical evaluation.
With the current example, Type “B” contract has been highlighted as the main efficient
contract type, when only 1 subcontract was deemed useful to include by the user, instead
of three sorted out by the tool.
Parameters
Study Sub-Group 3 received suggestions for parameters and concerning
observations. These parameters and observations have been reviewed and slightly
modified in the following table.
A definition of the parameters is included in the glossary.
No Parameter Observations
1 Requirement Successful outcome-based contracts had clearly
defined warfighter/end-user requirements.
2 Cost Analysis Cost is an important part of the contract. If the
contractor has a higher freedom of fulfilling the contract,
we assume that the cost will reduce.
When specific deliveries are necessary at specific
dates, the cost will increase.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-97
3 Metrics – Type For PBL-contracts, the use of top-level outcome metrics
was more valuable than output metrics in determining
successful results (e.g. availability of a system vs.
number of spares repaired).
4 Metrics – Number used Successful outcome-based arrangements typically
considered only 3-5 top-level metrics. More than 5
metrics usually indicates a lack of clarity of desired
output.
5 Complexity of System
Supported
The complexity of the system supported, often
determine the level of the outcome-based arrangements
(i.e. component, subsystem, or system level support).
6 Performance History /
Experience
The range of available and experienced
suppliers/providers relates to the maturity of the system
supported; impacting the level of competition.
7 Contract starting point in life
cycle
System maturity (e.g prototype, development,
production, support, or sundowning) had a direct effect
on type of contract/period of performance. Less mature
systems typically engaged OEMs on shorter, cost-type
contracts, and aged systems near end-of-life typically
were not conducive to outcome-based arrangements at
all.
8 Availability of Data and
Technology
Systems with significant performance data are more
conducive to outcome-based, fixed-price arrangements
– less risk. New-start programs, with little performance
data, were better managed with shorter-term and cost-
type arrangements.
9 Contractor Freedom of Solution
Choice
Outcome-based arrangements totally managed by
industry result in lower risk and greater
success. Imposed use of government agencies as part
of the solution resulted in less control of the options and
higher risk.
10 Provision for Performance
Improvement
The greater the freedom to perform reliability
improvements, and conduct configuration changes, the
greater the opportunity to improve performance and
cost.
11 Period of Performance: Return
on Investment Opportunity
Longer periods of performance (minimum of 5 years)
allow greater opportunity for return on investments
made by industry.
12 Scope – Extent of service
provided
The more control the contractor has over the integration
of support elements, the greater the opportunity for
success.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-98
13 Contract Type More successful arrangements utilized fixed price
contract structures with incentives.
14 Obsolescence Management /
Modernization and Upgrade
The broader the authorities for mitigating risk of
obsolescence, improving the capability, held by the
contractor, the more successful the program.
15 Number of Nations If the number of nations increases, on the one side the
complexities in coordination of funding and
requirements. On the other side increasing number of
nations increases the bargaining power.
With the number of nations as a single digital value, a
preference for PBL or Non-PBL cannot be given. This
parameter is subject for further refinement. It is kept, to
enable the user the adaptation for his special cases. In
the current tool implementation, for any value selection
the result “no preference” will occur at the stage of
development.
16 Contractor PBL Experience Success leads to success. Those contractors who have
previous successful experience with PBL are more likely
to leverage that experience in additional PBL
opportunities.
Notes:
General note for Parameters:
(1) We did not find main parameters for customers’ requirement to assess
performance during peacetime, higher threat level and full scale war. The reflexion of
circumstances for conflict period support activity needs to be handled in detail in the single
contracts. A detailed analysis is recommended for a follow-on study.
(2) Industry vision is not reflected in the parameters. A low risk for the customer
might increase the financial considerations from the contractor.
Note about risks:
Risks occur through all parameters and could not be handled as a single parameter.
The following table assigns the technical, operational, financial and managerial risk to the
different parameters.
Risk Element
Parameter (new) Tech Ops Fin Mgmt
Requirement X X X X
Cost Analysis X X X X
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-99
Metrics – Type
X X X
Metrics – Number used
X
Complexity of System Supported
X
Performance History / Experience X X
X
Contract starting point in life cycle X X X
Availability of Data and Technology X
X
Contractor Freedom of Solution Choice X
X
Provision for Performance Improvement
X X
Period of Performance: Return on Investment Opportunity X X X X
Scope – Extent of service provided
X X
Contract Type
X
X
Obsolescence Management / Modernization and Upgrade
X X
Number of Nations X X X
Contractor PBL Experience X X X X
Note for Parameter cost analysis:
Costs are mainly derived from the balance of the requirements and the risks.
Cost is a transverse factor, both input and output. It considers the financial impact
for the end user, customer, ordering nation, Procurement Agency, Contracting Officer…. In
general, project management costs to be handled by the Procurement Agency are already
studied as a separate item within the funding request, and agreed during the authorisation
process. This constitutes an external factor to the selection of the type of contract and not
considered in this study and decision support tool.
Note for Parameter number of nations:
The « number of nations » parameter includes several aspects such as:
o The number of participating nations; having one or several nations changes the
vision of the contract and may require coordination from the Contracting Officer.
From their past references, nations may want the Contracting Officer to privilege
specific type of contract.
o The involvement of nations. This sub-parameter describes the national
considerations when engaging into a project, namely their desired Return on
Investment for their national industry, along with their skills and areas of
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-100
competence, their particular specifications, time frames, industrial strategy. These
particulars are undertaken by the Contracting Officer as inputs to the other existing
Parameters.
Note for parameter contract type:
The “contract type” also covers the notion of “contracting management” (the fact of
contracting, in other words handling the administrative tasks necessary for contracting as
well as managing the contract). As an example, a specific type of contract may prove difficult
or costly to manage due to the nature of the project
Note for parameter contractor freedom of solution:
This includes the notion of “control over the asset”
o “control over the asset” describes the possibility to pilot the dedicated assets
required for the implementation of the contract and its freedom of handling, would it
be alone or in conjunction with another stakeholder, such as the Project Manager.
Remarks:
Initial parameter list
The initial parameter list considered the 3 point of view (ENDUSER- NATO-
INDUSTRY)
1. Scope included in the contract- topic – What´s in /out – workshare,- achievable?
2. Timeframe period of the contract
3. Total Cost of the contract
4. Commitment among the parties (KPIs and incentive /penalty models)
5. Availability of the Weapon System (Level of Service)
6. Risks
7. Control over the asset (management of the asset/authority to employ)
8. Surrounding parameter (Legal, interoperability & regulatory constrains)
9. Number of Customers (multinational, national, pool)
Reflection of our parameters against parameters from other sources
Our parameters have been reflected against parameters from other sources: from
a “US assessment tool” and against NATO AACP-2
Reflection against the Parameters (Factors) from “US assessment tool”
• Reduction in total cost of ownership, life cycle cost.
• Presence, determination of metrics, performances outcomes
• Improvement in performance
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-101
• Partnership, strategic alliance
• Length of contract
• Flexibility
• Better service, best value creation, best practices
• Successful implementation of BLK, PBL experience and training
• Innovation
• Data availability
• Candidates and competition
• Properties of system
• Up-front, early planning
Reflection of our parameters against possible Parameters from NATO AACP-2
(only headlines shown here)
• Abstract & key aspects: most salient points of the Contract at a glance
• Order of precedence (requirements and parameters ranking)
• Scope of Work;
• Content (definition / selection of equipment)
• Content (Performance – level of service)
• Execution of the contracts (Operational management – i.e. premises, dates,
consignees);
• Contract administration (administrative management) and financial guarantees
• Language.
Logic
As part of SSG 2 focus area, the Study Group has reviewed NATO documents to
identify policies and doctrines relevant to be updated to include concepts covering
contracting for capability. As part of this work the subgroup also identified requests/rules in
these documents relevant for SSG3 to consider for the decision tool.
SSG 2 findings included 3 suggested actions with 9 specific recommendations and
points to be aware of derived from the reviewed NATO documents.
Upon completed review and comparison, SSG3 did not find new parameters to be
added to the tool, but valuable confirmation and verification of the identified parameters was
possible.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-102
Conclusion on the provided list of findings from SSG 2:
No. SSG2 Finding SSG3 Conclusion
1 Solicitation: Earlier Identification of
(prospective) contractor.
Confirmed as covered by the decision
tool. This confirmation is based on the
tool being developed in the first place
to allow faster decision making which
will lead to earlier identification of
prospective contractors
2 “Evaluation and selection of proposals”: this
should be a part of the contract, and a
parameter. Evaluation criteria will differ through
different contracts due to their specificity and
the focus placed by the Contracting Officer (and
consumer) on some important features and
requirements. This is currently poorly described
in the official documents. As stated, the
criteria’s differ for each contract and
requirement type. This to an extent, where
SSG2 conclude is not possible to cover it to its
full extend by parameters in this tool.
This conclusion is agreed to by SSG3.
The decision tool allows the
Programme Manager to establish
relevant evaluation criteria’s based on
the initial parameter setting in the tool.
3 The decision-making tree should have already
been applied at the stage when the contract is
drafted. Contracting authority should already
know at this stage if outcome-based contracting
would be applied.
Correct and covered by the tool. The
tool is based on parameters possible to
judge as part of the initial assessment
of a requirement. The Contracting
Officer will have a tool-based
conclusion at this stage.
4 If and how contract length provisions would
influence industry and government rights
related to software.
Will not be addressed by the tool due
to the level of detail this relates to.
During use, the specification of the
parameters for the respective
requirement will lead to inclusion of
such if relevant.
5 Cost structure and cost sharing related to the
quality assurance process (in two scenarios i.e.
traditional contracting and outcome-based).
Quality Assurance Programs in place is
considered a pre-requisite regardless
the contract type and not included as a
parameter in the tool.
6 To what extent outcome-based contracting
would influence the “standard” GQA process,
are there additional costs involved? SSG2
suggest it should be considered what the
relation is between these additional costs vs.
overall benefit.
This reflection links to the previous
point and SSG3 conclude it cannot be
taken into consideration for the tool.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-103
7 National/International arrangements. SSG2
conclude this links to requirement and scope.
Correct and covered by the tool. This is
addressed in mentioned parameters as
well as Number of Nations.
8 Consideration of contracting type in the earlier
phases. Extent of the desired logistics
capability (Determined by defined logistics
requirements).
Correct and covered by the tool. The
aim of the tool is to guide the user
towards the best choice of contract
type. Parameters like Requirement,
Outcome or Traditional Contract Type
all drive the decision.
9 Obsolescence. Correct and covered by the tool.
10 Suggest which of the inputs should be handled
with additional parameters/characteristics/risk-
levels.
All relevant inputs are covered by the
tool.
11 Suggest which of the inputs should be handled
as “detailed level”, means investigated in a
further study when the tool can be extended.
Confirmed by SSG3 and covered by
the tool with guidance to which
parameters recommended for further
levels of parameters in line with each
specific contract/requirement.
General Process when to identify parameter values
SSG3 group took into consideration the overall process of the PBL decision support
tool requirement in order to identify the key Players and their expectations provided through
discussions, documents, specifications, criteria and finally “input parameters”.
The key players were identified to allow NATO to fully understand the requirements
at their source level:
• The end-Users and the contracting organisation
• The Customer organisation (logistic, …)
• NATO/NSPA
• The participating industrial future partners (for the project)
The requirement evolution and the values assigned to the parameters depend on
this level of comprehension obtained during the meetings.
It is most important to identify the main parameters that will influence a decision
making in the support, in order to feed the model.
The hereunder flowchart describes the process considered by SSG3 to reach a
good level of comprehension and be able to size the parameters level.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-104
Identification of the key players
This general flowchart highlights each level of requirement and place NATO
organisation in a position to understand the mutual relationship between the stakeholders.
It also allows NATO to take into consideration and include its own vision of the project, in a
NATO perspective.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-105
Identification of the Key documents
The different documents build during the process, when shared with NATO,
contribute to the comprehension of the end user’s requirements and the contracting Part
constraints. This is also key to allow NATO to select the relevant industry players that would
be foreseen to answer the RFQ of the Project.
End User requirement: is the source of the requirement as they will receive the
final support package.
Customer requirement: Is what has been taken into consideration by the
Contracting part representing the End-Users.
NATO / NSPA: is the entity in charge to gather the requirement and to prepare a
RFQ to the industry that will the best correspond to end-users, Customer and NATO policy.
Industry: will provide its proposal to NATO.
NATO/NSPA: Will select and contract with the best offer to cover the Customer
requirements.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-106
Identification of the Key set of parameters
The process is then viewed with a parameters perspective:
From each of the four above identified steps, a set of parameters will be extracted.
SoP1: is the set of parameters that is directly coming out of the Customer written
document (CCTP). This task is performed by the NATO team in charge of the project
SoP2: is the set of parameters that will be discussed with the Customer to make
sure of the good comprehension of the requirement and to be able to classify a kind of level
of priority and importance of each parameter. The classification is realised on two aspects
(Priority and influence on the type of support contract)
SoP3: Is a complementary set of parameters added by NATO team to take into
consideration NATO perspectives
SoP4: Is a set of parameters that takes into consideration the industry maturity and
possibility for all aspects of the Project.
These set of parameters are used as an input to the tool in order to provide a
recommendation based on existing contracts experience. The four sets of generic
parameters are described in the following pages.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-107
Key set of parameters
In order to design in the given time frame and resources perimeter a possible
decision support tool, it was decided to limit the number of parameters to the above list.
The demonstrator developed during this Study is based on the following list of
parameters.
The parameters are detailed in chapter “Parameters”.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-108
Set of Parameters 1 (SoP1), ambition/objective
This first set of parameters covers the general context of the Customer requirement
and the global situation.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-109
Set of Parameters 2 (SoP2), ambition/objective
The second set of parameters allows better understanding of the Customer and the
contracting Authority constraints and objectives. It allows sizing the priorities and starting a
possible view of the future types of solutions.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-110
Set of Parameters 3 (SoP3), ambition/objective
The third set of parameters is a performance set of metrics that allow confirming the
type of objectives and metrics to be used during the Contract.
It is NATO responsibility to size the effort acceptable to the Customer to enter or not
into a PBL contract type according to its organisation. This point can make a PBL contract
a success or not.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-111
Set of Parameters 4 (SoP4), ambition/objective
The fourth set of parameters is a performance set of metrics that allows confirming
the objectives and the way forward to setup the type of chosen support activity during the
Contract duration. It takes into consideration some fundamental parameters that can make
a PBL contract a success (e.g. duration/quality of the commitment between industry and
Customer/possibility to adapt the services to the real use of the system).
General exclusions
This vision of a Performance Based Logistic support solution does not take into
consideration the cost of the organisation to be setup to cover a different type of
management into the customer’s teams and the contracting authority (Different from spares
and repairs classical approach):
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-112
• Inside the contract
A set of metrics/KPIs, allowing to manage the PBL contract is to be created and
monitored with accuracy and regularity. This aspect is not negligible for customers
at operational level.
• Inside the systems
Some investment has to be foreseen from the early system definition to allow the
life management and KPI automatic generation and collection into a dedicated
management system.
• Specific contracts
That would generate specific parameter not covered in his tool.
These parameters are not taken into consideration in the tool perimeter.
Tool Description / Handling of the tool
General note:
• This tool is not taking the responsibility from the user in making his decision which
contract type he wants to use. The user will regard more circumstances, regulations
and pre-conditions.
• Therefore the tool shall only give the user support in making this decision. It is a
support tool!
For this Study, the tool was created with Microsoft® Excel® (Microsoft® Office
Professional Plus 2010)
The tool consists of 3 registers:
1. Register Main (user interface)
2. Register Parameters (setting of parameters and values for the register main, setting
of digital values in order to enable comparisons)
3. Register Evaluation (performing evaluations. The result of the evaluations will be
shown in the register Main)
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-113
Register Main
This register is the user interface for the standard user. The user enters his values
and receives the suggestion to which kind of contract is preferred.
16 parameters are listed in column “B” Parameter. In this version, a provision of 3
more parameters has been implemented. If there is a need to adapt/extend the parameters,
this shall be done in the register Parameters. Do not change any parameter in the register
Main.
For each of the different parameters, a value shall be selected in column “C” Value
from a given list. In order to enter a value, click on the cell. On the right of the cell a symbol
with a triangle appears. Click on the triangle and you will receive the installed values for the
parameter.
Example: 4 different values appear for the parameter Requirement.
Click on the value which fits best for your needs. If the pre-installed values do not
fit to your needs, click on unknown / not applicable.
Parameter Value Weight Prefered Contract
Requirement unknown / not applicable 3 No preference
Cost analysis unknown / not applicable 3 No preference
Metrics - Type unknown / not applicable 3 No preference
Metrics - Number used unknown / not applicable 3 No preference
Complexity of System Supported unknown / not applicable 3 No preference
Performance History / Experience unknown / not applicable 3 No preference
Contract starting point in life cycle unknown / not applicable 3 No preference
Availability of Data and Technology unknown / not applicable 3 No preference
Contractor Freedom of Solution Choice unknown / not applicable 3 No preference
Provision for Performance Improvement unknown / not applicable 3 No preference
Period of Performance: Return on Investment Opportunity unknown / not applicable 3 No preference
Scope - Extent of service provided unknown / not applicable 3 No preference
Contract Type unknown / not applicable 3 No preference
Obsolescence Management / Modernization and Upgrade unknown / not applicable 3 No preference
Number of Nations unknown / not applicable 3 No preference
Contractor PBL Experience unknown / not applicable 3 No preference
Reserved for further extension (1) unknown / not applicable 3 No preference
Reserved for further extension (2) unknown / not applicable 3 No preference
Reserved for further extension (3) unknown / not applicable 3 No preference
Overall Prefered Contract No preference
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-114
When you have chosen your value, the content of the cell in column “E” (Prefered
Contract) of the same line will change or even remain the same (depending on the chosen
value).
Example: For the parameter Requirement you have chosen the value Nebulous.
The content of the related cell in column “E” changes to Non-PBL preferred.
Also the overall evaluation will change. In cell E62 the Overall Preferred Contract
might change at the same time.
In our example all values are set to unknown / not applicable, only the value of
parameter requirement has been set to Nebulous. So the Overall Preferred Contract shows
the suggestion Non-PBL prefered:
Make your value selection for all 16 given parameters.
Some parameters might be of higher, others of lower importance. Therefore in
column “D” Weight, the importance can be expressed with one of the values from 1 to 5. 3
is the medium importance,1 is lowest and 5 is highest importance.
In order to change a parameters weight, click on referring cell in column “D”. The
triangle will appear. Make your selection.
Example:
Notes:
There are also some graphical indication in column “C” and “E”:
• Column “C” Value: When the selected value is unknown / not applicable it is marked
in light red.
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-115
• Column “E” Prefered Contract: The stronger the preference for a contract type, the
darker the colour. Pink colours are for Non-PBL contract preferences, brown colours
are for PBL contract preferences.
Register Parameters
In row 2 the value range for the selection boxes in row “D” Weight of register Main
is set from 1 to 5.
Starting with row 5, the title of the different columns for row 6-62 is set. Row 6-62
contains the different parameters with their values for the selection boxes and digital values
for comparison/evaluation.
Example:
In row 6 you find the parameter Requirement in column “B”, which is followed by the
installed values for the selection box (unknown /not applicable, nebulous, …)
Below each value you will find the digital value for PBL (row 7) and Non-PBL (row
8) which enables an evaluation/comparison.
The values have been set as a result of the discussion between the members of
subgroup 1 and subgroup 3. The higher a value in row 7 for PBL, the better for a PBL-
contract; the higher a value in row 8, the better for a Non-PBL-contract. Depending on the
difference of the digital PBL-value and the digital Non-PBL-Value, one of both contract types
is preferred. The stronger the difference, the stronger the preference. If the values are equal,
no preference is given.
Example:
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-116
If the value is set to unknown / not applicable, the difference is 0, so No preference
would be shown. This is the same case for value Defined.
For value Nebulous the difference is -2, so a Non-PBL contract would be suggested
for this parameters value.
Rows 54-62 enable the user to include additional parameters with values for the
selection boxes and digital values for comparison.
With the current setting, 5 values per parameter will be used. In order to establish a
new parameter, proceed in the following way:
• Change the parameters’ name in column “B”.
• Replace the selection box dummy values in column “D”-“H” with the new values. If
you have less than 5 values, just delete the content of the unused cells and set their
PBL- and Non-PBL-values to 0 (as it is set for unknown /not applicable).
• For each new value you need to enter the digital values for PBL and Non-PBL as
explained above.
Note:
The emptied unused cells will appear empty in the selection box in the register Main.
A change in register Main would be necessary, which should only be performed by
experienced Excel-users (e.g. using the function “Data->Data Validation”).
Register Evaluation:
Note:
This register Evaluation should not be changed. If changes are necessary, they
only should be performed by experienced Excel-users.
The register Evaluation looks as follows for columns “B”-“H”:
The register Evaluation looks as follows for columns “J”-“K”:
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-117
Explanation of the register, starting with row 5, picture 1:
• the parameters (column “B”) are taken from register Parameters
• the content of the selection boxes for the values (column “C”; row 6, 9, 12, …) is
taken from register Main
• The referring digital values (column “C”; row 7/8, 10/11, 13/14, …) are taken from
register Parameters
• The weight (column “D”; row 6, 9, 12, …) is taken from register Main
• The weight for PBL/Non-PBL (column “D”; row 7/8, 10/11, 13/14, …) is the product
of the referring digital value and the parameters’ weight
• The Prefered Contract for the individual parameter is evaluated from the related
difference of the digital values in column “C” in combination with the assignment of
the values seen in picture 2
• Column “F” regarded checks if the individual parameter shall be taken into account
for the overall evaluation. If the value is unknown / not applicable, the parameter will
be disregarded.
• Column “G” Reg Weight returns the weight, if the parameter will be taken into
account, else it returns 0.
• Cell C3 Weighted Sum for PBL is the sum of all weighted digital PBL-values
• Cell C4 Weighted Sum for Non-PBL is the sum of all weighted digital Non-PBL-
values
• Cell F4 Weighted Delta: PBL to Non-PBL returns the difference of cells C3 and C4
• Cell F3 Max Weighted Sum returns the amount of parameters taken into account
multiplied with the maximum difference the digital values can have (max. digital
value is 10, min. digital value is 1 when a parameter is taken into account)
• Cell H3 Relative Weighted Delta delivers the calculated value, needed for
determination of the overall preferred contract type
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-118
• Cell H6 Overall Preferred Contract delivers the overall preferred contract type from
picture 2, based on the value from H3. This result will be shown as Overall Preferred
Contract in register Main.
Tests / Test cases and results
Requirements from customer:
• One test case shall be for weapon systems (aircraft, vessel, vehicles, radar
system, etc.). -> This requirement is covered by the UK Sentry Project test case
• One test case shall be for large/complex Capability services (medical svc, airlift /
sealift svc, pilot training svc, Search + Rescue svc, fuel distribution svc) -> This
requirement is covered by the UK Aerial Refueling test case
Testcase UK Sentry Project:
Result of Test Case Sentry: PBL prefered
Testcase UK Aerial Refuelling:
Result of Test Case UK Aerial Refuelling: PBL prefered slightly
Parameter Value Weight Prefered Contract
Requirement Fully defined 3 No preference
Cost analysis Higher Cost Importance; Low Specific Deliveries 3 PBL prefered very strongly
Metrics - Type Outcome - availablilty measure 3 PBL prefered extremly
Metrics - Number used 1-2 3 PBL prefered very strongly
Complexity of System Supported Major System / tip-to-tail 3 Non-PBL prefered strongly
Performance History / Experience Medium level / few suppliers 3 No preference
Contract starting point in life cycle Support 3 PBL prefered very strongly
Availability of Data and Technology High 3 PBL prefered
Contractor Freedom of Solution Choice Best Value 3 PBL prefered
Provision for Performance Improvement unknown / not applicable 3 No preference
Period of Performance: Return on Investment Opportunity Long PoP (10+ year) 3 PBL prefered
Scope - Extent of service provided Full Spectrum Support / Turn-Key Solution 3 PBL prefered slightly
Contract Type Firm Fixed Price (pen + gainsharing) 3 PBL prefered
Obsolescence Management / Modernization and Upgrade unknown / not applicable 3 No preference
Number of Nations 1 3 No preference
Contractor PBL Experience unknown / not applicable 3 No preference
Reserved for further extension (1) unknown / not applicable 3 No preference
Reserved for further extension (2) unknown / not applicable 3 No preference
Reserved for further extension (3) unknown / not applicable 3 No preference
Overall Prefered Contract PBL prefered
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-119
Description of single parameter values:
General:
Use the value unknown / not applicable if the pre-installed values do not fit to your
needs. So they will not be taken into account.
Parameter Value Description
Requirement
Defined and fully defined requirements support both kind of contract. Since
the degree of freedom for PBL is higher than for Non-PBL, we assume a
higher risk for PBL than for Non-PBL when the requirements are nebulous.
Cost analysis
The values take into consideration if the contractor is bound to specific
deliveries. This limitation will reduce his possibilities to optimize his work and
therefor increase his cost.
Metrics – Type The metric types are distinguished if they refer more to output (good measure
for Non-PBL) or to outcome (good measure for PBL)
Metrics – Number used Takes into account the number of metrics used. Too many metrics (especially
for PBL) optimize in too many directions, which is contra-productive.
Complexity of System
Supported
The values reflect the systems from component to major system. We assume,
that smaller systems are easier to handle than complex ones. For major
systems, breaking down the contract to several smaller systems would reduce
the PBL risk significantly.
Performance History /
Experience
The higher the experience the lower the risk. A lot of supplies give you better
possibilities to select a good one.
Contract starting point in
life cycle
This value reflects the different stages in the life cycle of a product. PBL is
used best in the support phase.
Availability of Data and
Technology
The less data available the higher the risk for contract execution, especially in
reliable planning and prediction.
Contractor Freedom of
Solution Choice
A government directed workshare reduces the contractors possibilities in
achieving the best solution. A hybrid of directions and freedom improves his
possibilities and the best value gives him all opportunities for the best solution
Parameter Value Weight Prefered Contract
Requirement Fully defined 3 No preference
Cost analysis Medium Cost Importance; Medium Specific Deliveries 3 No preference
Metrics - Type Outcome - availablilty measure 3 PBL prefered extremly
Metrics - Number used 1-2 3 PBL prefered very strongly
Complexity of System Supported Major System / tip-to-tail 3 Non-PBL prefered strongly
Performance History / Experience New market/low-no experience 3 Non-PBL prefered
Contract starting point in life cycle Production 3 No preference
Availability of Data and Technology Medium 3 No preference
Contractor Freedom of Solution Choice Best Value 3 PBL prefered
Provision for Performance Improvement Discrete improvement targets 3 PBL prefered strongly
Period of Performance: Return on Investment Opportunity Long PoP (10+ year) 3 PBL prefered
Scope - Extent of service provided Full Spectrum Support / Turn-Key Solution 3 PBL prefered slightly
Contract Type unknown / not applicable 3 No preference
Obsolescence Management / Modernization and Upgrade Mods/Upgrades 3 PBL prefered strongly
Number of Nations 1 3 No preference
Contractor PBL Experience Low 3 Non-PBL prefered strongly
Reserved for further extension (1) unknown / not applicable 3 No preference
Reserved for further extension (2) unknown / not applicable 3 No preference
Reserved for further extension (3) unknown / not applicable 3 No preference
Overall Prefered Contract PBL prefered slightly
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-120
Provision for Performance
Improvement The higher the freedom to handle improvements, the better the overall result
Period of Performance:
Return on Investment
Opportunity
The longer the contract, the better the possibility to achieve ROI. Long-term
contracts enables PBL a good change for a return of investments
Scope – Extent of service
provided
Handling a single ILS-segment is easier than handling several of them due to
dependencies between each other. If the complete ILS-spectrum is covered,
harmonisation with other contracts is not needed.
Contract Type Fixed fees are not attractive for a PBL contract, but penalties/incentives and
gainsharing are interesting for PBL contracts.
Obsolescence
Management /
Modernization and
Upgrade
The lower the possibilities for technical investments in the future, the more a
Non-PBL becomes attractive
Number of Nations Number of involved nations (NSPA-customers)
Contractor PBL
Experience
The better the PBL experience of a contractor, the better for a new PBL
contract with this contractor
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-121
ANNEX III DECISION SUPPORT TOOL
The Tool itself is a protected Excel File and is posted together with a list of
passwords on the NATO DI Portal: https://diweb.hq.nato.int
(NIAG / Study Groups / SG-218 / Shared Documents):
Parameter Value Weight Preferred Contract
Requirement unknown / not applicable 5 No preference
Cost analysis unknown / not applicable 1 No preference
Metrics - Type unknown / not applicable 5 No preference
Metrics - Number used unknown / not applicable 3 No preference
Complexity of System Supported unknown / not applicable 3 No preference
Performance History / Experience unknown / not applicable 3 No preference
Contract starting point in life cycle unknown / not applicable 3 No preference
Availability of Data and Technology unknown / not applicable 4 No preference
Contractor Freedom of Solution Choice unknown / not applicable 5 No preference
Provision for Performance Improvement unknown / not applicable 4 No preference
Period of Performance: Return on Investment Opportunity unknown / not applicable 4 No preference
Scope - Extent of service provided unknown / not applicable 3 No preference
Contract Type unknown / not applicable 4 No preference
Obsolescence Management / Modernization and Upgrade unknown / not applicable 3 No preference
Number of Nations unknown / not applicable 1 No preference
Contractor OBC Experience unknown / not applicable 3 No preference
Reserved for further extension (1) unknown / not applicable 1 No preference
Reserved for further extension (2) unknown / not applicable 1 No preference
Reserved for further extension (3) unknown / not applicable 1 No preference
Overall Prefered Contract No preference
Weight 1 2 3 4 5
Parameter Value1 Value2 Value3 Value4 Value5 Value6 Value7 Value 8
Requirement unknown / not applicable Nebulous Partially defined Fully defined
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 1 5 10
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 3 5 10
Cost analysis unknown / not applicable Cost value 1 Cost value 2 Cost value 3
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 5 5 5
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 5 5 5
Metrics - Type unknown / not applicable Output - activity measure Mix - Outcome/Output Outcome - availablilty measure
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 1 4 10
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 10 6 2
Metrics - Number used unknown / not applicable 1-2 3-6 7-10 11+
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 10 8 3 1
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 4 6 8 8
Complexity of System Supported unknown / not applicable Major System / tip-to-tail Collection of Sub-systems/Minor System Sub-system Component / Commodity
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 8 4 4 10
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 2 5 7 10
Performance History / Experience unknown / not applicable New market/low-no experience Medium level / few suppliers High level /competitive market/OEM(SS)
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 1 5 10
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 3 5 10
Contract starting point in life cycle unknown / not applicable Prototype Development Production Support Sundowning/End-of-life
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 1 3 6 10 3
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 8 7 6 5 5
Availability of Data and Technology unknown / not applicable Low Medium High
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 1 5 10
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 3 5 8
Contractor Freedom of Solution Choice unknown / not applicable Workshare (gov't directed) Hybrid (PPP - best value decision) Complete Contractor Freedom
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 1 5 10
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 2 5 8
Provision for Performance Improvement unknown / not applicable No authority Restricted authorities Tacit improvement authorized Complete Improvement Freedom for Contractor
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 1 4 7 10
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 3 5 5 6
Period of Performance: Return on Investment Opportunity unknown / not applicable Extremely short (1 yr) Short (2-4 yr) Medium (5-9 yr) Long PoP (10+ year)
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 1 3 7 10
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 5 6 7 8
Scope - Extent of service provided unknown / not applicable Single ILS Element Low Hybrid - 2-5 High Hybrid - 6+ ILS elements Full Spectrum Support / Turn-Key Solution
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 10 6 5 8
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 10 7 5 7
Contract Type unknown / not applicable Cost Plus - Fixed Fee (Not Outcome based) Cost Plus Award Fee - subjective metrics Cost Plus Incentive Fee - (quantitative metrics) Fixed Price Incentive (cost/performance) Fixed Price Award Fee/Term (subj met) Firm Fixed Price (penalties) Firm Fixed Price (pen + gainsharing)
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 1 4 5 6 8 9 10
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 7 5 4 6 6 9 8
Obsolescence Management / Modernization and Upgrade unknown / not applicable No consideration Basic Obsolescence Mgmt Adv Obs Mgt - Tech Refresh Mods/Upgrades
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 1 4 8 10
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 3 4 5 6
Number of Nations unknown / not applicable 1 2 3 4+
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 5 5 5 5
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 5 5 5 5
Contractor OBC Experience unknown / not applicable Low Medium High
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 2 5 8
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 5 5 5
Reserved for further extension (1) unknown / not applicable Dummy value 1 Dummy value 2 Dummy value 3 Dummy value 4 Dummy value 5
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 5 5 5 5 5
Customer Risk Level for Non-OBC (10 best, 0 worst) 0 5 5 5 5 5
Reserved for further extension (2) unknown / not applicable Dummy value 1 Dummy value 2 Dummy value 3 Dummy value 4 Dummy value 5
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 5 5 5 5 5
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 5 5 5 5 5
Reserved for further extension (3) unknown / not applicable Dummy value 1 Dummy value 2 Dummy value 3 Dummy value 4 Dummy value 5
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 5 5 5 5 5
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 5 5 5 5 5
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-122
Weighted Sum for OBC 0 Max Weigthed Sum 0 Relative Weighted Delta
Weighted Sum for Non-OBC 0 Weighted Delta: OBC to Non-OBC 0 0
Parameter Value Weight Preferred Contract Regarded Reg Weight Overall Preferred Contract Delta OBC to Non-OBCSuggestion
Requirement unknown / not applicable 5 No preference 0 0 No preference -9 Non-OBC preferred extremly
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0 -8 Non-OBC preferred extremly
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0 -7 Non-OBC preferred extremly
Cost analysis unknown / not applicable 1 No preference 0 0 -6 Non-OBC preferred very strongly
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0 -5 Non-OBC preferred very strongly
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0 -4 Non-OBC preferred strongly
Metrics - Type unknown / not applicable 5 No preference 0 0 -3 Non-OBC preferred strongly
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0 -2 Non-OBC preferred
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0 -1 Non-OBC preferred slightly
Metrics - Number used unknown / not applicable 3 No preference 0 0 0 No preference
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0 1 OBC preferred slightly
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0 2 OBC preferred
Complexity of System Supported unknown / not applicable 3 No preference 0 0 3 OBC preferred strongly
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0 4 OBC preferred strongly
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0 5 OBC preferred very strongly
Performance History / Experience unknown / not applicable 3 No preference 0 0 6 OBC preferred very strongly
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0 7 OBC preferred extremly
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0 8 OBC preferred extremly
Contract starting point in life cycle unknown / not applicable 3 No preference 0 0 9 OBC preferred extremly
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
Availability of Data and Technology unknown / not applicable 4 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
Contractor Freedom of Solution Choice unknown / not applicable 5 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
Provision for Performance Improvement unknown / not applicable 4 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
Period of Performance: Return on Investment Opportunity unknown / not applicable 4 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
Scope - Extent of service provided unknown / not applicable 3 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
Contract Type unknown / not applicable 4 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
Obsolescence Management / Modernization and Upgrade unknown / not applicable 3 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
Number of Nations unknown / not applicable 1 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
Contractor OBC Experience unknown / not applicable 3 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
Reserved for further extension (1) unknown / not applicable 1 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Customer Risk Level for Non-OBC (10 best, 0 worst) 0 0
Reserved for further extension (2) unknown / not applicable 1 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
Reserved for further extension (3) unknown / not applicable 1 No preference 0 0
Digital value for OBC (10 best, 1 worst, 0 means disregard) 0 0
Digital value for Non-OBC (10 best, 1 worst, 0 means disregard) 0 0
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-123
ANNEX IV FINAL PROOF POINT
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-124
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-125
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-126
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-127
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-128
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-129
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-130
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-131
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-132
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-133
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-134
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-135
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-136
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-137
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-138
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-139
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-140
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-141
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-142
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-143
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-144
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-145
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-146
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-147
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-148
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-149
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-150
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-151
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-152
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-153
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-154
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-155
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-156
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-157
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-158
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-159
NATO UNCLASSIFIED
Releasable to PFP
NATO UNCLASSIFIED
Releasable to PFP
1-160
