Shifting Left With Developmental Testing for Systems of ... 1600_ITEA prese… · Internal –...

Engineering, Operations & Technology

Approval for public release 15-00006 Copyright © 2015Boeing. All rights reserved.

Boeing Test & Evaluation

Shifting Left With Developmental Testing for Systems of Systems

Pete De Salvo – Systems/Systems of Systems, Test Program Manager David Allsop - Systems/Systems of Systems, Systems Engineering

Author, 1/29/2015, Filename.ppt | 1

ITEA SoS Conference January 27-29, 2015

Approval for public release 15-00006 Copyright © 2014 Boeing. All rights reserved.

Boeing Test & Evaluation | Project Name

Late Discovery is Driving Program Cost and Schedule


DT&E Assessments recommending to proceed to IOT&E

(2012 DT&E Annual Report)

Proceed

Do Not Proceed

IOT&E

Significant Issues No Issues

25%

50%

Only 41% (5 of 12) programs made it to & through a successful IOT&E

Num

ber o

f Iss

ues

Rea

lized

Issue actual discovered Earliest opportunity for discovery

Nominal cost for fault removal (NIST data) 1x 5x 16x 40x 110x

Notional

Late discoveries are considerably more costly!

Resulting in production delays

The Cost The Impact

Copyright © 2014 Boeing. All rights reserved.

Sources NDIA ICOTE, Chief Developmental Tester & Industry Test Lead – Partnering for success, August 2013 NIDS Planning report 02-3. The economic Impacts of Inadequate Infrastructure for Software Testing May 2002. D.Galin, Software Quality Assurance From Theory to Implementation, Pearson/Addison-Wesley (2004) B.W. Boehm, Software Engineering Economics, Prentice Hall (1981)



▫ Systems and products are becoming ever more complex with their internal and external design approach evolving into systems of systems.

▫ Declining budgets and poor cost and schedule performance suggest new methods are needed.

▫ Verifying and validating the design early can prevent late problem discovery during test or during customer operation.

▫ Early validation and developmental test is commonly called “shift left”.

The Test Community Needs to Shift Left to Partner With Systems and Program Engineering to Improve Program Performance

Shift Left



Model-Based Concurrent Engineering Processes

Requirements-Analysis Systems

Analysis & Design

HW/SW Design HW/SW Implementation

Module Integration &

Test

System Integration &

Test System

Acceptance

4

Performing requirements and design validation as early as possible, as opposed to waiting until ”composition” activities begin, reduces cost and schedule risks.

Increase design stability by requirements validation and systems analysis prior to implementation

Cost of Design Change

Time Source: Use of a Model-Based Approach to Minimize System Development Risk and Time-to-Field for New Systems, US Army AMRDEC presentation at the 2010 NDIA Systems Engineering Conference, October 2010



Challenge – testers need ‘real’ requirements; developers need to prototype requirements (via test) to define ‘real’ requirements

Shifting Test to the Left Side of the ‘V



Key Concepts


Validation Build the right system Fitness for purpose Quality assurance External – stakeholder evaluation (suitability and effectiveness) Operational Test End to end focus Top down Did we get what we want?

Verification Build the system right Satisfies requirements Quality control Internal – contractor testing (demo, inspection, analysis) Development test Traceability Bottom up Did we get what we asked for?

Model Based Systems Engineering “…is the formalized application of modeling to support system requirements, design,

analysis, verification and validation activities. It begins at conceptual design & continues throughout the development life cycle.”

INCOSE Vision 2020 (Reference: “International Council on Systems Engineering, Systems Engineering Vision 2020,” INCOSE-TP-2004-004-02,

Version/Revision: 2.03, Dated September 2007)

Perform Validation and Verification as soon as Possible to Identify, Mitigate, and Retire Program Risks Early



Methods/Types of shift left Validation Test


Developmental Testing Includes • Modeling and Simulation • Model Based Systems Engineering • Physical prototyping • Brass Board • Virtualization • Actual Hardware Test • Combinations of above elements, eg, LVC testing

Early Risk Discovery and Retirement is Key to Project Execution

Source NDIA Final Report – Model Based Engineering (MBE) Subcommittee, October 2011



Considerations for shift-left-testing


• Partnership between test, systems engineering, and design engineering essential throughout development process is essential

• Tailor fidelity of simulation to particular need • Tailor level of test process discipline to match desired outcome • Validate the system and the system of systems interactions (typically in

build-up fashion)



Potential Organizational Impediments Chief Developmental Test role being defined to elevate the importance of shift left test. Address organizational Stove piping




Conclusions

• Testing is Both a Left and Right Side Vee Activity • Discover and Retire Risk Early Through Early Validation • Verification and Validation are Different


Discover and Retire Risk Early

Shifting Left With Developmental Testing for Systems of ... 1600_ITEA prese… · Internal –...

Documents

Transcript of Shifting Left With Developmental Testing for Systems of ... 1600_ITEA prese… · Internal –...