Developmental Test and Evaluation Committee Status and Plans December 2013

NDIA SE Division – Annual Planning MeetingDecember 11-12, 2013

1

Developmental Test and EvaluationCommittee

Status and Plans December 2013

Beth Wilson, RaytheonSteve Scukanec, Northrop Grumman

Industry Co-Chairs


2

DT&E Committee2013 Status

Activity Plans for 2013 Status/PlansScientific Test and Analysis TechniquesSTAT for T&E

Effective use of statistical approaches for test optimization (including DOE) with examples

White paper completePreparing journal publicationsPresented results at SE conference

Modeling & Simulation

Evaluate test as a stakeholder Joint meetings with SED meetingsPropose for 2014

ArchitectureNDIA T&E Conf DT&E panel, STAT report Conference cancelled

NDIA SE Conf DT&E tracks Joint tracks, metrics/STAT reports

Added Task Dec 2012: Metrics

Provide more information on leading indicators for T&E

Status: draft reportPlans: results at SE conference

Added Task Dec 2012: Reliability

Develop a better test strategy for reliability (collaborate with IEEE Reliability Society)

Discussed at June 2013 meetingLeads identified Nov 2013

Added Task Jun 2013:Cyber Testing

Develop recommendations for Cyber Guidelines 2.0 in 2014

Lead and team identifiedKicked off Nov 2013 (joint with ITEA)

Added Task Jun 2013:Chief Developmental Tester

Develop recommendations for industry interaction with Chief Developmental Tester role

Lead and team identified

Complete In Process Cancelled


3

DT&E Committee - 2014 Task PlanProjects Working Group

Proposed 2014 Tasks:• Cyber Testing Project• Chief Developmental Tester Project• Collaboration with IEEE Reliability• Joint meetings with SED Committees

• Modeling and Simulation• Architecture• Systems Security Engineering• Education and Training

Deliverables/Products• Cyber Testing Recommendations white paper• Chief Developmental Tester white paper

• Candidate KSABs for role (E&T)• Industry interaction recommendation• Tutorial on role for SE conference

• Inputs to reliability standards effort• Joint meeting notes

Schedule / Resources• Cyber Testing Draft Recommendations Feb 2014

• Weekly teleconferences started 12/3/2013• Full white paper later in 2014

• Chief Developmental Tester• Joint meeting with E&T with SED meeting• Tutorial at SE conference

• Reliability standards input for Jan/Aug meetings• Test as a Stakeholder joint meetings with SED

• Modeling and Simulation• Architecture• Systems Security Engineering

Issues / Concerns:• Industry engagement• NDIA T&E conference may be cancelled (Seattle

in March)• ITEA conference (Denver in September) may be

postponed and moved


4

Summary of DT&E Committee2013 Results/2014 Plans

Topic Activity

T&E

Test Optimization 2013: Scientific Test and Analysis Techniques report/articles

Cyber Testing 2013-2014: Industry recommendations for cyber testing

Chief Developmental Tester 2013-2014: Industry interaction with new role

SE

Performance Measurement 2013: Requirements Verification leading indicators report

Reliability 2013-2014: Recommendations for better reliability testing

Modeling and Simulation 2014: T&E Perspective for Modeling/Simulation products

Architecture 2014: T&E Perspective for Architecture views

Education and Training 2014: Chief Developmental Tester role

Systems Security Engineering

2014: Cyber testing guidelines connections to Program Protection Planning

Completed Current Proposed


5

DT&E Committee 2012-2013Statistical Test Optimization

2012 SE Conference summit/workshop thread- Tutorials on Monday 10/22- Presentations on Wednesday 10/24- Synthesis Panel on Wednesday 10/24

Follow-on Results: - DT&E Committee white paper - ITEA Journal article- SE Conference track Oct 2013

Activity Plans for 2013 Status/PlansScientific Test and Analysis TechniquesSTAT for T&E

Examples of effective use of statistical approaches for test optimization (including DOE) for an implementation framework

White paper completeSE Conference track: 2012 and 2013ITEA journal publication MORS conference Nov 2013

2012 - 2013


6

DT&E Committee 2012-2013Leading Indicator Metrics

2012 SE Conference workshop- Follow-on workshop with System Performance Measurement WG - Focused on Requirements Verification not addressed in first report

Potential Measures Identified: - System Maturity Level- Verification Requirement Maturity- Technical Measures and Stakeholder Need

Activity Plans for 2013 Status/PlansMetrics(with Performance Measurement WG)

Provide more information on leading indicators for T&E

Draft reportPresented results at SE conference

2012 - 2013


7

DT&E Committee 2012-2013Reliability Testing

Reliability Testing Focus- Software reliability standard- Cyber testing initiatives- Data for reliability growth measures

Input/Feedback Opportunities: - IEEE meetings January and August- Periodic revision periods

Activity Plans for 2013 Status/PlansReliability Testing(with IEEE)

Develop a better test strategy for reliability (collaborate with IEEE Reliability Society)

Solicit feedback on existing standardsIdentify recommendations to improve reliability testing

Project Leads: Lou Gullo, Taz Daughtrey


8

DT&E Committee 2013-2014Cyber Testing

Activity Plans for 2013 - 2014 Status/PlansCyber Testing Review what industry does for

cyber testing, expand focus to include interoperability

Recommendations for Cyber Testing guidelines

Project Lead: Dave DesjardinsJoint Project with ITEA


9

DT&E Committee 2013-2014Chief Developmental Tester

Activity Plans for 2013-2014 Status/PlansChief Developmental Tester

Review service policies for new role and industry implementations

Propose model for industry interactionWhite paperT&E/SE conference presentations

The Secretary of Defense shall require that each major defense acquisition program be supported by—‘‘(A) a chief developmental tester; and‘‘(B) a governmental test agency, serving as lead developmental test and evaluation organization for the program.

Coordinating DT&E activitiesInsight into Contractor activitiesOversee T&E activitiesInform government PM about contractor DT&E results

Project Lead: Joe Manas


10

Backup

DT&E Committee Results 2008-2013


11

Summary of DT&E Committee Test and Evaluation Efforts

Topic Activity

DT&E Committee

DoD T&E Policy Study 2006 – 2008: Workshops and Study ReportImproving T&E in the DoD Acquisition Process

Integrated Testing 2008 – 2010: Integrated Test Study NDIA presentations and tutorialITEA journal article

RFP Language for T&E 2010 – 2011: Comments for Update to OSD GuideIncorporating T&E into DoD Acquisition Contracts

Test Optimization 2012: Statistical Optimization Conference Thread2013: Scientific Test and Analysis Techniques report/articles

Cyber Testing Started 2013: Industry recommendations for cyber testing

Chief Developmental Tester Started 2013: Industry interaction with new role


12

Summary of DT&E Committee Systems Engineering Efforts

SED Committee Activity

SE Collaboration

Software 2009: SW T&E Summit Recommendations

System of Systems 2010 – 2011: T&E for SoS Workshop and Initiatives2012: Final Report on Best Practices Model

Performance Measurement

2012: Leading Indicators for T&E workshop

2013: Requirements Verification leading indicators report

Modeling and Simulation

2011: Effective Use of Modeling and Simulation for T&E 2012: Distributed Model-Based Testing

Reliability Started 2013: Recommendations for better reliability testing


13

T&E Themes

ForConferences

SEDCommittee

CollaborativeProjects

DT&E Committee Current Structure Since 2010

NDIA Systems Engineering Conference (Q4) Collaborative Projects

NDIA Test and Evaluation Annual Conference (Q1)Periodic T&E Committee-Focused Conferences

Cooperation with other NDIA Divisions

NDIA Industrial Committee on Test and Evaluation

(ICOTE)

DT&E Committee Focus: T&E initiatives aligned with SE, DT&E

T&E Themes & Issues

Project ResultsDT&E Themes

NDIATest and Evaluation

Division

NDIASystems Engineering

Division

NDIADT&E Committee

SE Themes & Issues


14

DoD T&E Policy Study August 2006: DT&E Committee Kickoff

Policy Study:“Improving T&E in the DoD Acquisition Process”Industry T&E policy recommendations

Workshops:August 2007January 2008

Focus Areas:1. Earlier contractor and tester involvement2. Integrated DT/OT and DT operational relevance3. Suitability

April 2008: Report Summarized Results:10 Findings15 Recommendations

2006-2008


15

Integrated Testing (CT/DT/OT)Implementation Framework

2008-2010


16

Software Test and EvaluationSoftware Summit

RFP Language

How Much Testing

is Enough?

Lifecycle and

End-to-End Software T&E

Changing Paradigms

Training & Competency ModelPolicy, Guidance, and Standards

Tools, Automation, Methodologies, Process2009


17

RFP LanguageIndustry

#

Master #

Reviewer Line# Comment and Rationale Recommended Input A/R/P

Industry Comments for Update:Incorporating Test and Evaluation

Into Department of DefenseAcquisition Contracts

Recommendations from SW Summit

2010-2011


18

Test and Evaluation forSystems of Systems

2010: Workshop2009: “Sleepless Nights” List of Issues

2010: “Sominex” Resulting Initiatives

InitiateSoS

PlanSoS

Update

EvolveSoSArch

EvolveSoSArch

ImplementSoS

Update

PlanSoS

Update

ContinueSoS Analysis

ImplementSoS

Update

PlanSoS

Update


ConductSoS Analysis


ImplementSoS

Update

DevelopSoSArch

External Environment

2011: Best PracticesWave Model

2009-2012 2012: Final Report


19

Effective Use of Modeling and Simulation for Test and Evaluation

Joint Meeting in August 2011• Distributed Testing, the Joint Mission

Environment Test Capability (JMETC) and the Test and Training Enabling Architecture (TENA)

• DoD M&S Community of Interest Data Management Working Group

• LVC Architecture Roadmap Implementation (LVCAR‐I) Gateways Effort Applicability to T&E

• OSD T&E Working Group • Raytheon Presentation on M&S for T&E • Potential Topics for November AMSWG

Meeting

2011


20

Modeling & Simulation forDistributed Testing

Benefits:• Find integration issues earlier• Test to learn in ‘safe’ environment• Protect proprietary information• Facilitate DT to OT transition• Increase performance testing range in operating environments • Support end to end studies throughout the programBarriers:• Security• Lack of persistent network• Early consideration of technical issues• Perceived value• Disconnect between the communities (M&S and T&E)Recommendations:• Harmonize the standards for M&S and Test for the life cycle perspective (HLA, TENA, Metadata)• Create a framework for reusing and repurposing M&S through the product model• Establish M&S as part of statistical test design

• Determine what tests are conducted to acquire data for model validation.• Fewer test events with better models.

• Recommend the use of M&S to do I&T • Recommend establishment of JMETC as a persistent node for industry to engage in MBDI&T

Joint TrackSE Conference October 2012

Joint MeetingAugust 2012

2012

Developmental Test and Evaluation Committee Status and Plans December 2013

Documents

Transcript of Developmental Test and Evaluation Committee Status and Plans December 2013