Successful Acquisition of FAASuccessful Acquisition of FAATerminal Doppler Weather RadarTerminal Doppler Weather Radar

James E. JonesC-130 AMP Chief Software Engineer

Cirrus Technology, Inc.

Third Annual Conference on the Acquisition of Software-IntensiveSystems – Track: Experience ReportsJanuary 26, 2004 Crystal City Marriott Arlington, Virginia

2

AgendaAgenda

• Introduction• Background• Contract Requirements• Software Development• Acquisition Team• Software Acquisition Process• Metrics• Conclusion

3

IntroductionIntroduction

• Air Traffic Control System Overview

• Capital Investment Plan (CIP)

• CIP Projects Cost and Schedule Problems

• Examples of CIP Projects

• Terminal Doppler Weather Radar (TDWR) System

• Key Elements for Successful Software Acquisitionand Development

4

Air Traffic Control SystemAir Traffic Control SystemOverviewOverview

5

Capital Investment Plan (CIP)Capital Investment Plan (CIP)

• The Federal Aviation Administration (FAA) CapitalInvestment Plan (CIP) [National Airspace System–(NAS) Plan] was established in late 1981.

• The purpose of the CIP is to modernize the nation’sair traffic control system for improvement incapacity, safety, and delays through the use of newtechnology.

• Currently, the CIP is a multibillion dollar investmentcomprising over 200 separate projects.

Source – GAO/T-RCED/AIMD-98-93, February 26,1998Source – GAO/T-RCED/AIMD-98-93, February 26,1998

6

CIP Projects Cost andCIP Projects Cost andSchedule ProblemsSchedule Problems

• Between 1982 and 1998 Congress hasappropriated over $25 billion1

– $5.3 billion on 81 completed projects– $15.7 billion on about 130 ongoing projects– $2.8 billion on projects that have been cancelled

or restructured– $1.6 billion for personnel-related expenses

associated systems acquisition

1 GAO/T-RCED/AIMD-98-93, February 26, 1998

7

Examples of CIP ProjectsExamples of CIP Projects

Example Project Current Status

• Microwave Landing System

• Radio Control Equipment

• Maintenance Control CenterProcessor / Maintenance MonitorConsole

• Advanced Automation System

• Voice Switching and Control System

• Terminal Doppler Weather Radar

• Terminated for Default (T4D)

• Terminated for Default (T4D)

• Terminated for Convenience(T4C)

• Restructured in 1994 afterestimated costs tripled from$2.5 billion to $7.6 billion

• Contract award – $1.3 billion;Production completed

• 100% on-time system delivery• Contractor of the Year award

• Delivered 6 months early• IEEE Computer Society Award

8

Terminal Doppler WeatherTerminal Doppler WeatherRadar (TDWR) SystemRadar (TDWR) System

• The TDWR systemenhance the safety of airtravel through the timelydetection and reporting ofhazardous wind shear inthe terminal area.

• Hazardous wind shearsdetected are microburstand gust fronts.

• TDWR displays areas ofwind shear and gust frontsto enable tower controllersto warn pilots of windshear conditions.

Weather is a primary factor in more than 35% of commercial aviation fatal accidents

9

Key Elements for Successful SoftwareKey Elements for Successful SoftwareAcquisition and DevelopmentAcquisition and Development

• Success in software acquisition anddevelopment depends on five key elements– The contract– The development process– The acquisition team– The acquisition process– Metrics

• To gain a better understanding of the keyelements of success, it helps to:– Understand the detailed requirements for each

element of success

10

BackgroundBackground

• Software Acquisition Management

• Key Software Acquisition Standards

11

Software Acquisition ManagementSoftware Acquisition Management

• Software was involved in essentially every CIPproject to resolve existing systems limitations.

• Effective management of software acquisition anddevelopment is unquestionably one of the greatestchallenges in the application of new technology.

• Software size, complexity, and diversity in air trafficcontrol systems applications make proper softwareacquisition management extremely critical.

The history of software acquisition and development has beenplagued with problems – many CIP projects have exceeded costand schedule.

12

Software AcquisitionSoftware AcquisitionManagement (cont’d)Management (cont’d)

• Quality of software is the result of– Development management techniques, controls, processes,

and tools

• Techniques, controls, and processes can bemanaged, measured, and progressively improved

• Software Acquisition Management methods andtechniques can be used to ensure compliance withtechniques, controls, and processes

• Software Acquisition Management methods andtechniques can also be used to verify softwarequality

13

Key Software AcquisitionKey Software AcquisitionStandardsStandards

• Standards allow organizations to communicate in commonterms – Backbone of a Successful Acquisition

FAA-STD-021A – Configuration Management,August 17, 1987

Softwareconfigurationmanagement

FAA-STD-018A - Computer Software QualityProgram Requirements, September 30, 1987

Software qualityassurance

MIL-STD-1521B – Technical Reviews andAudits for Systems, Equipments, andComputer Software, June 4, 1985

Technical reviewsand audits

DOD-STD-2167A – Defense System SoftwareDevelopment, February 29, 1988

Softwaredevelopment

StandardKey Process Area

Standards are essential….Key to acquisition and development success

14

ContractContract Requirements Requirements

• Contract Summary• Statement of Work (SOW)• Software Documentation• Software Contract Data Requirements List

Description• System Specification• Key Contract Modifications

15

Contract SummaryContract Summary

Raytheon Systems Company Nov 88- Develop, produce, and install 47 systems- 45 airports sites

Contract Award

Total F & E cost $322.2 MCost

Final Acceptance of the 1st System Testing - Contract Schedule Date Aug 93 - Incentive Target Date (at IAH) MEM MEM Feb 93Feb 93Final Acceptance of the 4th site - Contract Schedule Date Oct 93 - Incentive Target Date (at FLL) Apr 93

ScheduleIncentives

Firm Fixed Price Incentive (FFPI)Contract Type

An example of an FFPI contract type used to motive the contractor toincrease efficiency

GAO/RCED-99-25 FAA’s Modernization Program

16

Statement of Work (SOW)Statement of Work (SOW)

FAA-STD-018APerform software quality assurance

DOD-STD-2167APerform software development engineeringSoftware requirements analysis, preliminarydesign, detail design, code and unit testing, andtesting

FAA-STD-021APerform software configuration management

Contract DataRequirementsList (CDRL)

Prepare software documentationSoftware requirements specification, softwaredesign document, software test plan,etc.

DOD-STD-2167APerform software development managementPlanning, process, formal reviews/audits, riskmanagement, subcontract management

StandardExamples of Tasks

SOW – Basis for communicating management requirements Describes the tasks and how the project should be managed

17

Statement of Work (SOW) (cont’d)Statement of Work (SOW) (cont’d)

Governmentprovidedalgorithms

Develop meteorology algorithms

Provide Program Support Facility (PSF)software

Perform sizing and timing analysis

MIL-STD-1521BConduct technical reviews and auditsSystem Design Review (SDR)Software Specification Review (SSR)Preliminary Design Review (PDR)Critical Design Review (CDR)Test Readiness Review (TRR)Functional and Physical Configuration Audits(FCA/PCA)

StandardExamples of Tasks

18

Software DocumentationSoftware Documentation

• Software documentation is essential for managingthe development process.

• Software documentation is a natural by-product ofthe development effort to capture results for eachsoftware development activity.

• Software documentation was provided by theContract Data Requirements List (CDRL).

• CDRL items were referenced in the paragraphs ofthe SOW describing the development efforts.

19

Software DocumentationSoftware Documentation(cont’d)(cont’d)

• 16 Software CDRL items were defined:• CDRL Item key blocks

Approval Code – (A) Approved by the Contracting Officer8

Delivery RequirementsDelivery Requirements associated with Design Reviews

10, 11,12, and13

Contract ReferenceReference Statement of Work paragraphs

5

Authority (Data Acquisition Document No)Data Item Description (DID)1 – Defines format and contentpreparation instructions for the data product generated by taskrequirements as delineated in the contract.

1 Tailored to meet TDWR requirements1 Tailored to meet TDWR requirements

4

DescriptionBlock

CDRLs used to capture the results of software development activities

20

Development Management

P at PDR-1, F at CDRDI-MCCR-80014ASoftware Test Plan (STP)B025

5 days after new versionDI-MCCR-80013AVersion Description Document (VDD)B029

P at 5 DATC, F at 30 DARCDI-MCCR-80017ASoftware Test Report (STR)B028

F at PCA+1DI-MCCR-80029Software Product Specification (SPS)B034

DI-MCCR-80024A/T

DI-MCCR-80022A

DI-MCCR-80018A

DI-MCCR-80021A

DI-MCCR-80019A

DI-MCCR-80015A

DI-MCCR-80027A

DI-MCCR-80012A

DI-MCCR-80026A

DI-MCCR-80025A

DI-MCCR-80030A/T

DI-MCCR-80030A

DID

B053

B033

B032

B031

B030

B026

B037

B023

B036

B022

B022

B021

CDRL

P at PDR-1, F at CDR-1Interface Design Document (IDD)

P at SDR, F at SSRInterface Requirements Specification(IRS)

P at PDR, F at CDRComputer Resources IntegratedSupport Document

P at CDR+8, F at PCA+4Firmware Support Manual

P at CDR+8, F at PCA+4Computer System Operator’s Manual

P at CDR+8, F at PCA+4Software Programmer’s Manual

P at CDR+8, F at PCA+4Software User’s Manual

Support

Cases: P at CDR-3, F at CDRProcedures: P at 90 DPT, F at DPT

Software Test Description (STD)– Test Cases and Procedures

P at PDR-1, F at CDR-1Software Design Document (SDD)

P at SDR, F at SSRSoftware Requirements Specification(SRS)

P at SDR, F at SSRCommercial Product SoftwareDocumentation

Products

P at CA+2, F at SDRSoftware Development Plan (SDP)

DATE OF SUBMISSIONP – Preliminary F - Final

Description

SOFTWARE CONTRACT DATA REQUIREMENTS LIST (CDRL) ITEMS DESCRIPTION

21

System SpecificationSystem Specification

• Meteorological algorithms implemented in highorder language (HOL)

• Use of commercial software approved by the FAA• Programming Language (s)

– Use of a single higher order language (HOL)– Lower order language approved by Government– Programming language approved by Government

• Software Reliability

1/35 K SLOC3

1/70 K SLOC2

Zero1

Maximum NBR of UnresolvedErrors

Software ErrorCategory

22

Key Contract Modifications Key Contract Modifications

• Modification 0004: June 14, 1989– System specification– Microburst Detection Algorithm– Gust Front Detection Algorithm– TDWR Product Output Formats

• Modification 0010: June 25, 1990– Statement of work: Software documentation– Microburst Detection Algorithm– TDWR Product Output Formats

• Modification 0015: October 24, 1990– Microburst Detection Algorithm– Gust Front Detection Algorithm

• Modification 0020– Equitable adjustment under Mod 0004, 0010, 0015– FCA From 42 to 48 MACA (Nov 92)– PCA From 40 to 46 MACA (Sep 92)

Key Modifications –Microburst and GustFront Algorithmchanges

23

Software DevelopmentSoftware Development

• System Software Description

• Software Development Environment

• Software Development Life Cycle Models

• Software Development Methodology

24

System Software DescriptionSystem Software Description

• 11 Computer Software Configuration Items (CSCI)– Allocated to 6 Hardware Configuration Items (HWCI)

• 6 Applications– Digital Signal Processing, Radar Product Generation, Remote

Monitoring, Display, Antenna Control, Transmitter Control

• Distributed Microprocessors– 68020 SBC, Harris 3803 Nighthawk, 68030, Array Processor

• Multiple Operating Systems– Sun OS, Unix, VRTX32, CRT/RT

• Multiple Programming Languages– C, Assembly, Fortran, Microcode

• 304.2K - Source Lines of Code• 4,416 - Software Requirements

TDWR – Real-Time Software-Intensive System

25

Application CSCIsApplication CSCIs

DEVELOPMENT STATUS: NEW CODE• Command and control the transmitter• Performs performance fault monitoringfunction

1752.7KCAssembly

None68020 SBCTransmitterControl (XMT)

9

DEVELOPMENT STATUS: NEW CODECommand and control the pedestal.

1242.5KCAssembly

None68020 SBCAntennaControl

8

DEVELOPMENT STATUS: NEW CODEManages all graphic and alphanumericdisplay functions for display of radarproduct and status information.

77830CSUNOSUNIX

Sun 386iDisplayComputer(DPL)

4

DEVELOPMENT STATUS: NEW CODEPerforms performance monitoring andmaintenance alarm/alert generation.

176080CCX/RT• HARRIS 3803• ArrayProcessor

RemoteMonitoringSubsystem(RMS)

3

DEVELOPMENT STATUS: NEW CODE• Base data estimation/data conditioning• Weather product generation

78890CCX/RT• HARRIS 3803• ArrayProcessor

Radar ProductGeneration(RPG)

2

DEVELOPMENT STATUS: NEW CODEControls the basic radar timing function,performs clutter filtering and clutter filterselection, and implements the pulse-pairprocessing to generate the zeroth andfirst moment data.

56425CAssemblyMicrocode

VRT32• 68020 SBC• MicroProcessors

Digital SignalProcessor(DSP)

1

DescriptionNRBofRqmts

SLOCK

ProgrammingLanguage (s)

OSProcessorTitleCSCINo

26

Support CSCIsSupport CSCIs

DEVELOPMENT STATUS: COTS/NEWCODETools used for Sun workstations, dataprocessor system, and 68020 SBCs. AllCSCE tools, and programming utilities.

N/A7.5KN/AN/ASUN 386iSoftwareDevelopmentTools (SDV)

5

DEVELOPMENT STATUS: COTSThe Display System Operating Systemfor CSCI-4

N/AN/AN/ASunOS

Sun 386iDisplaySystemOperatingSystem (Unix)

11

DEVELOPMENT STATUS: COTSThe RPG/RMS Data ProcessingOperating System for CSCI-2 and 3.

N/A7.5KN/ACX/RT(Unix)

•HARRIS 3803(NIGHTHAWK)68030 CPUs

DataProcessingOperatingSystem (DPO)

10

DEVELOPMENT STATUS: COTSAll multi-tasking operating systemservices and related boots strap anddiagnostic function for the 68020SBC.

N/AN/AN/AN/A• 68020 SBCVRTX-32OperatingSystem (VTX)

7

DEVELOPMENT STATUS: NEW CODE,EXISTING, MODIFIEDSoftware to generate simulated base dataand moment data and to simulateexternal interfaces.

22759kCFORTRAN

CX/RTCX/UN(Unix)SunOSUnix

• HARRIS 3803(NIGHTHAWK)68030 CPUs• Sun 386i

Test ToolsLibrary (TTL)

6

DescriptionNRBofRqmts

SLOCProgrammingLanguage (s)

OSProcessorTitleCSCINo

27

Software DevelopmentSoftware DevelopmentEnvironment (SDE)Environment (SDE)

• Established to support the software engineeringprocess and provide production capability for theSoftware Requirements Specification (SRS) andSoftware Design Document (SDD) CDRL Items

• Computer-aided software engineering (CASE) tool(IDE Software Through Picture) used for structuredanalysis and design

• SDE augmented by management methods andpractices (i.e., measurements/metrics andmonitoring progress, judging the quality of the CDRLitems)

28

Software DevelopmentSoftware DevelopmentLife Cycle ModelsLife Cycle Models

System Design

CSCI-1Requirements

Analysis

CSCI-1Preliminary

Design

CSCI-4Requirements

Analysis

CSCI-4Preliminary

Design

Build 1• Detail Design • Code & Unit Test• CSC Integration• CSCI Testing

Build 4• Detail Design• Code & Unit Test• CSC Integration• CSCI Testing

System Testing

SDR PDRSSR CDR1 CDR4 TRR4TRR1 FCA/PCA

Prototype

Design Reviews used as Quality Gates

29

Software DevelopmentSoftware DevelopmentMethodologyMethodology

• Structured analysis and design performedduring software requirements analysis andpreliminary design.– Data flow diagrams used to model the software

requirements• Software requirements documented IAW DID (DI-MCCR-

80025A)

– Control flow diagrams used to model thepreliminary design

• Software design documented IAW DID (DI-MCCR-80012A)

30

Acquisition TeamAcquisition Team

• Program Management

• Technical Office

• Technical Support

31

Program ManagementProgram Management

• Program Management– Program Manager– Business Manager– Associate Program Manager

• Contracts• Logistics• Systems Engineering• Test and Evaluation• Flight Standards• Quality Assurance• General Counsel• Maintenance

Federal Aviation AdministrationProgram Manager for Weather Radar,ANR-500

Identified high-risk development efforts and structured themanagement process accordingly – to provide control of thedevelopment and to gain timely and accurate insight into its progress

32

Technical OfficeTechnical Office

• Technical Office– System Design– System Hardware Design– System Software Design– Test and Evaluation– Logistics– Configuration Management– Planning– Data Management– Integration

Martin Marietta Aerospace Air Traffic ControlDivisionSystems Engineering Integration Contractor

Ensure system requirements can be implemented within schedule andcost constraints

33

Technical SupportTechnical Support

• Technical Support– Meteorological Algorithm

• Massachusetts Institute of Technology (MIT) LincolnLaboratory

– Products Output Display• National Center for Atmospheric Research (NCAR)

– Weather Verification Tests• National Severe Strom Laboratory (NSSL)

– Engineering Support • NYMA, Inc.

Ensure accuracy of Meteorology Algorithm and Products OutputDisplay

34

Software Acquisition ProcessSoftware Acquisition Process

• Software Acquisition ManagementRelationships

• Software Acquisition Expertise

• Standards Applied

• Activities

35

Software AcquisitionSoftware AcquisitionManagement RelationshipsManagement Relationships

• Software Acquisition Management– Supported the assemble of the

contract requirements– Provided visibility to program

management• Software development

management– Managed the engineering

process– Provided processes and CDRL

items for evaluation• Software engineering

– Built the software product– Provided software work

products for evaluation

Program management

Software acquisition

management

Softwaredevelopment management

Software engineering

SW

wor

k P

rodu

ctsC

DR

L Ite

ms

Ass

essm

ents

/E

valu

atio

ns/

Rep

orts

Pr o

cess

es

SUPPORTSoftware configuration managementSoftware quality assurance

36

Software Acquisition ExpertiseSoftware Acquisition Expertise

• (6) Software engineering expertise– Software acquisition management– Software development process– Software development methods and techniques

using Government standards– Software design – mission critical, embedded,

real time, and distributed– Application of programming languages– Software test methods and techniques

• (8) Application Domain expertise– Microburst, Gust Fronts, Precipitation, Base Data

Processing, and Clutter Suppression

The focus is on what is done and the product being built

37

Standards AppliedStandards Applied

Standards are essential….Key to acquisitionand development success

Government standards were applied toestablish a uniform set of requirements andimplementation practices.

Government standards also provided insightinto the development management,engineering, testing, software configurationmanagement, and software qualityassurance processes and products

38

Standards AppliedStandards Applied

• Government standards

Preparation of Statement of WorkMIL-HDBK-245B

Defense System Software DevelopmentDOD-STD-2167A

Technical Reviews and Audits for Systems,Equipment and Computer Software

MIL-STD-1521B

Specification PracticesMIL-STD-490A

Work Breakdown Structure for Defense MaterialItems

MIL-STD-881A

Statement of Work PreparationFAA-STD-031A

Configuration Management PlanFAA-STD-021A

Computer Software Quality ProgramRequirements

FAA-STD-018A

Preparation of Specification DocumentsFAA-STD-005D

39

ActivitiesActivities

• Software Acquisition Planning• Request for Proposal Support• Management Assessments• Reviews / Meetings Participation• CDRL Items Evaluation• Software Test Evaluation• Requirements Accountability

Software Acquisition Management activities used to ensure complianceto standards and delivery of a quality software product

Software Acquisition Management activities used to ensure complianceto standards and delivery of a quality software product

40

Software Acquisition PlanningSoftware Acquisition Planning

• Defined plans for performing software acquisitionmanagement

• Content of detailed plan:– Defined organizational structure, responsibilities, and

required resources– Identified development risk and described criteria for risk

assessment– Defined Interfacing with the prime contractor’s

• Program Office Software Manager• Software Quality Assurance• Software Configuration Management

– Defined technical evaluation method and technique– CDRL item preparation standards and acceptance criteria

41

Software Acquisition PlanningSoftware Acquisition Planning(cont’d)(cont’d)

• Content of detailed plan (cont’d)– Established procedures for:

• Conducting management assessments• Evaluating CDRL items• Participating at reviews and meetings• Action items tracking and closure• Review Item Discrepancies (RID) processing

– Detailed schedule for CDRL item reviews– Established metrics for:

• Process, Product, Project, and Productivity

Procedures and metrics established to gain insight into the softwaredevelopment activities, to verify compliance, and to verify product quality.

42

Software Acquisition PlanningSoftware Acquisition Planning(concl)(concl)

• Review Item Discrepancy (RID) form– Use to capture CDRL items and work products

evaluations• Documentation Identification• Comment Location (Page, Paragraph, Figure, Table)• Comment (i.e., consistency, missing information,

correctness, ambiguous)• Recommendation / proposed solution

– Use to disposition review comments• Concur/non-concur• Action

– Track review comments closure• Date closed• Comment

43

Request for Proposal (RFP)Request for Proposal (RFP)SupportSupport

• Provided RFP support to assemble:– Solicitation package– Contract– Contract modifications

• Solicitation package included– Statement of Work– CDRL Items and DIDs– System Specification– Work Breakdown Structure (WBS)

Solicitation package -- Foundation of a successful software acquisition

44

Management AssessmentsManagement Assessments

• Management assessments were conducted to verifycompliance with contractual requirements.

• Management assessment were also conducted toensure that the development activities and productswere in accordance with the documenteddevelopment processes documented in the plans.

• Management assessments conducted for:– Software development management– Software configuration management– Software quality assurance

45

Software DevelopmentSoftware DevelopmentManagementManagement

• Verified the Software Development Plan (SDP)(CDRL B021) complied with DID DI-MCCR-80030A

• Verified software development managementactivities conducted for: 1) planning the softwareengineering, 2) managing the software project, 3)monitoring and controlling the development, 4) riskmanagement, and 5) subcontract management

• Examples of activities verified– Software development activities IAW DOD-STD-2167A– Design reviews IAW MIL-STD-1521B– Schedule consistent with the Program Master Schedule and

Contract Work Breakdown Structure

46

Software ConfigurationSoftware ConfigurationManagementManagement

• Verified the Configuration Management Plan (CMP)(CDRL A005) compiled with FAA-STD-021A

• Verified software configuration managementactivities conducted for establishing and maintainingthe integrity of the software products

• Examples of activities verified– Configuration identification is performed– Configuration control is established and performed– Configuration status accounting report generated on all

CDRL items comprising the Baselines (Allocated,Developmental Configuration, and Product)

– A software baseline library is established

47

Software Quality AssuranceSoftware Quality Assurance(SQA)(SQA)

• Verified the Computer Software Quality ProgramPlan (CSQPP) complied with FAA-STD-018A

• Verified software quality assurance activitiesconducted to provide management with visibility intothe process being used and the products being built– Examples of activities verified

• Software development activities IAW SDP• Software configuration management IAW SCMP• CDRL items internally evaluated prior to delivery• SQA audits are performed• Records maintained and presented at the design reviews

• SQA audits witnessed during preliminary and detaildesign phases

48

Reviews / MeetingsReviews / MeetingsParticipationParticipation

• Program Management Reviews

• Design Reviews

• In-Process Reviews

• Technical Interchange Meetings

Participated in reviews and meetings to gain insight and to providefeedback

Key focus --- What is done and the product being built

49

Program Management ReviewsProgram Management Reviews(PMR)(PMR)

• 39 PMRs conducted• Examples of items verified

– Status of work accomplished– Problems and impacts– Technical performance consistency with the

Program Status Report (CDRL item)

50

Design ReviewsDesign Reviews

• 57 informal (28) and formal (29) design reviews held– Determine visibility and status, measure progress, and assess the

integrity of the development processes and products

• Co-chaired all formal design reviews• Associated CDRL items RID provided prior to the

design review• Verified compliance with the SDP and MIL-STD-

1521B• Verified technical presentations• At the post review, attested to the Action Items• Monitored Action Items until closure

Design reviews used as quality gates

51

In-Process ReviewsIn-Process Reviews

• 14 In-Process Reviews (IPR) held to evaluate workproducts and provide feedback– 95 products (examples)

• Computer Software Unit (CSU) source code, Test cases, Testprocedures, Test reports

• Software Trouble Reports (STR)• Software development folder procedures• Program support library (PDL)• Cost/Schedule Status Report

• RIDs generated and disposition at the post review• SQA ensured action items were prepared and

tracked until closure

IPRs used to improve the process, product quality and content

52

Technical InterchangeTechnical InterchangeMeetings (TIM)Meetings (TIM)

• 52 TIMs conducted to resolve issues, RIDsresolution, and action items

• Examples of TIMs– Software development process– CDRL preparation (i.e., SRS, SDD, and STD)– SDF procedures– Meteorological algorithm changes– CX/RT Operating system argumentation

53

CDRL Items

Technical OfficeEvaluate

CDRL Item

RIDs

Technical OfficeRIDs

Consolidation

Technical SupportEvaluate

CDRL Item

COTRRIDs

Approval

RIDs

Consolidated RIDs

ApprovedRIDs(4,300)

ApprovedRIDs(4,300)

CDRL ApprovalLetter• Approved• Approved w/comment• Disapproved

CDRL ApprovalLetter• Approved• Approved w/comment• Disapproved

Software Development

Document FlowProcess Flow

Software Acquisition Management

CDRL Item EvaluationCDRL Item Evaluation

(45)

54

CDRL Item EvaluationCDRL Item Evaluation

• Techniques– Static analysis

• Detect errors through examination of the CDRL item• Focus -- form, structure, completeness, and consistency

– Dynamic analysis• Detect errors by studying the response to a set of input data• Focus -- functional processing, performance, interfaces, and

design constraints• Test coverage analysis

– Formal analysis• Processing accuracy, efficiency, and correctness of algorithm

– DOD-STD-2167A evaluation criteria– Requirements traceability

• System specification, software requirements specification,design, code, and test cases

Proper use of CDRL evaluation techniques provided visibility into thedevelopment effort.

55

Software Test EvaluationSoftware Test Evaluation

• Designated by the COTR as the Software TestDirector

• Evaluated software test CDRL Items– Software Test Plan, B025– Software Test Description (Test Cases and Procedures, B023)– Software Test Report, B028

• Participated at Test Readiness Reviews (TRR)– Updates for SDDs, source code, and Software Development

Folders (SDF)– CSU and CSC testing status– Source code in the Program Support Library (PSL)– Approved Engineering Change Proposals (ECP)

• Witnessed CSCI Formal Qualification Testing (FQT)– Ensured CSCI FQT conducted IAW approved STP and STD

Participation validated the “as-built” software.

56

Requirements AccountabilityRequirements Accountability

• Requirements database established duringthe CDRL Item and Software Test evaluationto capture requirements allocation andtraceability

• Requirements database also used to captureTest Case results (example)– Success– Failure/Error (Test Procedure Step)– Software Problem Report Number– Comment (e.g., not critical, needs immediate attention)

Requirements accountability database provided means for:• Validating the CSCI implementation of allocated system requirements• Validating “as-built” software satisfied the CSCI requirements

Requirements accountability database provided means for:• Validating the CSCI implementation of allocated system requirements• Validating “as-built” software satisfied the CSCI requirements

57

System Specification

Rgmts

Software RequirementsSpecification (SRS)

CSCI-1Rgmts

CSCI-NRgmts

Software Design Document (SDD)CSCCSU

Design

CSCCSU

Design

CSUSource Code

Test - XTest Procedures

Test - XTest Cases

Software Test Report (STR)

Software TestDescription (STD)

Test CasesResults

AllocationTraceability

Requirements AccountabilityRequirements Accountability

Requirements Flow

58

MetricsMetrics

• 45 Metrics provided insight into four Key AcquisitionAreas:– Process

• Provided insight into the software development process andhow it was working

– Product• Measured the quality of the product (e.g., frequency of

requirement changes, number of STRs, number of RIDs)– Project

• Provided progress-oriented measures (e.g. scheduleattainment, CDRL delivery)

– Productivity• Measured the rate at which the work was progressing

• Metrics reported to ALL to encourage performance.

Metrics provided feedback to refine the software development process

59

Metrics Mapping to KeyMetrics Mapping to KeyAcquisition AreasAcquisition Areas

XXRequirements Mutation

XXXIn-Process Reviews RIDS

XXManagement Assessments

XXCDRL Item RIDs

X XXSoftware Trouble Reports(Source Code)

XRPG/RMS HWCI-5

Resource Utilization

X XCDRL Item Status

X XFQT RPG CSCI-2 Progress

X XDevelopment Progress

XX XCost/Schedule Status

Report

ProductivityProjectProductProcessExamples of Metrics

60

Samples of MetricsSamples of Metrics

FQT Progress

CDRL RIDs

Requirements Mutation

Cost/Schedule Status Report

61

ConclusionConclusion

• Key Elements for Success

• TDWR Success Attributes

62

Key Elements For SuccessKey Elements For Success

• Success in software acquisition and developmentdepends on five key elements– The Contract

• Statement of Work, Contract Data Requirements List, SystemSpecification

– The Development Process• Software Development Plan, Software Configuration

Management Plan, Software Qualify Assurance Plan– The Acquisition Team

• Software engineering and application domain expertise– The Acquisition Process

• Activities used documented procedures and standards– Metrics

• Provided insight into four Key Acquisition Areas:– Process, Progress, Product, and Productivity

63

TDWR Success AttributesTDWR Success Attributes

• Software Completion– Build 2/3 FQT Completed June 6,

1991– Build 4 FQT Completed December

18, 1991

• Software Development Effort1

– Cost $17 million– 300,000 lines of code– 70 software and test engineers

• First Production– (MEM) Memphis, TN November

19921 (Schedule Incentive TargetDate December 1992)

1 Source: Aviation Week & Space Technology, January 27, 1992“TDWR Installation Begins; Sizable Fuel Savings Expected

64

TDWR Success AttributesTDWR Success Attributes

Radar Product Generation (RPG) CSCI-2

02.3_6.11

02.3_6.10

02.3_6.09

02.3_6.08

02.3_6.03

02.3_6.02

02.3_6.01

SRS #

1.9 sec5 secPrecipitation

1.2 sec5 secActiveRunwayResponse

.5 sec3 secError Report

4.68 sec5 secBase DataDisplay

.5 sec60 secArchiveRequest

.9 sec5 secGust Front

.9 sec5 secMicroburst

ACTUAL MEASUREMENTRQMTEVENT

Actual Key Performance MeasurementsActual Key Performance Measurements

65

TDWR Success Attributes

Raytheon Electronic Systems received IEEE Computer Society Award foroutstanding achievement in improving software processes

In 1991, Raytheon’s software process was evaluated at level three againstthe SEI Capability Maturity Model (CMM)…

“One notable project is the terminal doppler weather radar (TDWR) project werecently accomplished for the Federal Aviation Administration (FAA),” saidHaley . “Software development played a key role in achieving TDWR deliveryto the FAA six months ahead of schedule, an objective set to reflect a growingnational urgency for microburst detection . “

66

TDWR Success AttributesTDWR Success Attributes

TDWR Currently Operational at 45 Airports1

1 Source: Aviation Week & Space Technology, January 27, 1992“TDWR Installation Begins; Sizable Fuel Savings Expected

Questions??Questions??

James E. JonesC-130 AMP Chief Software EngineerCirrus Technology, Inc.109 Park DriveWarner Robins, GA 31088(478) 922-9234jjones@cirrusti.com