The DMCC Perspective on the The DMCC Perspective on the Application to Meteorological Application to Meteorological

Software of DOE’s SQA Software of DOE’s SQA RequirementsRequirements

Prepared by:Cliff Glantz (PNNL)

Carl Mazzola (Shaw Env.)Walt Schalk (NOAA/SRL)

Roadmap

Intro to the U.S. Department of Energy Meteorology Coordinating Council (DMCC) Why have a software quality assurance (SQA) program for safety software?DOE SQA requirements for safety softwareThe difference between SAFETY and SAFETY-RELATED softwareProblems applying SQA for SAFETY software to SAFETY-RELATED softwareAn alternate method of SQA for SAFETY-RELATED softwareApplication to meteorological software…

Department of Energy Meteorology Coordinating Council (DMCC)

Coordinates meteorological programs at all DOE sites. Promotes cost-effective meteorological support programs; facilitates use of common methods, procedures, and standards; and plans for future needs and missions Linked to DOE’s Emergency Management Issues Special Interest Group and Subcommittee on Consequence Assessment and Protective Actions (SCAPA) Resolves internal technical disputes with a mixture of violence and affection!

Background Info: SQA and Safety Software

In 2000 the Defense Nuclear Facilities Safety Board (DNFSB) issued a report critical of the SQA of safety software at DOE nuclear facilitiesAfter much back & forth, in 2005 DOE issued an Order and Guide establishing SQA requirements for safety software:

DOE Order 414.1cDOE Order 414.1c DOE Guide 414.1-4DOE Guide 414.1-4

Safety Software is Defined to Include:

Software that’s “performs a safety function” as part of a structure, system, or component (SSC) at a nuclear facility.Software that is used to classify, design, or analyze nuclear facilities. This software helps to ensure the proper accident or hazards analysis of nuclear facilities or an SSC with a safety functionSoftware that performs a hazard control function or a control function necessary to provide adequate protection from hazards. This software supports “eliminating, limiting, or mitigating nuclear hazards to workers, the public, or the environment…”

Establishing QA Requirements for Meteorological Programs

ANSI/ANS-3.11 and DOE/EH-0173T: Make the point that quality-assured site meteorological data are critical for atmospheric dispersion modeling DOE O 414.1C and DOE G 414.1-4 establish SQA requirements for SAFETY software based on “ASME NQA-1-2000, Quality Assurance Requirements for Nuclear Facility Applications, or other national or international consensus standards that provide an equivalent level of quality assurance.”Atmospheric dispersion models are SAFETY software if they are used for DOE hazard analyses or safety assessments at nuclear facilities

Establishing QA Requirements for Meteorological Programs (Cont.)

It is reasonable to assume that the meteorological software used to in these hazards analyses and safety assessments should be either SAFETY or SAFETY-RELATED software. DOE/HS has not yet decided how to classify meteorological software DOE/HS could decide met software that generates data for SAFETY software must also be SAFETY software.What are the potential implications of this decision?

*** HUGE ***

Why Not Apply Safety Software SQA Requirements?

The average cost to bring widely used simple dispersion codes (e.g., EPICode) into full compliance with DOE O 414.1C is >$300K/model! For more sophisticated models, these costs would be greater. Need to find a balance that allows the effective use of resources (i.e., balance technical development needs against SQA needs). Remember – SQA helps to ensure that software does what it is designed to do, not that it is doing the right thing!

A Solution…

Best Strategy: Answer the question for DOE before they grapple with this issue.How: Voluntarily adopt a standard for SQA, based on meteorological software being SAFETY-RELATED not SAFETY Software.Proactive: We define guidance for SAFETY-RELATED and applications before we are told what they should be.

DMCC Proposed Action

Improve current SQA practices for met softwareMet software should meet a “reasonable” set of SQA requirements applicable to SAFETY-RELATED software.Apply consistent standards across DOE (and vendors) for met software development and maintenance.Encourage technical innovation and avoid “stifling” SAFETY software SQA requirements \Be consistent with ANSI/ANS-3.11 and DOE/EH-0173T.Base SAFETY-RELATED SQA on the ten work activities required for SAFETY software, but use a more liberal approach.

Ten SQA Work Activities1. Software Project Management and Quality Planning

2. Software Risk Management

3. Software Configuration Management

4. Procurement & Vendor Management

5. Software Requirements Identification and Management

6. Software Design & Implementation

7. Software Safety Design

8. Verification & Validation

9. Problem Reporting & Corrective Action

10. Training of Personnel

In DOE’s SQA Requirements for Safety Software…

Requirements have been established for: 1. Custom developed2. Configurable3. Commercial off-the-shelf (COTS) software4. Utility calculations (e.g., spreadsheets)5. Commercial design and analysis tools.

For each type of software, three different levels (AA, BB, or CC) can be assigned based on how the software is being used. For each work activity specified in the SQA requirements, a full or graded approach may be required.

Work Activities CustomA B C

COTSA B C

1. Software Project Management and Quality Planning

FULLFULL FULLFULL GRADEGRADE

GRADEGRADE GRADEGRADE GRADEGRADE

2. Software Risk Management FULLFULL GRADEGRADE GRADEGRADE

FULLFULL GRADEGRADE GRADEGRADE

3. Software Configuration Mgmt FULLFULL FULLFULL GRADEGRADE

FULLFULL FULLFULL GRADEGRADE

4. Procurement & Vendor Mgmt FULLFULL FULLFULL FULLFULL

FULLFULL FULLFULL FULLFULL

5. Software Requirements Identification and Management

FULLFULL FULLFULL FULLFULL

FULLFULL FULLFULL FULLFULL

6. Software Design & Implementation FULLFULL FULLFULL FULLFULL

NANA NANA NANA

7. Software Safety Design FULLFULL FULLFULL GRADEGRADE

FULLFULL FULLFULL GRADEGRADE

8. Verification & Validation FULLFULL GRADEGRADE GRADEGRADE

FULLFULL GRADEGRADE GRADEGRADE

9. Problem Reporting & Corrective Action

FULLFULL FULLFULL FULLFULL

FULLFULL FULLFULL GRADEGRADE

10. Training of Personnel FULLFULL FULLFULL GRADEGRADE

FULLFULL FULLFULL GRADEGRADE

Safety Software SQA Level of Effort

Relative Importance of Work Activities for Safety Software -- Class A

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10

Work Activities

% E

ffo

rt

Relative Importance of Work Activities for Safety Software -- Class C

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10

Work Activities

% E

ffo

rt

Safety-Related Software SQA Level of Effort

Relative Importance of Work Activities for Safety-Related Software

0102030405060708090

100

1 2 3 4 5 6 7 8 9 10

Work Activities

% E

ffo

rt

Applying Work Activities

Develop SQA Project Management and Quality Planning documentation.

Document needed SQA activities Establish SQA milestonesAssign SQA responsibilities.

Develop a Configuration Management Plan to:Ensure configuration control during software development Ensure secure storage of the source code, executables, software documentation, V&V test procedures & results, and SQA documentation.

Prepare Design and Implementation Documentation to clearly detail how the software works. This should include:

sufficient information to support continuity in software development if there were an abrupt change in project personneladdress and provide users with sufficient documentation to efficiently use the software and understand what it is doing.

Design, conduct, and document V&V Testing to: focus on the portions of the code that were modifiedinclude baseline testing include independent testing by someone not on the software development team.

Develop and implement Problem Reporting/Tracking requirements Train the software development team Prepare User’s documentation/instructions

Applying Work Activities (cont)

Applicability of Safety-Related Software SQA Guidance

SQA must be applied during the development of new software and the modification of existing software. For legacy software that is not undergoing modification, there is no requirement to apply this SQA guidance retroactively. If resources are available, priorities are to ensure that code documentation and verification testing data sets are produced, testing conducted, and testing documentation placed into configuration-managed storage.

In applying the guidance to COTS software, SCAPA is setting up expectations for SQA that should be met easily by the existing SQA programs of commercial software developers. The proposed SQA guidance should be compatible with most existing SQA programs. In some cases, an existing SQA program may already incorporate all of the key aspects of the proposed guidance. In other instances, there may be components in the proposed guidance that are not currently addressed.

Applicability of Safety-Related Software SQA (cont)

Draft SQA Guidance for Safety-Related Software

…

Conclusion

DMCC will draft and propose SAFETY-RELATED SQA guidance for meteorological software.A graded approach using DOE G 414.1-4 ten work activities will be used for developing SQA guidance. Legacy software SQA will focus on code documentation and V & V activities. Upon acceptance, DMCC will distribute and promote.

Finalize draft DMCC SQA Plan for met software for by end of September and circulate for peer review

Issue SQA Guidelines and post on DMCC website by December 2008

Path Forward

Questions?Questions?

Cliff GlantzChair of DOE Subcommittee on Consequence Assessment and Protective Actions (SCAPA)

Pacific Northwest National LaboratoryPO Box 999

Richland, WA 99352509-375-2166

cliff.glantz@pnl.gov