Instrument Control System Project Management · PMCS-0021, Revision A Page 1 of 14 1. INTRODUCTION...

Project Documentation Document PMCS-0021

Revision A

Advanced Technology Solar Telescope 950 N. Cherry Avenue Tucson, AZ 85719 Phone 520-318-8108 [email protected] http://dkist.nso.edu Fax 520-318-8500

Instrument Control System Project Management

Bret Goodrich Software

September 2015


PMCS-0021, Revision A Page ii

REVISION SUMMARY: 1. Date: Aug 5, 2011

Revision: DRAFT Changes: Created

2. Date: Sep 1, 2015 Revision: Rev A Changes: Updated for Baseline.


PMCS-0021, Revision A Page iii

Table of Contents

1. INTRODUCTION ................................................................................................... 1 2. CONSTRUCTION ................................................................................................. 2 2.1 FABRICATION ......................................................................................................... 2 2.2 PURCHASE ............................................................................................................. 2 2.3 INTEGRATION, TESTING, AND COMMISSIONING .......................................................... 2 3. QUALITY CONTROL ............................................................................................ 3 3.1 VERIFICATION ......................................................................................................... 3 3.1.1 Requirements Verification .............................................................................................. 3 3.1.2 Design Verification .......................................................................................................... 3 3.1.3 As-Built Verification ........................................................................................................ 4 3.1.4 Documentation Verification ........................................................................................... 4 3.2 QUALITY ASSURANCE TASKS .................................................................................. 4 3.2.1 Unit Testing ..................................................................................................................... 4 3.2.2 Integration Testing .......................................................................................................... 5 3.2.3 User Acceptance (Verification) Testing ........................................................................ 5 3.3 VERIFICATION TEST PLAN ....................................................................................... 6 3.3.1 Unit Tests ......................................................................................................................... 6 3.3.2 Integration Tests ............................................................................................................. 7 4. RISK ...................................................................................................................... 8 4.1 RISK REGISTER ...................................................................................................... 8 4.2 RISK MITIGATION .................................................................................................... 8 5. COST AND SCHEDULE ESTIMATES ................................................................ 10 5.1 2009 COST ESTIMATE ........................................................................................... 10 5.1.1 Construction Labor ....................................................................................................... 10 5.1.2 Construction Non-labor ................................................................................................ 10 5.1.3 Contingency .................................................................................................................. 11 5.2 SCHEDULE ........................................................................................................... 12


PMCS-0021, Revision A Page 1 of 14

1. INTRODUCTION

This document discusses the project management of the DKIST Instrument Control System (ICS). It is concerned with construction and development plans, quality control and assurance, risks, costs, and schedule. Each of these areas is covered in sections of this document.



2. CONSTRUCTION

Construction phase planning involves the fabrication, purchase, and integration of the ICS. Planning for each of these areas is described here.

2.1 FABRICATION

Fabrication of the ICS will take place at NSO/Tucson. Fabrication involves the development of the ICS software and documentation. All ICS equipment resides in the Tucson NSO Computer Room in a computer rack with independent power and network management.

The ICS software development effort will develop three separate packages for DKIST. The ICS software controls the execution of observations on the instruments by issuing commands and data to the instrument in a synchronized and coordinated fashion. The Standard Instrument Framework (SIF) provides a framework for the instrument developers to build their applications. The Base package provides a set of standard components for software and hardware.

All fabrication activities continue through August 2016. At that time the ICS begins IT&C and maintenance support.

2.2 PURCHASE

The ICS software development requires the purchase of two development computers and a set of hardware components. The computers are commodity rack mount with available slots for hardware boards. The hardware purchased includes:

PTP grand master clocks; PTP time base cards; Power management controller; Power supply controller; Delta Tau motion control Network switches

2.3 INTEGRATION, TESTING, AND COMMISSIONING

ICS integration begins with the development of the instrument systems in their respective locations. The SIF and Base packages are delivered to the instrument developers during 2014 and are maintained through scheduled releases and patches for the duration of construction. SIF development continues through August 2016.

ICS integration and testing at the summit begins with the first instrument delivery in 2018. The ICS package is installed with the OCS in a computer room server.

The Test Architecture Framework (TAF) is also installed on the summit, allowing the regular regression testing of PROC-0017 to be implemented here. At this point, the OCS, ICS, DHS, and TCS should be installed at the site. Shortly after the DHS is installed, the Visible Broadband Interferometer (the first light DKIST instrument) should be installed. At this time the DKIST should be ready for the first instrument verification tests, per the VBI IT&C plan.



3. QUALITY CONTROL

Quality assurance and quality control are essential elements in the DKIST construction plan. Many aspects of the QA/QC plan are based upon past history of telescope development, along with modern systems engineering process control. The following definitions apply to the DKIST QA/QC.

Quality control, also known as verification, is a process used to evaluate whether or not a product, service, or system complies with regulations, specs, or conditions imposed at the start of development phase. Another way to think of this process is “Are you building it right?”

Quality assurance, also known as validation, is a process used to establish evidence that provides a high level of confidence that the product, service, or system accomplishes its intended requirements. Another way to think of this process is “Are you building the right thing?”

Change Control Board (CCB) is the name of the DKIST group that meets regularly to approve or reject requested changes in the DKIST specifications and interfaces.

Specification and Interface Control Documents define the specifications for each Work Breakdown Structure element and the public interfaces between each element. Both sets of documents are held under change control and changes must be approved by the CCB.

Design Document is the living document that defines the baseline design of the WBS element. This document is updated through the conceptual, preliminary, and final design phases to reflect the current design approach. During construction this document is updated to reflect the changes in scope or interfaces approved by the CCB. At the completion of construction the document should be a completed theory of operation manual for IT&C and operations.

3.1 VERIFICATION

The following tasks will be performed as part of the QA process. The work package manager and the DKIST QC/QA personnel will perform auditing of these tasks.

3.1.1 Requirements Verification

All new and changed requirements of the system will be verified to ensure the following:

They are directly related to an approved item from the CCB; They are consistent, feasible, and testable; They have been appropriately allocated to the correct mechanical, hardware, software, and

operational processes; and Those that are related to safety, security, and criticality have been verified by the rigorous

processes governing those areas.

The requirements verification process will be conducted by formal review. It will require participation and sign-off by the following team members:

Work package primary investigator; Work package manager; and Work package engineer responsible for the change.

3.1.2 Design Verification

All new and changed design elements will be verified to ensure the following:

Design is traceable to requirements;



Design provides details describing how requirements will be met; and Design implements safety, security, and other critical requirements correctly as shown by

suitably rigorous methods.

The design verification process will be conducted by formal review and requires participation of the following team members:

Work package manager; Work package engineer responsible for the change; and At least one engineering representative from a different DKIST work package.

3.1.3 As-Built Verification

All new and changed components will be verified once construction is complete to ensure the following:

Applicable standards are being followed; Applicable best practices standards are being followed; DKIST software coding and commenting standards are met; and DKIST software best practices are met per SPEC-0005.

The as-built verification process will be conducted by an informal review process such as email. The review process must be performed and signed off by at least one engineer from a different DKIST work package.

3.1.4 Documentation Verification

In conjunction with new or changed component being released the following documentation must be provided/updated:

Detailed design documentation, specifications, ICDs, etc.; Performance benchmarks (for performance critical modules); Test documentation (unit, component, integration, and user acceptance); and Operations document.

The documentation verification process will be conducted by informal review, such as email. The review process must be performed and signed off by the following team members:

DKIST QA/QC representative; and DKIST release manager.

3.2 QUALITY ASSURANCE TASKS

The following tasks shall be performed as part of the quality assurance process. The results of these tasks will be documented and reviewed as part of the “Document Verification” task of the QC process.

3.2.1 Unit Testing

Unit testing involves the testing of individual units of work to ensure they are fit for use. A new or changed component shall be unit tested. This testing shall be performed before proceeding to component level testing. The tasks that must be completed as part of unit testing are as follows:

Prepare new and/or changed unit tests and related documentation; Ensure traceability of new and/or changed tests to requirements;



Execute new, changed, and existing unit tests upon build of component; and Document unit test results.

Unit testing will be performed by the work package engineer responsible for the change or his/her designee.

3.2.2 Integration Testing

Integration testing involves testing an intended release to ensure it integrates correctly with all other DKIST systems for which it interfaces. Integration testing shall be performed in a qualified DKIST test environment that uses mechanical, hardware, and software systems equivalent to the production systems. Integration testing shall be performed before proceeding to user acceptance level testing. The tasks that must be completed as part of integration testing are as follows:

Prepare new and/or changed integration tests and related documentation; Ensure traceability of new and/or changed tests to requirements; Coordinate integration test schedule with test engineer of interfacing systems; Execute new, changed, and existing integration tests; and Document integration test results.

The work package engineer responsible for the change and the designated test engineer for each interfacing DKIST system will perform integration testing. Results shall be reviewed and signed off by the following team members before proceeding to user acceptance testing:

Work package manager; Work package manager(s) for all systems that interface with the released component; Work package engineer responsible for release; and Test engineer for all systems that interface with the released component.

3.2.2.1 Software Specific Tasks

The following tasks are specific to integration testing for software releases.

Any software defects (bugs) identified in testing will be logged in JIRA tracking system; All test cases impacted by the defect must be re-tested once the defect is resolved; Any un-resolved software defects must be approved by the review team before proceeding to

User Acceptance Testing; and Upon successful completion of integration testing the software source code will be tagged in

CVS to indicate it is part of a release. Test documentation will include reference to this release number.

3.2.3 User Acceptance (Verification) Testing

User acceptance testing involves performing tests for which the user will validate the output of the system to determine pass/fail status. User acceptance testing shall be performed in a production environment, or a qualified test environment that is approved by the user. User acceptance testing shall be performed before a release can be made operational for production use. The tasks that must be completed as part of user acceptance testing are as follows:

User to prepare new and/or changed integration tests and related documentation; Ensure traceability of new and/or changed tests to requirements; Coordinate user acceptance test schedule with production or test environments and systems; Execute new, changed, and existing user acceptance tests; and



Document test results.

User acceptance testing will be performed by the user, with the support of the work package engineer responsible for the release, and test engineers from other interfacing systems. Before proceeding to production, the release must be approved by the following team members:

Work package user (i.e., owner or primary investigator); Work package manager(s) for all systems that interface with the released component; Work package engineer responsible for release; Test engineer for all systems that interface with the released component; and DKIST release manager.

3.2.3.1 Software Specific Tasks

The following tasks are specific to user acceptance testing for software releases.

Only software source code from the CVS tag that matches the release number identified in the integration test documentation may be used for user acceptance testing;

Any software defects identified during testing will be logged in JIRA tracking system; All test cases impacted by the defect must be re-tested once the defect is resolved; Any un-resolved software defects must be approved by the review team before proceeding to

production release; and Test documentation should include reference to the CVS tag for this release.

3.3 VERIFICATION TEST PLAN

3.3.1 Unit Tests

Unit tests will be written with the Test Automation Framework (TAF), as described in PROC-0017. All unit tests will test against the ICS compliance matrix (CMX-0006). The major OCS system tests are described below.

3.3.1.1 ICS Observation Execution

Test procedure: A number of observations are created in the OCS and issued to the ICS. Simulation instruments are used to assure that the observations are executed at the correct times and with the required start and stop synchronizations.

Test results: The tests should show that the required execution times are achieved and that the observations complete correctly. The ICS should manage the observation queue are required.

3.3.1.2 ICS Observation Cancel and Abort

Test procedure: Ongoing and queued observations sent from the OCS are cancelled and aborted.

Test results: The ICS should correctly dequeue, cancel, or abort the observations and leave the system ready to execute the next set of observations.

3.3.1.3 SIF tests

Test procedure: All SIF tests defined by the ICS DRD are implemented as TAF tests. The SIF tests retrieve instrument programs from the database, execute their scripts, and return the results of the execution. The tests use a simple instrument simulator for the execution of the script.

Test results: All tests should return the correct results code.



3.3.1.4 SIF Instrument tests

Test procedure: The SIF is tested as part of the each instrument’s test procedures. The

Test results: Instrument test that fail are analyzed for the effects of the SIF. Tests where the SIF fail will be rewritten to run as part of the simulator testing.

3.3.1.5 Base

Test procedure: The Base package is tested as part of the TAF tests run for the Common Services Framework. Base requirements from TN-0088 are implemented as individual test procedures.

3.3.2 Integration Tests

Integration tests are performed both with the Tucson end-to-end test bed and at the summit.

3.3.2.1 Camera Software Systems

Test procedure: The camera is configured to operate at various rates and sizes that conform to the requirements. The ICS configures the camera according to the instrument program input.

Test results: Camera output should meet requirements. Multiple cameras and data streams should not impact the performance of each other.

3.3.2.2 Observatory Control System

Test procedure: This test must be performed with the cooperation of the OCS and the DKIST End-to-End test bed. The OCS is used to construct a simple experiment using an instruments display plug-in 'view'. A simple SIF simulator is then installed in the DKIST End-to-End test bed and the experiment is executed. The SIF simulator displays the configuration(s) is it given during execution.

Test result: The displayed configurations should contain the expected attributes and their values.

3.3.2.3 Visible Broadband Imager

Test procedure: This test should support the VBI integration tests as defined in SPEC-0107, VBI Critical Design Document, section 10.3.

Test results: The VBI requirements for instrument control should be met.



4. RISK

The DKIST Risk Management Plan defines the project approach to risk analysis and mitigation. The project maintains a risk register containing all identified risks at the beginning of the project and throughout the life of the project. Plans for mitigating each high-level risk and the subsequent results of the actions taken are also given.

The risk register exists to accomplish the following risk management goals:

• provide a tool for managing and reducing the risks identified before and during the project; • document risk mitigation strategies being pursued in response to the identified risks and their

grading in terms of likelihood and seriousness; • provide project management with a document from which status can be reported to oversight; • ensure the communication of risk management issues to key stakeholders; • provide a mechanism for seeking and acting on feedback to encourage the involvement of the key

stakeholders; and • identify the mitigation actions required for implementation of the risk management plan and

associated costs.

4.1 RISK REGISTER

Risk Item

Pro

bab

ilit

y

Non

-lab

or

Cos

t

Lab

or C

ost

HLSC-073 Scope refinements, specification documents in transition 80% $216K $173K

4.2 RISK MITIGATION

General risk mitigation strategies are applied to all risks in the ICS. These strategies include:

Early Development: The ICS will finish construction in 2016. The IT&C period will be used to address issues arising from later instrument development or unplanned integration needs.

End-to-end test bed: The end-to-end test bed will be used to test the interface to the CSS and OCS. As subassembly element test beds are added (e.g., mount, enclosure, VBI) the end-to-end test bed will be able to test full operational support. In addition, DKIST scientists can begin testing experiments and observations using the test bed.

The following risk mitigation strategies are envisioned for each of the major risks defined in the risk register.

4.2.1 HLSC-073: Scope refinement; specification documents in transition.

There may be additional work or delays in development due to formal specification changes.

The ICS Design Requirements Document (DRD, SPEC-0163) has not been finalized.



Mitigation: Complete the specification review and update the specification document. Update the DRD. Work with operations and management to determine the best ICS solutions for any new requirements using the existing budget and/or contingency.

Mitigation: Enforce current requirements. Initial operations at DKIST will need to be within the ICS specification and OCD use cases. The DKIST Change Control Board will need to deny scope increases.



5. COST AND SCHEDULE ESTIMATES

The DKIST budget for the ICS is $771,567. This escalated cost number was developed by a bottom-up cost analysis performed in August 2011, and assumed hardware technology available at that time. At that time, the definition of the DKIST instrument suite was not complete; it was uncertain how many cameras would be in operation and what their final data rates would be. Because of this and other technology factors, project contingency of 18% was set aside.

5.1 COST ESTIMATE

The August 2011 DKIST budget has created separate work packages for labor and materials of each ICS development area. New quotes were obtained for all materials required by the ICS design.

Work Package AmountS-WANS3-170 ICS – Management and Support (ARRA) $138,772S-WANS3-171 ICS – Final Design Staffing and Training (ARRA) $13,211S-WANS3-172 ICS – Base Components (ARRA) $142,692S-WANS3-173 ICS – Instrument Sequencer (ARRA) $63,771S-WANS3-174 ICS – ICS Observation Management (ARRA) $93,121S-WANS3-175 ICS – Simple Instrument (ARRA) $122,950S- WANS3-190 ICS – Instrument Control System Development (ARRA) $73,215S-WMNS3-190 ICS – Instrument Control System Development $123,835 Total $771,567

5.1.1 Construction Labor

Labor costs for the ICS were recalculated in August 2011 for the individual schedule items. Beginning in FY2012, the ICS provides 15% of its labor for instrument support; this includes assisting instrument developers with the SIF, instrument program development, and base hardware usage.

Resource Work

FY2011

FY2012

FY2013

FY014

FY2015

FY2016

SW.1 InstrumentControl 100% 85% 85% 85% 85% 85% InstrumentSupport 15% 15% 15% 15% 15%

Construction labor for the DHS ceases in August 2016, although instrument support activities continue through the end of construction.

5.1.2 Integration, Testing, and Commissioning Labor

After construction, the ICS labor remains in the project to perform IT&C activities. The IT&C planning for these activities is currently under development. The activities include:

Integration of the telescope subsystems; Construction and integration of the summit computer room; Integration and testing of the OCS and ICS; Integration and testing of the VBI; Integration and testing of the remaining instruments.



IT&C labor begins September 2016 and completes in July 2019.

5.1.3 Construction Non-labor

Hardware costs for the ICS were recalculated in July 2011 for the ICS design. All computers are commodity items that are expected to be purchase shortly before integration. Therefore the configuration and prices are expected to change over the course of the next three to four years. In some areas, such as network switches and disk drives, new technology may supplant our baseline hardware. In all cases except the real-time processing nodes, the current technology meets the ICS requirements.

Total cost for all materials is $142,692. This is comprised of both the computers and the test hardware.

5.1.4 Contingency

DKIST contingency is held by the project management and is not a part of any specific work package. However, to determine the project contingency, the ICS non-labor activities were assessed of possible risk factors and assigned a contingency percentage based upon the criteria of the DKIST Contingency Management Plan. The calculated ICS contingency was 18% of non-labor activities.



6. SCHEDULE

The ICS schedule started construction in FY2011 with the development of the instrument specifications, designs, and interfaces. During FY2012, the ICS developed the Base package software for the majority of standard software and hardware controllers. During FY2013, the ICS began interactions with the instrument developers to determine the specific implementation details of the instrument program execution and to help develop the Facility Instrument Definition, a document that defines the requirements for a facility-class instrument. The ICS began development of the Standard Instrument Framework in 2013 based upon the results of the instrument definition. This advance work also assisted the instrument schedule by providing critical software in a timely fashion.

In 2014, the ICS began development of the ICS package, including the instrument program database, sequencing, and SIF interface. This work, along with the improvements to the SIF and Base packages continues through the end of ICS fabrication in August 2016.

Integration of the ICS with the OCS and instruments is a part of the IT&C process and not covered by the ICS fabrication.



7. CONFIGURATION ITEM DATA LIST

The Configuration Item Data List (CIDL) describes the status of all major deliverable software and documentation items. At present, the CIDL for the DHS contains the following:

Source Code Packages

ICS Final Release 1.0 (zsh) Standard Instrument Framework Release (Canary 10) CSF Base Release (Canary 10)

Documentation

SPEC-0023, ICS Specification Document SPEC-0163, ICS Design Requirement Document TN-0102, ICS Critical Design Document CMX-0006, ICS Compliance Matrix MAN-0002, ICS Operations Manual PROC-0017, HLS Software Test Plan SPEC-0113, Facility Instrument Interface Specification TN-0088, DKIST Base Application Support Environment ICD-31.4/4.2, ICS to OCS Interface ICS Factory Acceptance Test Plan

Instrument Control System Project Management · PMCS-0021, Revision A Page 1 of 14 1. INTRODUCTION...

Documents

Transcript of Instrument Control System Project Management · PMCS-0021, Revision A Page 1 of 14 1. INTRODUCTION...