1 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations...

1 of XXFebruary 8-10, 2010

Section 09 – Aquarius Command and Control SystemMission Operations Review

Aquarius Instrument Operations

Aquarius/SAC-DMission Operations Review

February 8 – 10, 2010



Agenda

Peer Review Results Key RequirementsProcesses/Operations OverviewAquarius Command and Control System ArchitectureTesting StatusProcedures/Status/PlansRisksRisk MitigationDocumentation/Status/Plans for completion



PEER REVIEW RESULTSAquarius Instrument Operations



Recommended Actions

Clearly define the final set of instrument operations procedures Identify procedures to be written Identify what procedures must be verified on the Observatory before

flight

The project has been addressing these since the Peer Review.Concern remains that, due to project demands, personnel may not

be available to rehearse critical procedures



KEY REQUIREMENTSAquarius Instrument Operations



Key Requirements

Mechanics of the Ground System DesignUse of the Oceans Biology System

ACCS relies on its facilities, networks, security, and software supportUse of CONAE for Mission Operations

The ACCS Scheduler Client and Telemetry Viewer are CONAE deliverablesThe AQ Test-bed also interfaces with SInter, the CONAE commanding tool.ACCS processes and procedures must align with CONAE’s

Instrument Operations - Maximizing Quality and Quantity of Science Data CollectionScience data loss budget of 8%Science data accuracy to 0.2 PSU

Perform Aquarius command planning within flight rule constraintsMonitor Aquarius science and telemetry to determine state of healthPerform Aquarius anomaly handling



Key Requirements Flow-down Summary

All real-time tasks and all SAC-D tasks are the responsibility of the SAC-D Flight Operations Team

As captured in the documents:Aquarius Ground System Level-3 Requirements (AQ-326-0152)L4 AQ Command and Control System & Test Requirements (AQ-426-0197)



PROCESSESAquarius Operations



Aquarius Instrument Operations Processes

The Aquarius Command and Control System (ACCS) supports:

Aquarius command planningStandard: daily science data downloads; monthly Cold Sky CalibrationsSpecial: science requests such as Look Up Table changesEmergency: anomaly response

Aquarius offline state-of-health analysisReceipt of Aquarius/SAC-D science and housekeeping data from the ADPS. Receipt of Aquarius/SAC-D pass prediction files from the ADPS.Processing of Aquarius/SAC-D housekeeping and Aquarius science data for the purpose of monitoring instrument trends and state-of-health.Publishing of telemetry trends and alarms to Web-based tools to enable off-site health checks.

Aquarius anomaly handlingAquarius instrument anomaly tracking and resolution.



Command Planning Overview

Two standard command planning activities: Perform science data downloads at least once every 12 hours

Plan 2 weeks of downloads every weekCONAE has the flexibility to send a subset of these every few daysCONAE performs C&DH management

Perform a Cold Sky CalibrationNo Aquarius commands; purely a request for the S/P state of “Cold

Sky Calibration”

Science may request other command planning activities such as: Thermal set-point change Adjust the radiometer receive window



Command Planning Steps Daily: Check for science requests

Include any science requests in the upcoming command plan Create a stand-alone command plan if urgent

Weekly: Generate the command plan Two weeks of downloads

Command plan upload to the S/P is base-lined as being two weeks in the future Command plan execution by the S/P is base-lined as being 2 weeks + 3 days in the future

Any science requests Enter command plan into the Scheduler Client

A command plan is an “Action Request” to the Scheduler Client. An “Action Request” is composed of “Basic Action Components”, “Massive Upload Components” and

“Extended Condition Components”. Example Basic Action Component: start downlink/stop downlink command pair Example Massive Upload: Aquarius Flight Software Patch Example Extended Condition Component: X-band transmitter on/off command pair

Scheduler Client is a CONAE deliverable tool Interfaces with the CONAE Spacecraft Operations Planning tool (SOP)

CONAE SOP provides progress reports on Action Request planning via the Scheduler Client: Action Request Received by the SOP Action Request Accepted by the SOP

ACCS verifies reports, then verifies execution via telemetry/downlink file reception



Standard Command Planning Process at a Glance

W T F S S M T W T F S S M T W T F S S M T W T

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Aquarius Commands Planned at the ACCS

Aquarius Action Requests Submitted to the SOP

ACCS Receives Action Report: Action Request Received

ACCS Receives Action Report: Action Request Accepted



Command Planning - Emergency

The emergency command planning process is defined as follows: Utilize emergency channels to notify the CONAE MOC of an Aquarius anomaly:

PhoneEmail

And also enter a ticket into the:Aquarius Reporting and Tracking Tool

Generate the (contingency) Aquarius command plan as per standard procedure, but with accelerated implementation:

CONAE will schedule NEN supportCONAE will process received Aquarius command plans as quickly as possible



Daily Monitoring & Trending Process

Alarm and Error Notification Tool: Verify expected download files have been received

Download file times match commanded download times File sizes match expected file sizes No Cyclic Redundancy Check errors have occurred

Note: Data integrity (ICDS CRC and contiguous blocks) checked automatically Verify No Alarms present

If Alarms present, perform Anomaly Resolution process Web-based graphs

Examine science and telemetry trends on various scales: Daily Monthly, Seasonal Yearly, Mission

Telemetry Viewer Verify Alarms in agreement with Analysis Tool Verify instrument configuration is as expected Review data for software messages



Anomaly Response

The SAC-D and Aquarius teams generated a common anomaly response flow.Aquarius Operations has designed an anomaly process aligned with that flow.



AS Anomaly Process



I. Discovery a. CONAE FOT or AQ OPS detects telemetry Out of Limits

Team has procedures and authority to SAFE Aquariusb. AQ OPS detects abnormal science or telemetry trendc. Science detects abnormal science performance

II. Notificationa. If detected by CONAE or Science, party notifies AQ OPS via email/phoneb. AQ OPS notifies AQ GS Manager, CONAE Manager, AQ PI by submitting a TRAC anomaly

ticketIII. Gather Investigation Team

a. AQ OPS asks AQ GS Manager for additional supportb. AQ OPS updates TRAC with criticality, preliminary investigation report; updates TRAC status

to “team formation”. (Publishes TRAC ticket and report to AQ/Science/CONAE management)c. AQ GS Manager asks the AQ PI to authorize funds for JPL supportd. AQ PI asks JPL science liaison to recruit resolution teame. JPL science liaison recruits team, notifies AQ PI, AQ GS Mgr, AQ OPS of team membersf. AQ OPS updates TRAC with team member list; updates TRAC status to “in work”. (Publishes

TRAC ticket and report to entire team).

Aquarius Specific Anomaly Resolution Process



IV. Develop Solutiona. Technical team uses TRAC to confer, exchange data, develop solution and verification

criteria, exercise with Test-bed.b. When ready, technical team buy-in confirmed by setting TRAC status to “solution

proposed”.V. Obtain Approvals

a. Upon team acceptance, technical team leader updates TRAC status to “solution accepted, technical”.

b. CONAE exercises procedure against Observatory EM; forwards accept/reject to CONAE Mgmt

c. If “reject”, return to step V.d. Management team (AQ GS/AQ PI/CONAE) reviews solution; upon acceptance, AQ PI

updates TRAC status to “solution accepted, management”.VI. Execute and Verify

a. AQ OPS plans commands according to procedure b. CONAE FOT executes c. Full team verifies per step V criteria.

VII. ReportAQ OPS updates TRAC with Final Report and Resolution status: completed/reissued. (If

reissued, go to step V). Aquarius operations documents and procedures updated accordingly.

Aquarius Specific Anomaly Resolution Process



ARCHITECTUREAquarius Command and Control System



Aquarius Ground System

The Aquarius Ground System is contained within the Oceans Biology Data System The Aquarius Ground System is made up of 2 segments:

The Aquarius Data Processing System (ADPS) The Aquarius Command and Control System (ACCS)

The Aquarius Data Processing System receives Aquarius science data, SAC-D auxiliary data, and other science ancillary data to generate Aquarius science products.

The Aquarius Command and Control System receives Aquarius science data, SAC-D and Aquarius House Keeping Telemetry, and pass prediction files in order to monitor Aquarius health and status, plan commands, and respond to anomalies.



Oceans Biology Data System Support to the Aquarius Command and Control System

Functions/services provided:FacilitiesInfrastructureStaff IT network and securityInherent redundancy (except CPT; hot backup)Use of automationConfiguration ManagementSoftware maintenanceData storageUse of Oceans Biology Data System interface with CONAE



Aquarius Command and Control tools and interfaces

ACCS Architecture



TESTINGPlans and Status of



ACCS and Aquarius Instrument Operations Test List

ACCS Ground System TestingSoftware Tools

ComponentInternal Interfaces Functional External Interfaces

Command and Telemetry TestingCommand Database Telemetry DatabaseCross-Tool Verification

Mission Scenarios



Testing Status

COMPLETED TESTSSoftware Tools

Component – baseline completed January 11, 2010Internal Interfaces – baseline completed January 14, 2010Functional – baseline completed January 14, 2010

Command and Telemetry TestingCommand Database – completed June 15, 2009Telemetry Database – completed in instrument I&T at JPL

UPCOMING TESTSCross-Tool Verification – test design meeting scheduled February 15, 2010External Interfaces – will participate in CONAE led tests

Mission Scenarios – under development by the SAC-D and Aquarius Mission System Engineers.



Analysis Tool Alarm and Error Notification Tool Command Planning Tool Web-based graphs Reporting and Tracking Tool

JPL performed the Test-Bed functional test

Component Testing included the:

ACCS Components Tested



Interface testing included the:L0 file deliveryLevel 1A reportsAlarm NotificationsHKT file deliveryText files to Web graphsWeb graph publicationDownload time file delivery

ACCS Interfaces Tested



ACCS Functional Test DescriptionACCS Functional Test – encompasses component and internal interface tests

100% of Level 4 Requirements verifiedData Monitoring Test:

Correlated science and HKT data files from I&T processed through the full ACCS system Included a DPU memory download

Command Planning TestAll Aquarius commands and known sequences planned

Each singular commandDownlinksLUT programmingDPU memory downloadFlight software patchIntroduced illegal times, sequences, parameters

Anomaly Handling TestSimulated use of Recording and Tracking tool in anomaly resolution process

Included file exchanges, milestones, notificationsSimulated science and instrument team external access



Command and Telemetry Database Tests

Command and Telemetry Database TestingCommand Database

Worked with JPL to verify hex output of CONAE SInter tool matched output of I&T commanding toolExercised all legal commands and wide range of legal and illegal command parameters

Telemetry Database – completed in instrument I&T at JPLThe telemetry database is the telemetry dictionary as captured in Telemetry Viewer scripts.Scripts define how:

Telemetry bytes are extracted from telemetry framesHex values are converted to Engineering UnitsAlarms are applied

Telemetry Viewer pages display hex, EU values and alarmsUltimately compared to values from trusted sources:

Power metersCalibrated sensorsSoftware tools



AQ/SAC-D GS External Interface and Verification Test Lists

SDGS-GSD-I-0042-a Science data deliverySDGS-GSD-I-0044-a ST TM deliverySDGS-GSD-I-0024-a AR Reception and Confirmation Integration TestSDGS-GSD-I-0025-a Action Reports Request and Response Integration TestSDGS-GSD-I-0028-a Voice Communication Integration TestSDGS-GSD-I-0026-a Orbit Products delivery

SDGS-GSD-V-0002 X band data publicationSDGS-GSD-V-0004 MOC ancillary data production and deliverySDGS-GSD-V-0006 Pass plan generationSDGS-GSD-V-0008 Script validation and compilationSDGS-GSD-V-0012 Execution report generationSDGS-GSD-V-0011 Flight Software replacement

These external interface and verification tests are CONAE-led and will verify Aquarius Ground System Level 3 Requirements:



PROCEDURESStatus of



Key Aquarius Operations Procedures

The key procedures for Aquarius instrument operations are:

Aquarius Command PlanningAquarius Monitoring and TrendingAquarius Anomaly ResponseACCS Configuration Management

And the individual standard and contingency instrument procedures.



ACCS Procedures: Status and Plans for Completion

Key ACCS procedures are in preliminary form.Extracted from the ACCS Functional Test Procedure.Exception: Configuration Management Procedure

This procedure provides the steps for performing configuration managementIt also identifies what items are to be maintained under CM:

Processes, ProceduresTool software (including CONAE deliverables)Flight Software/patchesCommand and Telemetry scripts (databases)Tests and test reports

ACCS Procedures must be updated in lockstep with the Functional Test. This is expected with:

Expansion of Functional Test to encompass Joint TestsCorrections to Functional Test as Joint and Mission Tests progress



Aquarius Instrument Procedures, Standard and Contingency



Individual Aquarius procedures: Have been drafted by the instrument team as flowcharts (where

needed) with command sequences and telemetry responses identified. AND/OR Exist as ATLO scripts/procedures

Work to be done: The Aquarius team should come to consensus on which procedures

are operations-critical. The SAC-D Flight Operations Team and the Aquarius Instrument Ops

Team should rehearse/observe these critical procedures.

Instrument Operations Procedures Status



Aquarius flows and ATLO scripts will be transferred to operations procedures The transfer steps from ATLO scripts to operations procedures are:

Correlate specific ATLO/I&T scripts to each operations procedure Parse the scripts by decision point. ACCS procedures direct the use of the

CONAE Aquarius procedures. Thus: ACCS procedures include the decisions within a given ATLO script CONAE procedures include the steps between decisions

Write the ACCS procedures and the CONAE procedures (scripts) according to the parsed ATLO/I&T scripts

Rehearse/observe the ACCS and CONAE Aquarius procedures against the Observatory.

AND/OR Rehearse the ACCS and CONAE procedures in mission scenarios.

Generating Instrument Operations Procedures



ACCS vs CONAE Procedures

37

Two types of instrument procedures:ACCSCONAE Flight Operations

ACCS procedures are documents in step-by-step formatAll decision points occur within the ACCS procedures

CONAE procedures are scriptsCONAE Aquarius scripts are written in the Sinter scripting languageThe scripts consist of:

Aquarius commands established in known sequencesAssociated telemetry responsesAny other real-time activities required

Examples:Inserted delays between commandsSpecial data loggingInformation to note before proceeding



RISKS/CONCERNS



Risk Assessment

After an extensive risk/concern discussion during the Aquarius Instrument Peer Review and follow-up discussions with the Aquarius team, two risks/concerns remain.

The two risks/concerns are listed by priority from the instrument operations perspective.



Risk

RISK: Personnel availability may be too limited to allow for implementing the existing command sequences as procedures and rehearsing them.

IMPACT: The Operations Team would then be inadequately trained for Aquarius operations.

Mitigation: Project is planning a workshop in Bariloche in the April timeframe to rehearse critical procedures.



Concern

CONCERN: Awaiting Scheduler Client delivery.

IMPACT: Unable to develop adequate Aquarius command planning procedure in time for interface tests and scenarios. Interface testing of the command planning link could be delayed.

Mitigation: Scheduler Client User’s Guide and ongoing conversations with developers have provided baseline information for ACCS tool and procedure development.



DOCUMENTATIONDescription and Status of

42



Documentation/Status

Name Status Version Date

Aquarius Ground System Level-3 Requirements (AQ-326-0152)

Signed

ACCS Level-4 Requirements (AQ-426-0197)

Signed January 19, 2010

Aquarius Command and Control System Software Management Plan (AQ-316-0239)

Ready for Signature February 25, 2009

Aquarius Ground System Integration Test & Training Plan (AQ-336-0244)

Revision January 18, 2010

SAC-D Ground System to Aquarius Ground System ICD (AS-336-0151b)

Review July 14, 2008

Aquarius Ground System to Instrument Team Anomaly ICD (AQ-336-0522)

Ready for Signature December 7, 2009

Aquarius Science Team to Ground System ICD (AQ-336-0243)


Aquarius Flight Operations Plan (AQ-316-0157)


ACCS Operations Handbook Preliminary December 28, 2009

Aquarius Standard Procedures Preliminary January 4, 2010

Aquarius Anomaly Procedures Draft

ACCS User’s Guides Final January 6, 2010



ACCS – CONAE Rec/Dels - Status

Nº: TITLE ACCS Rec/Del

STATUS

1 CAS SCL Reference Manual Rec Received

2 CAS SCL User’s Guide Rec Received

3 Aquarius Telemetry Descriptions Del PendingDue: April 23, 2010

4 Aquarius Command Descriptions Del PendingDue: April 23, 2010

5 SAC-D HKTM Dictionary (TBC) Rec Pending

6 Aquarius Operating Procedures (TBC)

Del PendingDue: June 10, 2010

7 Scheduler Client User Manual (TBC)

Rec Received

8 TM Viewer User Manual (TBC) Rec Received

9 Aquarius Planning Restrictions (TBC)

Del PendingDue: April 23, 2010

10 Aquarius TM Pages and Decoding scripts (TBC)

Del Sent

11 Sinter Command Tool Rec Received

12 Telemetry Viewer Tool Rec Received

13 Scheduler Client Tool Rec Received



Instrument Team – ACCS Rec/Dels - Status

Nº: TITLE ACCS Rec/Del

STATUS

14 AQ Flight Software User’s Guide

Rec Received

15 AQ Instrument Operations Handbook

Rec Received

16 Aquarius Telemetry Dictionary

Rec Received

17 Aquarius Command Dictionary

Rec Received

18 Aquarius Test-Bed User’s Guide

Rec Received

19 Aquarius Test-Bed Maintenance Manual

Rec Received



Summary

The Aquarius instrument operations team and the ACCS tools are ready to move into the operations testing phase.

The testing, process development and procedure development is working in conjunction with SAC-D ground system and Aquarius instrument testing.

1 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations...

Documents

Transcript of 1 of XX February 8-10, 2010 Section 09 – Aquarius Command and Control System Mission Operations...