[American Institute of Aeronautics and Astronautics SpaceOps 2006 Conference - Rome, Italy ()]...

American Institute of Aeronautics and Astronautics

1

File transfer, Mass Memory and Mission Time Line – providing spacecraft remote commanding at Mars

Martin Shaw* VEGA, Darmstadt, Germany

Alan Moorhouse†, Michel Denis‡ European Space Agency, Darmstadt, Germany

and

Roberto Porta§, Zeina Mounzer** VEGA, Darmstadt, Germany

This paper presents the tools utilized on the Mars Express mission to realize remote commanding at Mars: The Solid State Mass memory recording device; the file transfer tele-command uplink methodology; the tele-command Mission Time Line; and on-board command files. The paper places such tools in context by identifying the need for commanding autonomy at Mars. We present specifics of the tools themselves, their respective interactions, and identify the operational constraints applicable to their usage. We also discuss anomaly resolution methods, and identify in-flight modifications to the commanding elements of the Mars Express mission since its launch. The paper concludes by presenting a summary of the success of this commanding methodology.

Nomenclature DMS = Data Management System ESOC = European Space Operations Centre FCT = Flight Control Team FDIR = Failure Detection Isolation and Recovery FT = File Transfer HKTM = House Keeping TeleMetry MEx = Mars Express MTL = Mission Time Line OBCF = On-Board Command File OBSM = On-Board Software Maintenance SSMM = Solid State Mass Memory TC = Tele-Command

I. Introduction He Mars Express (MEx) mission was launched on 2 June 2003, and entered Mars orbit on 25 December of the same year. Since that time, the MEx mission has been successful in returning a wealth of scientific data

concerning the geology/geomorphology, chemistry and atmospheric physics of Mars – including the Martian moons

*Mars Express Data Handling Engineer, ESA/ESOC Robert-Bosch Strasse 5, 64293 Darmstadt, Germany. †Mars Express deputy Spacecraft Operations Manager, ESA/ESOC Robert-Bosch Strasse 5, 64293 Darmstadt, Germany. ‡Mars Express Spacecraft Operations Manager, ESA/ESOC Robert-Bosch Strasse 5, 64293 Darmstadt, Germany. §Mars Express Data Handling Engineer, ESA/ESOC Robert-Bosch Strasse 5, 64293 Darmstadt, Germany. **Mars Express Data Handling Engineer, Aurora Space Exploration Programme Systems Engineer, ESA/ESOC Robert-Bosch Strasse 5, 64347 Darmstadt, Germany.

T

SpaceOps 2006 Conference AIAA 2006-5843

Copyright © 2006 by VEGA Group PLC. Published by the American Institute of Aeronautics and Astronautics, Inc., with permission.


2

- and of the diverse interactions between the planet and its surrounding inter-planetary medium1. During this period, it is estimated that - in the course of over 700 mission days since the end of payload commissioning - MEx has generated in excess of 800 Gbits of data. Operating (in non-real time) a complex spacecraft around Mars, supporting a complete suite of payload instruments which require demanding pointing strategies in often limiting power regimes, and which place a heavy load on methodologies for data storage and return, provide major challenges to spacecraft operations aiming to optimise such scientific data return at up to 400 million km and with such substantial data generation rates. That MEx has been able, so successfully, to satisfy these constraints, and provide such a rich scientific return, in part results from the successful implementation of considerable commanding autonomy given to the spacecraft by the Flight Control Team (FCT) at the European Space Operations Centre (ESOC). This has been possible by maximizing the combined use of a number of spacecraft functions: These functions, their usage and their respective interactions, are the subject of this paper.

II. The need for commanding autonomy in routine operations at Mars The need for a substantial degree of commanding autonomy is not unique to the MEx mission. However, taken

together, the operational constraints imposed are sufficiently stringent as to demand that a detailed operational strategy be defined, a strategy which is both easy to implement and support, but which is also robust and ensures the maximum operational safety. Such a methodology has naturally evolved during the course of the mission. The constraints which define it are listed below.

A. Two-way commanding light-time The communication time between the Earth and MEx, during routine science operations at Mars, varies between durations of 10-40 minutes. This obviates the possibility for real-time spacecraft commanding in all but the most exceptional of circumstances. One such exception is all platform on-board software maintenance (OBSM).

B. Multiple spacecraft pointings per orbit During a typical orbit of 6hrs 43min, MEx may perform any combination of (usually 1 or 2) nadir, inertial or across track slews in support of targeted scientific observations. Outside these periods, the spacecraft points back to Earth to enable the downlink of the acquired science data using the high gain antenna.

C. Parallel instrument operations During a typical peri-centre pass, as many as 6 payload instruments may need to be switched ON, and fully configured for data taking. All the necessary link establishment protocols to the Solid State Mass Memory (SSMM) must also be established.

D. X-band transmitter management The orbit chosen to optimize the scientific return from MEx requires that the X-band high gain antenna transmitter be periodically switched ON/OFF to take account of eclipses behind Mars (of up to 90 mins duration) and Martian occultations (of up to 60 mins duration). In addition, routine “spacecraft maintenance” windows of 90 mins per day are scheduled. These allow for the possibility to recover data lost on the downlink to Earth, and are characterized by a pre-programmed slew from Earth contact. MEx data are acquired from multiple daily Ground Station passes using ESA’s own Ground Station at New Norcia (Australia), and typically 1 to 2 passes using NASA’s Deep Space Network. The X-band transmitter is switched OFF outside such visibility periods.

E. Daily fine orbit maintenance Reaction wheel off-loadings, and/or orbit control manoeuvres, are required on a daily basis to maintain the accuracy and stability of the desired orbit.

F. Data storage/recovery Each day during contact periods with the concerned Ground Station, the downlink of both payload instrument science data, and platform/payload house-keeping telemetry (HKTM) data, is performed. Typically 1 to 5 Gbits of data are recovered each day.

G. Non-routine events (anomalies, and anomaly investigations) As on any mission, spacecraft anomaly investigation periodically requires the recovery of monitoring data, sometimes over periods of many months. Moreover, some anomalies cannot easily be rectified by software changes, and require a modified operational approach which may require periodic commanding. An example of the latter is a failing in the SSMM software


3

design. This prohibits the storage of data with a specific hex pattern within the on-board time, and requires a temporary suspension of the storage of data to the SSMM twice per year.

The tele-commands (TC’s) necessary to address points B-G above are loaded on the spacecraft one or more days

in advance of their scheduled execution time.

III. The remote commanding tools available on MEx A graphical overview of the commanding tools discussed in this paper is given in Fig. 1. Each element is further

described in this section.

Figure 1. MEx commanding tools concept. Graphical representation of the inter-relationships between the commanding tools available on MEx.

1) The SSMM The SSMM is a 12 Gbit (beginning of life) storage device, utilizing 64 Mbit SDRAM and comprising diverse memory types. It has a mass of 8.5 kg and power consumption of < 19 W. The concerned application software, controllers, internal/external interfaces and DC converters are all operated in cold redundancy (i.e. a backup chain is available, but the first level action to any failure detection, isolation and recovery (FDIR) triggering on an SSMM failure is to switch the unit OFF). Real-time error detection and correction from single event upsets is performed by a memory scrubbing background task. An 80C86 processor is responsible for the overall monitoring/control functions, access to the prime platform processors (via dedicated 1355 IEEE links), and in conducting data integrity checks. An 80C32 micro-controller manages access to the memory module (=data storage areas), performs the necessary file management functions and controls all data transfers. Each of the 3 memory modules has a capacity of 512 Mbytes, divided into 12 partitions. The memory is structured on 1 byte, with data physically stored in < 4112 byte sized packets (encoded using the ESA packet standard). The dedicated data storage area comprises 38 “packet store” partitions (in size ranging from 65 Kbytes to 540 Mbytes, and housing the science and HKTM data), a number of 8.2 Kbyte on-board command files (OBCF’s), and small (2048 or 4096 byte-sized) files used temporarily to store the contents of the MTL and of an active file transfer (FT). SSMM-dedicated HKTM is cyclically polled by the central processor (data management system - DMS) at 1/16 Hz. Event-based error detection is applied. Some anomalies detected in this way may merely result in the suspension of data storage (and/or termination of the link session) for the concerned unit, whilst more serious FDIR triggering originating from the SSMM results in a spacecraft software safe mode (resulting in the suspension of the mission time line (MTL), and therefore of the mission operations).

Ground TC stackdefinition

Uplink by FT

Insertion of TC’sinto SSMM MTLfiles

Insertion into MTL

Distribution touser at executiontime


4

Data on the SSMM can be stored in one of 3 file types. Plain files (in which data access by read and write pointers is not applied) are used to store the OBCF’s, and to act as intermediate repositories for FT’s. Non-cyclic packet stores (in which a read/write pointer strategy is applied) are used to store those data which are to be preserved (i.e. not overwritten) even when the end of the file is reached – e.g. platform and payload dump packets from memory. Cyclic packet stores (in which a read/write pointer strategy is also applied) are used to house the majority of data on the SSMM, including all the science and HKTM, and allow data overwrite once the end of the file is reached. Dedicated packet stores have been created to house science data for each payload instrument, with other packet stores containing more generic contents (e.g. event packets, TC acknowledgement packets, platform diagnostic packets, platform HKTM and payload HKTM). During routine operations, the more active instruments, and/or those generating the highest data volumes, fill their respective packet stores in typically 2 days of operation. Duplication of all but the science packet stores allows for the implementation of a daily flip-flop mechanism in the storage of data. The file sizes are optimized to ensure continual recording of both platform and payload TM over a 24 hour period given the telemetry data generation rate defined on-board for the concerned mission phase. The flip-flop mechanism is applied to ensure that all data for a 24 hour period can be recovered to Ground the following day. Rigorous configuration control is applied to the management and usage of the SSMM file store, and the routing of the data.

2) The Mission Time Line (MTL) The MTL is a scheduling mechanism for storing a time-ordered sequence of all the TC’s to be executed at the appropriate on-board time. The time resolution of all the TC’s contained in the MTL is 1 sec, although as many as 24 TC’s can execute per sec. The structure of the MTL is dynamically updated when the execution time of each TC has expired. In practice, there exists no constraint on the future execution time of any TC inserted into the MTL. The MTL can operate in two modes (selectable by Ground depending upon the mission phase or current spacecraft platform health status): The Nominal MTL allows for the storage of up to 3000 TC’s, and uses 3840 contiguous SSMM files for storage of up to 8 contained TC’s. The Short MTL does not use the SSMM – all data are stored in the DMS memory – and its size is limited to 119 TC’s. The Nominal MTL is used in all but the most exceptional situations. The MTL data are divided into several components: An index table (located in the DMS memory) is used to control the TC scheduling. For each TC is stored its execution time, its assigned sub-schedule identifier (which identifies the spacecraft sub-system or unit – e.g. payload instrument - to which the TC is associated), packet information and, in the case of the Nominal MTL, an SSMM index. A TC table (stored on the SSMM in the case of the Nominal MTL, and DMS memory when using the Short MTL) contains the actual TC’s to be executed. In Nominal state, the SSMM index pointer in the index table explicitly defines the location on the SSMM of each TC. A TC cache (Nominal state only) is located in DMS memory, and contains the next 300 TC’s fetched from the TC table and due for execution. A file status table (Nominal state only) identifies the number of TC’s stored within each intermediate SSMM file, and dynamically updates following TC deletion or execution. At filling time, up to 8 TC/sec can be inserted into the MTL if the time-tag of the first TC to be inserted is later than the last TC currently in the TC table, and if the inserted TC’s are time-ordered. An auto-regulation mechanism ensures that the rate of TC’s fetched from the TC table does not exceed the capacity of the intervening MTL buffer (which implies a rate of 1 TC/sec in the worst case). The Nominal MTL is cyclic, in the sense that the intermediate SSMM files populated by the oldest TC’s are overwritten once the final SSMM file is filled with TC’s. Control of the MTL is facilitated by commands to insert TC’s, and delete TC’s (either by specified execution time period, or assigned sub-schedule identifier). MTL monitoring is provided by daily requests for dumps of the entire MTL contents, and through the use of dedicated TM parameters and graphical displays. This is illustrated


5

in Fig. 2, in which we show the number of commands within the MTL at any time. In addition, we plot the cumulative number of loaded TC’s. The latter is observed to wrap-around following successful (cyclical) clearing of the intermediate files on the SSMM after the uplink of 30720 TC’s.

As the integrity of the MTL is critical to mission continuation, MTL processing is the subject of numerous FDIR checks. Timeouts are applied to the insertion of TC’s into the MTL, reading of TC’s from the index table, and the release of TC’s from the MTL at execution time. Additional FDIR exists to verify the integrity of contents of each SSMM intermediate MTL file. Should the need arise to temporarily disable the execution of TC’s for a particular destination user, this can easily be facilitated by the disabling of the appropriate sub-schedule identifier. All commands for the concerned user will be released from the TC cache at execution time, but will not be executed. Only when an explicit command is sent to re-enable that sub-schedule identifier are subsequent TC’s for that user executed.

3) The file transfer (FT) function The FT function is a method for ensuring complete delivery of a coherent list of TC’s to be executed on-board. It is encoded within the DMS software, allows for error detection and re-transmission, and is able to encode/deliver one TC per second. Although it is possible to queue multiple FT’s, only one FT will be managed by the DMS software at any time. FT’s are subdivided into parts, the number of which depend upon the number of TC’s to be sent. The first part (mandatory) contains a unique FT identifier, the number of sub-components to the FT, the final destination and name of the TC file on the SSMM in which the final TC’s will be stored. Intermediate parts (if required) contain

Figure 2. MTL Filling. Graphical representation of the filling of the Mission Time Line on Mars Express during a typical period of 50 mission days in 2006. The date is plotted along the X-axis, and either number of MTL commands (red line, range 0:3000 TC’s) or cumulative number of commands (yellow line, range 0:30000 TC’s) plotted along the Y-axis.


6

the embedded TC’s, including time-tags (if appropriate), and their corresponding identifier. The last part (mandatory) contains the final embedded TC’s (and time-tags if needed), together with the sequence number of the last TC, and the checksum calculated on the whole FT. The mechanism for processing of a FT follows. Whilst the FT is ongoing, the delivered TC’s are stored in 4096 intermediate (plain) files in a dedicated area of the SSMM. Following reception of successful acknowledgments for each part of the FT, a check is performed that all expected TC’s were received (as determined by their associated identifiers), and that the corresponding checksum is consistent. The contents of these intermediate files are then copied into the single SSMM destination file, the DMS returning an acknowledgement of the success of this copy. The intermediate SSMM files are then erased, awaiting the next FT. The SSMM destination file can be specified either as “immediate”- in which case it will be executed immediately upon completion of the copy – or “delayed” – in which case it will be executed only on request from the Ground. When the request for execution is made, each embedded TC is released in the order in which it is filed within the FT, and distributed to its destination. Time-tagged TC’s are loaded into the MTL at this time, whilst non time-tagged TC’s are executed immediately, at a rate of 1 per sec. On completion of this task, the destination file is deleted from the SSMM (allowing its re-use for another FT if required). Should any of the FT transfer TC’s not be received on-board, they are detected by their associated identifier, and flagged to the Ground. The file transfer enters a “running” state, awaiting re-transmission of the missing TC’s from the Ground. Only when all missing elements of the FT are successfully received on-board will the copy to the SSMM destination file proceed. The FT can be aborted at any stage, either by the Ground or by the internal DMS processing software. Cases of FT abort by the Ground are described in Section V below. FT abort by the DMS software occurs as a consequence of a number of failures (e.g. an SSMM failure, inconsistencies in the packet size, the sequence numbering, or number of segments received - compared to those expected – or in the associated FT checksum). Regardless of the origin of the FT abort request, all intermediate SSMM files are erased, the final copy to the SSMM destination file is cancelled, and the file transfer status returned to an “idle” state, allowing another FT (even using the same transfer ID) to be initiated from the Ground. If a FT is aborted, since the copy to the SSMM destination is not initiated, none of the embedded TC’s are delivered. This ensures that the status of the delivery process is explicit in all cases. To date, no FT’s have been aborted by the DMS software. The FT mechanism is applied in all cases when more than a single command is sent. Typically 2-3 FT’s are released from the Ground per day in routine operations (each containing 100-600 TC’s). During the mission lifetime, the FT process has proved to be robust to anomalies in the mission operations centre, the Ground Stations, or with the commanding link to the spacecraft, and in dissemination of the resulting commands to the DMS (Section V). In particular, its reliability has proven to be especially valuable for all OBSM conducted since launch, from the smallest patch to complete software reloads, and has never failed.

4) On-board command files (OBCF’s) OBCF’s are a mechanism for storing on the SSMM a dedicated set of related TC’s, which may require execution as a coherent whole more than once during the mission lifetime. The functions of such OBCF’s are distinguished on the basis of the following needs:

Routine and repetitive operations: To facilitate the daily swapping of packet stores used to archive the platform/payload HKTM, TC acknowledgment and event data in the SSMM. TM mode definitions: Five unique definitions of platform TM modes have been

defined, according to mission phase or emergency spacecraft recovery needs, and with a cumulative HKTM bit rate ranging from 32 bits/sec to 4.6 kbits/sec (that for 2.3 kbits/sec being used in routine operations).


7

Spacecraft emergency recovery: Re-definition of the TM real-time downlink routing to Ground following any software system re-initialization (e.g. during a spacecraft hardware safe mode).

Operationally, the dedicated set of TC’s are sent by FT to the SSMM, where they are stored as delayed destination files using a unique (and fully configuration controlled) naming convention. When needed, such files are copied to an interim file, and this interim file is then executed. Each one of the embedded TC’s (typically 100-200 per OBCF) are released to the destination user, at a rate of 1 per sec. The SSMM software then ensures deletion of this interim file. The delayed TC file remains for future re-use as and when required. Conceptually, OBCF’s fulfill a similar role to on-board command procedures used on other missions. However, on-board command procedures were never validated in on-ground testing on MEx, and have never been used in flight. By virtue of their simplicity, OBCF’s have the particular benefit of clarity of usage/operations, whilst minimizing the likelihood of human error during repetitive or mission critical commanding. More than 1000 OBCF’s have been executed on MEx since it entered routine operations, and no failure has yet been recorded.

IV. Operational constraints to remote commanding on MEx Operational constraints to commanding MEx during routine operations primarily result from the operability and

usability of the available tools presented in Section III, and in the visibility of the spacecraft. They are outlined explicitly below.

1) Constraints applicable to the commanding chain Ground station allocations dictate the frequency and duration of commanding access to the spacecraft. During

routine operations, a typical 24 hour period will include a single (uninterrupted) 4 hour visibility period, but may include a variable number of additional periods (each usually longer than 45 mins duration). Spacecraft commanding during this period must be executed in parallel to the acquisition of scientific data from the SSMM.

The TC commanding bitrate also places a constraint upon the commandability of the spacecraft. The default

value 2 kbits/sec results in a typical release time of 1 TC per sec. Finally, the end-to-end verification process in the commanding chain (dictated by the two-way light travel time

to the spacecraft) should be mentioned. This figure ranges from of order 10 to 40 minutes and, even for the shortest duration, defines the smallest usable Ground station commanding window to be of order 25 mins.

2) MTL operability The principal constraint of the MTL is the number of commands it can accommodate. Optimal re-filling of the

MTL, as commands are executed, must take account of the execution time of the commands already on-board, those to be sent, and on the Ground station visibility. This task is labour-intensive, and requires daily supervision by a dedicated member of the ESOC FCT2. The level of automation of this task is the subject of constant improvement by the use of various support tools, but it is likely that a “human” element will remain (to cope with the diversity of operations, or short term re-planning and minor anomalies, whilst maximizing the science return).

Although there exists the possibility to modify the execution time of a TC already loaded into the MTL, this

function has not been used in flight. This approach results from a desire to maintain the fidelity of the MTL contents at all times, regardless of whether the concerned TC’s are in the TC table or the TC cache. If the time of a TC must be changed, this TC is first deleted from the MTL, and reloaded with the new time. This operational approach is cumbersome, but is rarely needed and has always proved successful. Moreover, the “administrative” overhead resulting from the delete and re-load enforces (de facto) rigorous configuration control.

To avoid a potential processor overflow resulting from a conflict of equivalent priority tasks, successive non-

interruptable TC’s (such as file checksum requests) are not executed from the MTL. Instead, each command is executed only after the previous such command has been fully verified.

At any time, advanced tools within the mission control system maintain a running (or predicted) view of the

MTL status. Periodic reports of the entire MTL contents are also requested. The TC cache filling task has priority


8

over any such reporting. However, regular reports are performed to ensure consistency between the predicted MTL contents, and those known to be loaded on the spacecraft. To ensure a coherent view of the MTL contents at all times, reports are only requested when all commanding for the concerned Ground Station coverage is completed. This can be a significant constraint, as an MTL maintained close to capacity requires typically 50 mins to be reported.

Wherever possible, the recovery of (non-routine) diagnostic information from the SSMM, or OBSM, is not

performed from the MTL during periods of intense payload activity. This is to avoid task prioritization conflicts on the SSMM, but is rarely a severe operational constraint as few payload instruments are ON during the real-time commanding windows when such activities are normally performed.

3) FT operability and TC file execution In principal, only one FT, and one TC file execution, can be managed by the DMS software at any time.

Additional FT’s can be queued, and will be processed sequentially, although this approach is not adopted operationally to ease control and recovery should any failure in the FT process be encountered. Routine operational commanding of MEx is performed ensuring that one FT is not sent until its predecessor has been successfully verified, and by ensuring that only one TC file executes at any given time.

Any abort of a FT from Ground is immediately followed by the recovery of diagnostics to ensure the successful

deletion of the SSMM destination file, and the appropriate reset of the FT status flag in DMS memory. A number of background tasks are avoided in parallel to FT processing, or TC file execution, to minimize the

likelihood of SSMM task conflict. These include: Parallel data reads from the SSMM. Any OBSM activity, or SSMM checksum evaluations. Partial deletion of SSMM packet stores. Erasure of SSMM file contents. SSMM diagnostics monitoring.

4) The SSMM file store and “ghost” files The SSMM does not manage data access in the manner of file store on a conventional operating system. Instead,

the mapping of data to memory address locations is essentially static. This has a significant benefit in ensuring reliability of data recovery (from a configuration control perspective), but can sometimes be a limitation when modifications to the file storage structure have to be performed.

One consequence of this limitation is the case of “ghost” files. To minimize the likelihood of data fragmentation,

the SSMM software was designed in such a way as to release unused memory blocks only in the inverse order of the blocks creation. The deletion task for a data block references the concerned memory area using a sorted linked list, rather than dynamic memory allocation. Unfortunately, this means that the memory release task will be unable to free memory areas for re-use if another memory area (with a higher address) still exists on the same memory partition. As a consequence, on the rare occasions when file deletion/creation may be required (e.g. SSMM power cycling, or memory re-mapping as described in Section VI below) file deletion and re-creation is performed in a fully time-ordered sequence to avoid the creation of “ghost” files.

V. Anomalies and troubleshooting In the course of the routine science mission of MEx, operational workarounds have been devised to counter a

number of anomalies which do not originate from, or cannot be resolved by changes to, the on-board software. These have been fully documented in the concerned Flight Operations Procedures, and are outlined in detail below.

1) Ground failures impacting the commanding chain A number of failures can arise in the Ground Segment, which adversely impact the commanding chain.

Commanding loss on the space-Ground link can arise because of weak signal levels at low elevations, or due to bad atmospheric conditions at the Ground Station. Station anomalies (e.g. receiver or drive failures, inappropriate station configurations) can also adversely impact the commanding chain, as can a failure of the link between the Ground Station and ESOC. Troubleshooting, in collaboration with the Ground Segment support teams at ESOC, is routinely


9

undertaken in such cases, but such failures are not considered further in this paper as they have not resulted in the need for operational workarounds. Operationally, sufficient margin (typically 2-3 days) is exercised during the daily scheduling of command uplink and verification as to mitigate against the possible loss of one or more Ground Station visibility periods.

Two anomalies, based on real situations experienced on MEx, are considered here, although in both cases the

recovery procedures are similar: Namely TC failure or commanding dropout during FT uplink, and FT failures. Commanding dropout: The failure to radiate the contents of a FT to the spacecraft may result from a temporary dropout of the TC link (typically for a period of less than 1 hour), or from a major outage. Different recovery processes are initiated in these cases since, following a minor outage, there usually exists sufficient time during the concerned Ground Station visibility to resume the FT uplink without adverse impact, even to the point of receiving complete verification of the commanding process in real-time. Following a major outage, cases have been encountered where there existed insufficient time to resume the FT uplink before loss of commanding. In such cases, the ongoing FT was forcibly aborted, and re-uplinked in its entirety at a subsequent Ground Station visibility period.

FT failures: Situations have arisen in which the integrity of the delivered FT has been questioned following the reception of reports of corrupt or incomplete FT parts, with an inconsistent FT checksum result, or with a FT remaining in a “running” state long after the final embedded TC has been released from the Ground Station. In all cases, the SSMM destination TC file does not execute, and the TC’s are therefore not distributed to their destination user or loaded onto the MTL. The operational resolution of these anomalies was to abort the entire FT and re-uplink it at the next available opportunity. In addition, a confirmation was made that the SSMM destination file had been removed (if necessary this was done manually), and checks performed on the contents of the SSMM FT intermediate files (which were emptied, if required, by a single dedicated TC). In all cases, the final status of the MTL was confirmed by a full report.

2) SSMM anomalies impacting the commanding chain The observed cases of a destination file remaining (i.e. un-erased) in the SSMM after FT uplink, and of non-

empty FT intermediate files, have already been discussed above. An additional anomaly has been observed which involves the SSMM, namely corruption of the file allocation table.

The file allocation table is a static file in which are recorded the name, type, physical properties and memory

address locations of all other files on the SSMM. A situation was encountered in which a 3 byte offset lead to a corrupted file name, leading to failures in the reading of SSMM packet stores, and in the verification of a subsequent FT. Preliminary analysis involved the recovery of detailed diagnostics of the file allocation table contents, and dumps of the corresponding areas of SSMM memory. The problem was finally resolved by a complete reload of a new version of the concerned application software.

3) Recovery from human error A particular benefit of remote commanding, and the implementation of advanced planning in the commanding

cycle on MEx, is that it significantly mitigates against the adverse impact of (inevitable) human error. An attempt to insert TC’s into the MTL at the time of removal of the TC link to the Ground Station results in the

FT remaining in a “running” state, with the unsent FT parts awaiting release. Since the final part of the FT is never received on-board, the FT cannot be processed. It can therefore safely be aborted and re-sent at the next availability opportunity.

An attempt to insert more than 3000 TC’s into the Nominal MTL results in the assignment of a “no load” status

for those TC’s which could not be accommodated. In principal, these TC’s can merely be re-sent when sufficient space next becomes available in the MTL. However, this is a potentially dangerous situation, as the integrity of the delivered TC stack (which has been ensured from first generation, to checking within the FCT and uplink to the spacecraft) is no longer respected. As a consequence, numerous checks are implemented to ensure this situation is avoided.


10

Attempts to use an already assigned FT identifier, or SSMM destination file, which has not be purged from the FT ground model are flagged by the Mission Control System before the FT can be sent to the Ground Station. As a result, corresponding SSMM events which would flag such an anomaly have never been observed.

None of the above errors have had an adverse impact on the commanding of MEx during routine operations.

VI. In-flight modifications of the commanding elements on MEx since launch Inevitably, the changing needs of a demanding science mission, or variations in the available power, necessitate

an evolution of the mission profile. The commanding chain on MEx is no exception: It must be flexible to changing circumstances, and amenable to modifications which maintain optimized usability of key spacecraft commanding elements.

1) SSMM packet store modifications The increasing demands of the payload instruments during the mission lifetime have required a modification of

the assigned HKTM packet store dimensions in two cases, and significant increases (of 1 Mbyte, 10 Mbytes and 32 Mbytes) for 3 of the science packet stores. Moreover, the file type of a platform HKTM packet store was changed from non-cyclic to cyclic to avoid the repetition of a TM storage failure when this (rarely used) packet store becomes full.

Whilst any SSMM re-mapping is operationally straightforward, it requires considerable advanced planning, and

can only be achieved at the expense of mission outage for some – if not all – of the payload instruments. Firstly, an optimized re-definition of the SSMM packet store structure is sought. The concerned packet store is then deleted, together with as many other packet stores in the same associated area of memory as are necessary to obviate the creation of “ghost files”. Finally, all the deleted packet stores are re-created, with the appropriate characteristics, and in the appropriate order reflecting the optimized re-definition. The applied changes are also incorporated in the sequence of TC’s used to rebuild the SSMM following any subsequent power cycling of the unit.

2) SSMM software upgrades A number of SSMM in-flight anomalies could not be solved by operational workarounds, and necessitated

reloads of the application software on two separate occasions. The first reload, applied in April 2004, was restricted to the software of the file/packet controller unit. It sought to address the following anomalies:

1) A hardware logic error in the unit which performs the data integrity checks. This error manifested itself in a de-synchronization of subsequent TM during science data dumps from the SSMM.

2) A software error in the pointer management within the read queue management task. This error became evident when attempting to read a cyclic packet store at the time of pointer wrap-around, and resulted in reads which would not terminate, and/or which recovered only partial data.

The second reload, applied in May 2005, modified both the file/packet controller and memory management units

(essentially the entire SSMM application software). It sought to address the following anomalies: 1) A hardware design failure of the chip used in the 1355 link protocol management between the SSMM and

the DMS processor or certain payload instruments. This was manifest as a failure to exchange the necessary tokens to ensure establishment of the link between the SSMM and the concerned instrument, with a frequency of failure of typically 1% for all data exchanges. It resulted in the recording of no data for that observation.

2) Repetitive SSMM event generation in the case of write failure within a specified timeout period. 3) Read pointer management at the point of file wrap-around when this wrap-around is terminated

unexpectedly. The new software version also sought to improve the checking, and anomaly reporting, of the structure of packets written by payload instruments, and to improve the interrupt handling for file copies and checksum executions. Independently of the OBSM tasks required, both software reloads necessitated the definition of a detailed operational timeline for implementation. The activities in 2005, for example, required the use of commanding both directly, and by FT, over a period of 10 hours and required the use of 3 Ground Station visibility periods. In all cases, the activation of the new software was facilitated by a power cycling of the entire SSMM.


11

To date, the applied software modifications have been almost completely successful in correcting the anomalies to which they were addressed. The only remaining uncertainty concerns the hardware limitation of the 1355 chip, although even here the frequency of observance is now markedly lower than was the case prior to May 2005.

3) Partial SSMM activation to reduce power consumption As part of a suite of tests aiming to reduce the power consumption on MEx in advance of the (power limited) Aphelion/Solar Conjunction phase in September 2006, a period of SSMM operations was implemented using only 2 of the available 12 word groups. This task necessitated the establishment (and testing on a dedicated test bench) of a new procedure, the creation of dedicated TM mode, and a re-definition of the SSMM packet store structure. The entire task (including the subsequent return to nominal operations) was successfully undertaken in May 2006, and required a total period of over 13 hours commanding using 3 Ground Station visibility periods. The anticipated 30% power saving from such an activity was confirmed.

VII. Summary and conclusions In this paper we have sought to present those principal operational tools used on MEx which facilitate remote

commanding at Mars, and identified the interactions between them. We have shown how the commanding functions on MEx are closely coupled to optimize system performance. We have identified the need for – and benefits of - such remote commanding, and defined the operational constraints applicable to the mission.

Specifically, we have discussed how spacecraft operations since launch, spacecraft anomalies and

troubleshooting, or limitations in the available power have resulted in changes to the flight software realizing these functions or to a modification of such operational constraints and practice. Particular anomalies observed in the course of routine science operations have been identified. Operational workarounds, where possible and desirable, have been defined. Those anomalies which could not be circumvented by such a pragmatic approach have been addressed by in-flight modifications to the commanding chain, including software upgrades.

Implementation of this methodology has resulted in the evolution of a stable and robust commanding interface to

MEx, an interface which has been used routinely to uplink of order 900 TC’s per day over more than 700 mission days since MEx entered the routine science operations phase, and resulted in the successful return of more than 800 Gbits of date to Earth.

This success is manifest in Fig. 2: The applied commanding strategy on MEx is able to ensure an MTL filling

ratio continually in excess of 80%. In conclusion, the success of this “remote commanding” approach has been amply illustrated by the degree of

reliability offered by the spacecraft, with the resulting enormous scientific return that Mars Express continues to provide to the many diverse scientific institutions around the world.

Acknowledgments We acknowledge the support of our colleagues at ESA/ESOC, and in addition the excellent support and advice

of the MEx teams at EADS (Toulouse and Friedrichshafen).

References Proceedings

1Proceedings of the Lunar and Planetary Society meeting XXXVII, 2006. 2Z. Mounzer, M. Denis, A. Morehouse, and M. Shaw, “Flying Mars Express – A day in the Cockpit”. oral presentation, this

conference.

[American Institute of Aeronautics and Astronautics SpaceOps 2006 Conference - Rome, Italy ()]...

Documents

Transcript of [American Institute of Aeronautics and Astronautics SpaceOps 2006 Conference - Rome, Italy ()]...