Automating electronic data sheetsCCSDS Spring 2013

Sam Price

2 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Navigator Background• Missions

– GPM– MMS – High Altitude 12Re– Hubble servicing mission

• Reflected GPS signals

• Embedded orbit propagator– Geons– 3rd Party software– 3rd party commands/telemetry

• 45 commands• 43 telemetry messages

GPS orbits, 3Re

Main lobe

First side lobe

42.6 degrees (L1)

GEO, 6Re

LEO (below 3000 km) MMS, 12Re

3 / 23NASA GSFC – CCSDS Spring 2013

Document Count

Document Message Size

Descriptions Member Layout

Meta dataunits,polys,limits

TLM RatePer Mission

Responsible Person5+ people

1,2MMS,GPM

ICDMSWord

(1) Device

3,4GPM,MMS

Telemetry mockup (Ada)

Maybe??? (2) Mockup

5,6MMS,GPM

Ground system (asist)

(3) Ground System owner

7 C++ Headers YesDoxygen

YesDoxygen

(1) Device

8 3rd Party C++Orbit Prop

Yes3rd party

Yes3rd party

(4) 3rd Party C/C++

9 3rd party ICD Orbit Prop

Yes3rd party

Yes3rd party

Yes3rd party

Yes3rd party

(4) 3rd Party ICD

10,11MMS,GPM

Bandwidth Rates (excel)

(5) Bandwidth person

12 C++ Command Verification

(1) Device - Algorithms

13 CSV Logging (1) Device - Logging

14 Analysis (Matlab)

(1) Device - Analysis

15 Endianess converters for CSV Logging

(1) Device - Logging

Navigator Electronic Documents for Command / Telemetry

4 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Multiple documents are expensive $$$$

• 15 different electronic versions, – manually created

• Same data across all of the documents• 5+ different people maintaining separate

documents. • Changes propagate slowly throughout documents

($$$)• Errors occur from manual changes.

($$$$)• Naming conventions are not the same

5 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

• C/C++– Endianess conversion routines– Command validation– Header files

• Human readable Interface Control Document– Latex– HTML– Maybe MS Word …

• Ground system definitions• Bandwidth spreadsheet• Ada mockup of telemetry for testing.• Logging tools

– CSV Logging– Matlab import– Database logging

C++ headersTelemetry/Commands

3rd party C Headers

Some tools

XML Database

Why C/C++ as starting point– Widespread existence

• All of my Telemetry is in C• More people know C

– Machine readable– Exactly what is implemented– 3rd party libraries are in C– Many more reasons

A streamlined approach

6 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Convert C/C++ to XML

Workflow

C++ headersTelemetry/Commands

3rd party C Headers

XML

XML Database

xml_to_asist.py

xml_to_ctypes.py

Asist

Asist RDLS

Python

Python CTypes

xml_to_latex.py

Latex ICD

7 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

What does it take to mimic C/C++ in XML• Basic Types

– bools– integers -- Use stdint.h

• int8_t,int16_t,int32_t, int64_t

– unsigned Integers• uint8_t,uint16_t,uint32_t, uint64_t

– floating point memory• float, double, long double

– strings• char arrays

– Arrays• Multidimensional

– void pointer, pointers (16,32,64) sizes

– Structures/classes– unions– enums

• No child / parents / inheritance

• No templates• C/C++ can have

alignment/padding problems• Metadata stored in doxygen

style comments– Units, Polynomials, warning

ranges, etc• Bit fields defined in

comments• No C bitfield yet…

8 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Use a compiler for parsing• Clang/llvm

– C/C++ Compiler similar to GCC– Cross platform compatible(Windows,Linux,Mac)

• LibClang– Pass arguments to change C/C++ structure padding.

• -m32 , -m64 – Cross platform compatible (Linux, Mac, Windows)– Apple uses it in their developer tools for code completion,

syntax highlighting, etc…– Use it to find comments, structures, types, padding in

structure, etc.• http://clang.llvm.org/docs/Tooling.html• http://clang.llvm.org/get_started.html• https://github.com/trolldbois/clang <- padding/size info

9 / 23NASA GSFC – CCSDS Spring 2013

C Example CCSDS Header/*! * @brief CCSDS Telemetry Header * @TelemetryPacketHeader * */typedef struct CCSDS_Telemetry_Header{ /*! * @brief Destination address */ char p_sp_dest; /*! * @brief Protocol ID * @default 0xf2 */ char p_sp_prot; /*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @enum ENUM_PACKET_TYPE * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Secondary Header within this Space Packet. * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is not present. * @enum CCSDS_Secondary_Header * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135.0-b-1 application ID's 2040- 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */ uint16_t p_ident;/*! . . . . */ uint16_t p_seq_cont;/*! . . . . */ uint16_t p_data_len;/*! . . . . */ uint64_t s_time;} CCSDS_Telemetry_Header;

Doxygen comments start with @marker_name marker value

Indented comments belong to parent comment

Could have used C bitfields(See backup slides)

10 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

C/C++ -> clang_to_json.py -> json_to_xml.py -> XML

• Current status– 500 lines of code to go from C Header to XML– Need to add enums, meta data, bit fields.– Implemented structures, unions, bit structures, arrays,

padding bytes, integers, unsigned integers, floating point numbers, pointers, comment parsing.

• https://github.com/nasa/SOIS-CCSDS-XML– NonNasa Contact for access, currently private project

until XML format is well defined.– NASA: fill out a NAMS/IDMAX request

11 / 23NASA GSFC – CCSDS Spring 2013

C Example p_ident

/*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Secondary Header within this Space Packet. * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is not present. * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135.0-b-1 application ID's 2040- 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */ uint16_t p_ident;

12 / 23NASA GSFC – CCSDS Spring 2013

Clang to JSON Output for p_ident

…..{'array': [], 'basetype': 'uint16_t', 'basetype_name': None, 'bitarray': True,'data': <clang.cindex.Cursor object at 0x10a7969e0>, 'is_pointer': False, 'kind': CursorKind.FIELD_DECL, 'name': 'p_ident', 'offset': 16, 'size': 2L,

'comments': [{'bitarray': 'msbf', 'children': [{'bits': '3', 'children': ['name': 'packet_version_number'}, {'brief': 'Pacet Version Number’}, {'default': '0'} ]

}, ], …. More Member Comments …..]

'members': [{'array': [], 'basetype_name': None, 'bitarray': False, 'bits': 3, 'comments': [{'name': 'packet_version_number'}, {'brief': 'Pacet Version Number’}, {'default': '0'}], 'data': None, 'is_pointer': False, 'kind': None, 'members': [], 'name': 'packet_version_number', 'size': -1},… More Members..],},

> python clang_to_json.py ../examples/src_cpp/ccsds.h -m32 –I../examples/src_cpp

• Offset, Sizes include padding information.

• Clang object used for further processing if needed.

• Bit arrays become children for bitstructs.

13 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

XML Layout

• Type database– Basic types

• Integers, floats

– Structures– Unions– Bit structures

• Telemetry database– Similar to structures

• Command database– Similar to structures

<TypeDatabase>

<typeInt/>

<typeFloat/>

<typeBitStruct>

<Members>

<variable/>

</Members>

<typeBitStruct\>

<typeStruct>

<Members>

<variable/>

</Members>

<typeStruct/>

</TypeDatabase>

<Telemetry>

</Telemetry>

<Commands>

</Commands>

14 / 23NASA GSFC – CCSDS Spring 2013

C Example/*! * @brief CCSDS Telemetry Header * @TelemetryPacketHeader * */

typedef struct CCSDS_Telemetry_Header{ /*! * @brief Destination address */ char p_sp_dest; /*! * @brief Protocol ID * @default 0xf2 */ char p_sp_prot; /*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @enum ENUM_PACKET_TYPE * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Secondary Header within this Space Packet. * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is not present. * @enum CCSDS_Secondary_Header * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135.0-b-1 application ID's 2040- 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */

uint16_t p_ident;/*! . . . . */ uint16_t p_seq_cont;/*! . . . . */ uint16_t p_data_len;/*! . . . . */ uint64_t s_time;

} CCSDS_Telemetry_Header;

15 / 23NASA GSFC – CCSDS Spring 2013

Generated XML<TypeDatabase>

<typeInt bits="3" endianess="bigEndian" name="int3_be" signed="true"/>

…. More small types for int1, int11, int8, int16 .. etc ….

<typeBitStruct name="CCSDS_Telemetry_Header_p_ident" type="uint16_be">

<Members>

<variable name="packet_version_number" type="int3_be"/>

<variable name="packet_type" type="int1_be"/>

<variable name="secondary_header_present" type="int1_be"/>

<variable name="apid" type="int11_be"/>

</Members>

</typeBitStruct>

…. More typeBitStruct for p_seq_cont, s_time ….

<typeStruct name="CCSDS_Telemetry_Header">

<Members>

<variable bits="8" name="p_sp_dest" offset="0" type="int8_be"/>

<variable bits="8" name="p_sp_prot" offset="8" type="int8_be"/>

<variable bits="16" name="p_ident" offset="16" type="CCSDS_Telemetry_Header_p_ident"/>

<variable bits="16" name="p_seq_cont" offset="32" type="CCSDS_Telemetry_Header_p_seq_cont"/>

<variable bits="16" name="p_data_len" offset="48" type="uint16_be"/>

<variable bits="64" name="s_time" offset="64" type="CCSDS_Telemetry_Header_s_time"/>

</Members>

</typeStruct>

</TypeDatabase>

> python json_to_xml.py ../examples/src_cpp/messages.h -m32 –I../examples/src_cpp

16 / 23NASA GSFC – CCSDS Spring 2013

Writing XML with pythonfrom xml.etree.ElementTree import Element from xml.etree.ElementTree import ElementTree

parent_node = Element('EDS')

Messages = Element('Telemetry')parent_node.append(Messages)

#All attributes/text must textMessages.attrib['Count'] = str(5)Messages.text = “Hello”

document = ElementTree(parent_node)

#See json_to_xml for prettify functionprint prettify(document.getroot())

17 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

XML Output

<?xml version="1.0" ?><EDS> <Telemetry Count="5"> Hello </Telemetry></EDS>Or Root = document.getroot()print xml.etree.ElementTree.tostring(Root,'utf-8')'<EDS><Telemetry Count="5">Hello</Telemetry></EDS>'

18 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

json_to_xml.py

• XML to Python CTypes• Decode telemetry/send commands

C++ headersTelemetry/Commands

3rd party C Headers

XML

xml_to_ctypes.py

Python

Python CTypes

clang_to_json.py

19 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Reading XML with pythonimport xml.etree.ElementTree as ETdoc = ET.parse(filename)root = doc.getroot()for node in root.findall('TypeDatabase/typeStruct'): print node.attrib['name’] print node.text

See xml_to_ctypes.py for complete listing200 lines of code to convert from XML to python ctypesOrhttp://docs.python.org/2/library/xml.etree.elementtree.html

20 / 23NASA GSFC – CCSDS Spring 2013

Python ctypes example

import ctypes

class CCSDS_Telemetry_Header_p_ident ( ctypes.LittleEndianStructure ): """""" #_packed_ _fields_ = [ ("apid" , ctypes.c_uint16 , 11 ), ("secondary_header_present” , ctypes.c_uint16 , 1 ), ("packet_type" , ctypes.c_uint16 , 1 ), ("packet_version_number" , ctypes.c_uint16 , 3)]

class CCSDS_Telemetry_Header ( ctypes.LittleEndianStructure ): ""”This is a comment about the CCSDS Packet Header""" _pack_ = 1 _fields_ = [ ("p_sp_dest" , ctypes.c_int8 ), ("p_sp_prot" , ctypes.c_int8 ), ("p_ident" , CCSDS_Telemetry_Header_p_ident ), ("p_seq_cont" , CCSDS_Telemetry_Header_p_seq_cont ), ("p_data_len” , ctypes.c_int16 ), ("s_time" , CCSDS_Telemetry_Header_s_time )]

python xml_to_ctypes.py test.xml > test_ctypes.py 205 lines of code

21 / 23NASA GSFC – CCSDS Spring 2013

Using python ctypes to decode / encode dataimport test_ctypes as tcfid = open('log.raw’)

#Buffer big enough to hold any command / telemetry data

big_buffer = bytearray(1024*10) write_buffer = memoryview(big_buffer)

#header data directly into the buffer. 8bytes.

fid.readinto(write_buffer[0:8])#big_buffer[0:8]#bytearray(b'\x00\xf2E\n\x00\xc0\x17\x00\')

header = tc.CCSDS_Telemetry_Header.from_buffer(big_buffer)#Read in the rest of the CCSDS packet.

fid.readinto(write_buffer[8:(header.p_data_len+1)])print "Apid is %d decode it." % header.p_ident.apid

#changing ctypes changes the binary data. Good for commanding

header.p_sp_dest = 5#big_buffer[0:8]#bytearray(b'\x05\xf2E\n\x00\xc0\x17\x00')

fid.close()

22 / 23NASA GSFC – CCSDS Spring 2013

ctypes cont…class CCSDS_Telemetry_Header ( ctypes.LittleEndianStructure ): "””Documentation about CCSDS Telemetry Header""" _pack_ = 1 _fields_ = [ ("p_sp_dest" , ctypes.c_int8 ), ("p_sp_prot" , ctypes.c_int8 ), ("p_ident" , CCSDS_Telemetry_Header_p_ident ), ("p_seq_cont" , CCSDS_Telemetry_Header_p_seq_cont ), ("_p_data_len" , ctypes.c_int16 ), ("s_time" , CCSDS_Telemetry_Header_s_time )] @property def p_data_len(self): “””More Documentation””” return self._p_data_len + 1 @p_data_len.setter def p_data_len(self, new_len): self._p_data_len = new_len – 1

h = CCSDS_Telemetry_Headerh.p_data_len = 8print h._p_data_len7print h.p_data_len8

CCSDS data length is 1 minus actual length of the message.

Provide documentation via docstring on telemetry value.

Telemetry: Polynomials, conversions, etc

Commanding:limit values, throw errors if not within ranges.

23 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Lessoned learned• Provide all of the standard C/C++ types

– As listed on “mimic C/C++ in XML” slide• Standardized naming of Integers, floats, bools, pointers.

(int32,int16,int8, …) – Allows for creating dictionary mappings between XML and

desired language• Mimic the structure of existing languages.

– Makes mapping from XML to existing language more strait forward

• Keep it simple • Develop tools with XML standard

– Kept having to change scheme while making tools to make processing easier

24 / 23NASA GSFC – CCSDS Spring 2013

Backup slides

25 / 23NASA GSFC – CCSDS Spring 2013

C bit field example(Not supporting, yet …)

struct CCSDS_Telemetry_header_p_ident {

uint16_t apid:11;uint16_t secondary_header_present:1;uint16_t packet_type:1;uint16_t packet_version_number:1;

};

http://msdn.microsoft.com/en-us/library/ewwyfdbe(v=vs.71).aspx

Least significant field first

26 / 23NASA GSFC – CCSDS Spring 2013

C Example cont/*! * @ingroup MESSAGES * @brief An example message for Power point. * @apid 135 */typedef struct Basic_M{

CCSDS_Telemetry_Header packet_header; //CCSDS Packet Header/*! * @brief Temp of the baseplate the cpu is attached to. * @polynomial [1566.7676,-18.85822,-.17372, 0.00229, 0.00001] * @unit fahrenheit

* @mode nominal * @red_limit [-Inf,-20), (120,Inf] * @yellow_limit [-20,-15), (115,120] */int32_t cpu_temp;

/*! * @brief Receiver Position * @coordinate_frame ECEF * @unit meter */ double position[3];};

Doxygen comments start with @marker_name marker value

Indented comments belong to parent comment

27 / 23NASA GSFC – CCSDS Spring 2013

Example basic example

<Telemetry>

<Message name="Basic_M">

<Members>

<variable bits="128" name="packet_header" offset="0" type="CCSDS_Telemetry_Header"/>

<variable bits="32" name="cpu_temp" offset="128" type="int32_be"/>

<variable bits="192" name="position" offset="192" type="float64_be">

<array>

<size>

3

</size>

</array>

</variable>

</Members>

</Message>

</Telemetry>