Automating electronic data sheets CCSDS Spring 2013 Sam Price.

27
Automating electronic data sheets CCSDS Spring 2013 Sam Price

Transcript of Automating electronic data sheets CCSDS Spring 2013 Sam Price.

Page 1: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

Automating electronic data sheetsCCSDS Spring 2013

Sam Price

Page 2: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

2 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Navigator Background• Missions

– GPM– MMS – High Altitude 12Re– Hubble servicing mission

• Reflected GPS signals

• Embedded orbit propagator– Geons– 3rd Party software– 3rd party commands/telemetry

• 45 commands• 43 telemetry messages

GPS orbits, 3Re

Main lobe

First side lobe

42.6 degrees (L1)

GEO, 6Re

LEO (below 3000 km) MMS, 12Re

Page 3: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

3 / 23NASA GSFC – CCSDS Spring 2013

Document Count

Document Message Size

Descriptions Member Layout

Meta dataunits,polys,limits

TLM RatePer Mission

Responsible Person5+ people

1,2MMS,GPM

ICDMSWord

(1) Device

3,4GPM,MMS

Telemetry mockup (Ada)

Maybe??? (2) Mockup

5,6MMS,GPM

Ground system (asist)

(3) Ground System owner

7 C++ Headers YesDoxygen

YesDoxygen

(1) Device

8 3rd Party C++Orbit Prop

Yes3rd party

Yes3rd party

(4) 3rd Party C/C++

9 3rd party ICD Orbit Prop

Yes3rd party

Yes3rd party

Yes3rd party

Yes3rd party

(4) 3rd Party ICD

10,11MMS,GPM

Bandwidth Rates (excel)

(5) Bandwidth person

12 C++ Command Verification

(1) Device - Algorithms

13 CSV Logging (1) Device - Logging

14 Analysis (Matlab)

(1) Device - Analysis

15 Endianess converters for CSV Logging

(1) Device - Logging

Navigator Electronic Documents for Command / Telemetry

Page 4: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

4 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Multiple documents are expensive $$$$

• 15 different electronic versions, – manually created

• Same data across all of the documents• 5+ different people maintaining separate

documents. • Changes propagate slowly throughout documents

($$$)• Errors occur from manual changes.

($$$$)• Naming conventions are not the same

Page 5: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

5 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

• C/C++– Endianess conversion routines– Command validation– Header files

• Human readable Interface Control Document– Latex– HTML– Maybe MS Word …

• Ground system definitions• Bandwidth spreadsheet• Ada mockup of telemetry for testing.• Logging tools

– CSV Logging– Matlab import– Database logging

C++ headersTelemetry/Commands

3rd party C Headers

Some tools

XML Database

Why C/C++ as starting point– Widespread existence

• All of my Telemetry is in C• More people know C

– Machine readable– Exactly what is implemented– 3rd party libraries are in C– Many more reasons

A streamlined approach

Page 6: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

6 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Convert C/C++ to XML

Workflow

C++ headersTelemetry/Commands

3rd party C Headers

XML

XML Database

xml_to_asist.py

xml_to_ctypes.py

Asist

Asist RDLS

Python

Python CTypes

xml_to_latex.py

Latex ICD

Page 7: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

7 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

What does it take to mimic C/C++ in XML• Basic Types

– bools– integers -- Use stdint.h

• int8_t,int16_t,int32_t, int64_t

– unsigned Integers• uint8_t,uint16_t,uint32_t, uint64_t

– floating point memory• float, double, long double

– strings• char arrays

– Arrays• Multidimensional

– void pointer, pointers (16,32,64) sizes

– Structures/classes– unions– enums

• No child / parents / inheritance

• No templates• C/C++ can have

alignment/padding problems• Metadata stored in doxygen

style comments– Units, Polynomials, warning

ranges, etc• Bit fields defined in

comments• No C bitfield yet…

Page 8: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

8 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Use a compiler for parsing• Clang/llvm

– C/C++ Compiler similar to GCC– Cross platform compatible(Windows,Linux,Mac)

• LibClang– Pass arguments to change C/C++ structure padding.

• -m32 , -m64 – Cross platform compatible (Linux, Mac, Windows)– Apple uses it in their developer tools for code completion,

syntax highlighting, etc…– Use it to find comments, structures, types, padding in

structure, etc.• http://clang.llvm.org/docs/Tooling.html• http://clang.llvm.org/get_started.html• https://github.com/trolldbois/clang <- padding/size info

Page 9: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

9 / 23NASA GSFC – CCSDS Spring 2013

C Example CCSDS Header/*! * @brief CCSDS Telemetry Header * @TelemetryPacketHeader * */typedef struct CCSDS_Telemetry_Header{ /*! * @brief Destination address */ char p_sp_dest; /*! * @brief Protocol ID * @default 0xf2 */ char p_sp_prot; /*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @enum ENUM_PACKET_TYPE * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Secondary Header within this Space Packet. * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is not present. * @enum CCSDS_Secondary_Header * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135.0-b-1 application ID's 2040- 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */ uint16_t p_ident;/*! . . . . */ uint16_t p_seq_cont;/*! . . . . */ uint16_t p_data_len;/*! . . . . */ uint64_t s_time;} CCSDS_Telemetry_Header;

Doxygen comments start with @marker_name marker value

Indented comments belong to parent comment

Could have used C bitfields(See backup slides)

Page 10: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

10 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

C/C++ -> clang_to_json.py -> json_to_xml.py -> XML

• Current status– 500 lines of code to go from C Header to XML– Need to add enums, meta data, bit fields.– Implemented structures, unions, bit structures, arrays,

padding bytes, integers, unsigned integers, floating point numbers, pointers, comment parsing.

• https://github.com/nasa/SOIS-CCSDS-XML– NonNasa Contact for access, currently private project

until XML format is well defined.– NASA: fill out a NAMS/IDMAX request

Page 11: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

11 / 23NASA GSFC – CCSDS Spring 2013

C Example p_ident

/*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Secondary Header within this Space Packet. * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is not present. * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135.0-b-1 application ID's 2040- 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */ uint16_t p_ident;

Page 12: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

12 / 23NASA GSFC – CCSDS Spring 2013

Clang to JSON Output for p_ident

…..{'array': [], 'basetype': 'uint16_t', 'basetype_name': None, 'bitarray': True,'data': <clang.cindex.Cursor object at 0x10a7969e0>, 'is_pointer': False, 'kind': CursorKind.FIELD_DECL, 'name': 'p_ident', 'offset': 16, 'size': 2L,

'comments': [{'bitarray': 'msbf', 'children': [{'bits': '3', 'children': ['name': 'packet_version_number'}, {'brief': 'Pacet Version Number’}, {'default': '0'} ]

}, ], …. More Member Comments …..]

'members': [{'array': [], 'basetype_name': None, 'bitarray': False, 'bits': 3, 'comments': [{'name': 'packet_version_number'}, {'brief': 'Pacet Version Number’}, {'default': '0'}], 'data': None, 'is_pointer': False, 'kind': None, 'members': [], 'name': 'packet_version_number', 'size': -1},… More Members..],},

> python clang_to_json.py ../examples/src_cpp/ccsds.h -m32 –I../examples/src_cpp

• Offset, Sizes include padding information.

• Clang object used for further processing if needed.

• Bit arrays become children for bitstructs.

Page 13: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

13 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

XML Layout

• Type database– Basic types

• Integers, floats

– Structures– Unions– Bit structures

• Telemetry database– Similar to structures

• Command database– Similar to structures

<TypeDatabase>

<typeInt/>

<typeFloat/>

<typeBitStruct>

<Members>

<variable/>

</Members>

<typeBitStruct\>

<typeStruct>

<Members>

<variable/>

</Members>

<typeStruct/>

</TypeDatabase>

<Telemetry>

</Telemetry>

<Commands>

</Commands>

Page 14: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

14 / 23NASA GSFC – CCSDS Spring 2013

C Example/*! * @brief CCSDS Telemetry Header * @TelemetryPacketHeader * */

typedef struct CCSDS_Telemetry_Header{ /*! * @brief Destination address */ char p_sp_dest; /*! * @brief Protocol ID * @default 0xf2 */ char p_sp_prot; /*! *@bitarray msbf * @bits 3 * @name packet_version_number * @brief Pacet Version Number * @default 0 * @bits 1 * @name packet_type * @brief Packet type * @enum ENUM_PACKET_TYPE * @bits 1 * @name secondary_header_present * @brief Secondary Header flag * @detailed The Secondary Header Flag shall indicate the presence or absence of the Packet Secondary Header within this Space Packet. * It shall be 1 if a Packet Secondary Header is present; it shall be 0 if a Packet Secondary Header is not present. * @enum CCSDS_Secondary_Header * @bits 11 * @name apid * @brief Application Process Identifier (APID) * @detailed via 135.0-b-1 application ID's 2040- 2047 are reserved and should not be used. * The APID (possibly in conjunction with the optional APID Qualifier that * identifies the naming domain for the APID) shall provide the naming mechanism for the LDP. */

uint16_t p_ident;/*! . . . . */ uint16_t p_seq_cont;/*! . . . . */ uint16_t p_data_len;/*! . . . . */ uint64_t s_time;

} CCSDS_Telemetry_Header;

Page 15: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

15 / 23NASA GSFC – CCSDS Spring 2013

Generated XML<TypeDatabase>

<typeInt bits="3" endianess="bigEndian" name="int3_be" signed="true"/>

…. More small types for int1, int11, int8, int16 .. etc ….

<typeBitStruct name="CCSDS_Telemetry_Header_p_ident" type="uint16_be">

<Members>

<variable name="packet_version_number" type="int3_be"/>

<variable name="packet_type" type="int1_be"/>

<variable name="secondary_header_present" type="int1_be"/>

<variable name="apid" type="int11_be"/>

</Members>

</typeBitStruct>

…. More typeBitStruct for p_seq_cont, s_time ….

<typeStruct name="CCSDS_Telemetry_Header">

<Members>

<variable bits="8" name="p_sp_dest" offset="0" type="int8_be"/>

<variable bits="8" name="p_sp_prot" offset="8" type="int8_be"/>

<variable bits="16" name="p_ident" offset="16" type="CCSDS_Telemetry_Header_p_ident"/>

<variable bits="16" name="p_seq_cont" offset="32" type="CCSDS_Telemetry_Header_p_seq_cont"/>

<variable bits="16" name="p_data_len" offset="48" type="uint16_be"/>

<variable bits="64" name="s_time" offset="64" type="CCSDS_Telemetry_Header_s_time"/>

</Members>

</typeStruct>

</TypeDatabase>

> python json_to_xml.py ../examples/src_cpp/messages.h -m32 –I../examples/src_cpp

Page 16: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

16 / 23NASA GSFC – CCSDS Spring 2013

Writing XML with pythonfrom xml.etree.ElementTree import Element from xml.etree.ElementTree import ElementTree

parent_node = Element('EDS')

Messages = Element('Telemetry')parent_node.append(Messages)

#All attributes/text must textMessages.attrib['Count'] = str(5)Messages.text = “Hello”

document = ElementTree(parent_node)

#See json_to_xml for prettify functionprint prettify(document.getroot())

Page 17: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

17 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

XML Output

<?xml version="1.0" ?><EDS> <Telemetry Count="5"> Hello </Telemetry></EDS>Or Root = document.getroot()print xml.etree.ElementTree.tostring(Root,'utf-8')'<EDS><Telemetry Count="5">Hello</Telemetry></EDS>'

Page 18: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

18 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

json_to_xml.py

• XML to Python CTypes• Decode telemetry/send commands

C++ headersTelemetry/Commands

3rd party C Headers

XML

xml_to_ctypes.py

Python

Python CTypes

clang_to_json.py

Page 19: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

19 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Reading XML with pythonimport xml.etree.ElementTree as ETdoc = ET.parse(filename)root = doc.getroot()for node in root.findall('TypeDatabase/typeStruct'): print node.attrib['name’] print node.text

See xml_to_ctypes.py for complete listing200 lines of code to convert from XML to python ctypesOrhttp://docs.python.org/2/library/xml.etree.elementtree.html

Page 20: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

20 / 23NASA GSFC – CCSDS Spring 2013

Python ctypes example

import ctypes

class CCSDS_Telemetry_Header_p_ident ( ctypes.LittleEndianStructure ): """""" #_packed_ _fields_ = [ ("apid" , ctypes.c_uint16 , 11 ), ("secondary_header_present” , ctypes.c_uint16 , 1 ), ("packet_type" , ctypes.c_uint16 , 1 ), ("packet_version_number" , ctypes.c_uint16 , 3)]

class CCSDS_Telemetry_Header ( ctypes.LittleEndianStructure ): ""”This is a comment about the CCSDS Packet Header""" _pack_ = 1 _fields_ = [ ("p_sp_dest" , ctypes.c_int8 ), ("p_sp_prot" , ctypes.c_int8 ), ("p_ident" , CCSDS_Telemetry_Header_p_ident ), ("p_seq_cont" , CCSDS_Telemetry_Header_p_seq_cont ), ("p_data_len” , ctypes.c_int16 ), ("s_time" , CCSDS_Telemetry_Header_s_time )]

python xml_to_ctypes.py test.xml > test_ctypes.py 205 lines of code

Page 21: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

21 / 23NASA GSFC – CCSDS Spring 2013

Using python ctypes to decode / encode dataimport test_ctypes as tcfid = open('log.raw’)

#Buffer big enough to hold any command / telemetry data

big_buffer = bytearray(1024*10) write_buffer = memoryview(big_buffer)

#header data directly into the buffer. 8bytes.

fid.readinto(write_buffer[0:8])#big_buffer[0:8]#bytearray(b'\x00\xf2E\n\x00\xc0\x17\x00\')

header = tc.CCSDS_Telemetry_Header.from_buffer(big_buffer)#Read in the rest of the CCSDS packet.

fid.readinto(write_buffer[8:(header.p_data_len+1)])print "Apid is %d decode it." % header.p_ident.apid

#changing ctypes changes the binary data. Good for commanding

header.p_sp_dest = 5#big_buffer[0:8]#bytearray(b'\x05\xf2E\n\x00\xc0\x17\x00')

fid.close()

Page 22: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

22 / 23NASA GSFC – CCSDS Spring 2013

ctypes cont…class CCSDS_Telemetry_Header ( ctypes.LittleEndianStructure ): "””Documentation about CCSDS Telemetry Header""" _pack_ = 1 _fields_ = [ ("p_sp_dest" , ctypes.c_int8 ), ("p_sp_prot" , ctypes.c_int8 ), ("p_ident" , CCSDS_Telemetry_Header_p_ident ), ("p_seq_cont" , CCSDS_Telemetry_Header_p_seq_cont ), ("_p_data_len" , ctypes.c_int16 ), ("s_time" , CCSDS_Telemetry_Header_s_time )] @property def p_data_len(self): “””More Documentation””” return self._p_data_len + 1 @p_data_len.setter def p_data_len(self, new_len): self._p_data_len = new_len – 1

h = CCSDS_Telemetry_Headerh.p_data_len = 8print h._p_data_len7print h.p_data_len8

CCSDS data length is 1 minus actual length of the message.

Provide documentation via docstring on telemetry value.

Telemetry: Polynomials, conversions, etc

Commanding:limit values, throw errors if not within ranges.

Page 23: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

23 / 23NASA GSFC – CCSDS Spring 2013Background C++ XML CTypes Summary

Lessoned learned• Provide all of the standard C/C++ types

– As listed on “mimic C/C++ in XML” slide• Standardized naming of Integers, floats, bools, pointers.

(int32,int16,int8, …) – Allows for creating dictionary mappings between XML and

desired language• Mimic the structure of existing languages.

– Makes mapping from XML to existing language more strait forward

• Keep it simple • Develop tools with XML standard

– Kept having to change scheme while making tools to make processing easier

Page 24: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

24 / 23NASA GSFC – CCSDS Spring 2013

Backup slides

Page 25: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

25 / 23NASA GSFC – CCSDS Spring 2013

C bit field example(Not supporting, yet …)

struct CCSDS_Telemetry_header_p_ident {

uint16_t apid:11;uint16_t secondary_header_present:1;uint16_t packet_type:1;uint16_t packet_version_number:1;

};

http://msdn.microsoft.com/en-us/library/ewwyfdbe(v=vs.71).aspx

Least significant field first

Page 26: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

26 / 23NASA GSFC – CCSDS Spring 2013

C Example cont/*! * @ingroup MESSAGES * @brief An example message for Power point. * @apid 135 */typedef struct Basic_M{

CCSDS_Telemetry_Header packet_header; //CCSDS Packet Header/*! * @brief Temp of the baseplate the cpu is attached to. * @polynomial [1566.7676,-18.85822,-.17372, 0.00229, 0.00001] * @unit fahrenheit

* @mode nominal * @red_limit [-Inf,-20), (120,Inf] * @yellow_limit [-20,-15), (115,120] */int32_t cpu_temp;

/*! * @brief Receiver Position * @coordinate_frame ECEF * @unit meter */ double position[3];};

Doxygen comments start with @marker_name marker value

Indented comments belong to parent comment

Page 27: Automating electronic data sheets CCSDS Spring 2013 Sam Price.

27 / 23NASA GSFC – CCSDS Spring 2013

Example basic example

<Telemetry>

<Message name="Basic_M">

<Members>

<variable bits="128" name="packet_header" offset="0" type="CCSDS_Telemetry_Header"/>

<variable bits="32" name="cpu_temp" offset="128" type="int32_be"/>

<variable bits="192" name="position" offset="192" type="float64_be">

<array>

<size>

3

</size>

</array>

</variable>

</Members>

</Message>

</Telemetry>