CLARAty: Coupled Layer Architecture for Robotic Autonomy Issa A.D. Nesnas Ames Research Center...

CLARAty: Coupled Layer Architecture for Robotic Autonomy

Issa A.D. NesnasAmes Research Center

Carnegie Mellon UniversityJet Propulsion Laboratory

University of Minnesota (FY03)

CLARAty Workshop - January 13, 2003

Sponsors: Mars Technology ProgramIntelligent Systems Program

January 13, 2003 CLARAty Workshop - I.A.N. 2

Presentation Overview

• Architecture Overview

• Challenges of Interoperability

• Software Implementation and Status

• CLARAty testbed

• Selected Examples– Motion Control

– Locomotion

– Navigation

• Summary

Background and Overview


What is CLARAty?

CLARAty is a unified and reusable software that provides robotic functionality and simplifies the integration of new technologies on robotic platforms

A research tool for technology development and maturation


CLARAty Development TeamCLARAty/Rocky8/Rocky7

– Max Bajracharya (34)– Edward Barlow (34)– Caroline Chouinard (36)– Gene Chalfant (34)– Hari Das (34)– Tara Estlin (36)– Dan Gaines (36)– Mehran Gangianpour (34)– Dan Helmick (34)– Won Soo Kim (34)– Michael Mossey (31)– Issa A.D. Nesnas (34) (Task Manager)– Ashitey Trebi-Ollennu (35/34)– Richard Petras (34) – Stergios Roumeliotis (Univ. of Minnesota)– Kevin Watson (34)

• NASA Ames Research Center– Maria Bualat– Clay Kunz– Randy Sargent– Anne Wright (lead)

• Carnegie Mellon University– Kam Lasater

– Reid Simmons (lead)

– Chris Urmson

– David Wettergreen– Andy Yang

• Jet Propulsion Laboratory CLARAty on FIDO Team

– Hrand Aghazarian (35/34)– Terrance Huntsberger (35/34)– Chris Leger (35/34)– Matthew Robinson (35/34) Five Summer Students

http://claraty.jpl.nasa.gov/team











Working Groups

• CLARAty Navigation Group (Lead: Reid Simmons - CMU)

• CLARAty Vision Group (Lead: Max Bajracharya - JPL)

• CLARAty Mobility & Manipulation Group (Lead: TBD / Hari Das (former))

• CLARAty Estimation Group (Lead: Daniel Gaines - JPL)

• CLARAty Decision Layer Group (Lead: Tara Estlin - JPL) • CLARAty Test/Release Group (Lead: TDB - JPL)

• CLARAty Rocky Rover Group (Lead: Richard Petras - JPL)

• K9 CLARAty Developers (Lead: Anne Wright - ARC)

• CLARAty Science Working Group (Lead: Ted Roush ARC/Meemong Lee - JPL)

• CLARAty Simulation Group (Lead: Gene Chalfant - JPL)

http://claraty.jpl.nasa.gov/groups




THE DECISION LAYER: Declarative model-based Mission and system constraintsGlobal planning

CLARAty = Coupled Layer Architecture for Robotic Autonomy

INTERFACE:

Access to various levelsCommanding and updates

THE FUNCTIONAL LAYER: Object-oriented abstractionsAutonomous behaviorBasic system functionality

A Two-Layered Architecture

Adaptation to a system


Simulation Hardware Drivers

Adapting to a Rover

Generic

Functional Layer

Rocky 8 Specialized

Classes

&

Objects

Connector

Decision Layer

Multi-level access Connector

Rocky 8

Models/

Heuristics


The Decision Layer

Executives

(e.g. TDL)

General Planners

(e.g. CASPER)Activity Database

Rover Models

FL Interface

Plans

10CLARAty Workshop - I.A.N.January 13, 2003

ASPEN/CASPER System Architecture

TimelineManager

Activity infoConflict status

ADB modifications(e.g., add, modify, delete)

PCN

TCN

NetworkConstraint Propagation

ActivityDatabase

Real-timeSystem

RoverModel

Act, state & resource

updates

ActivitiesActivity

permissionupdates

Interface /GUI

Timelineupdates

Search commands

Activity, timeline, & conflict info

TDL Executive

CASPER

ASPEN

Activities to execute

Act status; updates

Goals & Init State

Domain-specific information

Legend

System Module

Socket connection

Can be socket connection or embedded

Pathplanning(outside module)

Heuristics

Repair Optimize

Search

11CLARAty Workshop - I.A.N.January 13, 2003

The Functional Layer

Transforms

Motion Control

VisionEstimation

Input/Output

Manipulation

Navigation

Communication

Math

Hardware Drivers

Locomotion

Rover

Behaviors

Path Planning

Rocky 8

Rocky 7

K9

FIDO

Science

Simulation

Sensor

Adaptations


Functional Layer Components

Pixel

MastWheeledLocLeggedLoc RBLoc

Locomotor

CoordMotors

SocketMotor

Camera

Instrument

BBMotorControlledMotor

Analog_IODigital_IO

IO

VisualNavigatorVisualTracker

Array_2D

Vector Location Point

Matrix

ImageColorImage HTrans

FeatureDetectorCorrelator

BehaviorState

Database Bit

Resource

Rover

SocketMsg

Manipulator

Arm

StereoGen

Specialized Data Structures

General Purpose Data Structures

Standard Template LibraryString

LinkedList

Container

Waypoint

PathCamera Image

Location

Connector


Top Level Abstractions

Rover_Impl

Rover

Sim_RoverR7_RoverR8_Rover

R8_Mast

R8_Power

Gestalt_Nav

D* Planner

Static or Object Owned

R8_Arm

R8_Locomotor

Mast

Arm

Power System

Camera x N

Path Planner

Mapper

Navigator

Locomotor

R8_Camera

Optional

Pose Estimator

Con

nect

or


Architectural Traverse Example

Stereo Camera

Stereo Engine<<active>>

Navigator

Gestalt Navigator

Grid Mapper

Mapper

JPL Stereo

Terrain Sensor

Locomotor

K9_Locomotor

Global Cost Func

D* Path Planner

Stereo Processor

Camera R Camera L

Synchronous/orAsynchronouse.g. Rate Set at: 10Hzused by other activities

Asynchronous

<<active>>Asynchronouse.g. Rate Set at: 5 Hz

<<active>>

Asynchronouse.g. Rate Set at: 8 Hz

EKF Pose Estimator

Pose Estimator

Path Information

Path Planner

K9 Rover

Rover

Decision Layer Commanding and

State Updates


Architectural Traverse Example

Stereo Camera

Stereo Engine<<active>>

Navigator

R7/Soj Navigator

Mapper

JPL Stereo

Terrain Sensor

Locomotor

K9_Locomotor

Tangent Graph

Stereo Processor

Camera R Camera L

Synchronous/orAsynchronouse.g. Rate Set at: 10Hzused by other activities

Asynchronous

<<active>>Asynchronouse.g. Rate Set at: 5 Hz

<<active>>

Asynchronouse.g. Rate Set at: 8 Hz

EKF Pose Estimator

Pose Estimator

Path Information

Path Planner

K9 Rover

Rover

Decision Layer Commanding and

State Updates

Obstacle Mapper


Navigation Architecture

1

1

1

1

1

0..*

1

1

NavigatorinterfaceNavigator Callback

Locomotor

interfaceWaypoint

State Estimator

Vehicle ModelMap Based Selector Fuzzy Logic Selector Rover Bug Selector

Action Selector

…


Architectural Components

• Data Structure Components (DSC)– Array, Vector, Matrix, Map, Container, LinkedList, Bit– Image, Message, Resource

• Generic Physical Components (GPC)– Locomotor, Arm, Mast, Spectrometer, …

• Specialized Physical Components (SPC)– K9_Locomotor, K9_Arm, R8_Mast, FIDO_Motor,…

• Generic Functional Components (GFC)– ObjectFinder, LayerDetector, VisualNavigator, StereoProcessor,

Localizer

• Specialized Functional Components (SFC)


Architectural Components Impl

• Defining base abstract classes for the generic types– DSC - abstract Data_Obj class– GPC- Device class– GFC - Behavior class

• Additional core classes– Template Mz<T> for measurement– Template Es<T> for estimated values– Template State<T> for state handling


Guiding Principles

• Use domain knowledge to guide design

• Make all assumptions explicit

• Use abstraction layers to help master complexity

• Use information hiding to protect implementation variability

• Use interfaces and documentation to define behavior

• Avoid ineffective over-generalization

• Encapsulate system specific runtime models– Capabilities vary considerably across platforms

– Hardware and hence runtime models vary across systems

– Need for both cooperative and pre-emptive scheduling

• Least common denominator solutions are unacceptable in hardware/firmware/software interactions


On Generic Algorithms

• Technologies that are generic by design should not be constrained by the software architecture & implementation

• Non-generic technologies should be accommodated on the appropriate platforms– Example (Generic): if you are working in navigation, you would

not care about H/W architecture difference among different rovers

– Example (Specific): if you are doing wheel/terrain interaction research, you might require specific hardware which one of the vehicles would support

Software Implementation & Status


Repository

Software Development Process

AFS Backbone

CMU JPL

CLARAty

Repository 3rd PartyReleases Web

0

5

10

15

20

25

30

35

40

45

FY00 FY01 FY02

Users/Collaborators

ExtendedTeam

JPL Team

UW...

VxWorks

ARC

Rocky 8

FIDO

Rocky 7

BenchtopsBenchtopsBenchtops

K9

Repository

ATRV

Some CLARAty Statistics~170 Modules (reusable entity)~31 Packages (module grps)~3 rovers ~250,000 lines of C++ code ~Java/scripts/ and models

Authentication

K9


Infrastructure & Platform Support

• Object-oriented Architecture and Design Patterns

• C++ Implementation (with some Java)

• AFS backbone

• YaM for collaborative distributed development environment

• ACE for OS independence

• Qt for Graphics Display

• Operating Systems & Architectures:

Operating System Arch CommentsVxWorks - Tornado 2.2 ppc, x86 Current Rocky 7VxWorks - Tornado 2.02 x86 Current Rocky 8 & FIDO VxWorks - Tornado 2.02 sparc Rocky 8/FIDO simulationSolaris 2.7 sparc Robotics networkLinux - Red Hat x86 K9, ARTV, ROAMS


Levels of Software Integration

• Level I - Encapsulated– Uses CLARAty components to interact with rover– Does not support a CLARAty API– Runs on at least one robot platform

• Level II - Encapsulated & Integrated– Conforms to a generic CLARAty API (or parent class)

• Level II - Integrated– Has no unsupported 3rd party dependencies– Runs on all applicable rover platforms

• Level III - Refactored and Reviewed– Software reviewed by committee to ensure internal/external consistency – Uses all applicable CLARAty classes– Internally conforms to CLARAty conventions and coding standards

• Level + - Reused– Re-used by other modules in CLARAty - dependent module– Provides access to all internal data products

Challenges in Interoperability


Solaris

x86Ames

JPL

Rocky 8x86

VxWorks

Linux

Rocky 7

K9

ROAMS

Currently Supported Platforms

FIDO

CMULinux

ATRVx86

JPL

FIDOVxWorks

x86

JPL

ppcVxWorks

JPL

Linux


Distributed Hardware Architecture

x86 ArchWireless ethernet1394 FireWireI2C Bus

Rocky 8

Compact PCI

K9 Controllers PIC Servos Current Sensing Digital & Analog I/O

Actuator/Encoders

Potentiometers

I2C / Serial Bus

1394 Bus

IMU

RS232 Serial

Sun Sensor

K9Rocky Widgets Single Axis Controllers Current Sensing Digital I/O Analog I/O


Custom Architecture/Variability

m68k ArchFramegrabbersDigital I/OAnalog I/OWireless ethernetRocky 7

VME Arch

Actuator/Encoders

Potentiometers

Parallel Custom InterfaceMUX/Handshaking

Video Switcher

GyrosAccels

AIO

AIO

PID Controllers


Centralized Hardware Mapped Architecture

x86 ArchFramegrabbersDigital I/OAnalog I/OWireless ethernetFido

PC104

Actuator/Encoders

Potentiometers

PID Control in Software

Video Switcher

IMU

RS232 Serial

Selected Examples


Examples of CLARAty Reusability

Controlled_Motor_Impl

ControlledMotor

Linear_AxisJoint

Non-Resuable Layer R8_MotorFido_Motor Sim_Motor

Trajectory

Trajectory_Generator

Mz<Type>

HCTL_Chip

Widget_Board

Widget_Motor

PID ControllerCounter

DIO

Analog Out Analog In

MSI P460

MSI P430 MSI P415

MSI P430

ResuableHW reusableNon-reusable

R7_Motor

LM629_Chip

R7_MC_Board


Rocky 7 ModulesLines of Code Status

Controlled Motor 2,652 Reusable

Input Output 2,690 Reusable

Bits 1,580 Reusable

Resources (Timers, etc) 725 Reusable

Rocky 7 Motor 927 Non-reusable

Rocky 7 H/W Maps 841 Non-reusable

Motor Controlller LM629 1,143 Reusable - LM629

Digital I/O Board (S720) 576 Reusable - S720PCI Components 329 Reusable - PCI

Total 11,463

Total Reusable 85%Total Reusable - Strict 67%

Code Reusability

Rocky 8 ModulesLines of Code Status


Trajectory Generator 691 Reusable


Bits 1,580 Reusable

Resources (Timers, etc.) 725 Reusable

Bits 756 Reusable

Rocky 8 Motor 1,180 Non-reusable

Rocky 8 H/W Maps 626 Non-reusable

Widget Board Software 2,126 Reusable - widget

Motor Controller HCTL 900 Reusable - HCTL

I2C Master 1,165 Reusable - I2CI2C Master Tracii 1,223 Reusable - Tracii

Total 16,314


FIDO ModulesLines of Code Status


Trajectory Generator 691 Reusable

PID Controller 997 Reusable


Bits 1,580 Reusable

Resources (Timers, etc) 725 Reusable

Common Definitions 2,380 Reusable

FIDO Motor 2,086 Non-reusable

FIDO H/W Maps 1,494 Non-reusable

Encoder Counter (ISA P 400) 463 Reusable - H/W

Analog Input Board (MSI P415) 519 Reusable - H/W

Analog Output Board (MSI P460) 462 Reusable - H/WDigital I/O Board (MSI P560) 602 Reusable - HCTL

Total 17,341



Capabilities of Wheel Locomotor

• Type of maneuvers:– Straight line motions (fwd / bkwd)– Crab maneuvers– Arc maneuvers– Arc crab maneuvers– Rotate-in-place maneuvers (arc turn r=0)

• Driving Operation – Non-blocking drive commands– Multi-threaded access to the Wheel_Locomotor class – e.g. one task can use

Wheel_Locomotor for driving while the other for position queries– Querying capabilities during all modes of operation. Examples include position

updates and state queries– Built-in rudimentary pose estimation that assumes vehicle follows commanded

motion


• Wheeled Locomotion – works for Rocky 8, Rocky 7, Fido, K9, ...

Front

xy

z

C

(a)Skid Steering

(no steering wheels)

x

yz

C

(b)Tricycle

(one steering wheel)

Front

x

y zC

(d)Partially Steerable

(e.g. Sojourner,Rocky 7)

Front

x

y zC

(e)All wheel steering

(e.g. MER, Rocky8, Fido, K9)

Front

x

yz

C

(c)Two –wheel steering

Generic Reusable Algorithms

(f)Steerable Axle (e.g.Hyperion)


Example: collaborative development for locomotor

JPL - 1998

CMU - 2002Future

ARC - 2003

• Designed for Rocky 7 • Used Motor class• Separated wheel

control from locomotion• Built-in pose estimation

• Generalized design for wheeled locomotors

• Full and partially steerable vehicle• Used generic motor classes• Implements fixed axle model• Developed continuous driving• Adapted to Rocky 8, Rocky 7, and Sim

Version 1.0 Version 2.0

• Separated model from control• Add separate locomotor state• Add concept of wheel and

steerable wheel, Drive Cmd, Drive Sequence

• Adapt to ATRV, Sim, Rocky 7, Rocky 8

Version 3.0

Redesign/ mature

Redesign/ mature Version 4.0

• Use device and telemetryinfrastructureAdd adaptation to K9

Add

JPL - 2001


Analysis of amount of resuable code across implementations:

-Reusable131Point 2D

Reusable2080Controlled Motor

-Reusable1083Matrix, Vector, Array

Rotation Matrix, Point 2DReusable341Location, Homogeneous Transforms

-Reusable435Rotation Matrices

584 (non-reusable)

Non-reusable

Non-reusable

Reusable

Reusable

Reusable

Status

Widget Motor, etc...

Rocky 8 Motor

-

Vector

Motion Sequence, 1D Solver, Homogeneous Transforms

Depends On

~92%Total Reusable

6995Total

334Rocky 8 Motor

250Rocky 8 Locomotor

3561D Solver

540Motion Sequence

1445Wheel Locomotor

Lines of CodeModule

Code Resuability Results

CLARAty Status and Future Work


Base Package

Developed Generic Physical Component Base classes• Math and Data Structure modules

– Merging array/matrix between ARC/JPL. Added support for sub-arrays & STL iterators

• Bits and I/O modules (JPL)– New implementation of bits, DIO, & AIO - more consistent, reentrant & efficient

• Telemetry module (ARC)– Added telemetry modules to represent data and parameter sets, handles general

serialization/deserialization, provides multi-threaded clients, & increases efficiency

• Device module (ARC)– Provide base classes for all devices such as cameras, arms, masts, locomotors, etc. Uses

telemetry classes and provides generic interfaces for accessing telemetry

• Power System module (ARC)– Provides information on devices such as power sources, batteries, and battery chargers.

Implemented specializations for K9 power hardware

• Converge implementations and adapt for all platforms• Add Mz, Es, and State logging capabilities


• Developed algorithms for partially-

steered and fully-steered vehicles• Developed continuous driving capability

(JPL)• Demonstrated on Rocky8 & Rocky7, and

FIDO (JPL)• Third generation redesign (CMU)

– Separated locomotor model from control – Added wheels, drive cmds and sequences.– Adapting R8, R7, ATRV, FIDO, and K9– Tested on ATRV, R8, R7, & ROAMS

Locomotion Package

Front

x

y zC

(d)Partially Steerable

(e.g. Rocky 7)

Front

x

y zC

(e)All wheel steering

(e.g. Rocky8, Fido, K9)

(c)Steerable Axle (e.g.Hyperion)

Continuous Driving and Separated Model from Control

• Converge with Device and refine APIand provide models for estimation

• Add ATRV adaptation to repository


Vision Package

• Developed generic infrastructure for vision– Generic Camera, Stereo Camera, Camera Models, Image, Image

operations (e.g. rectification, edge and corner detection), Image I/O, and Image transport

• Provided wrappers for vision algorithms– JPL Stereo, CAHVOR models, ARC Stereo, SVS Stereo, SLOG tracker

• Adapted package to various rovers– Rocky 8, K9, FIDO, Rocky 7 are currently using the same vision

infrastucture

• Worked with Machine Vision Group to integrate visual odometry

• Worked with RMSA team, Olson, Roush, M. Lee

Developed generic Image and Camera infrastructure

• Enhance operations/functionality for target tracking and stereovision• Integration of Wide-baseline Stereo (U. Washington)• Integration of Bundle Adjustment Techniques (JPL & Ohio State)


• Completed and tested FIDO-based EKF port to CLARAty

(JPL) – Two tier Kalman Filter

– Estimating heading (IMU + odometry), x, and y (odometry)

• Created second generation design for the Estimator module (JPL & Univ. of Minnesota)– Handle non-EKF based estimators

– Designed to better integrate with Locomotor and other model infrastructure

– Assume filter primary functions are to propagate state and update measurements

Estimation Package

3DOF FIDO EKF integrated and tested in CLARAty

• Implement Framework for more general estimation algorithm+(6DOF)• Integrate FIDO/R7 sun sensing algorithms


Navigation & Mapper Package

• New modular design for the Navigator (CMU)– Based on generic action-selector - can be adapted to support

navigation technologies– Specialized action selector to Map-based selector– Tightly integrated local and global cost functions (D*)– Navigator interfaces with Locomotor, Location Estimator, and

Terrain Map generator.

• Refactored Gestalt to separate terrain evaluation from action selection (JPL)

• Developed infrastructure to process multi-tiered panoramic images for terrain evaluation (JPL)

• Acquired complete 3-D map of Mars Yard

Developed Generic Navigation Infrastructure

• Integrate two navigation algorithms to exercise navigation framework


• Developed adaptation of the CLARAty

controlled motor classes to interface with ROAMS simulation (JPL)

• Tested Locomotor sending commands to wheels which in, turn, send commands to controlled motors (JPL)

• Specialized Sim_Motors turn commands into streams for socket-based communication

• Drove Simulated Rocky 8 rover with new locomotor (CMU)

• Interface between DL/FL (rover-level and ROAMS) is also available

Simulation Package

CLARAty Locomotor talking to ROAMS via motor cmds

• Provide adaptations for inertial sensors, I/Os, cameras, and science instruments

• Test functionality such as pose estimation, navigation and planning using ROAMS.

• Provide API for integration with MSF


• Interactions with T. Roush’s team for

integration of science analysis and spectrometer functionality

• Analysis code checked in the repository:– Carbonate Analysis– Edge Layer detection

• Interactions with Meemong Lee’s for interface with science instruments, analysis, and simulation. Code available in repository– Calibrated Reflectance Spectrum– Uncalibrated Spectrum– Spectrometer

Science Package

Infrastructure for Spectrometer and terrain analysis

• Provide adaptations to integrate with science instrument simulator and possibly real spectrometers

• Continue integration of science analysis


Communication Package

• Decision Layer/Functional Layer communication module

• K9 (ITAR-free) communication interface

• Communication mechanism between adaptations of CLARAty classes and ROAMS

• Uses CLARAty socket module

Interface between DLs and FL

• Provide easy and automated means for transporting CLARAty data structures (Data_Obj serialization and deserialization)

• Modularize communication components and improve reuse• Complete study regarding use of TAO for DL/FL interaction


Decision Layer Package

• Integrated D* Path Planner with Decision Layer• Developed infrastructure for sharing terrain map with FL• Migrating a version of CLEaR (CASPER/TDL - Linux)

into CLARAty repository. Now available are:– TCM - Task Control Management used by TDL– Tangent Graph Path Planner (as used by DL)– D* Path Planner (as used by DL)– TDL and Java utilities

• Unifying DL dependencies to use the CLARAty modules (e.g. path planners - tangent graph, D*), etc.

Decision Layer integrated D* Path Planner

• Converge more models between DL and FL• Interface with WITS and enhance interface with FL• Integrate another Decision Layer with Functional Layer (pending)

CLARAty Testbed


FIDO Benchtop

Rocky 8 Benchtop

Dexter ManipulatorsRocky 7 Benchtop

CLARAty Testbed (1/2)


CLARAty Testbed (2/2)

• 5 cPCI embedded targets (x86, ppc)

• 2 Linux and 5 Solaris hosts/targets

• AFS VxWorks installations (Tornado I, II, II.2, x86, ppc)

• Mockups for Rocky 8, FIDO, and Rocky 7

• Remotely accessible to CLARAty developers & users

• Web-based target status and control

• Small lab sandbox built for indoor testing

• Several remote and local users exercising testbed (MDS, CMU, ARC, JPL)

• Online sign up for target/rover usage

http://claraty.jpl.nasa.gov/testbed


Summary

• CLARAty provides a repository of reusable software components at various abstraction levels

• It attempts at capturing well-known robot technologies in a basic framework for researchers

• It publishes the behavior and interfaces of its components

• It allows researchers to integrate novel technologies at different levels of the architecture

• It is a collaborative effort within the robotics community

• It will run on multiple heterogeneous robots


Acknowledgements

CLARAty Team (multi-center)

Jet Propulsion Laboratory• ROAMS/Darts Team• CLEaR Team• Instrument Simulation Team • Machine Vision Team• FIDO Team

Ames Research Center• K9 Team

Carnegie Mellon University

Thank you for your Attention

CLARAty Software Packages


Decision Layer PackageDecision Layer Package

General Planners Basic planner infrastructure

...

Activity Database Generic Activity database structure

Planner Implementations CASPER ASPEN

Executives– TCM - Task Control Management used by TDL

– TDL - Task Description Language

Interfaces Functional Layer Interface ...


Base PackageBase Package

Share Common Definitions Math Constants Function Operators Display Functions

Array 1D Arrays 2D Arrays Sub Arrays

Matrix 1D matrices (math) 2D matrices (math) Fixed size matrices Rotation matrices

Transforms Homogeneous Transforms Quaternions

Frame Coordinate Frames Locations (Poses)

Point 2-D Points 3-D Points

Numeric Solvers Newton Rhapson Nedler-Mede

Resource Timers Tasks

...

State State data logging Database

Greyed Out – in progressOrange – non-JPL/non-CLARAty developmentGreen – JPL-non-CLARAty development

GPC & GFC Device Telemetry Periodic Objects


Input Output Package

Input Output Package

I/O I/O Digital I/O Composite Digital I/O Analog Input Analog Output

Bits Bits data structure Bit manipulation/masking


Motion Control Package

Motion Control Package

Motor Open Loop Motors Controlled Motors

Trajectory Trajectories Trajectory Generators Continuous Profiling

...

Coordinated Systems Motor Coordinators


Locomotion Package

Locomotion Package

Wheel Locomotion Wheels/Steerable Wheels Locomotor models Locomotor State Drive Commands Motion Sequences 2-D Wheel Locomotion algorithms 3-D Rocker Bogie Locomotion Continuous driving modules

Legged Locomotion Legged Locomotors

Hybrid Locomotion Wheel/Legged Locomotors

...


Vision PackageVision Package

Camera Framegrabbers Cameras (single, multi) Stereo Pairs Camera models

Data Structures Images Image Pyramids Image I/O Color Images Camera Images Point Clouds Meshes

Stereo Implementations JPL Stereo engine SRI Stereo engine (commercial) Ames Stereo engine

...

Stereo Vision StereoProcessor StereoEngine

Visual Tracking 2-D Feature extraction 3-D Feature extraction Affine Tracking 2-D Tracking algorithms Shaped based tracking Combined 2-D & 3-D tracking Visual odometry

Basic Image Processing 2-D Image processing

-Filtering & smoothing-Edge detection-Blob detection-Corner feature detection


Estimation PackageEstimation Package

Kalman Filter Algorithms 3 DOF EKF pose estimation 6 DOF EKF pose estimation

Pose Estimators Wheel-based Pose Estimators Legged-based Pose Estimators

...

Data Structure Measurements System models Generic Kalman Filter


Navigation PackageNavigation Package

Local Navigators Action Selectors Mapped-base Action Selector Waypoints Generic local navigator structure

Path Planner Generic path planner structure

...

Navigation Implementations MER/GESTALT Navigator Morphin (CMU) Navigator Fuzzy Navigator


Manipulation PackageManipulation Package

Manipulators Joints Links Generic Manipulators Generalized Fwd Kinematics Generalized Inverse Kinematics Limited DOF manipulators Redundant manipulators End effectors

...

Sensor-based Manipulation Stereovision based manipulation Vision/tactile manipulation


Simulation PackageSimulation Package

Simulator Connectors Interfaces to ROAMS simulation

Rover Sim Navigator Sim Locomotor Sim Motor Sim I/O Sim IMU Sim

...

Simulation Implementations ROAMS Rover Simulation (JPL)


Sensor PackageSensor Package

...

IMUs Generic inertial measurement

unit infrastructure

Tilt Sensors Generic Electronic Tilt sensors Generic Accelerometers

Rate Sensors Generic Gyros structures

Temperature Sensors

Voltage & Current Sensors


Science PackageScience Package

Spectrometers Generic Spectra classes

– Calibrated Reflectance Spectrum– Uncalibrated Spectrum

Generic Spectrometer classes

...

Science Cameras Special science cameras

(inherits for Cameras of Vision Package)

Analysis Algorithms Carbonate Analyzer Edge Layer detector

Science Database


Communication PackageCommunication Package

Ethernet Socket Communication

TCP CommUDP Comm

Messages Object serialization Socket messages

Connector Client/server comm. Executives

...

Serial I/O Serial Comm


Path Planning PackagePath Planning Package

Global Path Planners Generic path planner structure

...

Path Planner Implementations D* Path Planner (CMU) Tangent Graph Path Planner (JPL) ISE Path Planner (CMU)


Behavior PackageBehavior Package

Behaviors Basic rover behaviors

...

Behavior Coordination Arbiters Behavior combinators


Hardware DriversHardware Drivers

Motor Controllers

HCTL 1100 Motor Controller

LM 629 Motor Controller

...

I/O Boards/Chips

VPAR10 – Digital I/O Board

VADC20 – Analog Input Board

Framegrabbers/VisionMAX186 Analog to Digital Conv.

MAX528 Digital to Analog Conv.

Imagenation PX610 Framegrabber

Imagenation CX100 Framegrabber

Custom Boards

Widget Board – Motion Control

Other

I2C Master Controller

Crossbow DMU – Interial Measurement Unit

Sensoray S720 Digital I/O Board

1394 Camera Driver

... VISA – Vme to ISA Converter

Widget Board Firmware

PCI Controller

...

...


Rocky 8 PackageRocky 8 Package

Rocky 8 Specialized Physical Components Rocky 8 Motors Rocky 8 Cameras Rocky 8 IMUs Rocky 8 Hardware Map Rocky 8 Digital I/O Rocky 8 Analog I/O

...

Rocky 8 Specialized Functional Components R8 Locomotor R8 Contrained Locomotor R8 Navigator R8 Rover


K9 PackageK9 Package

K9 Specialized Physical Components K9 Motors K9 Cameras K9 IMUs K9 Hardware Map K9 Digital I/O K9 Analog I/O K9 Locomotor K9 Contrained Locomotor K9 Rover

...

K9 Specialized Functional Components

K9 Navigator


Rocky 7 PackageRocky 7 Package

Rocky 7 Specialized Physical Components Rocky 7 Motors Rocky 7 Cameras Rocky 7 Accels Rocky 7 Gyros Rocky 7 Hardware Map Rocky 7 Digital I/O Rocky 7 Analog I/O

...

Rocky 7 Specialized Functional Components R7 Locomotor R7 Contrained Locomotor R7 Navigator R7 Rover


FIDO PackageFIDO Package

FIDO Specialized Physical Components FIDO Motors FIDO Cameras FIDO Accels FIDO Gyros FIDO Hardware Map FIDO Digital I/O FIDO Analog I/O

...

FIDO Specialized Functional Components FIDO Locomotor FIDO Contrained Locomotor FIDO Navigator FIDO Rover


Map Based Selector

• Implements Morphin/D* & Gestalt type algorithms

• Enables easy exchange of terrain evaluation and cost functions

• Combine cost functions through arbitration or summation

11 1

0..11

Immediate CostDStar Cost

Action Selector

interfaceWaypoint

Map Based SelectorVehicle Model

Map<Plane_Fit_Moments>Plane Fit Map

Terrain EvaluatorinterfaceLocal Cost Function

Waypoint Cost

interfaceGlobal Cost Function


GESTALT Navigator on Rocky 8


Background & Objectives

Why are we doing this work?• To capture and preserve robotics expertise from

JPL and other centers• To provide a framework for future NASA rover

technology development and integration• To reduce the cost of integrating new technologies• To operate various robots from a unified

framework• To eliminate cost of redeveloping basic robotic

infrastructure


Goals

• Capture and integrate a wide range of technologies

• Leverage existing tools• Leverage experience and tools of the larger

software development community• Apply appropriate design patterns to the domain• Provide an infrastructure that enables rapid

robotic development• Capture experience of technologists

implementations


How?

• Study software implementation of various robots• Study interactions of elements in various systems• Identify reusable/recurring elements in robotic systems• Identify implicit assumptions made • Project potential advances to these elements• Design a generic/flexible implementation of these elements• Adapt to a number of robotic systems• Test and study the limitations of the design• Go back to design and iterate• Modify/extend/redesign to address limitations and

variability across systems

Your generic base is reusable


Approach

• Domain knowledge guides design• Layers of abstraction help master complexity• Abstractions also provide a classification of

various technology elements• Information hiding protects implementation

variability• Small modular components are more reusable

than monolithic blocks• Interfaces define behavior of various elements


Things to be aware of

• Over-generalizing leads to ineffectiveness– More general -> less functionality -> more work for

results

– Number of abstractions vs. complex hierarchies

– Modular elements with strongly typed interfaces

– Algorithm generality influences abstraction design

• Runtime models vary across systems– Challenges in combining hardware/firmware/software

architectures in most effective manner

– Need for both cooperative and pre-emptive scheduling


Component Analysis

StateState

Objects

Members

Generic Physical Comp

Estimator

State Machines

Object Services Creates

Private

Public

Links to

State Handler

Sub-object

Internal Implementation

Local Estimation

State 1

Queries

HW Object

- optional link


Relationships with Other Components

Locomotor

change_position(Location& loc)change_velocity(double vel)set_position(Location& loc)set_velocity(double vel)get_position()get_velocity()_compute_angles()

CoordMotionSystem

_position: Vector_velocity: double_acceleration: double

Provides trajectory planningbased on a selection criteria possibly by the Decision Layer

Functionality encodedwithin the objects

Wheeled_Locomotor

Legged_Locomotor

Manipulator

Leg

*Wheel

Steerable Wheel

**


R8 Specific Rover Implementation

MastLeggedLoc Wheeled Locomotor

Locomotor

CoordMotionSystem

Motor


Analog_IODigital_IO

IO

Manipulator

Arm

R8_Arm R8_Mast

Implements general fwd &inv. kinematics & joint ctrl

•Specialized inv. Kinematics (overrides default)•Attaches proper motors•Attaches proper cameras for mast•Adds filter wheel

R8_Locomotor

•Attaches proper motors•Restricts Steering to 2 wheels

R8_Motor

R8_Rover

R8_Arm

R8_MastR8_Locomotor

R8

Widget Board

Widget AIOWidget DIO

Widget Motor HCTL 1100 Chip

Non reusable Code Reusable Code

Trajectory

Trajectory_GeneratorTimers


R7 Specific Rover Implementation

MastWheeledLocLeggedLoc RBLoc

Locomotor

CoordMotionSystem

Motor


Analog_IODigital_IO

IO

Manipulator

Arm

R7_Arm R7_Mast

Implements general fwd &inv. kinematics & joint ctrl

•Specialized inv. Kinematics (overrides default)•Attaches proper motors•Attaches proper cameras for mast•Adds filter wheel

R7_Locomotor

•Attaches proper motors•Restricts Steering to 2 wheels

LM629Chip

LM629Motor

R7_Rover

R7_Arm

R7_MastR7_Locomotor

R7

Device Drivers

VPAR10Board

Non reusable Code Reusable Code


Example: Generic Controlled Motor

• Define generic capabilities independent of hardware

• Provide implementation for generic interfaces to the best capabilities of hardware

• Provide software simulation where hardware support is lacking

• Adapt functionality and interface to particular hardware by specialization inheritance

• Motor Example: public interface command groups:– Initialization and Setup– Motion and Trajectory– Queries– Monitors & Diagnostics

Time

Vel

ocit

y

S- Profile

Instantaneous Profile Change


Locomotor Design

0..*

1

Ctrl Motor

Wheel Locomotor Impl

Wheel Locomotor Old

Original Design New Design

0..1

0..1

1

0..*1

Vehicle Interface

Wheel Locomotor

Vehicle Model

Fixed Contact Model

R8 Vehicle Model

Wheel

Steerable Wheel

R8 Steerable Wheel

Ctrl Motor


Vehicle State

• Contains 6DOF pose

• Mechanism state contains variable vehicle parameters

• Mechanism information encapsulated in a separate class (bridging)

• Potential for dynamical state or other descendant state classes

0..1

Dynamic StateMechanism State

Vehicle State

#_position:Location#_time:double#_specific:Mechanism_State *

Ackerman State 4Wheel

#_wheel_base:float#_tire_diameter:float#_inter_axle_distance:float#_steering_angle:float

Skid State 4Wheel

#_wheel_base:float#_tire_diameter:float#_inter_axle_distance:float


Estimation Package

Acrobat Document


Navigator

• Public interface for navigation capability

• Provides a bridge between evaluation and execution

• Manages waypoints• Multi-threaded

– Interface available while executing a task

• Callback provides hook to trigger non-navigation events– Periodic science on the fly

Navigator

#queue < Waypoint* > _waypoints#Action_Selector & _sit_evaluator#Locomotor & _locomotor#State_Estimator & _estimator#Navigator_Callback * _callback#bool _active_waypoint#bool _stop_nav_loop

+Navigator(Locomotor &loc, Action_Selector &eval,StateEstimator & est)+void set_callback(Navigator_Callback * new_callback)+int goto_waypoint(Waypoint & wp)+void abort_waypoint(void)+bool is_waypoint_active(void)+void run(void)+void stop(void)+void nav_loop(void)


Action Selector

• Navigation algorithms provide this interface

• Used to determine the next motion the vehicle should take

Action Selector

#Vehicle_Model & _model#Waypoint * _cur_way_point

+Action_Selector(Vehicle_Model &mod)+void update(Vehicle_State & robotState)+void set_waypoint(Waypoint & wp)+Drive_Sequence get_best_action(Vehicle_State & curState)


Maps

• Generic base class extends matrix

• Iterators allow movement through maps in arbitrary directions

Map Origin

Map_Cell_Type:classMatrix<Map_Cell_Type>

Map

Plane Fit MapGoodness Map

Region

Circular RegionSquare Region

Coordinate Iterator

Global IteratorMap_Cell_Type:classMap Iterator

January 13, 2003 CLARAty Workshop - I.A.N. 93Nesnas/JPL

DECISION LAYER

FU

NC

TIO

NA

L

LA

YE

RTerrain Map from Laser

D* Path PlannerTerrain Map from Laser

Path

Rover

Navigator

Position Estimator

Estimator

Kalman Filter

Locomotor

IMU

R7_Rover

R8_Rover

R7_Locomotor

R8_Locomotor

MotorR7_Motor

R8_MotorWidget Board

Stereo Engine

Camera CameraPX610

Camera1394

FUNCTIONAL LAYER

WheelLocomotor Model

Camera_Image

D* Star

JPL Stereo

Connector

CLARAty Level I Milestone


Publications & Website

http://claraty.jpl.nasa.gov

• T. Estlin, F. Fisher, D. Gaines, C. Chouinard, S. Schaffer, and I. Nesnas, “Continuous Planning and Execution for an Autonomous Rover,” To appear in the Proceedings of the Third International Workshop on Planning and Scheduling for Space, Houston, TX, Oct 2002

• I.A.D. Nesnas, “CLARAty - An Architecture for Reusable Robotic Software,” CSMISS IT Spotlight Series, June 19, 2002

• I.A.D. Nesnas, R. Volpe, T. Estlin, H. Das, R. Petras D. Mutz, "Toward Developing Reusable Software Components for Robotic Applications" Proceedings of the International Conference on Intelligent Robots and Systems (IROS), Maui Hawaii, Oct. 29 - Nov. 3 2001. pdf (8 pages, 2MB)

• T. Estlin, R. Volpe, I.A.D. Nesnas, D. Mutz, F. Fisher, B. Engelhardt, S. Chien, "Decision-Making in a Robotic Architecture for Autonomy." Proceedings of 6th International Symposium on Artificial Intelligence, Robotics, and Automation in Space (i-SAIRAS), Montreal Canada, June 18-21 2001. pdf (8 pages, 72KB)

• R. Volpe, I.A.D. Nesnas, T. Estlin, D. Mutz, R. Petras, H. Das, "The CLARAty Architecture for Robotic Autonomy." Proceedings of the 2001 IEEE Aerospace Conference, Big Sky Montana, March 10-17 2001. pdf (12 pages, 470 KB)

• R. Volpe, I.A.D. Nesnas, T. Estlin, D. Mutz, R. Petras, H. Das, "CLARAty: Coupled Layer Architecture for Robotic Autonomy." JPL Technical Report D-19975, Dec 2000. pdf (116 pages, 904 KB)


FY03 PlansDate

ReadyNeeded

By Technology Description Receives FromDate

Needed

01-Dec-02 CLARAty/MDS, LT, IP

CLARAty testbed: Rocky 8, Rocky7, six embedded mockups for R8, R7, FIDO

01-Dec-02 LT, IP CLARAty/ROAMS open-loop locomotion with simulation of closed-loop wheel control

ROAMS with wheel control simulation

01-Nov-02

15-Dec-02 LT EKF position estimation for (x,y, heading)15-Jan-03 LT, IP Visual Odometry on Rocky 8 and Rocky 7 RMSA - Slope Navigation 01-Sep-0231-Jan-03 LT Sun sensor heading estimation FIDO - sun sensor code - legacy

Rocky 7 - sun sensor code - legacy15-Nov-02

01-Feb-03 IP 2D Visual Tracker with obstacle avoidance on Rocky 8 and Rocky 7

RMSA - Visual Tracking 01-Nov-02

15-Feb-03 LT ROAMS/CLARAty with reasonable R8 data (kinematic, mass inertias, etc) with locomotion, wheel control, IMU, and pose estimation

ROAMS version with IMU simulation 01-Dec-02

01-Mar-03 IP GESTALT navigator on Rocky 8 and Rocky 7 (encapsulated latest MER version)

MER - GESTALT navigation software 01-Dec-02

15-Mar-03 CLARAty/WITS - demonstrate uplink and downlink of a single command (e.g. Hazcam)

WITS version to support interface 01-Feb-03

01-Apr-03 LT, IP Two navigators designed with CLARAty infrastucture - Morphin++, Gestalt

Morphin++ - navigation softwareMER - GESTALT navigation software

01-Dec-02

15-May-03 IP Stereo-based manipulation infrastructure - control/coordination/sensing for Rocky 8, PDM, and K9 Arms and masts

PDM - legacy instrument placement

01-Jun-03 LT CLARAty/WITS - demonstrate commanding for locomotion (driving) and navigation

15-Jun-03 IP CLARAty/ROAMS interface for rover-based manipulators

ROAMS simulation of manipulator kinematics and interface

01-Apr-03

30-Jun-03 IP CLARAty/WITS - demonstrate manipulator commanding for Rocky 8Define format for shared models

WITS version to support Rocky 8 commanding interface

01-Mar-03


CLARAty Collaborations (1/2)

• JPL - MDS Team– Participated with MDS team on CLARAty/MDS infusion task– Supported MDS by upgrading Rocky 7 to MSL-like hardware and

provided working version of CLARAty software for migration to MDS

• JPL - ROAMS Team (A. Jain)– Close collaborations for the integration of the Functional and Decision

Layers to the ROAM rover simulation

• JPL - CLEaR Team (F. Fisher)– Provided an instantiation of the CLARAty Decision Layer, integrated,

tested and debugged on Rocky 7 and Rocky 8 rovers

• JPL - Instrument Simulation Team (M. Lee)– Worked closely with instrument simulation team to develop spectrometer

and spectra infrastructure for CLARAty.



• RMSA Teams:– ARC - Science Analysis Team (T. Roush)

• Close interactions to support integration of science analysis into the CLARAty environment.

– JPL - Slope Navigation (L. Matthies)• Collaboration for integration of visual odometry on rover platforms

– University of Washington (C. Olson & R. Li)• Close collaboration for integration of bundle adjustment and wide

baseline stereo– Carnegie Mellon University (A. Stentz)

• Integration of D* path planner for long range traverse– Ames Research Center (E. Bandari)

• Integration of visual tracking

• ALERT Team (former REE)– Providing CLARAty communication and rover control infra-structure to

REE for a navigation demonstration on Rocky 8



• JPL - MDS Team– Participated with MDS team on CLARAty/MDS infusion task– Supported MDS by upgrading Rocky 7 to MSL-like hardware and

provided working version of CLARAty software for migration to MDS

• JPL - ROAMS Team (A. Jain)– Close collaborations for the integration of the Functional and Decision

Layers to the ROAM rover simulation

• JPL - CLEaR Team (F. Fisher)– Provided an instantiation of the CLARAty Decision Layer, integrated,

tested and debugged on Rocky 7 and Rocky 8 rovers

• JPL - Instrument Simulation Team (M. Lee)– Worked closely with instrument simulation team to develop spectrometer

and spectra infrastructure for CLARAty.



• RMSA Teams:– ARC - Science Analysis Team (T. Roush)

• Close interactions to support integration of science analysis into the CLARAty environment.

– JPL - Slope Navigation (L. Matthies)• Collaboration for integration of visual odometry on rover platforms

– University of Washington (C. Olson & R. Li)• Close collaboration for integration of bundle adjustment and wide

baseline stereo– Carnegie Mellon University (A. Stentz)

• Integration of D* path planner for long range traverse– Ames Research Center (E. Bandari)

• Integration of visual tracking

• ALERT Team (former REE)– Providing CLARAty communication and rover control infra-structure to

REE for a navigation demonstration on Rocky 8

CLARAty: Coupled Layer Architecture for Robotic Autonomy Issa A.D. Nesnas Ames Research Center...

Documents

Transcript of CLARAty: Coupled Layer Architecture for Robotic Autonomy Issa A.D. Nesnas Ames Research Center...