Achieving Next-Level Robotic Intelligence with SoS Implications

14th International Conference onSystem of Systems Engineering

Sheraton Anchorage20 May 2019

Associate Director, Robotics

Dr. Edward Tunstel, FIEEEtunstel@ieee.org

President

2 E. Tunstel

Robots are here

3 E. Tunstel

High-Tech

4 E. Tunstel

Transdisciplinary

5 E. Tunstel

Robotic intelligence pipeline

§ Advances to date are impacting individual pipeline elements§ How to effectively integrate the pipes to realize practical

intelligent robots and robotic systems is unclear.

§ I do not have all of the answers…§ Will pull the thread on the issue as food for thought:

Ø Some select robotics research topics and applications Ø Some next-level intelligence considerations for each

SENSE

ACTPLAN

INTEGRATED INTELLIGENCE

PER

CEI

VE

PLAN

ACT

CO

NTR

OL

LEARN INTERACT

REASON

6 E. Tunstel

Early Automata at APL (1960s)

HopkinsBeast

APL News, March 1964

Ferdinand

E. Tunstel

7 E. Tunstel

Fast forward…2018Got robot? …and Internet?

§ RoboEarth, WWW for robots: giant repository where robots can share information; learn from each other about their behavior & environment –similarly, Robo Brain

§ Cloud Robotics – cloud computing centered around shared robot computing resources (central to robotics within Industry 4.0)

§ Robot-App Store – an apps marketplace intent on enabling apps sharing among robots

http://www.robotappstore.com/

8 E. Tunstel

Robots and IoT

§ Robots as systems connected to IoT in and around intelligent homes and buildings (including hotels, universities and the workplace) – delivery and other services

StarshipTechnologies (UK)

Amazon Scout

PepsiCo & Robby Technologies

Alibaba

Singapore

9 E. Tunstel

Robots and IoT

§ Robots as systems connected to IoT in manufacturing and logistics environments, future (smart) factories, precision agriculture, etc

§ Robots as the truly physical or apparently tangible nodes in Cyber-Physical Systems

IoT-enabled smart robot: KUKA Connect

10 E. Tunstel

Robotics Research Context

Modeling &Simulation

Hardware+ Software

Theoreticalfoundation

Practicalexperience

Mobility AutonomousNavigation Manipulation Sensing &

Perception

Learning, Adaptation, Deliberative planning, Robust field operation, Systems-thinking,

Transdisciplinary ideas…

...

Breakthroughs

I N N

O V

A T

I O

N

RoboticsResearch

InnovationSpace

11 E. Tunstel

Command Sequencing

Engineering Assessment

Uplink

CommandSequences

Downlink

Telemetry

Science Team

scienceactivities

autonomousexecution

• Best health knowledge

•Recommendations• engineering & image data• science data

Spirit / Opportunity

Semi-autonomous operations from Earth

Intelligence and Autonomy• Mission intelligence (science/exploration) is largely human while remote autonomy

is necessarily robotic• Sequencing and analysis teams plan and assess robotic activities using their

perception of the rover surroundings and knowledge of rover state and behavior

12 E. Tunstel

Mars Rover Technology & Robotic Intelligence

§ Autonomous navigation§ Local/global waypoint planning§ Dense stereo vision§ Autonomous terrain assessment§ Visual odometry§ Goal-driven visual servoing§ Robotic arm motion planning§ Precision arm placement§ Terrain sampling and handling§ Autonomous fault response§ Cmd sequencing & visualization

M. Maimone, J. Morrison, JPL

13 E. Tunstel

Next up…https://mars.nasa.gov/mars2020/

§ Faster navigation§ More modern path planning

(RRT, particle filters)

14 E. Tunstel

Next-Level rover intelligence: Mars Sample Return

§ Richer set of capabilitiesØ Sample acquisition & handlingØ Sample fetch & retrievalØ Lander detection & rendezvous

§ Mobility & manipulation capability maturation for sample caching andfetch rovers (prototypes demonstrated in field tests > a decade ago *)

§ Mars 2020 rover is representative of the sample-caching rover in Mars sample return mission concepts

§ Extreme terrain access (cliffs, caves, subsurface lava tubes, etc)

JPL

Schenker, Huntsberger, Pirjanian, Baumgartner and Tunstel, "Planetary Rover Developments Supporting Mars Exploration, Sample Return and Future Human-Robotic Colonization," Autonomous Robots, Vol. 14, 2003.

*

15 E. Tunstel

Intelligent Co-Robots

Beyond mobile sensing and toward environment manipulation and intuitive & physical HRI

Background imgae: http://gp-email.brtapp.com

16 E. Tunstel

Intelligent Co-Robots – Comms. network topology§ UGVs, UAVs, and a ground

link to the user are connected using a wireless mesh Mobile Ad hoc NETwork (MANET)

§ Mini-MACSS uses a COTS comms module that cannot directly link to the Wave Relay mesho One UAV assigned to act as

bridge to mesh (Mini-MACSS is outside of MANET)

17 E. Tunstel

DARPA Robotics Challenge Tech Exposition 2015

MULTI-ROBOT SEARCH & SAMPLINGIN INCREASINGLY CONSTRAINED ENVIRONMENTS

“Russian Doll” scenario

UGV à UAV à micro-UGV

§ A unique demonstration scenario that focused our development of underlying capabilities in key IRAD areasØ Autonomous UAV and UGV mobility/navigationØ Intelligent co-robots and human-robot teamingØ Dexterous manipulationØ Robot vision and perceptionØ Data fusion, distribution, and display

RoboSally

Pelican

Mini-MACCS

Moore, J., Wolfe, K.C., Johannes, M.S., Katyal, K.D., Para, M.P., Murphy, R.J., Hatch, J., Taylor, C.J., Bamberger Jr., R.J. and Tunstel, E., "Nested Marsupial Robotic System for Search and Sampling in Increasingly Constrained Environments," 2016 IEEE Intl. Conf. on Systems, Man, and Cybernetics, Budapest, Hungary, pp. 2279-2286. Oct. 2016.

18 E. Tunstel

Mini-MACSSMiniature Autonomous Crawling Surveillance System

19 E. Tunstel

Human Interface

Operator Console

Keyboard and Mouse

Multi-monitor Display

State Info. & Situational Awareness: maps, state estimates, video, point clouds

User Intent: motion primitives, desired poses

User Confirmation and/or Refinement of Plan

Planned Behavior

Remote Robot Team

UGV: Robo-Sally

UAV: Modified AscTec Pelican

Miniature Ground Vehicle Joystick

20 E. Tunstel

DRC Tech. Expo. demo scenario

21 E. Tunstel

Demo video

https://www.youtube.com/watch?v=Hvh20ySwgPw

22 E. Tunstel

Next-Level manipulation intelligence§ Enhancing perception capabilities beyond

the visual modality

§ Moving beyond object recognition and grasping to knowledge and reasoning about object properties

§ Dense tactile arrays / e-skin, and sensor processing (e.g., neuromorphic)

CEA LIST, FranceKing’s College, London

23 E. Tunstel

Beyond ISR toward environment manipulationand intuitive & physical HRI

Autonomy for Marsupial Robot Team

24 E. Tunstel

Autonomous Team

Planning

Collision-free motion planning

Marsupial-team motion planning

Whole-body planning

Collaborative mapping

Autonomous Team Planning

Vision and Object Recognition

https://www.youtube.com/watch?v=7Gz7yjYqEEM

P.G. Stankiewicz, S. Jenkins, G.E. Mullins, K.C. Wolfe, M.S. Johannes and J.L. Moore, "A Motion Planning Approach for Marsupial Robotic Systems," 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems, Sept. 2018.

25 E. Tunstel

Next-Level multi-robot system intelligence

§ Realizing smart behavior for not only singular robots but for multi-robot systems

§ Tactical behaviors with greater, situational intelligence

§ Coupling w/high-level reasoning systems/ architectures

§ Move beyond swarm intelligence to enable multi-functional swarms that do more than distributed sensing – e.g., manipulation of their environment

26 E. Tunstel

User Interaction & Interfaces

Next Generation First ResponderDHS

27 E. Tunstel

Head-Up, Hands-Off HRI

Current robots require oversight from a trained operator, resulting in reduced

alertness and higher cognitive workload for one or more human teammates. This results in a less effective team.

A conversational human-robot interface using standard, unambiguous military language would allow more intuitive robot C2 and would reduce

the human cognitive burden. This results in more efficient and safer operations.

“Head-Down, Hands-On” “Head-Up, Hands-Off”

28 E. Tunstel

Motivating Use Cases

29 E. Tunstel

Multi-modal and Robot agnostic

{Voice Cmds} {Gestures} {BMI} {AR}

{Behaviors}

BRUCE

Set of common high-level

commands

Other Robots

Multi-modal Head-Up, Hands-Off HRI

BMI ARVoice Text

Gesture

30 E. Tunstel

EWT-30

DARPA RP-2009 (to present):Modular Prosthetic Limb§ New approach to limb systems closely

mimicking human natural form and movements (developed for amputees)

§ Control options range from non-invasive or minimally-invasive techniques to use of sets of wireless implants (targeted muscle reinnervationto thought control)

§ Modular design (hand only, hand+ forearm, whole arm); all power, actuation, and control self contained

§ APL served as lead & systems integrator for a large multi-institutional team

31 E. Tunstel

EWT-31

Thought control…

§ “Breakthrough: Robotic limbs moved by the mind”

§ Scott Pelley reports on movement of robotic limbs using human thought; reception of sensory feedback from a robotic hand

§ Subject, Jan Scheuermann, suffers from a genetic disease that severs the brain-body connection (spinocerebellar degeneration)

http://www.jhuapl.edu/newscenter/stories/

CBS’60 MinutesDec. 2012

32 E. Tunstel

EWT-32

Targeted muscle reinnervation…

§ Breakthrough: Robotic prosthetic attachment to implant in bone

§ Subject, John Matheny, lost arm to cancer in 2008§ Implant inserted in marrow space of bone replaces constricting and

potentially uncomfortable harnesses§ Prosthetic guided by brain signals via nerve reassignment surgery

https://hub.jhu.edu/2016/01/12/prosthetic-limb-more-mobility-apl/

33 E. Tunstel

Robotic Exoskeletons

§ Human Augmentation (able bodied users)Ø Provides user augmentation or assistance to

accomplish generalized tasking

§ Technical ChallengesØ Other than engineering (power, actuation,

controls, ergonomics, physical constraints)

Ø Control software that can assist operators in highly dynamic behaviors.

Ø Exoskeleton control to seamlessly sense and interpret the operator’s intended behavior and then introduce assistive power that coordinates with the operator's motions.

34 E. Tunstel

Applications

§ Health CareØ Functional RestorationØ AssistanceØ Rehabilitation

§ IndustrialØ LogisticsØ Load CarriageØ Heavy Machine Operations

§ Military & Law EnforcementØ Operator Assist Carrying Heavy LoadØ Improve Operator Performance Under

Stressed ConditionsØ Tactical ProtectionØ Heavy Load Carrying

35 E. Tunstel

Next-Level interaction / interface intelligence

§Smart human-collaborative robots that are responsive to intuitive, physical, and brain-interfaced interaction

§Human-robot fusion – Neural interfaces, BMI, etc

§ Tighter feedback loops with brain, EMG signals, etc enabling the robotics to make smart decisions and take appropriate shared control actions leveraging the human as a sensor & supervisory controller

36 E. Tunstel

Modular Open Systems Architecture:Example: AEODRS Standards-based Common Open Architecture

Autonomous BehaviorsVisual SensorsManipulator End Effector Master Comm Link Power

System

InfrastructureUGV

DismountedUGV

TacticalUGV

AEODRS standardized wireless link

Handheld OCU Common OCU

AEODRSElectrical Interface

AEODRSPhysical Interface

AEODRSLogical Interface

AEODRS standardized

interfaces

System Capabilities

Interchangeable and Interoperable

37 E. Tunstel

AEODRS – A Modular Open System

38 E. Tunstel

AEODRS – A Modular Open System

39

AEODRS Capability Modules and Distributed Architecture

Implication for next-level robotic intelligence…but intelligence is non-trivial to modularize in a similar manner

Softweare/Hardware System Test Bed

40 E. Tunstel

Next-Level systems intelligence

§ MOSA provides interoperability, but that is not enough; need “conversability” or dialog as well, enabling next-level robotic intelligence to interoperate with humans & IoTØperhaps borrowing techniques from MAS wherein

agents communicate, not solely to pass data and messages, but particularly in order to achieve their goals

§ Greater leverage and improved adaptation of cognitive architectures (ACT-R, Soar, DUAL, CLARION, etc)

§ Better ways to modularize intelligence

41 E. Tunstel

Robot Learning – Next-Level

§ Need to advance from learning for X (perception, control, etc) Ø autonomous/developmental learningØ knowledge/skill transfer

Next-level intelligence for robot learning moves toward:

§ Autonomous learning (e.g. book by P Angelov)

§ Empirical machine learning (new book. P. Angelov)

§ Broad learning (P. Chen)

§ Robot memetics

PER

CEI

VE

PLAN

ACT

CO

NTR

OL

LEARN INTERACT

REASON

42 E. Tunstel

Robot Memetics

Forthcoming book (colleagues and I):

§ Concept of memes and memetics as elements of collective robotic intelligence and implications for emergence of a hybrid community of humans and intelligent robots

§ Illustration of robot memetics ideas in the context of a space exploration scenario (development and operation of a human-robot settlement on Mars)

§ New ways of thinking about how to realize higher levels of intelligence and learning in robots and robotic communities

43 E. Tunstel

44 E. Tunstel

National GeographicIllustration by Jason Treat, NGM Staff, and Dylan ColeSources: Robert Braun, Georgia Institute of Technology; NASA/JPL

45 E. Tunstel

NASA

46 E. Tunstel

Achieving Next-Level robotic intelligence

Sharp focus on

§ Increased robustness

§Cognitive facilities

§More sophisticated behavior

§System-/colony-level situational awareness

47 E. Tunstel

Need for Basic Robotic-SE

§ Systems engineering (SE) for research products should serve to bridge the research-applications gap

§ Like systems engineering, robotics is inter-disciplinary; thus can benefit from structured approach SE offers

§ A more structured development stage, during or following fundamental research, will allow us to field robots and expect them to be reliable for extended periods

SE – Building robotic systems right

48 E. Tunstel

Some Robotic-SE Considerations

Robotics practice needs more attention to:§Requirements definition/analysis

Ø Customer needs and objectivesØ Requirements V&V (Does design meet requirements? Can it perform

the required mission?)

§Capability characterizationØ What are the capabilities?Ø How reliable/risky are they, and under what conditions?Ø Can the customer trust the system? (Big issue for autonomy!)Ø What specific investment(s) will make it better?

§SpecificationsØ What is the customer really getting? What’s on the spec sheet?Ø Is it supported by statistically significant test results?

49 E. Tunstel

More Robotic-SE Considerations

Robotic-systems development also needs:§Performance evaluation/benchmarking

Ø Metrics/measures of performance over life-cycleØ How does the system measure up to the state of the art?Ø Methods to evaluate performance to facilitate system comparisons

§Test planning/logisticsØ System tests in relevant operating environmentsØ Why is the chosen test environment relevant? To what degree?Ø Support equipment? Costs? Environment regulations?

§People-oriented collaborationØ Complex/large systems call for more personnel from different groups

or organizations (“people skills” – communication, social, etc)Ø Beyond the documents, people facilitate subsystem interface and

configuration control, their negotiation, and cross-system transparency

50 E. Tunstel

SoSE Implications / Challenges

§ Robots (like people) will fail...how to engineer the capacity to self-recover, recover with human assistance without re-programmingØ autonomicity, Ø conversational co-troubleshooting (explainable AI) Ø real-time re-directioning / re-purposing

§ How to evolve from moving data across interfaces to moving information, intelligence, and contextual dialog

§ How to construct design and concept exploration environments that enable non-experts to configure, use and re-use robotic systems as needed

SoSE – Right systems & Interactions

51 E. Tunstel

Robotics & Systems Foci in the IEEE SMC SocietySELECTED ROBOTICS & SYSTEMS TECHNICAL COMMITTEES

§ Autonomous Bionic Robotic Aircraft

§ Bio-mechatronics and Bio-robotics Systems

§ Brain-Machine Interface Systems

§ Computational Cybernetics

§ Cyber-Medical Systems

§ Intelligent Learning in Control Systems

§ Model-Based Systems Engineering

§ Robotics and Intelligent Sensing

§ Shared Control

§ Systems of Systems

§ Unmanned Maritime Systems Engineering

http://ieeesmc.org/

IEEE SMCis where it all

comes together!

Join Us!

52 E. Tunstel

Acknowledgments

Research and real-world applications discussed are based

on past work as PI or contributor, or owing to influence on

or exposure to work by fellow group members at NASA JPL

(Advanced Robotic Controls Group; Robotic Intelligence

Group, Mars rover project teams) and Johns Hopkins APL

(Intelligent Systems Center).

53

5353

Thank You!

Sunset as imaged by the Spirit rover from a hilltop on the surface of Mars

Q U E S T I O N S ?