Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan...
Transcript of Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan...
Plan
• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio
• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions
Télécom Bretagne
A graduate Engineering Schooland a Research Centre in the field of
Science and Information Technologies
Institut Mines-Télécom
4
Télécom Bretagne
The Engineering School of the (French) Far West
Télécom Bretagne in few numbers350 Researchers (faculty, masters, PhD)9 Research DepartmentsAffiliations to French National Research Agencies (CNRS)International prices (Marconi, SPIE, IEEE) and National representativeness (French Academy of Sciences)
1200 students
160 Academics
45% International students from 50+ countries
Telecom Bretagne ranked in Top 15 among 200+ Graduate Engineering Schools in France
International exchanges
• Out: Our student have to spend between 4 and 6 months abroad
• In: Master of Sciences• One semester with English spoken teaching + intensive classes
of French language
• In: Internship• Research / engineering project in one of our labs
• In/Out: Co-tutelle PhD
Lab-STICCCNRS UMR 6285
From Sensors to Knowledge:Communicate and Decide
Télécom Bretagne
UBO
UBS
ENIB
ENSTA-Bretagne
The Research Unit of the (French) Far West
Lab-STICC teams
Short bio
Airborne systems engineering (Thales - 1986/2000)PhD in Computer Sciences - 1999Professor at Telecom Bretagne - 2000 / presentHead of Department (Cognitive Sciences / Usages)Visiting professor MIT - Aero-Astro Department 2006/2007French representative NATO group HF - UAV 2008/2012Deputy director Lab-STICC Research Unit
Plan
• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio
• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions
Human supervisory control of complex systems
Human supervisory control
Increasing complexity and autonomy of systems and subsystems have led to • a separation in time and space between command /control and
execution of the task• a mediation through computers between the operator and the
actuators and sensors
Human supervisory control
Operator Situation
Operator’s space
Command
Display
System
Space of task
Informationmanagement
TaskSharing
Coordination
Informationcollection
Command system
• Task sharing and allocation• Human - System InteractionDimension 1
Referring to OODA
• Functional decomposition• Information processing in systemsDimension 2 Functional decomposition
Supervisory control and OODA
Operator Situation
Operator’s space
Command
Display
System
Space of task
Commandsystem
Informationmanagement
TaskSharing
Coordination
Informationcollection
Space of task
Ob
Space of taskSpace of task
A
Operator
Or
Operator’Operator’Operator
D
• Usual functional sharing• ~ Fitts tableDimension 1 + Dimension 2
Plan
• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio
• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions
Human factors and UVs control: a brief state-of-the-art
RTO-TR-HFM-170 Supervisory Control of Multiple Uninhabited Systems -
Methodologies and Enabling Human-Robot Interface Technologies
www.cso.nato.int/pubs/rdp.asp?RDP=RTO-TR-HFM-170
Annual Workshops on Human Factors of Unmanned Aerial
VehiclesCERI
http://www.cerici.org/ceri_workshops.htm
Operator Situation
Operator’s space
Command
Display
System
Space of task
Commandsystem
Informationmanagement
TaskSharing
Coordination
Space of task
Ob
Space of taskSpace of task
A
OperatorOperator
Ors spaceOperator’Operator’Operator s space
D
Supervisory control and OODA
Focusing on Observation
Informationcollection
Remote perception (Observe)
Relative orientation in space
Stability of images
ref. Cooper & al., CERI-UAV 2007
by courtesy of TNO / L. van Breda
North-up Platform up UAV-up
MovingOperatorCentered
MovingUAV
Centered
Remote perception (Observe)
Telepresence UGV (TNO Netherlands / US Army): acuity, contrast sensibility, stereo vision and audio
by courtesy of TNO / L. van Breda
EyeRobot
Operator Situation
Operator’s space
Command
Display
System
Space of task
Commandsystem
Informationmanagement
TaskSharing
Coordination
Informationcollection
Space of task
Ob
Space of taskSpace of task
A
OperatorOperator’OperatorOperator
Ors spaceOperator’Operator’Operator s space
D
Supervisory control and OODA
Focusing on Action
Remote action
ref. Maj. Martin, CERI-UAV 2007
Operator Situation
Operator’s space
Command
Display
System
Space of task
Commandsystem
Informationmanagement
TaskSharing
Coordination
Informationcollection
Space of task
Ob
Space of taskSpace of task
A
Operator
Ors spaceOperator’Operator’Operator s space
D
Supervisory Control and OODA
Focusing on Orientation
Support to situation assessment
Cognitive task analysis• Identification of data structure that are meaningful (or salient) for the
operator
Situation awareness models• Mica Endsley 3 stages process (perception, understanding,
projection)• Gary Klein RPD (Recognition Primed Decision)• and many others ...
Hunn, CERI 2006
Graphical interfaces, multimodality
MUSCIT (US Army / USAF) : monitoring ground scenes from multiple UAVs
• Graphical interfaces, attention allocation, multimodal presentation
Left Control Station displaytactical situation display, payload control
Right Control Station displayaircraft video, sensor management
by courtesy of AFRL / M Patzek
Supervisory Control and OODAGoing ahead: teaming!!!
Operator Situation
Operator’s space
Command
Display
System
Space of task
Commandsystem
Informationmanagement
TaskSharing
Coordination
Informationcollection
Space of task
Ob
Space of taskSpace of task
A
OperatorOperator
Ors spaceOperator’Operator’Operator s space
D
collection
D
collection
OrOb A
Cognitive assistant unitsCognitive Assistant Units (Germany / UBM Münich)From shared knowledge representation, mimicking human operators and pilots to handle selected tasks
by courtesy of Schulte / IFS
Assistant systems
by courtesy of Donath & al. / IFS
Assistant systems
by courtesy of Donath & al. / IFS
Playbook (Miller)
Delegation: one way humans manage supervisory control with heterogeneous, intelligent assets
Require a shared knowledge of domain goals, tasks and actions
Plays reference a defined range of plan/behavior alternatives (that can be further constrained or simulated)
Supervisor calls plays; agents have autonomy within the play’s scope
by courtesy of Shively / AMRDEC
Playbook
Overwatch: sustained surveillance of a fixed target or area
Track target: sustained surveillance of a moving target
Area recon one-pass reconnaissance of an area
Route recon: one pass reconnaissance of a route
Encircle patrol: circle a target
Protect: surveillance of an area while moving
“Plays”
Miller, Goldman, Funk, Wu & Pate 2004
Constraints on Plays
Example of constraints (for overwatch function)• earliest / latest acceptable “time on target”• earliest / latest acceptable “time off target”• stealth• priority• sensors• etc...
ref: Miller, Goldman, Funk, Wu & Pate 2004
AI constraint basedcontrol algorithms
Human supervisory control
• Next step is autonomy handled by pro-active automata (also called agents)
• From “Human-in-the-loop” to “Human--the-loop” to “Human-on-the-loop”• From supervisory control to authority sharing (a.k.a.
mixed initiative)
Supervisory Control and OODAHow to cooperate?
? ? ? ?
Operator Situation
Operator’s space
Command
Display
System
Space of task
Commandsystem
Informationmanagement
TaskSharing
Coordination
Informationcollection
How to cooperate?
Space of task
Ob
Space of taskSpace of task
A
OperatorOperator
Ors spaceOperator’Operator’Operator s space
D
collection
D
collection
OrOb A
Interaction?
Most of the approaches focus on task sharing, in terms of delegation or responsibilityBut very few address the communication and interaction protocol that supports this delegation
Following: a detailed example from the domain of swarm control
Plan
• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio
• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions
Playing with swarms
Collective Intelligence & Swarms
• Collective Intelligence paradigm : an intelligent system may be composed of a large number of simple entities interacting together (agents) • Natural collectives systems: social insects, multi-cellular organisms,
communities...• Physical systems: mobile robots, sensors networks, distributed
systems
• Computational systems: multi-agents systems, grid computing, cellular automata
• Properties• Robustness, reactivity, self-organisation, adaptativity
by courtesy of LORIA France
Artificial pheromone
Mobile multi-agent system based on social insect mimickingAgents communicate through the environment where they virtually d r o p a m o u n t s o f a r t i f i c i a l pheromoneA pheromone can have
a repulsive effect so that agents avoid to patrol on the same areaan attractive effect so that agents follow the track opened by others
environment where they virtually a r t i f i c i a l
Principles of swarm
by courtesy of MAIA team LORIA - France
Principles of swarms
by courtesy of MAIA team LORIA - France
SMAARTSwarms for surveillance and security
Context: intrusion surveillance
Interaction management for swarms
• A swarm is an autonomous complex system• Main characteristics of UAVs system:
• Many vehicles• Intrinsic collective automation, local decision• Complex interactions
• Several concurrent tasks• Manage threads of interaction• Priorities, turn-taking, interruption, change of
context, etc.
HMI: a global view
HMI details
Pheromones
2 pheromones in order to ensure patrol and tracking functions• Repulsive pheromone for self organization and covering the
area• Attractive pheromone to concentrate UAVs on tracking (and
possible future identification functions)
HMI details (2)
Patrolling with swarms
Results
Between +6,5% and -13% on the first 20 minutes of pure
surveillance
Fail:
1. to control system
2. to self evaluate
Interacting with a swarm?
Objects out of usual representations• Interactions through the environment, fields of pheromone, diffusion,
evaporation, random moves• What happens? Why? What is to happen?How to control?How to set systems parameters?
SUSIESupervising Intelligent Systems based
on Swarms
Interaction management
• Pheromon grids, large amount of subsystems, multiple localized events,”fuzzy” areas ...
• Need to increase the power of expression for the interpretation and for the generation of interaction• New modalities to interact on these
objects• Reflection on an adequate model of
interaction
Control the self-organized systems
Keep human in/on the loopAdapt the algorithm to the new modes of interaction• Speak the language of the
operator• A : Deal with topological +
operational concepts• Control more accurately but
not too much• B : Mix different modes of
control
Changing the point of viewA : Dealing with topological / technical aspects
Mobile Point Open lineClosed
line Area
Watch see followbear regul. a sensor on a given point
bear regul. a sensor on all points / line
bear regul. a sensor on all points /cl. line
bear regul. a sensor on all points / area
Avoid e.g. coll. avoidance
see closed line reduced to a point
do not cross a line
do not cross a given
closed line
get out area if inside
Find find targets systematic search of a
point
find at least one point of
a line
see open line with inside/
outside
see closed line
Follow track N/Amaneuver in
order to align on a line
see open line N.A
Intercept cross mobiletrajectory
N/Amaneuver so as to meet an
open linesee open line N/A
Changing the point of viewB : Mixing modes of control
• Keep pheromone as a basic principle for self organization and surveillance
• Add use of way-points for specific tasks• reach directly the area to patrol• come back to area when shifting because of speed• watch a point (see previous table)
• Add other use of pheromones to extend the expressiveness of commands
• complete exclusion zone (avoid an area - see previous table)• dynamic map of pheromone (work in progress)
Presentation of SUSIE
see video at:
recherche.telecom-bretagne.eu/susieor
deev-interaction.com/project-susie/
Human factors experiment (show)
Human factors experiment (no show)
Experiment configuration
• 47 people
• Non expert but trained
• 15 minutes scenario
• No show (24 people) / Show (23 people)
Same age mean
Experiments conclusions
Show No show
Better time of identification Better results in identification
Less manual transfer of UAVs Less failure in identification
Better usability perceived Lower perceived workload
Lower perceived cognitive load
Lower perceived time pressure
???
!!!
And a NATO price!
http://www.dailymotion.com/video/xjfmut_recherche-exploratoire-susie-supervision-de-systemes-d-intelligence-en-essaim_news
Demonstration French Air Show Le Bourget 2011
Demonstration French Air Show Le Bourget 2011
http://recherche.telecom-bretagne.eu/susie
Plan
• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio
• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions
New trends and challenges in dialogue management for human supervisory
control
Did we fix everything?We proposed:
! A user-centered level of language! A way to make algorithms (and machines) compatible! A new intuitive mode of interaction (tabletop)
! but
! what about the cooperation?! what about more complex systems ... needing more complex commands?
Supervisory control is an InteractionSupervisory control is an interaction…
! Between a supervisory and one or more subordinates
! Nature of supervisor! Nature of subordinates! Nature of tasks! Nature/range of subordinate behaviors! Nature of world (operating conditions)
! Reliability! Trust! Criticality/Importance! Social/Emotional issues! Autonomy
from Chris Miller / SIFT 2012
Human-Human Supervisory Control Examples
A Parking ValetA Postal clerk/Fast Food ChefShepherd to sheepdogA young child (~3 yrs)A teenager (~16 yrs)A new secretary/asstA conciergeQuarterback (Am.) to team Project Manager to teamCEO to corporationPresidential AideMovie Director’s GopherRadar O’Reilly
from Chris Miller / SIFT 2012
Describing the Interaction (Simply)
• How frequently do I have to interact?• How explicitly/expressively?• What kinds of things can I ask for?• Two dimensions proposed:
• Intervention Demand = how much time/attention do I have to spend commanding and monitoring for useful work?• How much time must I spend managing the automation vs.
how much time does it perform independently• Scope = what range of functions can the automation provide?
from Chris Miller / SIFT 2012
Scoring Supervisory Control Examples
A Parking ValetA Postal clerk/Fast Food ChefShepherd to sheepdogA young child (~3 yrs)A teenager (~16 yrs)A new secretary/asstA conciergeQuarterback (Am.) to team Project Manager to teamCEO to corporationPresidential AideMovie Director’s GopherRadar O’Reilly
Frequency11610873243331
Scope12326644689910
Useful?+++----++++
++++++
Ratio1.525
1.31.1.75.5.67.38.33.33.1
from Chris Miller / SIFT 2012
A (Partial) Utility Threshold?
Radar O’ReilyValet
Postal Clerk
Sheepdog
Child
Teenager
New Secretary
Concierge
QB to Team
PM to Team
CEO to Corp
Presidential Aide
Gopher
1
10
110Scope
Frequency from Chris Miller / SIFT 2012
New kind of Operator Support System
Vehicles Command & ControlSituation AwarenessDSS & Automation
New kind of Operator Support System (II)
Vehicle Command & ControlSituation Awareness
Automation
Interaction ManagementSemantic Bridge
Automation
Interaction Management+
Situation AwarenessTwo tasks:
1. mission
2. interaction
Need for Semantic Bridge
same principles upstream
interface manipulation
Need for Interaction Management
• New characteristics of UVSs and their OSSs, e.g.• Several vehicles• More automation, decision support• More complex interactions
• Several concurrent tasks• Manage threads of interaction• Priorities, turn-taking, interruption, change of
context, etc.
interactive
trajectory
planner
inter-UV CoordinationMore automation, decision support
multiple failures or events
Priorities, turn-taking, interruption, change of
multiple payloads + automation
Interaction as a Collaborative Activity
Traditional view [Shannon]: unidirectional, one-m e s s a g e , a d d r e s s e e i s p a s s i v e , n o n -understandings are errors
Collaborative view [Clark]: shared goal, shared effort, interactive refinement, feedbacks
Interaction is a Subordinate Activity
Execute the mission Interact with the OSS
The operator does two things:
>more important
Perfect understanding is not requiredSufficient for the current purpose: mission & context
Shared effort, but unequal contributions
Adjusting the Interaction Load
Non-Understanding
Give up / postpone
“Hold on. Intruder detected at XY”
Ask for recast “Please rephrase”
Request refinement
“Which building?”
Propose refinement
“Do you mean the building North of the airport?”
Understanding
understanding
= positive
feedback
Mor
e co
oper
ativ
e st
rate
gies
Adjusting the Interaction Load
Non-Understanding
Generation & Interpretation
Give up / postpone Keywords
Ask for recast Selfish
Request refinement Cooperative
Propose refinement Mutual awareness
Understanding
understanding
= positive
feedback
only one’s
beliefs
only the addressee’s
beliefs
Mutual awarenesscommon beliefs
Mor
e co
oper
ativ
e st
rate
gies
Why Not Always Be Cooperative?
Maintain the operator’s SAImprove the feeling to be working with the OSSConstruct and improve a common language & representations (e.g. “selfish” at first enables “cooperative” later)Develop more accurate models of each otherUnusual operator’s demand, critical task, etc.
with the OSS
similar to “adverse effects of
automation”
*loss of SA
*complacency
*skill degradation
[Parasuraman et al.]
Summary
• Consider interaction as a task, not only as a an overhead
• Principles similar to human-human interaction/dialogue
• The cooperative attitude of the system can be adjusted
• Interaction load adjustment could be useful to mitigate mission workload
Next step: Trust management
Trust
• A major indirect effect of dialog management• Within a team:
• mutual modeling (task driven)• interaction (speech turn, style, etc. impact trust)
• Constructive / based upon observations
Lee’s Model
Trust and interaction
Trust and interaction
Plan
• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio
• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions
Conclusions
Main asserts
• Interaction is a key concept in the control of autonomous systems
• Interaction must be processed as a task by itself• Interaction has to remain intuitive and close to
operators high-level representations as the autonomous system has to interpret these representations in its own OODA space
(Many) Remaining challenges
Operator multi-tasking / cognitive load• Attention and focus
• Quantitative models
Trust in automation• Extrinsic factors (role of alarms, task environment)
• Intrinsic factors (self-confidence, sharing mental models)
• Situation awareness• Task and mode switching
• Error diagnosis and recovery
(Many) Remaining challenges
• Understanding the operator(s)
• Physiological and cognitive assessment of the operator’s state
• Assessing team cohesion and performance
• Systems of systems
• UAVs within the complete system
• Insertion of UAVs in civil air traffic / regulation
• Psycho-sociological aspects: UAV operators
• Lack of link between UAV operators and field forces
• Ruptures of contexts
Any questions?