M.  De  Cecco  -­ Robotic Perception and  Action

Robotic Perception and  Action  -­‐ Project

Augmented Reality  for  the  “Man  in  the  Loop”

M.  De  Cecco  -­ Robotic Perception and  Action

• Introduction  to  AR• AR  interaction  technologies• ToF cameras  (HW)• UNITY  basics• C#• State  machine  with  UNITY• How  to  display  an  image  on  the  projector  with  UNITY• A  software  based  on  a  state  machine  structure  able  to  interact  with  

the  user  with  UNITY• Identification  and  localization  module:

– Superquadrics– Optimization  for  fitting– RANSAC– Exercitation  with  matlab– PCL  C++

• Calibration  of  Tof camera,  traditional  cameras  and  projectors  on  a  plane

Project – prerequisites

M.  De  Cecco  -­ Robotic Perception and  Action

• A  projector  to  display  on  a  table  indications  to  the  user  (HW)

• A  table  with  retro-­‐reflective  sheets?  (HW)• A  Kinect  v2  ToF camera  (HW)• A  software  based  on  a  state  machine  structure  able  to  interact  with  the  user  (sw)

• An  identification  and  localization  module  (sw)• Calibration  of  Tof camera  with  the  projector  and  with  the  table  (table  and  projector  could  simply  fit  by  moving  properly  the  projector)

Project – components

M.  De  Cecco  -­ Robotic Perception and  Action

An  example  of  setup  from  literature

Jaakko  Hyry,  Max  Krichenbauer,  GoshiroYamamoto,  TakafumiTaketomi,  Christian  Sandor,  Hirokazu Kato,  Petri  Pulli,  Design  of  Assistive  TabletopProjector-­‐Camera  System  for  the  Elderly  with  Cognitive  and  Motor  Skill  Impairments  

M.  De  Cecco  -­ Robotic Perception and  Action

• The  user  quest  for  help  (voice,  gesture  or  virtual  button  on  the  table)

• The  system:1. Item  :=  1;2. Detect  the  item  to  manipulate3. Highlight  it  to  the  user4. Animate  the  action  to  be  performed5. Verify  that  the  item  was  moved6. Ask  if  the  first  step  was  performed  correctly7. Item  =  item  +  18. If  not  end-­‐procedure,  goto point  2

Project – interaction sequence

M.  De  Cecco  -­ Robotic Perception and  Action

Project – modules to develop

Project  Modules Description Software  to  use Groups

State  Machine

>  Main  software  that  manages  the  HMI  interaction.>  Main  projected  interface.HW:  virtual  buttons  with  retroreflective  material C#  -­‐ UNITY

Objects  Localization  with  AR  Toolkit

>  Markers  placed  on  the  objects  extraction  (position  and  location  on  the  projector  space).>  Calibration  of  camera/projector/furnituresHW:  kitchen  simulated  furnitures C#  -­‐ UNITY

Skeleton  acquisition>  Skeleton  acquisition  with  ToF  (kinect)>  Control  of  task  execution UNITY

AnimationsDevelopment  of  proper  anmations  (for  example  with  Blender) UNITY  -­‐ Blender

Objects  Localization  with  ToFcamera

>  Point  cloud  extraction  and  its  use  for  objects  recognition  and  localization C++  (PCL)

Sensor  Fusion

>  Sensor  Fusion  of  Objects  locations  estimated  with  Ar  Toolkit  and  the  ToF  camera UNITY

M.  De  Cecco  -­ Robotic Perception and  Action

1.  State  Machine

M.  De  Cecco  -­ Robotic Perception and  Action

Objects  Localization  with  AR  Toolkit

M.  De  Cecco  -­ Robotic Perception and  Action

Skeleton  acquisition

M.  De  Cecco  -­ Robotic Perception and  Action

Animations

Link  to  some  examples:  https://www.youtube.com/watch?annotation_id=annotation_974074&feature=iv&src_vid=l-­‐gMnB3XsDM&v=VVQ7UiX8HTk

2D:  https://www.youtube.com/watch?v=RKEuxDXpcFA

M.  De  Cecco  -­ Robotic Perception and  Action

Objects  Localization  with  ToF camera

Link  to  our  project:  http://www.robosense.it/index.php/component/allvideoshare/video/tof-­‐pallet-­‐identification-­‐mocup?Itemid=172

M.  De  Cecco  -­ Robotic Perception and  Action

Sensor  Fusion