Supporting Remote Manipulation: An Ecological Approach

Supporting Remote Manipulation with an Ecological Augmented Virtuality InterfaceJ. Alan AthertonMichael GoodrichBrigham Young UniversityDepartment of Computer ScienceApril 9, 2009

Funded in part by Idaho National LaboratoryAnd Army Research Laboratory1Exploratory study1OutlineBackgroundRelated WorkEcological InterfaceUser StudyInterface Changes from StudyConclusions and Future Work

2BackgroundWhat is a remote manipulator?ApplicationsUSAREODPlanetary Exploration

3We show a remote manipulator and a mobile manipulator. Mobile manipulators are used in these applications. Our research is a step toward supporting mobile manipulation, although right now we are focused on only the manipulation aspect.Mobile manipulation involves two phases for most tasks: navigation and manipulation. Operators must drive the robot to a particular place and then manipulate on objects there.3ProblemRemotely operating a robot is difficultSoda straw Maintaining situation awarenessTime delayMental workloadWhy is this a problem?CollisionsSlow Stressful

Foster-Miller Talon4Hard to get depth perception from 2D camera video4Older Interfaces

All images adopted fromYanco, H. A.; Drury, J. L. & Scholtz, J.Beyond usability evaluation: analysis of human-robot interaction at a major robotics competitionHum.-Comput. Interact., L. Erlbaum Associates Inc., 2004, 19, 117-149 5Problems with these older interfacesHard to get contextHard to integrate informationUnnatural interaction5OutlineBackgroundRelated WorkEcological InterfaceUser StudyInterface Changes from StudyConclusions and Future Work

6Related Work

Idaho National Laboratory

UMass LowellBruemmer, D. J. et al.Shared understanding for collaborative control.IEEE Transactions on Systems, Man and Cybernetics, Part A, 2005, 35, 494-504 Yanco, H. A. et al.Analysis of Human-Robot Interaction for Urban Search and Rescue.Proceedings of the IEEE International Workshop on Safety, Security and Rescue Robotics, 2006 7Improvements on 2D OCU styleGive much more information than just camerasGood for engineering7Related Work

INL / BYU AV InterfaceFerland et al. - SherbrookeC. W. Nielsen, M. A. Goodrich, and B. Ricks. Ecological Interfaces for Improving Mobile Robot Teleoperation. IEEE Transactions on Robotics and Automation. Vol 23, No 5, pp. 927-941, October 2007. Ferland, F.; Pomerleau, F.; Dinh, C. T. L. & Michaud, F.Egocentric and exocentric teleoperation interface using real-time, 3D video projection.Proceedings of the 4th ACM/IEEE international conference on Human robot interaction, ACM, 2009, 37-44 8Augmented virtuality interfacesShow real elements in a virtual sceneAllow for viewpoints you cant get with camerasReduce workload for associating data, reducing transformsOur work represents an extension of AV interfaces to support manipulation.8Related Work

NASA VizNguyen, L. A.; Bualat, M.; Edwards, L. J.; Flueckiger, L.; Neveu, C.; Schwehr, K. ..; Wagner, M. D. & Zbinden, E.Virtual Reality Interfaces for visualization and control of remote vehiclesAutonomous Robots, 2001, 11, 59-68 9

Kelly, A.; Anderson, D.; Capstick, E.; Herman, H. & Rander, P.Photogeometric Sensing for Mobile Robot Control and Visualisation TasksProceedings of the AISB Symposium on New Frontiers in Human-Robot Interaction, 2009

CMU Robotics InstituteExtremely high detailCareful planningIntegrate data from multiple sourcesOur work is like applying the Nasa style interface to a real-time task, with real-time controls9OutlineBackgroundRelated WorkEcological InterfaceUser StudyInterface Changes from StudyConclusions and Future Work

10Interface DesignRequirementsEcologicalIncrease situation awarenessManage workload Existing InterfacesLack depth informationNo manipulation supportNot designed for real-time operation

11SA multiple viewpoints, depth informationWorkload Integrate information, grounded frame of reference

11Ecological Augmented Virtuality Interface

12Real-time remote manipulationDefine ecological make relationships perceptually evident.Define AV real inside virtual env.Explain arm graphic, 3D scan, video, video rotationSay how we have multiple perspectives, integrated information, depth information, grounded frame of reference12Ecological Interface

13Video rotates for view-dependent controlCan get views from the top, sideColor indicates distance from the ranging camera13InfrastructureRobotBuild from kit, modifyPlayer driverMotion planning for armSwiss Ranger driverCommunicationIntegrate with INLs systemNetwork data transferUser InterfaceOpenGL displayExperiment automation

14

Robot ControllerUser InterfaceJust the key points for what we had to do14OutlineBackgroundRelated WorkEcological InterfaceUser StudyInterface Changes from StudyConclusions and Future Work

15Experiment SetupVariant 1Variant 2Variant 3Variant 4Variant 5Variant 6

3D + VideoEnd EffectorVideo 3D Joint Robot ControlVisualizationTask: collect yellow blocks30 participantsBetween-subject comparison1630 participants2x6Each participant tested 3 variantsBlock collection task, similar to collecting rock samplesSwiss Ranger camera to get depth and build up modelWebcam mounted to armReconfigurable bases16ReconfigurationReduce memorization effectsMinimize damage to armQuick change

17Task

18Visualization

193 VisualizationsNot everyone tested all 3Calibration of 3D model not perfect, as bad as 2cm off in some places19Robot ControlJoint controlView-dependent end effector control

20Some people tested one, some people mixedJoint control moves each joint independently very commonEE control Blue pad on the left controls the view angle20Robot View-Dependent Control

21Robot reaches for pointUser moves point with joystickPoint movement depends on view orientationResults Time

3D + Vid.End eff.3D + Vid.Joint3DEnd eff.3DJointVideoEnd eff.VideoJoint22People work slower with the 3d scan, faster with videoMight have something to do with adjusting the virtual viewpoint22Results Collisions

3D + Vid.End eff.3D + Vid.Joint3DEnd eff.3DJointVideoEnd eff.VideoJoint3D + Vid.End eff.3D + Vid.Joint3DEnd eff.3DJointVideoEnd eff.VideoJoint23Collisions with posts, box, tableCollisions with block in final adjustmentsWhen video is present, people bump into things more (sensor FOV?)Joint control correlates with more collisions (faster, coarser though)Video helps with final alignment when poorly calibrated 3d23Results Comparison InterfaceMeasure3D + Vid. End eff.3D + Vid.Joint3DEnd eff.3DJointVideoEnd eff.VideoJointTime to completion6th1stWorld collisions3rd6th1st3rd6thBlock collisions1st6th6thPreference1st1st6th1stSubjective mental workload1st6th2nd3rd2424OutlineBackgroundRelated WorkEcological InterfaceUser StudyInterface Changes from StudyConclusions and Future Work

25Changes Inspired by User StudyProblemsAlignmentTime lagCluttered 3D scan modelChangesStereo camera exterior orientationInteractive robot arm calibrationSimple QuickeningScan Pruning

26First study brought out some problems to work on.26Camera and Robot CalibrationInteractive stereo camera calibration

Live robot arm calibration

27Camera calibration was just guess and check before, now we get the human to specify correspondences and then use linear least squares to find the best transformationRobot arm was measured as carefully as possible, but ended up being nonlinear.We calibrate only the graphic of the arm, as the piecewise transformation can cause the arm to behave erratically.27Simple Quickening28

Time lag leads to a move, stop, check cycle for controlQuickening shows a model-based prediction of the actual current stateWe show the predicted end effector point.28Scan Pruning29

Original 3D scan had clutter. Floor, curtain, robot arm noise.The only things essential to this task are the shapes, posts, and box.Caveat is that the simple filter sometimes did not prune all of the floor, so some participants mistook the floor for the box.29Second User Study30

Important thing to notice is that everything is faster and more consistent.Even video-only improved due to smoother arm control.30OutlineBackgroundRelated WorkEcological InterfaceUser StudyInterface Changes from StudyConclusions and Future Work

31Conclusions3D visualization supports SAVideo is faster3D + video is a good tradeoff3D + video might reduce workload32We have already fixed several of the major design flaws and are currently running a follow-up studyAlso poor coverage of experiment parameter space 6 variations, only test 3 per person, basically random order32Future Work

33Head trackingEcological camera videoHaptics

Head tracking with wii remote from Johnny Lee at CMUPresent video more integrated, hide when zoomed out

33