Cross-Modal Interpretation of Multi-Modal Sensor Streams inInteractive Perception Based on Coupled Recursion

Roberto Martın-Martın Oliver Brock

Abstract— We present an online system to perceive kine-matic properties of articulated objects from multi-modal sensorstreams. The novelty of our system is that it leverages multi-modal information in a cross-modal manner: instead of simplyfusing information from different modalities, sensor streams areinterpreted by leveraging information from another modality.We realize each cross-modal information extraction processusing recursive estimation, with each process addressing aperceptual subproblem. Several estimators are then coupledin a cross-modal network, leading to efficient and robustonline perception. We demonstrate experimentally that ourcross-modal system improves over its uni-modal counterparts,increasing the variability of environments and task conditionsin which the robot can robustly perceive the articulated objects.We further demonstrate that these perceptual abilities are suf-ficiently fast to provide feedback during manipulation actionsand sufficiently comprehensive to allow the generation of newmanipulation actions.

I. INTRODUCTION

We propose a system for the cross-modal perceptionof kinematic properties of articulated objects. Cross-modalperception is a form of multi-modal perception that leveragesthe information obtained in one modality to facilitate theinterpretation of another modality (and possibly vice-versa).Cross-modal integration is able to leverage regularities inthe combined multi-modal signal space, whereas traditionalmulti-modal perception interprets the individual sensor sig-nals independently and then combines the results.

An example of cross-modal perception is the McGurkeffect [1]. In this perceptual illusion, a subject watches avideo of a person pronouncing syllables. The subject isconvinced to hear different utterances, such as ba-ba or ga-ga,when in fact the sounds are identical. The subjects misjudgesthe sound because the video in fact shows the person sayingdifferent syllables but the sound has been altered to playthe identical syllable. The illusion occurs because the visualcue produced by the facial motions influences the perceptionof the sound. As a result, identical sounds are perceived asbeing different. This illusion demonstrates that visual cuesaffect hearing. This cross-modal interpretation of multiplemodalities is necessary for robust perception of speech [2].If humans were simply merging modalities, the subject wouldnotice the contradiction in the visual and audio signal.

To leverage cross-modality in robot perception, we pro-pose to realize the robot’s perceptual system as a network

All authors are with the Robotics and Biology Laboratory, TechnischeUniversitat Berlin, Germany. We gratefully acknowledge the funding pro-vided by the Alexander von Humboldt foundation and the Federal Ministryof Education and Research (BMBF), by the European Commision (EC,SOMA, H2020-ICT-645599) and the German Research Foundation (DFG,Exploration Challenge, BR 2248/3-1).

Fig. 1: Our robot manipulates three articulated objects (a cupboarddoor, a glass door, and a camera tripod) based on the informationfrom our proposed multi-modal perception system exploiting cross-modal information; the system perceives online the kinematicproperties of the interacted object by integrating visual (RGB-Dimages) and proprioceptive signals (joint encoder values, wrencheson robot’s wrist and air-pressure signals in the pneumatic hand)using information from one modality to interpret the other.

of coupled recursive estimation filters. Each estimation filteraddresses a perceptual subproblem. The coupling of thesecomponents allows us to use the estimated value of onerecursive estimation loop as a prior for others, even acrossdifferent modalities. For example, our system predicts motionfrom proprioception and uses it to interpret visual perception.The system also combines proprioception and vision toperceive the type of grasp achieved by the robot hand, thenusing this information as a prior to disambiguate propriocep-tive signals. These examples illustrate that information frommultiple modalities and multiple perceptual subproblemspropagates through the network, leading to robust, onlineperception.

We evaluate the proposed cross-modal interactive percep-tion system on different articulated objects while varying thetask and environmental conditions. Our method increasesrobustness and accuracy of perception by extracting moreinformation from the multi-modal stream. We also demon-strate that the information perceived during an interactioncan be used to monitor manipulation and also to generatenew interactions (Fig. 1).

II. RELATED WORK

The work we present here is a) a new interactive per-ception system, and b) a multi-modal perceptual systemleveraging cross-modal information. On our path towards

2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)September 24–28, 2017, Vancouver, BC, Canada

978-1-5386-2682-5/17/$31.00 ©2017 IEEE 3289

a multi-modal system, we will also propose c) a novelperceptual approach to perceive kinematic structures basedonly on proprioception. We will now discuss these three areasof related work.

a) Interactive Perception: Interactive perception meth-ods exploit correlations between actions and changes in thesensor signals for robot perception [3]. They enhance robotperception by exploiting interaction as an additional sourceof information for the perceptual process. This strategyhas proved successful for object segmentation and recogni-tion [4, 5], shape reconstruction [6, 7], and the perception ofdynamic [8, 9] and kinematic properties [10, 11] of articu-lated objects. However, these methods are based on a singlemodality or use multiple sensor modalities but they applyone independently to each perceptual sub-task. This neglectsthe benefits of a tighter integration and exploitation of thecorrelations between sub-tasks. The multi-modal interactiveperception system we present here improves robustness andversatility over previous approaches by using cross-modalcommunication to acquire from one modality priors for theinterpretation of the other.

b) Multi-Modal Perception: Multi-modality has beenapplied previously in recursive filters to overcome limita-tions of uni-modal robotic perceptual systems. The commonmethodology is to estimate a correction by fusing the multi-modal signal into a single estimate [12, 13]. This approachdoes not leverage information from one modality to helpinterpret the other. Differently, we exploit the results fromone recursive filter as priors in the others to obtain moreinformation. This cross-modal exploitation was applied suc-cessfully by Garcia Cifuentes et al. [14] to track a robot armand an object from a multi-modal stream. However, theirmethod requires models of the arm and the object and cannotbe applied to perceive previously unseen articulated objects.

c) Perceiving Kinematic Models from Proprioception:Previous approaches showed that the kinematic propertiesof an articulated object can be perceived from end-effectortrajectories [15] and applied wrenches [16] during inter-action. These methods are based on two assumptions thatlimit their applicability: 1) there is only one moving partconnected with a joint to the static environment, and 2) thereis no translation between the end-effector and the movingpart during the interaction. We leverage information fromvision to correctly interpret proprioception and to overcomethese limitations, estimating the correct grasp model andperceiving more complex kinematic structures.

III. VISION-BASED PERCEPTION OF KINEMATICPROPERTIES

Our goal is to integrate vision and proprioception into asingle multi-modal system that exploits cross-modal infor-mation. In this section, we will summarize our previouslypresented perceptual system based only on vision. In thenext section, we present a novel perceptual system based onproprioception. Then, in Section V, we will explain how tointegrate both systems leveraging cross-modal information.

Kinematic Model Grasping Model Soft-Hand Bending

Joint Encoder ReadingsWrenches and

Air Pressure Signals

Rigid Body Motion End-Effector Motion

FeatureMotion

RGB-D Stream

Fig. 2: Our proposed system for interactive perception of articu-lated objects based on cross-modal information between coupledrecursive filters; bottom: input sensor signals; arrows: informationflow between filters and across modalities (blue: input measure-ments, red: alternative predictions)

The integrated system with its most relevant recursive filtersis depicted in Fig. 2.

The interactive perception system we presented in [10]perceives kinematic models (structure and state) of unknownobjects in an online manner from RGB-D streams. The key toachieve robust online perception was 1) to factorize the orig-inal problem into subproblems that can be solved robustlyand efficiently using recursive estimation based on task-specific priors, and 2) to reuse priors of one recursive filterin the others by communicating estimations and predictionsbetween the estimators.

Before we summarize the recursive filters used in thatwork [10], let us review the most important elements ofrecursive state estimation. In recursive estimation, the goal isto infer the current state of a dynamic system, xt , based on theprevious state, xt−1, an observation, zt , and a control input,ut . When state and observation are stochastic processes,the recursive estimation filter can be implemented as arecursive Bayesian filter. In this case, the filter estimates theposterior p(xt |z1:t ,u1:t) over the state, based on the prior statep(xt−1|z1:t−1,u1:t−1) [17]. Implementations require a forwardprocess model for predicting the next state, p(xt |xt−1,ut),and a measurement model, p(z|x), to be able to mergeobservations and predictions. These models can be definedbased on domain knowledge (i.e. task-specific priors), orlearned from regularities in the sensor-action space (e.g. [18]or the alternative models in [10]).

Our prior work factorizes the perception of articulatedobjects into three recursive estimation subproblems: 1) theestimation of the location of N 3D point features (xfm ∈R3N) from a tracking process in RGB-D images leveragingmotion continuity, 2) the estimation of the motion (poseand velocity) of M moving rigid bodies (xrbm

m = (p,v), p ∈

3290

SE(3),v ∈ R6,m ∈ {0, . . . ,M− 1})1 from assigned subsetsof moving features leveraging rigid body physics, and 3)the estimation of motion constraints and degrees of freedombetween pairs of moving bodies leveraging kinematics ofarticulated objects. The solution to this last subproblem isthe kinematic model.

To estimate a joint of the kinematic model we main-tain four joint filters, one for each perceivable joint type:prismatic, revolute, rigidly connected, and disconnected.The filters estimate the distribution of continuous randomvariables over the joint parameters and state, xjoint, that isdifferent for each type. At each time step the perceivedkinematic model is composed of the most likely joint filterestimate between each pair of bodies, given the observedrelative motion between bodies. See [10] for a more in depthexplanation.

The way our system reuses priors from one recursive filterin the others is by passing 1) their estimated state to the filterabove (see left part of Fig. 2, blue arrows) as measurements,and 2) their predicted measurements to the filter below asalternative predicted state (red arrows). In the first case,the estimated state from the filter i is used as (virtual)measurement by the filter j, z j = xi, using the appropriatemeasurement model, p(z j|x j). We used this communicationpattern to “inject” more priors at each filter until we solvethe original perceptual problem.

In the second case, predictions from filter j – statepredictions, x j

t that lead to measurement predictions, z jt – are

used as state predictions by the filter i, xit = z j

t . To resolve formultiple predictions at one filter, we select the one that bestexplains the measurements based on maximum-likelihoodmodel selection: xt = arg maxxk

tp(zt |xk

t ). Exploiting this com-munication pattern we restrict the space of possible solutionsof one subproblem using the other processes (and their pri-ors) as alternative forward and measurement models. We willexploit similar intercommunication patterns in the perceptualsystem based on proprioception and to exploit cross-modalinformation in the multi-modal system.

Limitations: This system fails if the interaction cannotbe observed visually, e.g. due to poor lighting conditions,occlusions, or insufficient texture. Later, we will leverageinformation from proprioception through cross-modal inte-gration to improve the interpretation of the visual streamand to overcome these limitations.

IV. PROPRIOCEPTION-BASED PERCEPTION OFKINEMATIC PROPERTIES

Proprioception refers to sensory information about theconfiguration of the robot’s own body (kinaesthetic) and theforces it exerts (haptics). Our robot obtains proprioceptivesignals from a force-torque sensor on its wrist, from air-pressure sensors monitoring the chambers of its soft hand,

1We changed the representation of rigid body poses to avoid the singular-ities of the exponential coordinates at 2kπ rotations. Poses are now elementsof the Special Euclidean group p ∈ SE(3) affected by a small randomperturbation in the tangential Lie space ξ ∈ R6 so that their compositionppp = exp(ξ )p is also a random variable [19]

Fig. 3: Effect of the deformation of the soft-hand; left: hand inthe nominal state; middle and right: hand in the bent state after amotion of the end-effector without motion of the interacted body(a door handle)

and from measurements of the configuration of the jointsof its arm. The goal is to use these signals to perceive themotion of the object the robot is interacting with as wellas its motion constraints, leading to the object’s kinematicmodel.

The motion of the interacted body and the robot’s end-effector are coupled, as their relative motion is constrainedby their contact. Because our robot uses a soft hand forthe interaction, the relative motion between the hand andthe object depends on the deformation of the hand and onthe remaining degrees of freedom of the contact interaction(grasp).

We factorize the perception of articulated bodies into thefollowing five subproblems: the estimation of A) the motionof the end-effector, B) the bending state of the soft-hand,C) the kinematic model of the grasp, D) the motion ofthe interacted body, and E) the constraints in the motionof the interacted body. Fig. 2 depicts the recursive filtersaddressing these subproblems, together with the filters ofthe vision-based system. The estimation of motion of theinteracted body is subsumed with the estimation of otherbodies from vision in the box “Rigid Body Motion”. In thefollowing, we will explain how we solve the subproblemsof the proprioception-based system using coupled recursionestimation (the last subproblem is solved the same way asin the previous section).

A. Estimation of End-Effector Motion

The first recursive filter estimates the motion of the end-effector. The state of the end-effector is represented byits pose and velocity, xee = (pee,vee). To predict the nextstate, we use a velocity-based kinematic update: pee

t =exp(vee∆t)pee

t−1, veet = vee

t−1. The measurements for the esti-mation of the end-effector motion are the pose and velocityof each robot joint provided by robot’s joint encoders, zee =(qk, qk), k ∈ {0, . . . ,N− 1}, N = robot’s number of joints.We combine them with prior knowledge about the robot’sembodiment and forward kinematics and obtain a directmeasurement of the end-effector’s pose and velocity that weintegrate recursively: z′ee = (pee

z ,veez ).

B. Estimation of Hand Bending

When the robot interacts with an object, the soft handdeforms. This changes their relative pose (see Fig. 3). In

3291

the second recursive filter, we estimate the consequences ofthis bending effect. We represent the bending state of the softhand as the relative transformation between the nominal end-effector pose (estimated by the filter described above) and thepose of a virtual body we call bent end-effector (defining thehand’s physical pose), xbent = pee−bee = pee pbee, where is the inverse composition of elements of SE(3). We assumethat the bending state remains constant between consecutivetime steps, xbent

t = xbentt−1 . We use as measurements the signals

of the proprioceptive stream that correlate to the bendingof the hand. These are the wrenches measured at robot’swrist and the pressure values in the four air chambers ofthe soft-hand: zbent = (w,P), where the wrenches w ∈ R6

and the air pressure signals P ∈ R4. However, defining anexplicit measurement model relating bending and proprio-ceptive signals for a complex soft-manipulator as the RBOHand 2 [20] is a difficult problem [21]. We will adopt adata-driven approach and learn from experiences a model thattransforms the proprioceptive signals into direct observationsof the bending state: f (w,P) = z′bent = pee−bee

z .We approximate f using an artificial neural network. To

obtain labeled data to train the model, we execute 15 interac-tions of the robot grasping an object that is rigidly attachedto the environment. We recorded the wrenches and pressuresignals at different relative poses of the bent soft hand withrespect to the nominal pose during these interactions. Wethen trained a multi-layered perceptron regressor (MLPR1)to map from wrenches and pressure signals to the 6D relativepose observations.

To integrate the observations recursively, we also needto learn their uncertainty. Following the approach proposedin [22], we trained several partial MLPRs, leaving out groupsof two trials, and computing the standard deviation betweenpredictions from these partial MLPRs and the fully trainedMLPR. We then train a second MLPR (MLPR2), mappingwrenches and pressure signals to standard deviation of theregressor. With this procedure, the second MLPR learns thedifficulty of the transformation problem for each input signaland allows us to filter proprioceptive signals into a robustestimate of the hand bending state.

C. Estimation of Grasp Model

In the third recursive filter, we estimate a kinematic modelof the grasp. The grasp model explains the kinematic con-straints between the motion of the bent end-effector and theinteracted body. We maintain and estimate independently theparameters of four filters for grasp models, one for each typeof grasp that our anthropomorphic soft-hand can perform:(i) perfect grasp (no relative motion), (ii) revolute grasp(allowing rotation around the grasping axis), (iii) cylindricalgrasp (allowing rotation around and translation along thegrasping axis), and (iv) failed grasp (no motion constraint).For revolute and cylindrical grasps, the estimated parametersare the position and orientation of the grasping axis: xgr

rev =xgr

cyl = (ap, ao), ap, ao ∈ R3 and |ao| = 1. For the perfectgrasp, the parameters are the fixed relative pose betweenthe bent end-effector and the interacted body: xgr

per = pbee−ib.

The failed grasp does not impose any motion constraints andtherefore does not have any parameters to estimate, xgr

f ail = /0.We initialize these parameters based on the morphology ofthe hand.

The estimation of the grasp model leverages the couplingbetween filters to obtain measurements. The estimates ofthe pose of the bent hand (from the previous two filters)and the interacted body (from the next filter) are combinedto generate a measurement, zgr = f (xee,xbent,xib) = pee ⊕pee−bee pib = pbee−ib. The estimation of the parameters andthe most likely type are performed similarly to the estimationof joint parameters of a kinematic model in [10].

D. Estimation of Interacted Body Motion

The fourth recursive filter estimates the motion of the bodythe robot interacts with. The state of the interacted body isrepresented by its pose, xib = pib. The prediction of its nextstate leverages also the coupling between filters: the changein pose depends on the motion of the end-effector, correctedwith the bending effect and propagated through the graspingmodel, pib

t = exp(xgrAdpee−beevee∆t)pibt−1, where xgr is a 6×

6 matrix representation of the kinematic constraints of thegrasping model and Adpee−bee is the adjoint transformationassociated to the bending effect. None of the proprioceptivesignals can be used as observations of the motion of theinteracted body, and thus the prediction becomes the beliefstate.

Limitations: The first limitation of the system basedonly on proprioception is due to the mutual dependencybetween the estimation of the interacted body motion andthe grasp model. This effectively reaffirms the initial priordistribution over the grasp model. The accuracy of theestimated interacted body motion depends on this prior.

The second limitation is that the proprioceptive signals,because of their limited range, only provide measurementsabout the state of the robot and the responses from theinteracted body. The system can only perceive a singlebody connected by a joint to the environment defining thekinematic model. Overcoming both limitations will requireadditional prior knowledge that our integrated system willobtain from vision by leveraging cross-modal information.

V. INTEGRATION OF VISION AND PROPRIOCEPTION

Once we have explained how to extract information fromeach modality, we will explain how to leverage informationfrom one modality to help interpret the other. The pro-posed multi-modal system exploits cross-modal informationto overcome the limitations of uni-modal perception.

Predictions about the motion of the interacted body fromproprioception are leveraged to assign correctly visual pointfeatures, even under challenging visual conditions. The fea-tures can be used as the missing observations to correctthe proprioceptive predictions, zib = xfm|ib, where xfm|ib arethe visual point features assigned to the interacted body.The cross-modal predictions from proprioception to visionand corrections from vision to proprioception lead to a

3292

new estimate that breaks the mutual dependency of theproprioception-only system, xib = pib.

Using the interacted body motion perceived from cross-modal information, our system can correctly interpret theconstraints in the bent end-effector motion perceived fromproprioception, and that define the kinematic grasp model,xgr. The type and parameters of the grasp model are inferredfrom the relative motion between bent end-effector andcross-modal estimates of interacted body motion (section IV-C):zgr = pbeepib.

The system can use grasp model estimates from cross-modal information as prior to further interpret proprioceptivesignals when the visual modality degenerates (e.g. the objectgoes out of the field of view, is occluded, or due to extremelybad lighting conditions or not enough visual texture). Theprior obtained from cross-modal information is sufficient toestimate the kinematic model of the interacted body usingonly proprioceptive signals.

Finally, the integrated system interprets correctly theconstraints in the motion of the interacted body perceivedfrom proprioception leveraging information from vision. Thesystem perceives from vision the motion of other bodiesapart from the directly interacted and uses this prior toanalyze the motion constraints of the interacted body fromproprioception. The integrated system based on cross-modalinformation can perceive complex kinematic models withmultiple joints or when the interacted body is not connectedto the static environment, xjoint.

VI. ROBOT MOTION GENERATION AND CONTROL

Generating a multi-modal stream rich on informationdepends on the strategy to control robot’s interaction. Ourgoal is to generate motion in the dimensions allowed bythe (initially unknown) kinematic structure. This adaptivebehavior can be achieved using a compliant controller basedon the force-torque signals [23]. We use an operationalspace impedance controller on the Lie Group SE(3) [24]to adapt a desired trajectory of the 6D pose of the end-effector, pee|des(t), based on the signals from the force-torquesensor. The behavior of this controller is parametrized bythree 6× 6 matrices –stiffness (K), damping (D) and mass(M)– that transform virtually the end-effector into a spring-mass damped system with different reactive behavior foreach dimension.

The aforementioned controller can adapt an initial ex-ploratory trajectory. To generate such a trajectory for artic-ulated objects we propose a velocity-based controller thatsets at each step a new goal for the end-effector pose, pee|des,based on the error between the measured and desired velocitytwists: pee|des

t = exp(kp(vee|meas− vee|des))pee|dest−1 . Combining

both controllers the robot can explore articulated objects withdifferent dynamic properties and create rich multi-modalsignals for perception.

Once the kinematic structure has been revealed and per-ceived leveraging cross-modal information, the robot shouldbe able to use this information to improve the interaction orgenerate new manipulations. We implemented this skill as

an online trajectory generator that computes an end-effectoroperational space trajectory to achieve a manipulation task(i.e. reaching a desired object configuration). The trajectorygenerator uses the perceived kinematic model of the objectand interpolates its joint configurations towards the goal con-figuration. Then it computes the trajectory of the interactedbody, and from that, the trajectory of the end-effector thatwill generate the desired change in the object’s configuration.

VII. EXPERIMENTS

We conducted two sets of experiments. In the first set,we evaluated quantitatively the performance of our systemwhen perceiving different articulated objects and comparethe use of only vision, proprioception, or the multi-modalstream leveraging cross-modal information. We measurethe robustness, accuracy, and convergence of the kinematicmodel estimation by comparing to ground truth for the jointparameters and state. In the second set, we made use of theonline information from the cross-modal system to controlrobot’s motion and fulfill a manipulation task. The robotexplores an articulated object until it discovers a joint andperceives that it reaches a desired joint configuration. Then,the robot exploits the perceived information to plan a newtrajectory to return the object to its initial configuration. Wemeasure the accuracy of the execution (final joint state) ofboth the explorative and the exploitative interactions.

A. Experimental Setup

In our experiments, we use a robot manipulator composedof a Barrett WAM arm and a RBO Soft Hand 2 [20]. Thejoint configurations of the arm are measured at 200 Hz byencoders placed at the motors controlling the cables. Thestretching of the cables introduces uncertainty about the end-effector’s pose that we model with a covariance of 1 cmand 3◦ in the end-effector pose measurements, resultingfrom an offset calibration. The visual input is an RGB-Dstream (640×480 pixels at 30 Hz) provided by a Carminesensor rigidly attached and registered to the robot’s base. Theforce-torque signals are provided by an ATI 6-DoF sensormounted on robot’s wrist and delivering signals at 100 Hz.Air pressure in the chambers of the soft-hand are delivered at100 Hz. To compensate for the disparity in sensor frequencieswe accumulate signals and process them at 15 Hz. Thisestimation rate can be maintained on an Intel Xeon E5520PC at 2.27 GHz.

The estimated states of each filter are assumed to be Gaus-sian distributions. Both process and measurement models areof the form xt = g(xt−1,ut)+εt and zt = h(xt)+δt , where gand h are possibly non-linear (but linearizable) functions,and ε and δ are process and measurement additive Gaussiannoise. This allows us to implement the recursive estimationfilters as Kalman filters or their variant for non-linear models,extended Kalman filters.

The neural network regressors (MLPR) have a topologyof three layers with 10-10-10 fully connected neurons. Thistopology was selected in a hyperparameter search by aleave-one-out cross-validation process, selecting between 1

3293

t=0sq=0cm

t=10sq=14cm

t=26.5sq=22.3cm

0 5 10 15 20 25 30Time [s]

0

10

20

30

40

50

60

70

Orie

ntat

ion

Err

or [± ]

rismatic oint stimation

orientation error visionorientation error proprioceptionorientation error -modal

J

cross

P E

t=0sq=0◦

t=10sq=57◦

t=40sq=80◦

0 5 10 15 20 25 30 35 40 45Time [s]

0

5

10

15

20

25

30

35

Orie

ntat

ion

Err

or [◦]

evolute oint stimation

orientation visionorientation proprioceptionorientation multi-modal

JR E

0

5

10

15

20

25

30

35

40

Pos

ition

Err

or [c

m]

orientation visionorientation proprioceptionorientation -modalposition visionposition proprioceptionposition -modalcross

cross

t=0sq=0◦

t=40sq=44◦

t=57sq=90◦

0 10 20 30 40 50 60Time [s]

0

10

20

30

40

50

60

70

80

90

± ]O

rient

atio

n E

rror

[

evolute oint stimationJR E

0

5

10

15

20

25

30

35

Pos

ition

Err

or [c

m]

orientation error proprioceptionorientation error -modalposition error proprioceptionposition error -modalcross

cross

Fig. 4: Experiments of the estimation of kinematic models (each row represents a different object): initial (first column), intermediate(second column), and final frame (third column) of the estimation, including error plot (fourth column) of estimated joint parametersrelative to ground truth; the insets in the three images show the time t and the estimated joint configuration q using the cross-modalvariant; estimated prismatic joints are shown as solid green cylinders, revolute joints as solid red cylinders

and 100 neurons per layer in networks of one, two, orthree layers. The vision-based system tracks N = 200 pointfeatures. To focus the attention on the estimation of thekinematic model of the articulated object and not on therobot’s arm, we project a model of the robot on the cameraplane and subtract this part from the visual analysis.

In our experiments, we parametrized the controller to becompliant in all dimensions (main diagonal elements of K =0.1, D = 1, and M = 1) except in the pulling direction of theend-effector (main diagonal elements of K = 400, D = 200,and M = 1). These parameters perform well in all objectswe evaluated. To generate an exploratory behavior we guidethe robot’s end-effector into a grasping distance of the object,command the robot to close the hand and command a desiredvelocity, vee|des, of 2 cm s−1 in the pulling direction (kp =0.1). As a result, the robot reveals the kinematic structure bypulling with increasing force and adapting to the dimensionof allowed motion of the articulated object.

We evaluate our system on articulated objects with differ-ent types of joints, size, color, and surface properties. Wedid not add artificial visual markers that could facilitate thevisual perception. The objects are placed at different posewith respect to the robot and sensors. In some experimentswe also change abruptly the lighting conditions to evaluatethe robustness of the perceptual systems. To obtain theground truth for the joints, we manually measured the jointparameters and the final joint state.

B. Results

Uni-Modal vs Cross-Modal Perception: We evaluate theaccuracy and convergence of the kinematic model estimatesfrom the three perceptual systems: only vision, only propri-oception and cross-modal integration. Figure 4 shows threeimages from the RGB-D sensor (initial, middle, final steps)and graphs of the estimation error to ground truth over time.

In the first experiment the robot interacts with a drawer.After 6.5 s (indicated with a vertical line in the plot) wechange abruptly the lighting conditions by switching off thelights. The vision-only system stops perceiving the objectwhile the proprioception-only and the cross-modal systemcontinue the estimation. The final joint state estimated bythe cross-modal system is the most accurate (22.3 cm, groundtruth 22.5 cm), followed by the proprioception-only (23 cm).The vision-only system stops tracking at (9.8 cm). The cross-modal system achieves the best performance because itleverages vision to estimate a more accurate grasping model,which lead to more accurate body motion estimates androbustness against vision failures from the interpretation ofproprioception.

In the second experiment, the robot interacts with a doorthat rotates around a revolute joint. The robot almost com-pletely occludes the object during the first 25 s of interaction(indicated with a vertical bar in the plot). The proprioception-only and the cross-modal system perceive the object duringthe entire interaction. The vision-only system perceives theinteracted object only when it becomes clearly visible. Thefinal joint state estimation from the cross-modal system (80◦,

3294

TABLE I: Robot Interaction Based on Online Multi-ModalPerception

Exploration ExploitationSliding Door 2.2 cm±1.6 cm 1.8 cm ±1.6 cmCamera Tripod 7.8◦±2.3◦ 2.6◦±2.24◦

Glass Door 1.3◦±0.73◦ 0.6◦±0.5◦

ground truth 85◦) is the most accurate, followed by theproprioception-only (78◦). The final estimation of the visualsystem (43◦) is affected by the delayed start. The cross-modalsystem achieves the best performance because it uses theproprioceptive signals to interpret the visible motion in thesmall non-occluded parts of the object.

In the third experiment, the robot interacts with a card-board box and closes one of its lids. As a result from theexplorative interaction, the entire box translates. We focusthe analysis on the estimation of the relative revolute jointbetween the box and the lid. Both uni-modal systems fail todetect this joint. The vision system only perceives the lowerpart of the box, while the proprioception system detects onlythe motion of the lid and interprets it as a revolute jointwith respect to the environment. The cross-modal systemcorrectly perceives the relative joint between the box andthe lid because it uses the motion of the box perceived fromvision to correctly interpret the motion constraints of the lidperceived from proprioception. The final joint state estimatefrom multi-modality is 90◦ (ground truth 100◦).

d) Controlling Interaction with Online Interactive Per-ception:: We tested our cross-modal perceptual system andonline trajectory generator for the manipulation of threepreviously unseen objects (see objects in Fig.1): openinga glass door (GD) 20◦, turning camera tripod (CT) 45◦

and opening a sliding door (SD) 30 cm. These objects arechallenging because they do not present strong textured sur-faces and because the hand cannot grasp them perfectly. Werepeated the interactions 5 times on each object with differentinitial robot-object pose. The results (mean and standarddeviation on the error to ground truth) are depicted inTable I. The interaction succeeded in the 15 trials (see videoattachment) indicating that the information perceived onlinecan be used to generate new successful trajectories. Ourproposed perceptual system leverages information betweenvision and proprioception to estimate accurately the jointstate at the turning point and the end of the manipulation,which indicates that the system can be applied to monitorongoing interactions.

VIII. CONCLUSION

We presented a perceptual system for online multi-modalinteractive perception of articulated objects based on coupledrecursive estimation. The key novelty of our system is theconcept of cross-modality, when information extracted fromone sensor modality is leveraged to extract information fromthe sensor stream of another modality. This approach tomulti-modal perception is present in human perception [1]and as we show also benefits robot perception. We demon-strate this in the context of perceiving articulated objectscross-modally, using information from vision (RGB-D sen-

sor) and proprioception (from joint encoders, a force-torquesensor, and air-pressure sensors). Our cross-modal approachincreases robustness and versatility of interactive perceptionunder varying environmental conditions and overcomes lim-itations of uni-modal systems. We also show that our systemcan be used to monitor ongoing interactions and to generatenew manipulation actions.

REFERENCES[1] H. McGurk and J. MacDonald, “Hearing lips and seeing voices,”

Nature, vol. 264, no. 5588, pp. 746–748, 1976.[2] L. D. Rosenblum, R. M. Miller, and K. Sanchez, “Lip-read me now,

hear me better later,” Psychological Science, vol. 18, 2007.[3] J. Bohg, K. Hausman, B. Sankaran, O. Brock, D. Kragic, S. Schaal,

and G. Sukhatme, “Interactive perception: Leveraging action in per-ception and perception in action,” Transactions on Robotics, 2017.

[4] H. van Hoof, O. Kroemer, H. Amor, and J. Peters, “Maximallyinformative interaction learning for scene exploration,” in IROS, 2012.

[5] J. Sinapov, T. Bergquist, C. Schenck, U. Ohiri, S. Griffith, andA. Stoytchev, “Interactive object recognition using proprioceptive andauditory feedback,” IJRR, vol. 30, no. 10, 2011.

[6] R. Martın-Martın, S. Hofer, and O. Brock, “An integrated approach tovisual perception of articulated objects,” in ICRA, 2016.

[7] K. Xu, H. Huang, Y. Shi, H. Li, P. Long, J. Caichen, W. Sun,and B. Chen, “Autoscanning for coupled scene reconstruction andproactive object analysis,” ACM Transactions on Graphics (TOG),vol. 34, no. 6, p. 177, 2015.

[8] F. Endres, J. Trinkle, and W. Burgard, “Learning the Dynamics ofDoors for Robotic Manipulation,” in IROS, 2013.

[9] R. Martın-Martın and O. Brock, “Building kinematic and dynamicmodels of articulated objects with multi-modal interactive perception,”in AAAI Symposium on Interactive Multi-Sensory Object Perceptionfor Embodied Agents, AAAI, Ed., 2017.

[10] R. Martın-Martın and O. Brock, “Online Interactive Perception ofArticulated Objects with Multi-Level Recursive Estimation Based onTask-Specific Priors,” in IROS, 2014.

[11] “Active articulation model estimation through interactive perception.”[12] J. Illonen, J. Bohg, and V. Kyrki, “3-d object reconstruction of

symmetric objects by fusing visual and tactile sensing,” IJRR, vol. 33,no. 2, pp. 321–341, Oct. 2013.

[13] P. Hebert, N. Hudson, J. Ma, T. Howard, T. Fuchs, M. Bajracharya,and J. Burdick, “Combined shape, appearance and silhouette forsimultaneous manipulator and object tracking,” in ICRA, 2012.

[14] C. Garcia Cifuentes, J. Issac, M. Wuthrich, S. Schaal, and J. Bohg,“Probabilistic articulated real-time tracking for robot manipulation,”IEEE Robotics and Automation Letters (RA-L), vol. 2, Apr. 2017.

[15] J. Sturm, A. Jain, C. Stachniss, C. C. Kemp, and W. Burgard,“Operating articulated objects based on experience,” in IROS, 2010.

[16] Y. Karayiannidis, C. Smith, F. E. V. Barrientos, P. Ogren, andD. Kragic, “An adaptive control approach for opening doors anddrawers under uncertainties,” Transactions on Robotics, vol. 32, 2016.

[17] S. Thrun, W. Burgard, and D. Fox, Probabilistic Robotics (IntelligentRobotics and Autonomous Agents). The MIT Press, 2005.

[18] R. Martın-Martın, A. Sieverling, and O. Brock, “Estimating therelation of perception and action during interaction,” in Workshop onRobotics in the 21st century: Challenges and Promises, 2016.

[19] T. D. Barfoot and P. T. Furgale, “Associating uncertainty with three-dimensional poses for use in estimation problems,” Transactions onRobotics, vol. 30, no. 3, pp. 679–693, June 2014.

[20] R. Deimel and O. Brock, “A novel type of compliant and underactuatedrobotic hand for dexterous grasping,” IJRR, 2015.

[21] G. Smoljkic, G. Borghesan, D. Reynaerts, J. D. Schutter, J. V.Sloten, and E. V. Poorten, “Constraint-based interaction control ofrobots featuring large compliance and deformation,” Transactions onRobotics, vol. 31, no. 5, pp. 1252–1260, Oct 2015.

[22] R. Rojas, Neural Networks: A Systematic Introduction. New York,NY, USA: Springer-Verlag New York, Inc., 1996.

[23] H. Bruyninckx and J. D. Schutter, “Specification of force-controlledactions in the task frame formalism-a synthesis,” Transactions onRobotics, vol. 12, no. 4, pp. 581–589, Aug 1996.

[24] J. Park and K. Kim, “Tracking on lie group for robot manipulators,”in 2014 11th International Conference on Ubiquitous Robots andAmbient Intelligence (URAI), Nov 2014, pp. 579–584.

3295